Im facing the following challenge. I try to subscribe to "newPendingTransactions" via websocket. I can successfully connect to the websocket. When connected, I would expect a stream of new incoming pending transactions, which I can read out and propagate it to some channel (chan json.RawMessage).
...
c, httpResponse, err := websocket.DefaultDialer.Dial(u.String(), req.Header)
...
for {
messageType, message, err := c.ReadMessage()
if err != nil {
log.Println("ERROR:", err)
os.Exit(1)
return
}
_, _ = message, messageType
// s.Out is the outgoing chan json.RawMessage
s.Out <- message
}
...
sadly I dont receive any message (pending tx).. only one on closing the whole construct. When I check on my node directly with "txpool.status" in console, then I can see that there are new pending txs incoming all the time. They just dont wanna get propagated to my websocket connection. Is there anyone who can help me out here? Maybe I am missing a parameter for starting the geth node itself?
here is how I start my geth node:
geth --http.api eth,web3,debug,txpool,net,shh,db,admin,debug --http --ws --ws.api eth,web3,debug,txpool,net,shh,db,admin,debug --ws.origins localhost --gcmode full --http.port=8545 --maxpeers 120
here is my "admin.nodeInfo":
Geth/v1.10.16-stable-20356e57/linux-amd64/go1.17.5
I found out what I did wrong:
It was not about any forgotten geth parameter and even that the "newPendingTransactions" were propagated correctly..
I tested the websocket connection with another tool called "wscat" and sent the necessary rpc via console (resulting in a stream of tx hashes!)
{"id": 1, "method": "eth_subscribe", "params": ["newPendingTransactions"]}
It showed me that the error must be about the golang code itself..
The last line inside the for loop
s.Out <- message
was sending messages into an unbuffered channel. In order for this to work, some other go routine must consume the channels messages on the other side.
It also helps to use a buffered channel like this:
s.Out = make(chan Message, 1000)
... so at least a 1000 messages will be sent out
Related
Bit of a newb to both Go and GRPC, so bear with me.
Using go version go1.14.4 windows/amd64, proto3, and latest grpc (1.31 i think). I'm trying to set up a bidi streaming connection that will likely be open for longer periods of time. Everything works locally, except if I terminate the client (or one of them) it kills the server as well with the following error:
Unable to trade data rpc error: code = Canceled desc = context canceled
This error comes out of this code server side
func (s *exchangeserver) Trade(stream proto.ExchageService_TradeServer) error {
endchan := make(chan int)
defer close(endchan)
go func() {
for {
req, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
log.Fatal("Unable to trade data ", err)
break
}
fmt.Println("Got ", req.GetNumber())
}
endchan <- 1
}()
go func() {
for {
resp := &proto.WordResponse{Word: "Hello again "}
err := stream.Send(resp)
if err != nil {
log.Fatal("Unable to send from server ", err)
break
}
time.Sleep(time.Duration(500 * time.Millisecond))
}
endchan <- 1
}()
<-endchan
return nil
}
And the Trade() RPC is so simple it isn't worth posting the .proto.
The error is clearly coming out of the Recv() call, but that call blocks until it sees a message, like the client disconnect, at which point I would expect it to kill the stream, not the whole process. I've tried adding a service handler with HandleConn(context, stats.ConnStats) and it does catch the disconnect before the server dies, but I can't do anything with it. I've even tried creating a global channel that the serve handler pushes a value into when HandleRPC(context, stats.RPCStats) is called and only allowing Recv() to be called when there's a value in the channel, but that can't be right, that's like blocking a blocking function for safety and it didn't work anyway.
This has to be one of those real stupid mistakes that beginner's make. Of what use would GPRC be if it couldn't handle a client disconnect without dying? Yet I have read probably a trillion (ish) posts from every corner of the internet and noone else is having this issue. On the contrary, the more popular version of this question is "My client stream stays open after disconnect". I'd expect that issue. Not this one.
Im not 100% sure how this is supposed to behave but I note that you are starting separate receive and send goroutines up at the same time. This might be valid but is not the typical approach. Instead you would usually receive what you want to process and then start a nested loop to handle the reply .
See an example of typical bidirectional streaming implementation from here: https://grpc.io/docs/languages/go/basics/
func (s *routeGuideServer) RouteChat(stream pb.RouteGuide_RouteChatServer) error {
for {
in, err := stream.Recv()
if err == io.EOF {
return nil
}
if err != nil {
return err
}
key := serialize(in.Location)
... // look for notes to be sent to client
for _, note := range s.routeNotes[key] {
if err := stream.Send(note); err != nil {
return err
}
}
}
}
sending and receiving at the same time might be valid for your use case but if that is what you are trying to do then I believe your handling of the channels is incorrect. Either way, please read on to understand the issue as it is a common one in go.
You have a single channel which only blocks until it receives a single message, once it unblocks the function ends and the channel is closed (by defer).
You are trying to send to this channel from both your send and receive
loop.
When the last one to finish tries to send to the channel it will have been closed (by the first to finish) and the server will panic. Annoyingly, you wont actually see any sign of this as the server will exit before the goroutine can dump its panic (no clues - probably why you landed here)
see an example of the issue here (grpc code stripped out):
https://play.golang.org/p/GjfgDDAWNYr
Note: comment out the last pause in the main func to stop showing the panic reliably (as in your case)
So one simple fix would probably be to simply create two separate channels (one for send, one for receive) and block on both - this however would leave the send loop open necessarily if you don't get a chance to respond so probably better to structure like the example above unless you have good reason to pursue something different.
Another possibility is some sort server/request context mix up but I'm pretty sure the above will fix - drop an update with your server setup code if your still having issues after the above changes
I am using the chat app example from gorilla websockets, but I have an issue, sometimes, when the backend needs to send two different messages to the client, they are send in just one Message Event, this is bad for me because JSON.parse will fail parsing 2 jsons from one string.
I can do a split by newline and get every json from the message, but I prefer not to.
If I put a timeout on backend everything works.
Can I do something to prevent this? If not can you please explain me why?
This is the chat example:
https://github.com/gorilla/websocket/tree/master/examples/chat
This is my code where I broadcast 2 messages:
if err == nil {
c.SendMessageWithOrders(DB)
data := models.EventSuccess{
Event: utils.EventOrdersCreateSuccess,
}
toReturnBytes, err := json.Marshal(data)
if err == nil {
toReturn := BroadcastOne{
ID: c.ID,
Message: toReturnBytes,
}
NewHub.broadcastOne <- &toReturn
}
}
c.SendMessageWithOrders(DB) Is making NewHub.broadcastOne <- &toReturn with different data
The following code in client.go reduces the amount of data sent over the network by sending queued chat messages as a single WebSocket message:
n := len(c.send)
for i := 0; i < n; i++ {
w.Write(newline)
w.Write(<-c.send)
}
Fix the problem by deleting the code from the example. The optimization is not required.
I'm trying to start an HTTP server in Go, and when the server is started, a message should be printed, in case of an error, an error message should be printed.
Given the following code:
const (
HTTPServerPort = ":4000"
)
func main() {
var httpServerError = make(chan error)
var waitGroup sync.WaitGroup
setupHTTPHandlers()
waitGroup.Add(1)
go func() {
defer waitGroup.Done()
httpServerError <- http.ListenAndServe(HTTPServerPort, nil)
}()
if <-httpServerError != nil {
fmt.Println("The Logging API service could not be started.", <-httpServerError)
} else {
fmt.Println("Logging API Service running # http://localhost" + HTTPServerPort)
}
waitGroup.Wait()
}
When I do start the application, I don't see anything printed to the console, where I would like to see:
Logging API Service running # http://localhost:4000
When I change the port to an invalid one, the following output is printed to the console:
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan receive]:
main.main()
...app.go:45 +0x107
exit status 2
Could anyone point me in the right direction so that I know what I'm doing wrong with this implementation?
You can't do this unless you change the logic in your code or use Listen and Serve separately. Because ListenAndServe is a blocking function. If there something unexpected happens, it will return you an error. Provided it is not, it will keep blocking running the server. There is neither an event that is triggered whenever a server is started.
Let's run Listen and Serve separately then.
l, err := net.Listen("tcp", ":8080")
if err != nil {
// handle error
}
// Signal that server is open for business.
if err := http.Serve(l, rootHandler); err != nil {
// handle error
}
See https://stackoverflow.com/a/44598343/4792552.
P.S. net.Listen doesn't block because it runs in background. In other means, it simply spawns a socket connection in OS level and returns you with the details/ID of it. Thus, you use that ID to proxy orders to that socket.
The issue is that your if statement will always read from the httpServerError channel. However the only time something writes to that is if the server fails.
Try a select statement:
select{
case err := <-httpServerError
fmt.Println("The Logging API service could not be started.", err)
default:
fmt.Println("Logging API Service running # http://localhost" + HTTPServerPort
}
The default case will be ran if the channel does not have anything on it.
Notice this does not read from the channel twice like your example. Once you read a value from a channel, its gone. Think of it as a queue.
I'm new to goroutines and trying to work out the idiomatic way to organise this code. My program will generate async status events that I want to transmit to a server over a websocket. Right now I have a global channel messagesToServer to receive the status messages. The idea is it that will send the data if we currently have a websocket open, or quietly drop it if the connection to the server is currently closed or unavailable.
Relevant snippets are below. I don't really like the non-blocking send - if for some reason my writer goroutine took a while to process a message I think it could end up dropping a quick second message for no reason?
But if I use a blocking send, sendStatusToServer could block something that shouldn't be blocked if the connection is offline. I could try to track connected/disconnected state but if a message was sent at the same time as the disconnection occurred I think there would be a race condition.
Is there a tidy way I can write this?
var (
messagesToServer chan common.StationStatus
)
// ...
func sendStatusToServer(msg common.StationStatus) {
// Must be non-blocking in case we're not connected
select {
case messagesToServer <- msg:
break
default:
break
}
}
// ...
// after making websocket connection
log.Println("Connected to central server");
finished := make(chan struct{})
// Writer
go func() {
for {
select {
case msg := <-messagesToServer:
var buff bytes.Buffer
enc := gob.NewEncoder(&buff)
err = enc.Encode(msg)
conn.WriteMessage(websocket.BinaryMessage, buff.Bytes()); // ignore errors by design
case <-finished:
return;
}
}
}()
// Reader as busy loop on this goroutine
for {
messageType, p, err := conn.ReadMessage()
I am trying to write a RabbitMQ Consumer in Go. Which is suppose to take the 5 objects at a time from the queue and process them. Moreover, it is suppose to acknowledge if successfully processed else send to the dead-letter queue for 5 times and then discard, it should be running infinitely and handling the cancellation event of the consumer.
I have few questions :
Is there any concept of BasicConsumer vs EventingBasicConsumer in RabbitMq-go Reference?
What is Model in RabbitMQ and is it there in RabbitMq-go?
How to send the objects when failed to dead-letter queue and again re-queue them after ttl
What is the significance of consumerTag argument in the ch.Consume function in the below code
Should we use the channel.Get() or channel.Consume() for this scenario?
What are the changes i need to make in the below code to meet above requirement. I am asking this because i couldn't find decent documentation of RabbitMq-Go.
func main() {
consumer()
}
func consumer() {
objConsumerConn := &rabbitMQConn{queueName: "EventCaptureData", conn: nil}
initializeConn(&objConsumerConn.conn)
ch, err := objConsumerConn.conn.Channel()
failOnError(err, "Failed to open a channel")
defer ch.Close()
msgs, err := ch.Consume(
objConsumerConn.queueName, // queue
"demo1", // consumerTag
false, // auto-ack
false, // exclusive
false, // no-local
false, // no-wait
nil, // args
)
failOnError(err, "Failed to register a consumer")
forever := make(chan bool)
go func() {
for d := range msgs {
k := new(EventCaptureData)
b := bytes.Buffer{}
b.Write(d.Body)
dec := gob.NewDecoder(&b)
err := dec.Decode(&k)
d.Ack(true)
if err != nil { fmt.Println("failed to fetch the data from consumer", err); }
fmt.Println(k)
}
}()
log.Printf(" Waiting for Messages to process. To exit press CTRL+C ")
<-forever
}
Edited question:
I have delayed the processing of the messages as suggested in the links link1 link2. But the problem is messages are getting back to their original queue from dead-lettered queue even after ttl. I am using RabbitMQ 3.0.0. Can anyone point out what is the problem?
Is there any concept of BasicConsumer vs EventingBasicConsumer in
RabbitMq-go Reference?
Not exactly, but the Channel.Get and Channel.Consume calls serve a similar concept. With Channel.Get you have a non-blocking call that gets the first message if there's any available, or returns ok=false. With Channel.Consume the queued messages are delivered to a channel.
What is Model in RabbitMQ and is it there in RabbitMq-go?
If you're referring to the IModel and Connection.CreateModel in C# RabbitMQ, that's something from the C# lib, not from RabbitMQ itself. It was just an attempt to abstract away from the RabbitMQ "Channel" terminology, but it never caught on.
How to send the objects when failed to dead-letter queue and again
re-queue them after ttl
Use the delivery.Nack method with requeue=false.
What is the significance of consumerTag argument in the ch.Consume
function in the below code
The ConsumerTag is just a consumer identifier. It can be used to cancel the channel with channel.Cancel, and to identify the consumer responsible for the delivery. All messages delivered with the channel.Consume will have the ConsumerTag field set.
Should we use the channel.Get() or channel.Consume() for this scenario?
I think channel.Get() is almost never preferable over channel.Consume(). With channel.Get you'll be polling the queue and wasting CPU doing nothing, which doesn't make sense in Go.
What are the changes i need to make in the below code to meet above
requirement.
Since you're batch processing 5 at a time, you can have a goroutine that receives from the consumer channel and once it gets the 5 deliveries you call another function to process them.
To acknowledge or send to the dead-letter queue you'll use the delivery.Ack or delivery.Nack functions. You can use multiple=true and call it once for the batch. Once the message goes to the dead letter queue, you have to check the delivery.Headers["x-death"] header for how many times its been dead-lettered and call delivery.Reject when its been retried for 5 times already.
Use channel.NotifyCancel to handle the cancellation event.