I'm new to goroutines and trying to work out the idiomatic way to organise this code. My program will generate async status events that I want to transmit to a server over a websocket. Right now I have a global channel messagesToServer to receive the status messages. The idea is it that will send the data if we currently have a websocket open, or quietly drop it if the connection to the server is currently closed or unavailable.
Relevant snippets are below. I don't really like the non-blocking send - if for some reason my writer goroutine took a while to process a message I think it could end up dropping a quick second message for no reason?
But if I use a blocking send, sendStatusToServer could block something that shouldn't be blocked if the connection is offline. I could try to track connected/disconnected state but if a message was sent at the same time as the disconnection occurred I think there would be a race condition.
Is there a tidy way I can write this?
var (
messagesToServer chan common.StationStatus
)
// ...
func sendStatusToServer(msg common.StationStatus) {
// Must be non-blocking in case we're not connected
select {
case messagesToServer <- msg:
break
default:
break
}
}
// ...
// after making websocket connection
log.Println("Connected to central server");
finished := make(chan struct{})
// Writer
go func() {
for {
select {
case msg := <-messagesToServer:
var buff bytes.Buffer
enc := gob.NewEncoder(&buff)
err = enc.Encode(msg)
conn.WriteMessage(websocket.BinaryMessage, buff.Bytes()); // ignore errors by design
case <-finished:
return;
}
}
}()
// Reader as busy loop on this goroutine
for {
messageType, p, err := conn.ReadMessage()
Related
I was following the chat client/server example for the gorilla websocket library.
https://github.com/gorilla/websocket/blob/master/examples/chat/hub.go#L36
I tried modifying the code to notify other clients when a new client connects, like so:
for {
select {
case client := <-h.register:
h.clients[client] = true
// My addition. Hangs after this (no further register/unregister events are processed):
h.broadcast <- []byte("Another client connected!")
case client := <-h.unregister:
if _, ok := h.clients[client]; ok {
delete(h.clients, client)
close(client.send)
}
case message := <-h.broadcast:
for client := range h.clients {
select {
case client.send <- message:
default:
close(client.send)
delete(h.clients, client)
}
}
}
}
}
My understanding is on the next iteration of the outer for loop, the broadcast channel should receive that data and follow the logic in the message case, but it just hangs.
Why? I can't spot any reason. No further channel events are processed (nothing on register/unregister or broadcast), which makes me think it's some kind of unbuffered channel mechanism it's stuck on, but I don't see how?
The hub's broadcast channel is unbuffered. Communication on an unbuffered channel waits for a ready sender and a ready receiver. The hub goroutine blocks because the goroutine cannot be ready to send and receive at the same time.
Changing the channel from an unbuffered channel to a buffered channel does not fix the problem. Consider the case where the buffer capacity is one:
return &Hub{
broadcast: make(chan []byte, 1),
...
}
with this timeline:
1 clientA: client.hub.register <- client
2 clientB: c.hub.broadcast <- message
3 hub: case client := <-h.register:
4 hub: h.broadcast <- []byte("Another client connected!")
The hub blocks at #4 because the channel was filled to capacity at #2. Increasing the channel capacity to two or more does not fix the problem because any number of clients can broadcast a message while another client is registering.
To fix the problem, move the broadcast code to a function and call that function from both cases in the select:
// sendAll sends message to all registered clients.
// This method must only be called by Hub.run.
func (h *Hub) sendAll(message []byte) {
for client := range h.clients {
select {
case client.send <- message:
default:
close(client.send)
delete(h.clients, client)
}
}
}
func (h *Hub) run() {
for {
select {
case client := <-h.register:
h.clients[client] = true
h.sendAll([]byte("Another client connected!"))
case client := <-h.unregister:
if _, ok := h.clients[client]; ok {
delete(h.clients, client)
close(client.send)
}
case message := <-h.broadcast:
h.sendAll(message)
}
}
}
Your channels are unbuffered, this means that each read/write blocks until an other goroutine performs the opposite operation on the same channel.
When you try to write to h.broadcast the goroutine stops, waiting for a reader. But the same goroutine is supposed to act as a reader of this channel, which never happens because the goroutine is blocked by the write. Thus the program deadlock.
Yea, this won't work. You cannot send/receive on the same unbuffered channel in the same go routine.
The line h.broadcast <- []byte("Another client connected!") blocks until another go routine pops off that queue. An easy solution is to make the broadcast channel have a length 1 buffer. broadcast := make(chan []byte, 1)
You can see it in this playground example
// c := make(chan int) <- This will hang
c := make(chan int, 1)
c <- 1
fmt.Println(<-c)
Remove the length 1 buffer and the whole system deadlocks. One issue you can run into is if 2 clients register at the same time, then you can have the case where 2 items are trying to be stuffed into the broadcast channel, and we're back to the same problem with the unbuffered channel. You can avoid this and keep 1 go routine like so:
for {
select {
case message := <-h.broadcast:
// ...
default:
}
select { // This select statement can only add 1 item to broadcast at most
case client := <-h.register:
// ...
h.broadcast <- []byte("Another client connected!")
}
}
}
However, this will still break if another go routine is also adding to the broadcast channel. So I'd go with Cerise Limon's solution, or buffer the channel enough that other go routines won't ever fill the buffer.
I have code that looks like this, where I'm listening to a channel up until a timeout interval. Let's say this goroutine 1
select {
case <-time.After(TimeoutInterval):
mu.Lock()
defer mu.Unlock()
delete(msgChMap, index)
return ""
case msg := <-msgCh:
return msg
}
Elsewhere, I have a goroutine 2 that runs something like this where it grabs the appropriate msgCh from a Map, deletes the entry in the map and then sends a message through the channel.
mu.Lock()
msgCh, ok := msgChMap[index]
delete(msgChMap, index)
mu.Unlock()
if ok {
msgCh <- "yay"
}
It seems like it is possible for me to grab the message channel msgCh from the Map, try to send a message but because TimeoutInterval has already passed, there will be nothing listening to the channel, and my code will get stuck waiting for a listener. If I put the lock after sending yay to the msgCh, it seems possible that I could deadlock as 2 will be waiting for a listener to the channel and is not releasing the lock, but 1 is no longer listening but requires the lock.
What is a general pattern to avoid getting stuck waiting for a listener? Perhaps go is smart enough to not get stuck here.
You can prevent getting stuck when waiting for a listener by using select for the sender.
By using select you can use more case for sender in this situation
mu.Lock()
msgCh, ok := msgChMap[index]
delete(msgChMap, index)
mu.Unlock()
if ok {
select {
// listener is available
case msgCh <- "yay":
fmt.Println("sent")
// if not avalable (execute immediately)
default:
fmt.Println("no available listener")
// ...just ignore or do something else
}
}
Or waiting for a short time
mu.Lock()
msgCh, ok := msgChMap[index]
delete(msgChMap, index)
mu.Unlock()
if ok {
select {
// listener is available
case msgCh <- "yay":
fmt.Println("sent")
// if not available, waiting for listener
case <-time.After(30 * time.Second):
fmt.Println("after 30 seconds, still no available listener")
// ...just ignore or do something else
}
}
The problem here is that the channel reader my stop without the writer knowing it. It should be possible to structure this solution so that this situation never happens, but ignoring that for now, for this specific problem what you need is atomic access to the channel itself, along with a flag for channel status:
type channel struct {
sync.Mutex
msgCh chan Msg
active bool
}
Writing to the channel is now done by locking it:
ch.Lock()
if ch.active {
ch.msgCh<-data
}
ch.Unlock()
And when you "inactivate" the channel, reset the flag:
case <-time.After(TimeoutInterval):
mu.Lock()
defer mu.Unlock()
ch.Lock()
defer ch.Unlock()
delete(msgChMap, index)
ch.active=false
return ""
And of course, with this now you have to keep a *channel in your map.
I am trying to create an intermediate layer between user and tcp, with Send and Receive functions. Currently, I am trying to integrate a context, so that the Send and Receive respects a context. However, I don't know how to make them respect the context's cancellation.
Until now, I got the following.
// c.underlying is a net.Conn
func (c *tcpConn) Receive(ctx context.Context) ([]byte, error) {
if deadline, ok := ctx.Deadline(); ok {
// Set the read deadline on the underlying connection according to the
// given context. This read deadline applies to the whole function, so
// we only set it once here. On the next read-call, it will be set
// again, or will be reset in the else block, to not keep an old
// deadline.
c.underlying.SetReadDeadline(deadline)
} else {
c.underlying.SetReadDeadline(time.Time{}) // remove the read deadline
}
// perform reads with
// c.underlying.Read(myBuffer)
return frameData, nil
}
However, as far as I understand that code, this only respects a context.WithTimeout or context.WithDeadline, and not a context.WithCancel.
If possible, I would like to pass that into the connection somehow, or actually abort the reading process.
How can I do that?
Note: If possible, I would like to avoid another function that reads in another goroutine and pushed a result back on a channel, because then, when calling cancel, and I am reading 2GB over the network, that doesn't actually cancel the read, and the resources are still used. If not possible in another way however, I would like to know if there is a better way of doing that than a function with two channels, one for a []byte result and one for an error.
EDIT:
With the following code, I can respect a cancel, but it doesn't abort the read.
// apply deadline ...
result := make(chan interface{})
defer close(result)
go c.receiveAsync(result)
select {
case res := <-result:
if err, ok := res.(error); ok {
return nil, err
}
return res.([]byte), nil
case <-ctx.Done():
return nil, ErrTimeout
}
}
func (c *tcpConn) receiveAsync(result chan interface{}) {
// perform the reads and push either an error or the
// read bytes to the result channel
If the connection can be closed on cancellation, you can setup a goroutine to shutdown the connection on cancellation within the Receive method. If the connection must be reused again later, then there is no way to cancel a Read in progress.
recvDone := make(chan struct{})
defer close(recvDone)
// setup the cancellation to abort reads in process
go func() {
select {
case <-ctx.Done():
c.underlying.CloseRead()
// Close() can be used if this isn't necessarily a TCP connection
case <-recvDone:
}
}()
It will be a little more work if you want to communicate the cancelation error back, but the CloseRead will provide a clean way to stop any pending TCP Read calls.
I have a websocket client which spins a goroutine, using Conn.ReadJSON to read incoming JSON messages.
I would like the goroutine to be able to nicely react to ctx.Done(), hence I wrote the following code:
for {
msg := new(message)
select {
case <- ctx.Done():
fmt.Println("Halting gracefully")
return
default:
err := Conn.ReadJSON(&msg)
if err != nil {
fmt.Println(err)
break
}
c.inbound <- *msg
}
}
Obviously, in the current state, ReadJSON is a non-blocking function, hence it is launched and then the execution stops somewhere within (not sure where) as it waits for a message to be received.
This means that it will never block at the select statement and hence it will never handle ctx.Done() appropriately.
Gorilla documentation doesn't show any case <- IsThereAnyNewMessage() available.
How can it be handled elegantly?
I have some code that is a job dispatcher and is collating a large amount of data from lots of TCP sockets. This code is a result of an approach to Large number of transient objects - avoiding contention and it largely works with CPU usage down a huge amount and locking not an issue now either.
From time to time my application locks up and the "Channel length" log is the only thing that keeps repeating as data is still coming in from my sockets. However the count remains at 5000 and no downstream processing is taking place.
I think the issue might be a race condition and the line it is possibly getting hung up on is channel <- msg within the select of the jobDispatcher. Trouble is I can't work out how to verify this.
I suspect that as select can take items at random the goroutine is returning and the shutdownChan doesn't have a chance to process. Then data hits inboundFromTCP and it blocks!
Someone might spot something really obviously wrong here. And offer a solution hopefully!?
var MessageQueue = make(chan *trackingPacket_v1, 5000)
func init() {
go jobDispatcher(MessageQueue)
}
func addMessage(trackingPacket *trackingPacket_v1) {
// Send the packet to the buffered queue!
log.Println("Channel length:", len(MessageQueue))
MessageQueue <- trackingPacket
}
func jobDispatcher(inboundFromTCP chan *trackingPacket_v1) {
var channelMap = make(map[string]chan *trackingPacket_v1)
// Channel that listens for the strings that want to exit
shutdownChan := make(chan string)
for {
select {
case msg := <-inboundFromTCP:
log.Println("Got packet", msg.Avr)
channel, ok := channelMap[msg.Avr]
if !ok {
packetChan := make(chan *trackingPacket_v1)
channelMap[msg.Avr] = packetChan
go processPackets(packetChan, shutdownChan, msg.Avr)
packetChan <- msg
continue
}
channel <- msg
case shutdownString := <-shutdownChan:
log.Println("Shutting down:", shutdownString)
channel, ok := channelMap[shutdownString]
if ok {
delete(channelMap, shutdownString)
close(channel)
}
}
}
}
func processPackets(ch chan *trackingPacket_v1, shutdown chan string, id string) {
var messages = []*trackingPacket_v1{}
tickChan := time.NewTicker(time.Second * 1)
defer tickChan.Stop()
hasCheckedData := false
for {
select {
case msg := <-ch:
log.Println("Got a messages for", id)
messages = append(messages, msg)
hasCheckedData = false
case <-tickChan.C:
messages = cullChanMessages(messages)
if len(messages) == 0 {
messages = nil
shutdown <- id
return
}
// No point running checking when packets have not changed!!
if hasCheckedData == false {
processMLATCandidatesFromChan(messages)
hasCheckedData = true
}
case <-time.After(time.Duration(time.Second * 60)):
log.Println("This channel has been around for 60 seconds which is too much, kill it")
messages = nil
shutdown <- id
return
}
}
}
Update 01/20/16
I tried to rework with the channelMap as a global with some mutex locking but it ended up deadlocking still.
Slightly tweaked the code, still locks but I don't see how this one does!!
https://play.golang.org/p/PGpISU4XBJ
Update 01/21/17
After some recommendations I put this into a standalone working example so people can see. https://play.golang.org/p/88zT7hBLeD
It is a long running process so will need running locally on a machine as the playground kills it. Hopefully this will help get to the bottom of it!
I'm guessing that your problem is getting stuck doing this channel <- msg at the same time as the other goroutine is doing shutdown <- id.
Since neither the channel nor the shutdown channels are buffered, they block waiting for a receiver. And they can deadlock waiting for the other side to become available.
There are a couple of ways to fix it. You could declare both of those channels with a buffer of 1.
Or instead of signalling by sending a shutdown message, you could do what Google's context package does and send a shutdown signal by closing the shutdown channel. Look at https://golang.org/pkg/context/ especially WithCancel, WithDeadline and the Done functions.
You might be able to use context to remove your own shutdown channel and timeout code.
And JimB has a point about shutting down the goroutine while it might still be receiving on the channel. What you should do is send the shutdown message (or close, or cancel the context) and continue to process messages until your ch channel is closed (detect that with case msg, ok := <-ch:), which would happen after the shutdown is received by the sender.
That way you get all of the messages that were incoming until the shutdown actually happened, and should avoid a second deadlock.
I'm new to Go but in this code here
case msg := <-inboundFromTCP:
log.Println("Got packet", msg.Avr)
channel, ok := channelMap[msg.Avr]
if !ok {
packetChan := make(chan *trackingPacket_v1)
channelMap[msg.Avr] = packetChan
go processPackets(packetChan, shutdownChan, msg.Avr)
packetChan <- msg
continue
}
channel <- msg
Aren't you putting something in channel (unbuffered?) here
channel, ok := channelMap[msg.Avr]
So wouldn't you need to empty out that channel before you can add the msg here?
channel <- msg
Like I said, I'm new to Go so I hope I'm not being goofy. :)