I am seeing an inconsistency in the way unbuffered channels appear to work - this is either an inconsistency in Go, or in my understanding of Go...
Here is a simple example with output. The 'inconsistency' is with the 'make channel' lines.
package main
import (
"fmt"
)
func send(sendto chan string) {
fmt.Println("send 1")
sendto <- "Hello"
fmt.Println("send 2")
sendto <- "World"
fmt.Println("send 3")
sendto <- ""
fmt.Println("send() exit")
}
func main() {
//hole := make(chan string)
//hole := make(chan string, 0)
hole := make(chan string, 1)
go send(hole)
fmt.Println("main loop")
carryon := true
for carryon {
msg := <- hole
if msg == "" {
carryon = false
} else {
fmt.Println(" recd ", msg)
}
}
}
When I run as above, the output is as expected (and also as expected for a buffer size of 2). i.e. the channel has a buffer of 1, which holds one value - on next attempting to write, there is a context switch to the main to allow it to consume the first value.
main loop
send 1
send 2
recd Hello
send 3
recd World
send() exit
When I then change the make channel line to:
hole := make(chan string, 0)
The output is:
main loop
send 1
send 2
recd Hello
recd World
send 3
send() exit
I would have expected the send 2 and the recd Hello to be the other way around...
I get the same output for hole := make(chan string)
I checked the specification and it says
The capacity, in number of elements, sets the size of the buffer in the channel. If the capacity is zero or absent, the channel is unbuffered and communication succeeds only when both a sender and receiver are ready. Otherwise, the channel is buffered and communication succeeds without blocking if the buffer is not full (sends) or not empty (receives).
Please can someone explain either
Why my expectations are wrong - please be kind
or whether Go is actually wrong
Thank you
Roughly: Send and receive happen concurrently. The details are explained in the Go Memory Model which determines this behaviour. Concurrent code is complicated...
This timeline for the two goroutines shows what's going on:
send() main()
fmt.Println("send 1")
sendto <- "Hello" msg := <- hole // sender and receiver both ready
fmt.Println("send 2")
fmt.Println(" recd ", msg) // msg is "Hello"
sendto <- "World" msg := <- hole // sender and receiver both ready
fmt.Println(" recd ", msg) // msg is "World"
fmt.Println("send 3")
sendto <- ""
fmt.Println("send() exit")
send 2 is printed before recd Hello because send() runs to the print statement before the runtime schedules main() to run again.
There is no happens before relationship for printing the two messages. They can be printed in either order.
communication succeeds only when both a sender and receiver are ready
The key is that this does not require the receiver to immediately start processing the message it has received. Specifically in your case, it is ready, so it receives the value without invoking the scheduler (no context switch). The goroutine continues running until it tries to send again, at which point the receiver is not ready, so the scheduler is invoked etc.
Related
Is there a priority rule between sent and receive operation in a go select statement ?
Since a "send" operation is always ready, not like a "receive" operation that wait for something to come from the channel, I always have the feeling that the "send" will be executed first on a select.
I tried a little code to test what happens when both send and receive are ready:
package main
import (
"fmt"
"time"
)
func main() {
ch1 := make(chan string)
ch2 := make(chan string)
go goOne(ch1)
go goTwo(ch2)
time.Sleep(time.Second * 2)
select {
case ch2 <- "To goTwo goroutine":
case msg1 := <-ch1:
fmt.Println(msg1)
}
}
func goOne(ch chan string) {
ch <- "From goOne goroutine"
}
func goTwo(ch chan string) {
msg := <-ch
fmt.Println(msg)
}
The result seems to always be "From goOne goroutine". So it seems the receive operation has the priority.
But is it by design effect ? Or could it happen that the sent got executed first? I couldn't find the info in the doc
If I want the receive operation to have the priority, can I rely on that or should I do something else ?
Thanks!
Is there a priority rule between sent and receive operation in a go select statement?
No. When more than one case is ready at the same time, one at random is executed.
Since a "send" operation is always ready
Not true. A send operation may just block (i.e. not ready) when nothing is receiving on the other side, or when a buffered channel's buffer is full.
Or could it happen that the sent got executed first?
Yes, but you may see no output when this case is selected because your program resumes execution in main and exits immediately before the goTwo goroutine can actually print anything.
If I want the receive operation to have the priority [...]
The very semantics of a select statement are: "execute whichever is ready first". If one case must have priority over the other, change the other one to default (runs if nothing else is ready):
select {
case msg1 := <-ch1:
fmt.Println(msg1)
default:
ch2 <- "To goTwo goroutine"
}
I am new to golang and have a hard time understanding how the channels work. My understanding is that by default, channels are supposed to be blocking, so I expect a goroutine that writes into a channel to be frozen by the scheduler until an other goroutine reads the channel content. So I tried the following code, which gives the corresponding output:
package main
import (
"fmt"
"sync"
)
var wg sync.WaitGroup
var v int
func main() {
ch := make(chan string)
wg.Add(2)
go func(ch chan string) {
fmt.Println("Ready to receive")
for msg := range ch {
fmt.Println("received: ", msg)
fmt.Println(v)
}
wg.Done()
}(ch)
go func(ch chan string) {
fmt.Println("Will send the SMS to mama")
ch <- "msg 1"
v += 1
fmt.Println("Done! sent the message 1")
ch <- "msg 2"
v += 1
fmt.Println("Done! sent the message 2")
ch <- "msg 3"
v += 1
fmt.Println("Done! sent the message 3")
close(ch)
wg.Done()
}(ch)
wg.Wait()
}
output:
Will send the SMS to mama
Ready to receive
received: msg 1
0
Done! sent the message 1
Done! sent the message 2
received: msg 2
2
received: msg 3
2
Done! sent the message 3
I am a bit surprised as I was expecting the following order:
msg 1 sent
msg 1 received
msg 2 sent
msg 2 received
and so on. But this is obviously not the case.
Does someone have any idea on why Go behaves this way? Many thanks,
Here is a link to the code https://play.golang.org/p/O6SXf0CslPf. And here are my sources for stating what I said earlier: https://medium.com/rungo/anatomy-of-channels-in-go-concurrency-in-go-1ec336086adb https://rakyll.org/scheduler/
This behaviour is completely normal so to answer your question
so I expect a goroutine that writes into a channel to be frozen by the scheduler until an other goroutine reads the channel content
Schedular may or may not continue that same goroutine after value is sent through channel unless further synchronization is needed.
So for example After "msg 2" is sent to ch and is read in another goroutine in the following line
ch <- "msg 2"
goroutine can continue to execute v += 1 and call fmt.Println before other goroutine calls it.
Also calls to fmt.Println from different goroutines requires synchronization and possibly mutex calls which may also reorder print statements.
Further more there is data race on variable v
I have some code that is a job dispatcher and is collating a large amount of data from lots of TCP sockets. This code is a result of an approach to Large number of transient objects - avoiding contention and it largely works with CPU usage down a huge amount and locking not an issue now either.
From time to time my application locks up and the "Channel length" log is the only thing that keeps repeating as data is still coming in from my sockets. However the count remains at 5000 and no downstream processing is taking place.
I think the issue might be a race condition and the line it is possibly getting hung up on is channel <- msg within the select of the jobDispatcher. Trouble is I can't work out how to verify this.
I suspect that as select can take items at random the goroutine is returning and the shutdownChan doesn't have a chance to process. Then data hits inboundFromTCP and it blocks!
Someone might spot something really obviously wrong here. And offer a solution hopefully!?
var MessageQueue = make(chan *trackingPacket_v1, 5000)
func init() {
go jobDispatcher(MessageQueue)
}
func addMessage(trackingPacket *trackingPacket_v1) {
// Send the packet to the buffered queue!
log.Println("Channel length:", len(MessageQueue))
MessageQueue <- trackingPacket
}
func jobDispatcher(inboundFromTCP chan *trackingPacket_v1) {
var channelMap = make(map[string]chan *trackingPacket_v1)
// Channel that listens for the strings that want to exit
shutdownChan := make(chan string)
for {
select {
case msg := <-inboundFromTCP:
log.Println("Got packet", msg.Avr)
channel, ok := channelMap[msg.Avr]
if !ok {
packetChan := make(chan *trackingPacket_v1)
channelMap[msg.Avr] = packetChan
go processPackets(packetChan, shutdownChan, msg.Avr)
packetChan <- msg
continue
}
channel <- msg
case shutdownString := <-shutdownChan:
log.Println("Shutting down:", shutdownString)
channel, ok := channelMap[shutdownString]
if ok {
delete(channelMap, shutdownString)
close(channel)
}
}
}
}
func processPackets(ch chan *trackingPacket_v1, shutdown chan string, id string) {
var messages = []*trackingPacket_v1{}
tickChan := time.NewTicker(time.Second * 1)
defer tickChan.Stop()
hasCheckedData := false
for {
select {
case msg := <-ch:
log.Println("Got a messages for", id)
messages = append(messages, msg)
hasCheckedData = false
case <-tickChan.C:
messages = cullChanMessages(messages)
if len(messages) == 0 {
messages = nil
shutdown <- id
return
}
// No point running checking when packets have not changed!!
if hasCheckedData == false {
processMLATCandidatesFromChan(messages)
hasCheckedData = true
}
case <-time.After(time.Duration(time.Second * 60)):
log.Println("This channel has been around for 60 seconds which is too much, kill it")
messages = nil
shutdown <- id
return
}
}
}
Update 01/20/16
I tried to rework with the channelMap as a global with some mutex locking but it ended up deadlocking still.
Slightly tweaked the code, still locks but I don't see how this one does!!
https://play.golang.org/p/PGpISU4XBJ
Update 01/21/17
After some recommendations I put this into a standalone working example so people can see. https://play.golang.org/p/88zT7hBLeD
It is a long running process so will need running locally on a machine as the playground kills it. Hopefully this will help get to the bottom of it!
I'm guessing that your problem is getting stuck doing this channel <- msg at the same time as the other goroutine is doing shutdown <- id.
Since neither the channel nor the shutdown channels are buffered, they block waiting for a receiver. And they can deadlock waiting for the other side to become available.
There are a couple of ways to fix it. You could declare both of those channels with a buffer of 1.
Or instead of signalling by sending a shutdown message, you could do what Google's context package does and send a shutdown signal by closing the shutdown channel. Look at https://golang.org/pkg/context/ especially WithCancel, WithDeadline and the Done functions.
You might be able to use context to remove your own shutdown channel and timeout code.
And JimB has a point about shutting down the goroutine while it might still be receiving on the channel. What you should do is send the shutdown message (or close, or cancel the context) and continue to process messages until your ch channel is closed (detect that with case msg, ok := <-ch:), which would happen after the shutdown is received by the sender.
That way you get all of the messages that were incoming until the shutdown actually happened, and should avoid a second deadlock.
I'm new to Go but in this code here
case msg := <-inboundFromTCP:
log.Println("Got packet", msg.Avr)
channel, ok := channelMap[msg.Avr]
if !ok {
packetChan := make(chan *trackingPacket_v1)
channelMap[msg.Avr] = packetChan
go processPackets(packetChan, shutdownChan, msg.Avr)
packetChan <- msg
continue
}
channel <- msg
Aren't you putting something in channel (unbuffered?) here
channel, ok := channelMap[msg.Avr]
So wouldn't you need to empty out that channel before you can add the msg here?
channel <- msg
Like I said, I'm new to Go so I hope I'm not being goofy. :)
Why is it not sending on the channel and blocking the execution? How can I make this constellation work so that I can send a signal into MoneyDive() and continue execution?
package main
import (
"fmt"
)
type Quack func(ch chan bool)
type DagobertDuck struct {
quack Quack
}
func (self *DagobertDuck) MoneyDive() {
ch := make(chan bool)
self.quack(ch)
b := <-ch
if b {
fmt.Println("true")
} else {
fmt.Println("false")
}
}
func mockQuack(ch chan bool) {
fmt.Println("mockQuack start")
ch <- true
fmt.Println("mockQuack done")
}
func main() {
dd := DagobertDuck{quack: mockQuack}
dd.MoneyDive()
}
https://play.golang.org/p/1omlb7u6-A
Because you have an unbuffered channel, and you can only send a value on an unbuffered channel without blocking if there is another goroutine which is ready to receive from it.
Since you only have 1 goroutine, it gets blocked. Solution is simple: launch your Quack.quack() method in a new goroutine:
go self.quack(ch)
Then the output (try it on the Go Playground):
mockQuack start
mockQuack done
true
Another option is to not launch a new goroutine but make a buffered channel, so it can hold some values without any receivers ready to receive from it:
ch := make(chan bool, 1) // buffered channel, buffer for 1 value
This creates a channel which is capable of "storing" one value without any receivers ready to receive it. A second send on the channel would also block, unless the value is received from it first (or a receiver ready to receive a value from it).
Try this buffered channel version on the Go Playground.
Relevant section from the spec: Send statements:
Both the channel and the value expression are evaluated before communication begins. Communication blocks until the send can proceed. A send on an unbuffered channel can proceed if a receiver is ready. A send on a buffered channel can proceed if there is room in the buffer. A send on a closed channel proceeds by causing a run-time panic. A send on a nil channel blocks forever.
Notes:
Based on the received value you print true or false. This can be done with a single line, without the if statement:
fmt.Println(b)
You can even get rid of the b local variable, and print the received value right away:
fmt.Println(<-ch)
Also I assume you used channels because you wanted to play with them, but in your case mockQuack() could simply return the bool value, without the use of channels.
I am trying to understand why my code does not work, so I managed to recreate the problem in a simpler example.
I expected this code to output the string "broadcasted", but it does not output anything.
package main
import (
"fmt"
"time"
)
type hub struct {
handle chan []byte
broadcast chan []byte
}
func (h *hub) init() {
for {
select {
case m := <-h.handle:
handler(m)
case _ = <-h.broadcast:
fmt.Println("broadcasted")
}
}
}
var socketHub = hub{
handle: make(chan []byte),
broadcast: make(chan []byte),
}
func main() {
go socketHub.init()
m := []byte("this is the message")
socketHub.handle <- m
time.Sleep(time.Second * 2)
}
func handler(m []byte) {
// Do some stuff.
socketHub.broadcast <- m
}
Why does this not work?
Your broadcast channel is unbuffered. This means that:
You send the message to the handle channel: main goroutine blocks until...
In the goroutine, the select case gets the message...
And calls (in the goroutine) handler, which sends the message to the broadcast channel, blocking until...
And that's it: your child goroutine is blocking, waiting on itself to pick the message. Concurrently, your main goroutine sleeps, then reaches the end of main, exits and terminates the program.
You can 'solve' it in several ways:
Make your broadcast channel buffered: this way, goroutine sends the message, which succeeds immediately, and returns in the for loop, selecting it back and printing as intended
Make your send a (new) goroutine, either in handler, or by calling go handler(m) in your loop
Have two different goroutines listen to handler and broadcast
Which one you choose depends on the exact problem you try to solve: in this tiny example, it's hard to find a 'best' one.