Buffered channel and closing it - go

I have a snippet of code that I am trying to understand based on how I put the close call and the location
func main() {
ch := make(chan int, 2)
go func(ch chan int) {
for i := 1; i <= 5; i++ {
ch <- i
fmt.Println("Func goroutine sends data: ", i)
}
//Pos1 - Works perfectly
//close(ch)
}(ch)
fmt.Println("Main goroutine sleeps 2 seconds")
time.Sleep(time.Second * 2)
fmt.Println("Main goroutine begins receiving data")
//Pos2 - Puts in only 2 ints 1 and 2 and then prints only that
//close(ch)
for d := range ch {
fmt.Println("Main goroutine received data:", d)
}
//Pos3 - Throws fatal error
close(ch)
}
I have been trying to understand and read blogs on this but not able to understand somethings still
When I place the close at Pos1, it works fine. But I am not sure why
it works. The buffer cannot hold more than 2 elements at any given
time, so when 2 elements are written, the loop will block until the
main routing does a read. But I thought to do a range over a
buffered channel, the range function has to know beforehand how many
elements to iterate over and for that channel must be closed. Why
does close work at this position?
When I put it as position 2, it prints only 2 elements which kind of makes sense but why is the for loop not throwing an exception when it is trying to write more elements to the channel that is closed?
When I close at Pos3, I get an exception fatal error: all goroutines are asleep - deadlock! though all 5 ints are printed. Why is that?

But I thought to do a range over a buffered channel, the range function has to know beforehand how many elements to iterate over and for that channel must be closed. 
That assumption is wrong and the root of all misunderstandings.
The behavior of ranging over a channel is described in the Go Spec: https://golang.org/ref/spec#For_statements
For channels, the iteration values produced are the successive values sent on the channel until the channel is closed. If the channel is nil, the range expression blocks forever.
The channel does not need to be closed when the for statement is evaluated and the statement does not need to know number of elements.
So, in your code, when you put close in Pos1, it is indeed the right way to do it. When you put it in Pos3, the for loop waits the channel to be closed, which can only happen after the for loop itself, so it is a deadlock.
Putting close in Pos2 is buggy and the behavior is a little tricky. It is possible to raise an error, yet it is possible to just output two numbers. This is because when the channel is closed before the for loop, the loop can run without block and then main() returns. And when main() returns, the Go program ends. Whether it raises an error solely depends on if the scheduler switch to goroutine between the process, which is not guaranteed.

Modifying the code snippets with some peppered print statements to add on to the excellent answer by #leaf bebop, to clarify what's happening.
Pos 2, but make main wait a bit
func main() {
ch := make(chan int, 2)
go func(ch chan int) {
for i := 1; i <= 5; i++ {
ch <- i
fmt.Println("Func goroutine sends data: ", i)
}
fmt.Println("goR work is done")
}(ch)
fmt.Println("Main goroutine sleeps 2 seconds")
time.Sleep(time.Second * 2)
fmt.Println("Main goroutine begins receiving data")
close(ch)
for d := range ch {
fmt.Println("Main goroutine received data:", d)
}
fmt.Println("Main waiting, will the other goR panic?")
time.Sleep(time.Second * 10)
}
Main goroutine sleeps 2 seconds
Func goroutine sends data: 1
Func goroutine sends data: 2
Main goroutine begins receiving data
Main goroutine received data: 1
panic: send on closed channel
goroutine 6 [running]:
main.main.func1(0x0?)
/tmp/sandbox3746599466/prog.go:15 +0x3b
created by main.main
/tmp/sandbox3746599466/prog.go:13 +0x85
Program exited.
So, given enough time (main routine didn't exit), the scheduler went back to the go routine, where we wrote to a closed (closed by main) channel - Panic!
Pos 2, but only send 2 values, i.e not writing to a closed channel
func main() {
ch := make(chan int, 2)
go func(ch chan int) {
// Send only 2 values
for i := 1; i <= 2; i++ {
ch <- i
fmt.Println("Func goroutine sends data: ", i)
}
fmt.Println("goR work is done")
}(ch)
fmt.Println("Main goroutine sleeps 2 seconds")
time.Sleep(time.Second * 2)
fmt.Println("Main goroutine begins receiving data")
close(ch)
for d := range ch {
fmt.Println("Main goroutine received data:", d)
}
fmt.Println("Main waiting, will the other goR panic?")
time.Sleep(time.Second * 10)
fmt.Println("Main exiting")
}
Main goroutine sleeps 2 seconds
Func goroutine sends data: 1
Func goroutine sends data: 2
goR work is done
Main goroutine begins receiving data
Main goroutine received data: 1
Main goroutine received data: 2
Main waiting, will the other goR panic?
Main exiting
Program exited.
No panic, no spinning for ever. Main routine just exits.
Pos 3, check if main got unblocked?
func main() {
ch := make(chan int, 2)
go func(ch chan int) {
for i := 1; i <= 5; i++ {
ch <- i
fmt.Println("Func goroutine sends data: ", i)
}
fmt.Println("Routine done with work")
}(ch)
fmt.Println("Main goroutine sleeps 2 seconds")
time.Sleep(time.Second * 2)
fmt.Println("Main goroutine begins receiving data")
for d := range ch {
fmt.Println("Main goroutine received data:", d)
}
fmt.Println("Have I closed this channel?")
close(ch)
fmt.Println("I have")
}
Main goroutine sleeps 2 seconds
Func goroutine sends data: 1
Func goroutine sends data: 2
Main goroutine begins receiving data
Main goroutine received data: 1
Main goroutine received data: 2
Main goroutine received data: 3
Func goroutine sends data: 3
Func goroutine sends data: 4
Func goroutine sends data: 5
Routine done with work
Main goroutine received data: 4
Main goroutine received data: 5
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan receive]:
main.main()
/tmp/sandbox3301646152/prog.go:26 +0x194
Program exited.
Main never got even to close the channel, it was blocked in the range loop. It'll stay here forever, until it gets a value, or someone closes the channel. We don't have any other routine to do that, so deadlock.

Related

Gorountines causing a deadlock

I am trying my hands on goroutines and came up this example - https://go.dev/play/p/mWHUmALk-1_K
But I am having this error - fatal error: all goroutines are asleep - deadlock!
I have tried to fix this but no luck. Please how do I fix this?
The error seem to be on lines 15, 23 and 32.
The problem is that your program starts 3 separate goroutines that send to the same channel. And you have only the main goroutine receive from that channel only once. This causes the second channel send (ch <- fmt.Sprintf("...) to block indefinitely. With unbuffered channels you need to do as many receives as you do sends.
One approach to ensure all sends are received would be to use a range loop over the channel.
func getLength(dd []string, wg *sync.WaitGroup) {
wg.Add(len(dd))
c := make(chan string)
for _, d := range dd {
d1 := d
go computeLength(d1, c, wg)
}
// close c once all goroutines are done to
// ensure the for-range loop below exits.
go func() { wg.Wait(); close(c) }()
// Use for-range loop on the channel to receive all the sends.
//
// But note that a for-range loop over a channel exits only
// when the channel is closed or the loop is exited from within.
//
// So to exit you can close c once wg.Wait() returns,
// that's why there's that extra goroutine above.
for v := range c {
fmt.Println(v)
}
}
https://go.dev/play/p/BUb7NHrq2B0

golang why can not pass value to channel on the main thread

case1
package main
func main() {
dogChan := make(chan int)
dogChan <- 1
}
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.main()
/Users/xuzhongwei/Source/awesomeProject/main.go:5 +0x50
case2
package main
func main() {
dogChan := make(chan int)
go func(ch chan int) {
}(dogChan)
dogChan <- 1
}
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.main()
/Users/xuzhongwei/Source/awesomeProject/main.go:9 +0x72
case3
package main
func main() {
dogChan := make(chan int)
go func(ch chan int) {
<- ch
}(dogChan)
dogChan <- 1
}
case4
package main
func main() {
dogChan := make(chan int)
go func(ch chan int) {
<- ch
}(dogChan)
dogChan <- 1
dogChan <- 2
}
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.main()
/Users/xuzhongwei/Source/awesomeProject/main.go:10 +0x90
case5
package main
func main() {
dogChan := make(chan int)
go func(ch chan int) {
for {
select {
case <- ch:
}
}
}(dogChan)
dogChan <- 1
dogChan <- 2
dogChan <- 3
dogChan <- 4
dogChan <- 5
}
Could anyone tell me why case1, case2 got errors while case3 is ok?
In case1 my guess is that dogChan is not used in goroutine so it is treated to be closed.
In case2 my guess is that although dogChan is passed in goroutine but it is not used in goroutine so it is treated to be closed
Could anyone tell me why case4 got errors while case5 is ok?
Why do you think thats happening in case1 and case2? The channels are meant to behave as a synchronisation primitive between a sender and receiver. You have a sender sending on a channel, dogChan but none is receiving on it.
Without a receiving goroutine or a receive operation on a goroutine, the sender simply blocks (being a unbuffered channel)
Same problem on case4, you have two sends on the channel, but a single receive on the goroutine. The dogChan <- 2 will block forever. In case5, if your intention was to read from the channel, simply use a range loop to iterate over the successive values sent over it.
Golang expects program to read message(s) placed into channel.
Consumer (reader) needs to drain (read) all messages from channel, either using simple for-read, or for-select. Channel send and receive both block until sender and receiver are ready.
case1, case2 = send one message to channel, block awaiting reader, read zero messages
case4 = send one message to channel, block awaiting reader, reader does not consume (read) message
case3 = send one message to channel, consume one message from channel, sender blocks awaiting reader
case5 = send five messages to channel, consume all (five) messages, each send blocks until reader receives
// for range over channel
for msg := range ch {
// process msg
}
// for select
done := false
for !done {
select {
case msg := <-ch: {
// process msg
}
case ch == nil: {
done = true
}
}
}
// producer should close channel
close(ch)
Note:
channel can be buffered, specify a channel (queue) size
channel size default = 1 (unbuffered), writer blocks when channel full

Code for stopping a loop doesn't work

I am trying to implement a stop to a loop in Go.
The inspiration from the code I have right now is from here:
how to kill goroutine
However, I've not been able to get my code to behave as expected. The simplified version of my code from a complex code base is this:
package main
import (
"fmt"
"time"
)
var quit chan struct{}
var send chan int
func quitroutine() {
for {
select {
case cnt := <-send:
fmt.Println(cnt)
if cnt == 5 {
quit <- struct{}{}
}
}
}
}
func runLoop() {
cnt := 0
for {
select {
case <-quit:
fmt.Println("quit!")
default:
fmt.Println("default")
}
fmt.Println("inloop")
time.Sleep(1 * time.Second)
cnt++
send <- cnt
}
}
func main() {
quit = make(chan struct{})
send = make(chan int)
go quitroutine()
runLoop()
fmt.Println("terminated")
}
This code crashes:
default
inloop
5
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.runLoop()
/tmp/t.go:37 +0x1a6
main.main()
/tmp/t.go:45 +0xa4
goroutine 5 [chan send]:
main.quitroutine()
/tmp/t.go:18 +0x10e
created by main.main
/tmp/t.go:44 +0x9f
exit status 2
Questions:
Why does it crash at all after cnt is 5? quitroutine only writes to the quitchannel if cnt == 5, but doesn't terminate itself. While runLoop, if it receives on the quit channel, should just print "quit!" (which it doesn't), but not terminate itself.
Why don't I get the "quit!" output? Do I even get the quit channel?
How does this need to be implemented correctly
Just as Adrian has said, one of your goroutines is trying to send on quit, while the other is trying to send on send. To answer your questions:
When cnt == 5 the quitroutine begins attempting to send on quit. Because quit <- struct{}{} is not a select case, the goroutine will block until another tries to read from quit. The other goroutine is similarly stuck trying to do send <- cnt (when cnt = 6).
You never get the "quit!" output because that goroutine is stuck trying to do send <-cnt.
The simplest solution I see would be to adjust runLoop() so that the send <- cnt is a case in the select.
I'd change runLoop() to look like this:
func runLoop() {
cnt := 0
for {
select {
case <-quit:
fmt.Println("quit!")
case send <- cnt: // moved stuff here
fmt.Println("inloop")
time.Sleep(1 * time.Second)
cnt++
default:
fmt.Println("default")
}
// stuff used to be here
}
}
This gives me output (until I killed the program):
default
inloop
0
inloop
1
inloop
2
inloop
3
inloop
4
inloop
5
quit!
default
inloop
6
inloop
7
Which seems to mostly be what you were after.
I'd also like to note that the select block in quitroutine() is unnecessary because it only has one case. Cleaning that up may make it more clear that that goroutine is stuck trying to send, and never takes the input from the send channel.
When you try to send on the quit channel, the quitroutine blocks until something reads from it.
At the same time, the main routine, in runloop, is trying to send the next number on the send channel. This also blocks, because the routine that would be reading from it is currently blocked trying to send on the quit channel.
Both routines are blocked, which is a deadlock, so the program crashes.
This could be fixed by either putting one or both channel send in a select, or making one or both channels buffered (even a buffer length of 1 would suffice).

How can I stop the goroutine based on the returned value from that goroutine

Like here I created a go playground sample: sGgxEh40ev, but cannot get it work.
quit := make(chan bool)
res := make(chan int)
go func() {
idx := 0
for {
select {
case <-quit:
fmt.Println("Detected quit signal!")
return
default:
fmt.Println("goroutine is doing stuff..")
res <- idx
idx++
}
}
}()
for r := range res {
if r == 6 {
quit <- true
}
fmt.Println("I received: ", r)
}
Output:
goroutine is doing stuff..
goroutine is doing stuff..
I received: 0
I received: 1
goroutine is doing stuff..
goroutine is doing stuff..
I received: 2
I received: 3
goroutine is doing stuff..
goroutine is doing stuff..
I received: 4
I received: 5
goroutine is doing stuff..
goroutine is doing stuff..
fatal error: all goroutines are asleep - deadlock!
Is this possible? Where am I wrong
The problem is that in the goroutine you use a select to check if it should abort, but you use the default branch to do the work otherwise.
The default branch is executed if no communications (listed in case branches) can proceed. So in each iteration quit channel is checked, but if it cannot be received from (no need to quit yet), default branch is executed, which unconditionally tries to send a value on res. Now if the main goroutine is not ready to receive from it, this will be a deadlock. And this is exactly what happens when the sent value is 6, because then the main goroutine tries to send a value on quit, but if the worker goroutine is in the default branch trying to send on res, then both goroutines try to send a value, and none is trying to receive! Both channels are unbuffered, so this is a deadlock.
In the worker goroutine you must send the value on res using a proper case branch, and not in the default branch:
select {
case <-quit:
fmt.Println("Detected quit signal!")
return
case res <- idx:
fmt.Println("goroutine is doing stuff..")
idx++
}
And in the main goroutine you must break out from the for loop so the main goroutine can end and so the program can end as well:
if r == 6 {
quit <- true
break
}
Output this time (try it on the Go Playground):
goroutine is doing stuff..
I received: 0
I received: 1
goroutine is doing stuff..
goroutine is doing stuff..
I received: 2
I received: 3
goroutine is doing stuff..
goroutine is doing stuff..
I received: 4
I received: 5
goroutine is doing stuff..
goroutine is doing stuff..
The fundamental issue is that producer must always check in between sending values if the consumer (main in your case) has decided to quit reading (in your code this is optional). What's happening is even before the value of quit is sent (and received), the producer goes ahead and sends the next value on res which the consumer never is able to read - the consumer is in fact trying to send the value on the quit channel expecting the producer to read. Added a debug statement which can help you understand : https://play.golang.org/p/mP_4VYrkZZ, - producer is trying to send 7 on res and blocking, and then consumer trying to send value on quit and blocking. Deadlock!
One possible solution is as follows (using a Waitgroup is optional, needed only if you need a clean exit from producer side before return):
package main
import (
"fmt"
"sync"
)
func main() {
//WaitGroup is needed only if need a clean exit for producer
//that is the producer should have exited before consumer (main)
//exits - the code works even without the WaitGroup
var wg sync.WaitGroup
quit := make(chan bool)
res := make(chan int)
go func() {
idx := 0
for {
fmt.Println("goroutine is doing stuff..", idx)
res <- idx
idx++
if <-quit {
fmt.Println("Producer quitting..")
wg.Done()
return
}
//select {
//case <-quit:
//fmt.Println("Detected quit signal!")
//time.Sleep(1000 * time.Millisecond)
// return
//default:
//fmt.Println("goroutine is doing stuff..", idx)
//res <- idx
//idx++
//}
}
}()
wg.Add(1)
for r := range res {
if r == 6 {
fmt.Println("Consumer exit condition met: ", r)
quit <- true
break
}
quit <- false
fmt.Println("I received: ", r)
}
wg.Wait()
}
Output:
goroutine is doing stuff.. 0
I received: 0
goroutine is doing stuff.. 1
I received: 1
goroutine is doing stuff.. 2
I received: 2
goroutine is doing stuff.. 3
I received: 3
goroutine is doing stuff.. 4
I received: 4
goroutine is doing stuff.. 5
I received: 5
goroutine is doing stuff.. 6
Consumer exit condition met: 6
Producer quitting..
On playground : https://play.golang.org/p/N8WSPvnqqM
As #icza's answer is pretty clean, which #Ravi's goes to the synchronised way.
But coz I don't want to spend that much effort to restructure the code, and also I don't want to go to the synchronised way, so eventually went to the defer panic recover flow control, as below:
func test(ch chan<- int, data []byte) {
defer func() {
recover()
}()
defer close(ch)
// do your logic as normal ...
// send back your res as normal `ch <- res`
}
// Then in the caller goroutine
ch := make(chan int)
data := []byte{1, 2, 3}
go test(ch, data)
for res := range ch {
// When you want to terminate the test goroutine:
// deliberately close the channel
//
// `go -race` will report potential race condition, but it is fine
//
// then test goroutine will be panic due to try sending on the closed channel,
// then recover, then quit, perfect :)
close(ch)
break
}
any potential risk with this approach?

Closing channel of unknown length

I'm not able to close channel when there is no knowledge about its
length
package main
import (
"fmt"
"time"
)
func gen(ch chan int) {
var i int
for {
time.Sleep(time.Millisecond * 10)
ch <- i
i++
// when no more data (e.g. from db, or event stream)
if i > 100 {
break
}
}
// hot to close it properly?
close(ch)
}
func receiver(ch chan int) {
for i := range ch {
fmt.Println("received:", i)
}
}
func main() {
ch := make(chan int)
for i := 0; i < 10; i++ {
go gen(ch)
}
receiver(ch)
}
It gives me error
panic: send on closed channel
goroutine 8 [running]:
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:12 +0x57
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:35 +0xbd
goroutine 1 [panicwait]:
runtime.gopark(0x0, 0x0, 0x50b8e0, 0x9, 0x10, 0x1)
/usr/lib/go/src/runtime/proc.go:185 +0x163
runtime.main()
/usr/lib/go/src/runtime/proc.go:121 +0x2f4
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:1696 +0x1
goroutine 6 [sleep]:
time.Sleep(0x989680)
/usr/lib/go/src/runtime/time.go:59 +0xf9
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:11 +0x29
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:33 +0x79
goroutine 7 [sleep]:
time.Sleep(0x989680)
/usr/lib/go/src/runtime/time.go:59 +0xf9
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:11 +0x29
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:34 +0x9b
exit status 2
It's logical - first goroutine closing channel when the second one tries to send to it. What will be the best approach to close channel in this situation?
Once a channel is closed, you can't send further values on it else it panics. This is what you experience.
This is because you start multiple goroutines that use the same channel and they send values on it. And you close the channel in each of it. And since they are not synchronized, once the first goroutine reaches the point where it closes it, others may (and they will) still continue to send values on it: panic!
You can close the channel only once (attempting to close an already closed channel also panics). And you should do it when all the goroutines that send values on it are done. In order to do this, you need to detect when all the sender goroutines are done. An idiomatic way to detect this is to use sync.WaitGroup.
For each started sender goroutine we add 1 to the WaitGroup using WaitGroup.Add(). And each goroutine that is done sending the values can signal this by calling WaitGroup.Done(). Best to do this as a deferred statement, so if your goroutine would terminate abruptly (e.g. panics), WaitGroup.Done() would still be called, and would not leave other goroutines hanging (waiting for an absolution - a "missing" WaitGroup.Done() call that would never come...).
And WaitGroup.Wait() will wait until all sender goroutines are done, and only after this and only once will it close the channel. We want to detect this "global" done event and close the channel while processing the values sent on it is in progress, so we have to do this in its own goroutine.
The receiver goroutine will run until the channel is closed since we used the for ... range construct on the channel. And since it runs in the main goroutine, the program will not exit until all the values are properly received and processed from the channel. The for ... range construct loops until all the values are received that were sent before the channel was closed.
Note that the solution below works with buffered and unbuffered channel too without modification (try using a buffered channel with ch := make(chan int, 100)).
Correct solution (try it on the Go Playground):
func gen(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
var i int
for {
time.Sleep(time.Millisecond * 10)
ch <- i
i++
// when no more data (e.g. from db, or event stream)
if i > 100 {
break
}
}
}
func receiver(ch chan int) {
for i := range ch {
fmt.Println("received:", i)
}
}
func main() {
ch := make(chan int)
wg := &sync.WaitGroup{}
for i := 0; i < 10; i++ {
wg.Add(1)
go gen(ch, wg)
}
go func() {
wg.Wait()
close(ch)
}()
receiver(ch)
}
Note:
Note that it's important that receiver(ch) runs in the main goroutine, and the code what waits for the WaitGroup and closes the channel in its own (non-main) goroutine; and not the other way around. If you would switch these 2, it might cause an "early exit", that is not all values might be received and processed from the channel. The reason for this is because a Go program exits when the main goroutine finishes (spec: Program execution). It does not wait for other (non-main) goroutines to finish. So if waiting and closing the channel would be in the main goroutine, after closing the channel the program could exit at any moment, not waiting for the other goroutine that in this case would loop to receive values from the channel.
"One general principle of using Go channels is don't close a channel from the receiver side and don't close a channel if the channel has multiple concurrent senders."
Every channel will be GCed eventually once it is marked for cleanup, so it is okay to leave channel un-closed the only difference it will make is that that channel will be available for gc after a few cycles maybe if not closed explicitly.
Nevertheless it is always good if you can close the channel off. Please go through the following links for detailed explaination.
Articles this and this shows various ways to close a channel in case of 1:N, N:1 or M:N (senders:receivers)

Resources