I'm not able to close channel when there is no knowledge about its
length
package main
import (
"fmt"
"time"
)
func gen(ch chan int) {
var i int
for {
time.Sleep(time.Millisecond * 10)
ch <- i
i++
// when no more data (e.g. from db, or event stream)
if i > 100 {
break
}
}
// hot to close it properly?
close(ch)
}
func receiver(ch chan int) {
for i := range ch {
fmt.Println("received:", i)
}
}
func main() {
ch := make(chan int)
for i := 0; i < 10; i++ {
go gen(ch)
}
receiver(ch)
}
It gives me error
panic: send on closed channel
goroutine 8 [running]:
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:12 +0x57
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:35 +0xbd
goroutine 1 [panicwait]:
runtime.gopark(0x0, 0x0, 0x50b8e0, 0x9, 0x10, 0x1)
/usr/lib/go/src/runtime/proc.go:185 +0x163
runtime.main()
/usr/lib/go/src/runtime/proc.go:121 +0x2f4
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:1696 +0x1
goroutine 6 [sleep]:
time.Sleep(0x989680)
/usr/lib/go/src/runtime/time.go:59 +0xf9
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:11 +0x29
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:33 +0x79
goroutine 7 [sleep]:
time.Sleep(0x989680)
/usr/lib/go/src/runtime/time.go:59 +0xf9
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:11 +0x29
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:34 +0x9b
exit status 2
It's logical - first goroutine closing channel when the second one tries to send to it. What will be the best approach to close channel in this situation?
Once a channel is closed, you can't send further values on it else it panics. This is what you experience.
This is because you start multiple goroutines that use the same channel and they send values on it. And you close the channel in each of it. And since they are not synchronized, once the first goroutine reaches the point where it closes it, others may (and they will) still continue to send values on it: panic!
You can close the channel only once (attempting to close an already closed channel also panics). And you should do it when all the goroutines that send values on it are done. In order to do this, you need to detect when all the sender goroutines are done. An idiomatic way to detect this is to use sync.WaitGroup.
For each started sender goroutine we add 1 to the WaitGroup using WaitGroup.Add(). And each goroutine that is done sending the values can signal this by calling WaitGroup.Done(). Best to do this as a deferred statement, so if your goroutine would terminate abruptly (e.g. panics), WaitGroup.Done() would still be called, and would not leave other goroutines hanging (waiting for an absolution - a "missing" WaitGroup.Done() call that would never come...).
And WaitGroup.Wait() will wait until all sender goroutines are done, and only after this and only once will it close the channel. We want to detect this "global" done event and close the channel while processing the values sent on it is in progress, so we have to do this in its own goroutine.
The receiver goroutine will run until the channel is closed since we used the for ... range construct on the channel. And since it runs in the main goroutine, the program will not exit until all the values are properly received and processed from the channel. The for ... range construct loops until all the values are received that were sent before the channel was closed.
Note that the solution below works with buffered and unbuffered channel too without modification (try using a buffered channel with ch := make(chan int, 100)).
Correct solution (try it on the Go Playground):
func gen(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
var i int
for {
time.Sleep(time.Millisecond * 10)
ch <- i
i++
// when no more data (e.g. from db, or event stream)
if i > 100 {
break
}
}
}
func receiver(ch chan int) {
for i := range ch {
fmt.Println("received:", i)
}
}
func main() {
ch := make(chan int)
wg := &sync.WaitGroup{}
for i := 0; i < 10; i++ {
wg.Add(1)
go gen(ch, wg)
}
go func() {
wg.Wait()
close(ch)
}()
receiver(ch)
}
Note:
Note that it's important that receiver(ch) runs in the main goroutine, and the code what waits for the WaitGroup and closes the channel in its own (non-main) goroutine; and not the other way around. If you would switch these 2, it might cause an "early exit", that is not all values might be received and processed from the channel. The reason for this is because a Go program exits when the main goroutine finishes (spec: Program execution). It does not wait for other (non-main) goroutines to finish. So if waiting and closing the channel would be in the main goroutine, after closing the channel the program could exit at any moment, not waiting for the other goroutine that in this case would loop to receive values from the channel.
"One general principle of using Go channels is don't close a channel from the receiver side and don't close a channel if the channel has multiple concurrent senders."
Every channel will be GCed eventually once it is marked for cleanup, so it is okay to leave channel un-closed the only difference it will make is that that channel will be available for gc after a few cycles maybe if not closed explicitly.
Nevertheless it is always good if you can close the channel off. Please go through the following links for detailed explaination.
Articles this and this shows various ways to close a channel in case of 1:N, N:1 or M:N (senders:receivers)
Related
This question already has answers here:
Buffered/Unbuffered channel
(3 answers)
fatal error: all goroutines are asleep - deadlock! when channel is not buffered
(1 answer)
Why is the order of channels receiving causing/resolving a deadlock in Golang?
(3 answers)
In the Go select construct, can I have send and receive to unbuffered channel in two case statements?
(2 answers)
Closed 7 days ago.
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan struct{})
wg := &sync.WaitGroup{}
wg.Add(1)
go func() {
defer wg.Done()
ch <- struct{}{}
}()
wg.Wait()
<-ch
fmt.Println("finished")
}
result
fnished will not be printed
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan struct{}, 1)
wg := &sync.WaitGroup{}
wg.Add(1)
go func() {
defer wg.Done()
ch <- struct{}{}
}()
wg.Wait()
<-ch
fmt.Println("finished")
}
result
finished can be printed immediately
The only difference between the two code is that the first code use no buffer channel, but the second code use channel with a buffer.
In the first code, ch <- struct{}{} can not be run. message can not be sent to the channel.
Why does that happen? Is there any official explanation or material?
In the first example wg.Done() can only be called after a value on the channel has been sent. But since the channel is unbuffered, the send can only happen if there's a ready receiver. The main goroutine would be the receiver, but it only receives after wg.Wait(). But wg.Wait() blocks until wg.Done() is called, so this will never happen, it's a deadlock.
This is in Spec: Send statements:
Communication blocks until the send can proceed. A send on an unbuffered channel can proceed if a receiver is ready.
When using non-buffered channels, sending or receiving commands from the channel are blocking.
This means with non-buffered channel, your go routine that sends the message is not done, hence it won't call wg.Done() which will keep the wg.Wait() method in blocking state.
If you run your code on the play server https://go.dev/play/p/oDpyL6-dARP, it will panic with an informative message: fatal error: all goroutines are asleep - deadlock!
When your using buffered channels, the go routine runs because the channel has a buffer, then wg.Done() is called, which release wg.Wait() afterwards.
I am trying my hands on goroutines and came up this example - https://go.dev/play/p/mWHUmALk-1_K
But I am having this error - fatal error: all goroutines are asleep - deadlock!
I have tried to fix this but no luck. Please how do I fix this?
The error seem to be on lines 15, 23 and 32.
The problem is that your program starts 3 separate goroutines that send to the same channel. And you have only the main goroutine receive from that channel only once. This causes the second channel send (ch <- fmt.Sprintf("...) to block indefinitely. With unbuffered channels you need to do as many receives as you do sends.
One approach to ensure all sends are received would be to use a range loop over the channel.
func getLength(dd []string, wg *sync.WaitGroup) {
wg.Add(len(dd))
c := make(chan string)
for _, d := range dd {
d1 := d
go computeLength(d1, c, wg)
}
// close c once all goroutines are done to
// ensure the for-range loop below exits.
go func() { wg.Wait(); close(c) }()
// Use for-range loop on the channel to receive all the sends.
//
// But note that a for-range loop over a channel exits only
// when the channel is closed or the loop is exited from within.
//
// So to exit you can close c once wg.Wait() returns,
// that's why there's that extra goroutine above.
for v := range c {
fmt.Println(v)
}
}
https://go.dev/play/p/BUb7NHrq2B0
package main
import (
"fmt"
"sync"
)
func main(){
ch1 := make(chan int,100)
ct := 0
var wg sync.WaitGroup
wg.Add(1)
go func(){
//defer close(ch1)
for i:= 0; i < 10;i ++{
ch1 <- i
}
}()
go func(){
defer wg.Done()
for x := range ch1{
fmt.Println(x)
}
}()
wg.Wait()
fmt.Println("numbers:",ct)
}
Why this code will return
fatal error: all goroutines are asleep - deadlock!
I've found if I closed the channel there will be no deadlock, but I don't know why it's like that.
Do I have to close the channel after I inputted all items into the channel?
for range over a channel only terminates if / when the channel is closed. If you don't close the channel and you don't send more values on it, the for range statement will block forever, and so will the main goroutine at wg.Wait().
The "sender" party should close the channel once it sent all values, signalling the "receiver" party that no more values will come on the channel.
So yes, you should close the channel:
go func() {
defer close(ch1)
for i := 0; i < 10; i++ {
ch1 <- i
}
}()
The for loop that reads from the channel will continue reading unless you close the channel. That is the reason for the deadlock, because the goroutine that reads from the channel is the only active goroutine, and there is no other goroutines that can write to it. When you close, the for loop terminates.
range unclosed ch1 will causing block;
even close ch1, you can still receive data from ch1;
It can be said, close(ch1) will send a special message to ch1, which can be used to notify the ch1 receiver that no more data will be received. So even if there is data in the ch1, you can close() without causing the receiver to not receive the remaining data.
The channel does not need to release resources through close. As long as there is no goroutine holding the channel, the related resources will be automatically released.
uncomment defer close(ch1) will solve this problems.
here
TL;DR: A typical case of all goroutines are asleep, deadlock! but can't figure it out
I'm parsing the Wiktionary XML dump to build a DB of words. I defer the parsing of each article's text to a goroutine hoping that it will speed up the process.
It's 7GB and is processed in under 2 minutes in my machine when doing it serially, but if I can take advantage of all cores, why not.
I'm new to threading in general, I'm getting a all goroutines are asleep, deadlock! error.
What's wrong here?
This may not be performant at all, as it uses an unbuffered channel, so all goroutines effectively end up executing serially, but my idea is to learn and understand threading and to benchmark how long it takes with different alternatives:
unbuffered channel
different sized buffered channel
only calling as many goroutines at a time as there are runtime.NumCPU()
The summary of my code in pseudocode:
while tag := xml.getNextTag() {
wg.Add(1)
go parseTagText(chan, wg, tag.text)
// consume a channel message if available
select {
case msg := <-chan:
// do something with msg
default:
}
}
// reading tags finished, wait for running goroutines, consume what's left on the channel
for msg := range chan {
// do something with msg
}
// Sometimes this point is never reached, I get a deadlock
wg.Wait()
----
func parseTagText(chan, wg, tag.text) {
defer wg.Done()
// parse tag.text
chan <- whatever // just inform that the text has been parsed
}
Complete code:
https://play.golang.org/p/0t2EqptJBXE
In your complete example on the Go Playground, you:
Create a channel (line 39, results := make(chan langs)) and a wait-group (line 40, var wait sync.WaitGroup). So far so good.
Loop: in the loop, sometimes spin off a task:
if ...various conditions... {
wait.Add(1)
go parseTerm(results, &wait, text)
}
In the loop, sometimes do a non-blocking read from the channel (as shown in your question). No problem here either. But...
At the end of the loop, use:
for res := range results {
...
}
without ever calling close(results) in exactly one place, after all writers finish. This loop uses a blocking read from the channel. As long as some writer goroutine is still running, the blocking read can block without having the whole system stop, but when the last writer finishes writing and exits, there are no remaining writer goroutines. Any other remaining goroutines might rescue you, but there are none.
Since you use the var wait correctly (adding 1 in the right place, and calling Done() in the right place in the writer), the solution is to add one more goroutine, which will be the one to rescue you:
go func() {
wait.Wait()
close(results)
}()
You should spin off this rescuer goroutine just before entering the for res := range results loop. (If you spin it off any earlier, it might see the wait variable count down to zero too soon, just before it gets counted up again by spinning off another parseTerm.)
This anonymous function will block in the wait variable's Wait() function until the last writer goroutine has called the final wait.Done(), which will unblock this goroutine. Then this goroutine will call close(results), which will arrange for the for loop in your main goroutine to finish, unblocking that goroutine. When this goroutine (the rescuer) returns and thus terminates, there are no more rescuers, but we no longer need any.
(This main code then calls wait.Wait() unnecessarily: Since the for didn't terminate until the wait.Wait() in the new goroutine already unblocked, we know that this next wait.Wait() will return immediately. So we can drop this second call, although leaving it in is harmless.)
The problem is that nothing is closing the results channel, yet the range loop only exits when it closes. I've simplified your code to illustrate this and propsed a solution - basically consume the data in a goroutine:
// This is our producer
func foo(i int, ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
ch <- i
fmt.Println(i, "done")
}
// This is our consumer - it uses a different WG to signal it's done
func consumeData(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
for x := range ch {
fmt.Println(x)
}
fmt.Println("ALL DONE")
}
func main() {
ch := make(chan int)
wg := sync.WaitGroup{}
// create the producers
for i := 0; i < 10; i++ {
wg.Add(1)
go foo(i, ch, &wg)
}
// create the consumer on a different goroutine, and sync using another WG
consumeWg := sync.WaitGroup{}
consumeWg.Add(1)
go consumeData(ch,&consumeWg)
wg.Wait() // <<<< means that the producers are done
close(ch) // << Signal the consumer to exit
consumeWg.Wait() // << Wait for the consumer to exit
}
I have a snippet of code that I am trying to understand based on how I put the close call and the location
func main() {
ch := make(chan int, 2)
go func(ch chan int) {
for i := 1; i <= 5; i++ {
ch <- i
fmt.Println("Func goroutine sends data: ", i)
}
//Pos1 - Works perfectly
//close(ch)
}(ch)
fmt.Println("Main goroutine sleeps 2 seconds")
time.Sleep(time.Second * 2)
fmt.Println("Main goroutine begins receiving data")
//Pos2 - Puts in only 2 ints 1 and 2 and then prints only that
//close(ch)
for d := range ch {
fmt.Println("Main goroutine received data:", d)
}
//Pos3 - Throws fatal error
close(ch)
}
I have been trying to understand and read blogs on this but not able to understand somethings still
When I place the close at Pos1, it works fine. But I am not sure why
it works. The buffer cannot hold more than 2 elements at any given
time, so when 2 elements are written, the loop will block until the
main routing does a read. But I thought to do a range over a
buffered channel, the range function has to know beforehand how many
elements to iterate over and for that channel must be closed. Why
does close work at this position?
When I put it as position 2, it prints only 2 elements which kind of makes sense but why is the for loop not throwing an exception when it is trying to write more elements to the channel that is closed?
When I close at Pos3, I get an exception fatal error: all goroutines are asleep - deadlock! though all 5 ints are printed. Why is that?
But I thought to do a range over a buffered channel, the range function has to know beforehand how many elements to iterate over and for that channel must be closed.
That assumption is wrong and the root of all misunderstandings.
The behavior of ranging over a channel is described in the Go Spec: https://golang.org/ref/spec#For_statements
For channels, the iteration values produced are the successive values sent on the channel until the channel is closed. If the channel is nil, the range expression blocks forever.
The channel does not need to be closed when the for statement is evaluated and the statement does not need to know number of elements.
So, in your code, when you put close in Pos1, it is indeed the right way to do it. When you put it in Pos3, the for loop waits the channel to be closed, which can only happen after the for loop itself, so it is a deadlock.
Putting close in Pos2 is buggy and the behavior is a little tricky. It is possible to raise an error, yet it is possible to just output two numbers. This is because when the channel is closed before the for loop, the loop can run without block and then main() returns. And when main() returns, the Go program ends. Whether it raises an error solely depends on if the scheduler switch to goroutine between the process, which is not guaranteed.
Modifying the code snippets with some peppered print statements to add on to the excellent answer by #leaf bebop, to clarify what's happening.
Pos 2, but make main wait a bit
func main() {
ch := make(chan int, 2)
go func(ch chan int) {
for i := 1; i <= 5; i++ {
ch <- i
fmt.Println("Func goroutine sends data: ", i)
}
fmt.Println("goR work is done")
}(ch)
fmt.Println("Main goroutine sleeps 2 seconds")
time.Sleep(time.Second * 2)
fmt.Println("Main goroutine begins receiving data")
close(ch)
for d := range ch {
fmt.Println("Main goroutine received data:", d)
}
fmt.Println("Main waiting, will the other goR panic?")
time.Sleep(time.Second * 10)
}
Main goroutine sleeps 2 seconds
Func goroutine sends data: 1
Func goroutine sends data: 2
Main goroutine begins receiving data
Main goroutine received data: 1
panic: send on closed channel
goroutine 6 [running]:
main.main.func1(0x0?)
/tmp/sandbox3746599466/prog.go:15 +0x3b
created by main.main
/tmp/sandbox3746599466/prog.go:13 +0x85
Program exited.
So, given enough time (main routine didn't exit), the scheduler went back to the go routine, where we wrote to a closed (closed by main) channel - Panic!
Pos 2, but only send 2 values, i.e not writing to a closed channel
func main() {
ch := make(chan int, 2)
go func(ch chan int) {
// Send only 2 values
for i := 1; i <= 2; i++ {
ch <- i
fmt.Println("Func goroutine sends data: ", i)
}
fmt.Println("goR work is done")
}(ch)
fmt.Println("Main goroutine sleeps 2 seconds")
time.Sleep(time.Second * 2)
fmt.Println("Main goroutine begins receiving data")
close(ch)
for d := range ch {
fmt.Println("Main goroutine received data:", d)
}
fmt.Println("Main waiting, will the other goR panic?")
time.Sleep(time.Second * 10)
fmt.Println("Main exiting")
}
Main goroutine sleeps 2 seconds
Func goroutine sends data: 1
Func goroutine sends data: 2
goR work is done
Main goroutine begins receiving data
Main goroutine received data: 1
Main goroutine received data: 2
Main waiting, will the other goR panic?
Main exiting
Program exited.
No panic, no spinning for ever. Main routine just exits.
Pos 3, check if main got unblocked?
func main() {
ch := make(chan int, 2)
go func(ch chan int) {
for i := 1; i <= 5; i++ {
ch <- i
fmt.Println("Func goroutine sends data: ", i)
}
fmt.Println("Routine done with work")
}(ch)
fmt.Println("Main goroutine sleeps 2 seconds")
time.Sleep(time.Second * 2)
fmt.Println("Main goroutine begins receiving data")
for d := range ch {
fmt.Println("Main goroutine received data:", d)
}
fmt.Println("Have I closed this channel?")
close(ch)
fmt.Println("I have")
}
Main goroutine sleeps 2 seconds
Func goroutine sends data: 1
Func goroutine sends data: 2
Main goroutine begins receiving data
Main goroutine received data: 1
Main goroutine received data: 2
Main goroutine received data: 3
Func goroutine sends data: 3
Func goroutine sends data: 4
Func goroutine sends data: 5
Routine done with work
Main goroutine received data: 4
Main goroutine received data: 5
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan receive]:
main.main()
/tmp/sandbox3301646152/prog.go:26 +0x194
Program exited.
Main never got even to close the channel, it was blocked in the range loop. It'll stay here forever, until it gets a value, or someone closes the channel. We don't have any other routine to do that, so deadlock.