I am trying my hands on goroutines and came up this example - https://go.dev/play/p/mWHUmALk-1_K
But I am having this error - fatal error: all goroutines are asleep - deadlock!
I have tried to fix this but no luck. Please how do I fix this?
The error seem to be on lines 15, 23 and 32.
The problem is that your program starts 3 separate goroutines that send to the same channel. And you have only the main goroutine receive from that channel only once. This causes the second channel send (ch <- fmt.Sprintf("...) to block indefinitely. With unbuffered channels you need to do as many receives as you do sends.
One approach to ensure all sends are received would be to use a range loop over the channel.
func getLength(dd []string, wg *sync.WaitGroup) {
wg.Add(len(dd))
c := make(chan string)
for _, d := range dd {
d1 := d
go computeLength(d1, c, wg)
}
// close c once all goroutines are done to
// ensure the for-range loop below exits.
go func() { wg.Wait(); close(c) }()
// Use for-range loop on the channel to receive all the sends.
//
// But note that a for-range loop over a channel exits only
// when the channel is closed or the loop is exited from within.
//
// So to exit you can close c once wg.Wait() returns,
// that's why there's that extra goroutine above.
for v := range c {
fmt.Println(v)
}
}
https://go.dev/play/p/BUb7NHrq2B0
Related
I've read a pipeline article from https://go.dev/blog/pipelines.
I've tried to remove the goroutine that covers wg.Wait() but it turns out that caused all goroutines are asleep - deadlock!
I really don't understand why.
func merge(cs ...<-chan int) <-chan int {
var wg sync.WaitGroup
out := make(chan int)
// Start an output goroutine for each input channel in cs. output
// copies values from c to out until c is closed, then calls wg.Done.
output := func(c <-chan int) {
for n := range c {
out <- n
}
wg.Done()
}
wg.Add(len(cs))
for _, c := range cs {
go output(c)
}
// Start a goroutine to close out once all the output goroutines are
// done. This must start after the wg.Add call.
go func() {
wg.Wait()
close(out)
}()
return out
}
out is a blocking channel. You need to return out and let something consume from out, or else the output goroutines will block as soon as they try to write anything. If merge waits on the waitgroup (i.e. waits for everything to finish) before the consumer even has a chance to start, that's an inevitable deadlock.
TL;DR: A typical case of all goroutines are asleep, deadlock! but can't figure it out
I'm parsing the Wiktionary XML dump to build a DB of words. I defer the parsing of each article's text to a goroutine hoping that it will speed up the process.
It's 7GB and is processed in under 2 minutes in my machine when doing it serially, but if I can take advantage of all cores, why not.
I'm new to threading in general, I'm getting a all goroutines are asleep, deadlock! error.
What's wrong here?
This may not be performant at all, as it uses an unbuffered channel, so all goroutines effectively end up executing serially, but my idea is to learn and understand threading and to benchmark how long it takes with different alternatives:
unbuffered channel
different sized buffered channel
only calling as many goroutines at a time as there are runtime.NumCPU()
The summary of my code in pseudocode:
while tag := xml.getNextTag() {
wg.Add(1)
go parseTagText(chan, wg, tag.text)
// consume a channel message if available
select {
case msg := <-chan:
// do something with msg
default:
}
}
// reading tags finished, wait for running goroutines, consume what's left on the channel
for msg := range chan {
// do something with msg
}
// Sometimes this point is never reached, I get a deadlock
wg.Wait()
----
func parseTagText(chan, wg, tag.text) {
defer wg.Done()
// parse tag.text
chan <- whatever // just inform that the text has been parsed
}
Complete code:
https://play.golang.org/p/0t2EqptJBXE
In your complete example on the Go Playground, you:
Create a channel (line 39, results := make(chan langs)) and a wait-group (line 40, var wait sync.WaitGroup). So far so good.
Loop: in the loop, sometimes spin off a task:
if ...various conditions... {
wait.Add(1)
go parseTerm(results, &wait, text)
}
In the loop, sometimes do a non-blocking read from the channel (as shown in your question). No problem here either. But...
At the end of the loop, use:
for res := range results {
...
}
without ever calling close(results) in exactly one place, after all writers finish. This loop uses a blocking read from the channel. As long as some writer goroutine is still running, the blocking read can block without having the whole system stop, but when the last writer finishes writing and exits, there are no remaining writer goroutines. Any other remaining goroutines might rescue you, but there are none.
Since you use the var wait correctly (adding 1 in the right place, and calling Done() in the right place in the writer), the solution is to add one more goroutine, which will be the one to rescue you:
go func() {
wait.Wait()
close(results)
}()
You should spin off this rescuer goroutine just before entering the for res := range results loop. (If you spin it off any earlier, it might see the wait variable count down to zero too soon, just before it gets counted up again by spinning off another parseTerm.)
This anonymous function will block in the wait variable's Wait() function until the last writer goroutine has called the final wait.Done(), which will unblock this goroutine. Then this goroutine will call close(results), which will arrange for the for loop in your main goroutine to finish, unblocking that goroutine. When this goroutine (the rescuer) returns and thus terminates, there are no more rescuers, but we no longer need any.
(This main code then calls wait.Wait() unnecessarily: Since the for didn't terminate until the wait.Wait() in the new goroutine already unblocked, we know that this next wait.Wait() will return immediately. So we can drop this second call, although leaving it in is harmless.)
The problem is that nothing is closing the results channel, yet the range loop only exits when it closes. I've simplified your code to illustrate this and propsed a solution - basically consume the data in a goroutine:
// This is our producer
func foo(i int, ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
ch <- i
fmt.Println(i, "done")
}
// This is our consumer - it uses a different WG to signal it's done
func consumeData(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
for x := range ch {
fmt.Println(x)
}
fmt.Println("ALL DONE")
}
func main() {
ch := make(chan int)
wg := sync.WaitGroup{}
// create the producers
for i := 0; i < 10; i++ {
wg.Add(1)
go foo(i, ch, &wg)
}
// create the consumer on a different goroutine, and sync using another WG
consumeWg := sync.WaitGroup{}
consumeWg.Add(1)
go consumeData(ch,&consumeWg)
wg.Wait() // <<<< means that the producers are done
close(ch) // << Signal the consumer to exit
consumeWg.Wait() // << Wait for the consumer to exit
}
This is homework and beginner question. I edited the question since I made a mistake that I found.
I am trying to do parallel frequency maps from a text and I get an error in the last operation (reducing maps)
The code seem to work up to this point.
If I close the channel I get error:
"panic: send on closed channel"
If I do not close the channel I get:
"fatal error: all goroutines are asleep - deadlock!"
func WordCount(text string) {
text = strings.ToLower(text)
re := regexp.MustCompile("\\w+")
sentence := re.FindAllString(text, -1)
numberOfGroups := 4
piece := len(sentence) / numberOfGroups
wordChannel := make(chan map[string]int)
wg := new(sync.WaitGroup)
wg.Add(numberOfGroups)
for i := 0; i < numberOfGroups; i ++ {
go processToCounting(sentence[i*piece:(i+1)*piece], wordChannel, wg)
}
wg.Wait()
fmt.Print(<-wordChannel)
fmt.Print("\n")
finalMap := make(map[string]int)
close(wordChannel)
for i := 0; i < numberOfGroups; i++ {
for k, v := range <- wordChannel {
finalMap[k] += v
}
}
}
func processToCounting(textSlice []string, wordChannel chan map[string]int, wg *sync.WaitGroup) {
freq := make(map[string]int)
for _, v := range textSlice {
freq[v]++
}
wg.Done()
wordChannel <- freq
}
1. First question: panic
If I close the channel I get error: "panic: send on closed channel"
Why? One of you goroutines is trying to write to the channel which you already closed in the calling (main) goroutine. In your case in WordCount function.
In the current version of your code the panic is not reproducible with my test sentence, but you can easily cause this e.g. if you would call close(wordChannel) before wg.Wait().
Lets look at the Bug in processToCounting which may cause the panic:
wg.Done() // tells to the WaitGroup that the gouroutine is Done (decrements the counter of goroutines)
wordChannel <- freq // trying to write to the channel
Here wg.Done() signals to the WaitGroup that the goroutine is Done before actual writing to the channel has happened. The calling goroutine (WordCount function) at some point thinks that all gouroutines are done (wg.Wait() line) and closes the channel. But one of your goroutine which have not finished writing, will try to write to the closed channel. Then you will get the panic.
How to fix:
Use defer in processToCounting function
defer wg.Done() // triggers wg.Done right before the function returns (but after writing to the channel in your case)
What to read:
See for beginners in A Tour of Go / Concurrency
Sending on a closed channel will cause a panic.
and documentation for: close
Sending to or closing a closed channel causes a run-time panic.
2. Second question: deadlock
If I do not close the channel I get: "fatal error: all goroutines are asleep - deadlock!"
You have a for-loop which reads from the channel. This for-loop is locked forever. Waiting for new values from the channel, but nobody will write there anymore.
See in A Tour of Go / Concurrency
The loop for i := range c receives values from the channel repeatedly until it is closed.
and see documentation for Channels
Receivers always block until there is data to receive
Q1: why close(wordChannel) triggers a panic
A:
you put wg.Done() before pushing result into the channel
the channel is a non-buffer channel, which means that it will be stuck to write until there is a read. but the read happens after you close channel. the sequence is close channel -> read channel -> write channel -> panic
Q2: why there is a deadlock.
A: fmt.Print(<-wordChannel) read a msg from the channel, so that the last loop can't read numberOfGroups message. it waits for the last message forever.
for i := 0; i < numberOfGroups; i++ {
for k, v := range <- wordChannel {
finalMap[k] += v
}
}
I was just experimenting with Go channels on my Ubuntu 64 bit environment and got confused with the output the following program produced.
I got the output:
0
1
2
3
Exit
The output when I uncommented the two commented lines:
0
1
2
3
4
Exit
Please explain the behavior.
TIA.
package main
import (
"fmt"
//"time"
)
func main() {
ch := make(chan int)
done := make(chan bool)
go func() {
for i := 0; i < 5; i++ {
ch <- i
}
//time.Sleep(1 * time.Second)
done <- false
}()
go func() {
for {
select {
case message := <-ch:
fmt.Println(message)
case <-done:
return
}
}
}()
<-done
fmt.Println("Exit")
}
Your main thread is waiting on done, then exiting. Meanwhile your first go function pipes 5 values into ch, then sends to done.
The value in done then gets read from the main thread and happens to occur before the second go function reads the last value from ch. When it does so, it exits the program.
Note that if your second thread did happen to both read from ch and done, then your program would deadlock since the main thread would never receive on done and all running go threads would be blocked waiting to receive on channels.
You're not waiting for both goroutines, and only sending a single value over done to 2 receivers, which will deadlock if the second receiver happens to be main.
Using a WaitGroup simplifies the code, and allows you to easily wait for as many goroutines as needed. https://play.golang.org/p/MWknv_9AFKp
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
defer close(ch)
for i := 0; i < 5; i++ {
ch <- i
}
}()
wg.Add(1)
go func() {
defer wg.Done()
for message := range ch {
fmt.Println(message)
}
}()
wg.Wait()
fmt.Println("Exit")
You have two go routines running in parallel. One inserts 5 numbers into the channel and then signals the main thread to exit, and another one reads the numbers from the channel.
Note that once the go routine that is responsible for enqueuing the numbers into the channel finishes, it signals the main thread to exit, regardless of whether the go routine that reads the numbers finished or not. So you could end up with a case where the enqueueing go routine finished before the dequeuing finished, and the main thread exited.
By adding the sleep, you make the enqueueing go routine live a bit longer and give a chance to the dequeueing go routine to read and print all the numbers before the enqueueing go routine signals the main thread to exit.
To solve that, you could just run the dequeuing code in the main thread. No need to run it in a go routine in this case.
I'm not able to close channel when there is no knowledge about its
length
package main
import (
"fmt"
"time"
)
func gen(ch chan int) {
var i int
for {
time.Sleep(time.Millisecond * 10)
ch <- i
i++
// when no more data (e.g. from db, or event stream)
if i > 100 {
break
}
}
// hot to close it properly?
close(ch)
}
func receiver(ch chan int) {
for i := range ch {
fmt.Println("received:", i)
}
}
func main() {
ch := make(chan int)
for i := 0; i < 10; i++ {
go gen(ch)
}
receiver(ch)
}
It gives me error
panic: send on closed channel
goroutine 8 [running]:
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:12 +0x57
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:35 +0xbd
goroutine 1 [panicwait]:
runtime.gopark(0x0, 0x0, 0x50b8e0, 0x9, 0x10, 0x1)
/usr/lib/go/src/runtime/proc.go:185 +0x163
runtime.main()
/usr/lib/go/src/runtime/proc.go:121 +0x2f4
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:1696 +0x1
goroutine 6 [sleep]:
time.Sleep(0x989680)
/usr/lib/go/src/runtime/time.go:59 +0xf9
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:11 +0x29
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:33 +0x79
goroutine 7 [sleep]:
time.Sleep(0x989680)
/usr/lib/go/src/runtime/time.go:59 +0xf9
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:11 +0x29
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:34 +0x9b
exit status 2
It's logical - first goroutine closing channel when the second one tries to send to it. What will be the best approach to close channel in this situation?
Once a channel is closed, you can't send further values on it else it panics. This is what you experience.
This is because you start multiple goroutines that use the same channel and they send values on it. And you close the channel in each of it. And since they are not synchronized, once the first goroutine reaches the point where it closes it, others may (and they will) still continue to send values on it: panic!
You can close the channel only once (attempting to close an already closed channel also panics). And you should do it when all the goroutines that send values on it are done. In order to do this, you need to detect when all the sender goroutines are done. An idiomatic way to detect this is to use sync.WaitGroup.
For each started sender goroutine we add 1 to the WaitGroup using WaitGroup.Add(). And each goroutine that is done sending the values can signal this by calling WaitGroup.Done(). Best to do this as a deferred statement, so if your goroutine would terminate abruptly (e.g. panics), WaitGroup.Done() would still be called, and would not leave other goroutines hanging (waiting for an absolution - a "missing" WaitGroup.Done() call that would never come...).
And WaitGroup.Wait() will wait until all sender goroutines are done, and only after this and only once will it close the channel. We want to detect this "global" done event and close the channel while processing the values sent on it is in progress, so we have to do this in its own goroutine.
The receiver goroutine will run until the channel is closed since we used the for ... range construct on the channel. And since it runs in the main goroutine, the program will not exit until all the values are properly received and processed from the channel. The for ... range construct loops until all the values are received that were sent before the channel was closed.
Note that the solution below works with buffered and unbuffered channel too without modification (try using a buffered channel with ch := make(chan int, 100)).
Correct solution (try it on the Go Playground):
func gen(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
var i int
for {
time.Sleep(time.Millisecond * 10)
ch <- i
i++
// when no more data (e.g. from db, or event stream)
if i > 100 {
break
}
}
}
func receiver(ch chan int) {
for i := range ch {
fmt.Println("received:", i)
}
}
func main() {
ch := make(chan int)
wg := &sync.WaitGroup{}
for i := 0; i < 10; i++ {
wg.Add(1)
go gen(ch, wg)
}
go func() {
wg.Wait()
close(ch)
}()
receiver(ch)
}
Note:
Note that it's important that receiver(ch) runs in the main goroutine, and the code what waits for the WaitGroup and closes the channel in its own (non-main) goroutine; and not the other way around. If you would switch these 2, it might cause an "early exit", that is not all values might be received and processed from the channel. The reason for this is because a Go program exits when the main goroutine finishes (spec: Program execution). It does not wait for other (non-main) goroutines to finish. So if waiting and closing the channel would be in the main goroutine, after closing the channel the program could exit at any moment, not waiting for the other goroutine that in this case would loop to receive values from the channel.
"One general principle of using Go channels is don't close a channel from the receiver side and don't close a channel if the channel has multiple concurrent senders."
Every channel will be GCed eventually once it is marked for cleanup, so it is okay to leave channel un-closed the only difference it will make is that that channel will be available for gc after a few cycles maybe if not closed explicitly.
Nevertheless it is always good if you can close the channel off. Please go through the following links for detailed explaination.
Articles this and this shows various ways to close a channel in case of 1:N, N:1 or M:N (senders:receivers)