I'm learning channel and below is a test I tried, but deadlock happens
func main() {
ch := make(chan int)
go func() {
select {
case ch <- 1:
fmt.Println("send suc")
default: // if comment this line, it will run smoothly
fmt.Println("default")
}
}()
time.Sleep(2) // do some time consuming thing...
fmt.Printf("receive val: %d", <-ch)
}
I expected there is no deadlock, but result is:
default
fatal error: all goroutines are asleep - deadlock!
But if I remove default or time.Sleep(2), code will run smoothly, result:
send suc
receive val: 1
Can someone explain why deadlock happen?
You have a select with default, that means if none of the communication ops are ready, select will not wait, will not block, but execute default immediately. The channel is unbuffered, so the send on it is not ready (because there are no receivers ready – main is sleeping).
So once the sleep is over in main, it tries to receive from the channel, but by then there is nobody trying to send on it. Deadlock.
Related
I am trying my hands on goroutines and came up this example - https://go.dev/play/p/mWHUmALk-1_K
But I am having this error - fatal error: all goroutines are asleep - deadlock!
I have tried to fix this but no luck. Please how do I fix this?
The error seem to be on lines 15, 23 and 32.
The problem is that your program starts 3 separate goroutines that send to the same channel. And you have only the main goroutine receive from that channel only once. This causes the second channel send (ch <- fmt.Sprintf("...) to block indefinitely. With unbuffered channels you need to do as many receives as you do sends.
One approach to ensure all sends are received would be to use a range loop over the channel.
func getLength(dd []string, wg *sync.WaitGroup) {
wg.Add(len(dd))
c := make(chan string)
for _, d := range dd {
d1 := d
go computeLength(d1, c, wg)
}
// close c once all goroutines are done to
// ensure the for-range loop below exits.
go func() { wg.Wait(); close(c) }()
// Use for-range loop on the channel to receive all the sends.
//
// But note that a for-range loop over a channel exits only
// when the channel is closed or the loop is exited from within.
//
// So to exit you can close c once wg.Wait() returns,
// that's why there's that extra goroutine above.
for v := range c {
fmt.Println(v)
}
}
https://go.dev/play/p/BUb7NHrq2B0
TL;DR: A typical case of all goroutines are asleep, deadlock! but can't figure it out
I'm parsing the Wiktionary XML dump to build a DB of words. I defer the parsing of each article's text to a goroutine hoping that it will speed up the process.
It's 7GB and is processed in under 2 minutes in my machine when doing it serially, but if I can take advantage of all cores, why not.
I'm new to threading in general, I'm getting a all goroutines are asleep, deadlock! error.
What's wrong here?
This may not be performant at all, as it uses an unbuffered channel, so all goroutines effectively end up executing serially, but my idea is to learn and understand threading and to benchmark how long it takes with different alternatives:
unbuffered channel
different sized buffered channel
only calling as many goroutines at a time as there are runtime.NumCPU()
The summary of my code in pseudocode:
while tag := xml.getNextTag() {
wg.Add(1)
go parseTagText(chan, wg, tag.text)
// consume a channel message if available
select {
case msg := <-chan:
// do something with msg
default:
}
}
// reading tags finished, wait for running goroutines, consume what's left on the channel
for msg := range chan {
// do something with msg
}
// Sometimes this point is never reached, I get a deadlock
wg.Wait()
----
func parseTagText(chan, wg, tag.text) {
defer wg.Done()
// parse tag.text
chan <- whatever // just inform that the text has been parsed
}
Complete code:
https://play.golang.org/p/0t2EqptJBXE
In your complete example on the Go Playground, you:
Create a channel (line 39, results := make(chan langs)) and a wait-group (line 40, var wait sync.WaitGroup). So far so good.
Loop: in the loop, sometimes spin off a task:
if ...various conditions... {
wait.Add(1)
go parseTerm(results, &wait, text)
}
In the loop, sometimes do a non-blocking read from the channel (as shown in your question). No problem here either. But...
At the end of the loop, use:
for res := range results {
...
}
without ever calling close(results) in exactly one place, after all writers finish. This loop uses a blocking read from the channel. As long as some writer goroutine is still running, the blocking read can block without having the whole system stop, but when the last writer finishes writing and exits, there are no remaining writer goroutines. Any other remaining goroutines might rescue you, but there are none.
Since you use the var wait correctly (adding 1 in the right place, and calling Done() in the right place in the writer), the solution is to add one more goroutine, which will be the one to rescue you:
go func() {
wait.Wait()
close(results)
}()
You should spin off this rescuer goroutine just before entering the for res := range results loop. (If you spin it off any earlier, it might see the wait variable count down to zero too soon, just before it gets counted up again by spinning off another parseTerm.)
This anonymous function will block in the wait variable's Wait() function until the last writer goroutine has called the final wait.Done(), which will unblock this goroutine. Then this goroutine will call close(results), which will arrange for the for loop in your main goroutine to finish, unblocking that goroutine. When this goroutine (the rescuer) returns and thus terminates, there are no more rescuers, but we no longer need any.
(This main code then calls wait.Wait() unnecessarily: Since the for didn't terminate until the wait.Wait() in the new goroutine already unblocked, we know that this next wait.Wait() will return immediately. So we can drop this second call, although leaving it in is harmless.)
The problem is that nothing is closing the results channel, yet the range loop only exits when it closes. I've simplified your code to illustrate this and propsed a solution - basically consume the data in a goroutine:
// This is our producer
func foo(i int, ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
ch <- i
fmt.Println(i, "done")
}
// This is our consumer - it uses a different WG to signal it's done
func consumeData(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
for x := range ch {
fmt.Println(x)
}
fmt.Println("ALL DONE")
}
func main() {
ch := make(chan int)
wg := sync.WaitGroup{}
// create the producers
for i := 0; i < 10; i++ {
wg.Add(1)
go foo(i, ch, &wg)
}
// create the consumer on a different goroutine, and sync using another WG
consumeWg := sync.WaitGroup{}
consumeWg.Add(1)
go consumeData(ch,&consumeWg)
wg.Wait() // <<<< means that the producers are done
close(ch) // << Signal the consumer to exit
consumeWg.Wait() // << Wait for the consumer to exit
}
I am trying to understand channels in Go. I have read that by default sends and receives block until both the sender and receiver are ready. But how do we figure out readyness of sender and receiver.
For example in the following code
package main
import "fmt"
func main() {
ch := make(chan int)
ch <- 1
fmt.Println(<-ch)
}
The program will get stuck on the channel send operation waiting forever for someone to read the value. Even though we have a receive operation in println statement it ends up in a deadlock.
But for the following program
package main
import "fmt"
func main() {
ch := make(chan int)
go func () {
ch <- 1
}()
fmt.Println(<-ch)
}
The integer is passed successfully from go routine to main program. What made this program work? Why second works but first do not? Is go routine causing some difference?
Let's step through the first program:
// My notes here
ch := make(chan int) // make a new int channel
ch <- 1 // block until we can send to that channel
// keep blocking
// keep blocking
// still waiting for a receiver
// no reason to stop blocking yet...
// this line is never reached, because it blocks above forever.
fmt.Println(<-ch)
The second program splits the send off into its own line of execution, so now we have:
ch := make(chan int) // make a new int channel
go func () { // start a new line of execution
ch <- 1 // block this second execution thread until we can send to that channel
}()
fmt.Println(<-ch) // block the main line of execution until we can read from that channel
Since those two lines of execution can work independently, the main line can get down to fmt.Println and try and receive from the channel. The second thread will wait to send until it has.
The go routine absolutely makes a difference. The go routine that writes to the channel will be blocked until your main function is ready to read from the channel in the print statement. Having two concurrent threads, one that reads and one that writes fulfills the readiness on both sides.
In your first example, the single thread gets blocked by the channel write statement and will never reach the channel read.
You need to have a concurrent go routine to read from a channel whenever you write to it. Concurrency goes hand-in-hand with channel usage.
I am trying to implement a stop to a loop in Go.
The inspiration from the code I have right now is from here:
how to kill goroutine
However, I've not been able to get my code to behave as expected. The simplified version of my code from a complex code base is this:
package main
import (
"fmt"
"time"
)
var quit chan struct{}
var send chan int
func quitroutine() {
for {
select {
case cnt := <-send:
fmt.Println(cnt)
if cnt == 5 {
quit <- struct{}{}
}
}
}
}
func runLoop() {
cnt := 0
for {
select {
case <-quit:
fmt.Println("quit!")
default:
fmt.Println("default")
}
fmt.Println("inloop")
time.Sleep(1 * time.Second)
cnt++
send <- cnt
}
}
func main() {
quit = make(chan struct{})
send = make(chan int)
go quitroutine()
runLoop()
fmt.Println("terminated")
}
This code crashes:
default
inloop
5
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.runLoop()
/tmp/t.go:37 +0x1a6
main.main()
/tmp/t.go:45 +0xa4
goroutine 5 [chan send]:
main.quitroutine()
/tmp/t.go:18 +0x10e
created by main.main
/tmp/t.go:44 +0x9f
exit status 2
Questions:
Why does it crash at all after cnt is 5? quitroutine only writes to the quitchannel if cnt == 5, but doesn't terminate itself. While runLoop, if it receives on the quit channel, should just print "quit!" (which it doesn't), but not terminate itself.
Why don't I get the "quit!" output? Do I even get the quit channel?
How does this need to be implemented correctly
Just as Adrian has said, one of your goroutines is trying to send on quit, while the other is trying to send on send. To answer your questions:
When cnt == 5 the quitroutine begins attempting to send on quit. Because quit <- struct{}{} is not a select case, the goroutine will block until another tries to read from quit. The other goroutine is similarly stuck trying to do send <- cnt (when cnt = 6).
You never get the "quit!" output because that goroutine is stuck trying to do send <-cnt.
The simplest solution I see would be to adjust runLoop() so that the send <- cnt is a case in the select.
I'd change runLoop() to look like this:
func runLoop() {
cnt := 0
for {
select {
case <-quit:
fmt.Println("quit!")
case send <- cnt: // moved stuff here
fmt.Println("inloop")
time.Sleep(1 * time.Second)
cnt++
default:
fmt.Println("default")
}
// stuff used to be here
}
}
This gives me output (until I killed the program):
default
inloop
0
inloop
1
inloop
2
inloop
3
inloop
4
inloop
5
quit!
default
inloop
6
inloop
7
Which seems to mostly be what you were after.
I'd also like to note that the select block in quitroutine() is unnecessary because it only has one case. Cleaning that up may make it more clear that that goroutine is stuck trying to send, and never takes the input from the send channel.
When you try to send on the quit channel, the quitroutine blocks until something reads from it.
At the same time, the main routine, in runloop, is trying to send the next number on the send channel. This also blocks, because the routine that would be reading from it is currently blocked trying to send on the quit channel.
Both routines are blocked, which is a deadlock, so the program crashes.
This could be fixed by either putting one or both channel send in a select, or making one or both channels buffered (even a buffer length of 1 would suffice).
package main
import (
"fmt"
"time"
)
func main() {
p := producer()
for c := range p {
fmt.Println(c)
}
}
func producer() <-chan string {
ch := make(chan string)
go func() {
for i := 0; i < 5; i++ {
ch <- fmt.Sprint("hello", i)
time.Sleep(1 * time.Second)
}
// commented the below to show the issue
// close(ch)
}()
return ch
}
Running the above code will print 5 messages and then give a "all go routines are a sleep - deadlock error". I understand that if I close the channel the error is gone.
The thing I would like to understand is how does go runtime know that the code will be waiting infinitely on the channel and that there is nothing else that will be sending data into the channel.
Now if I add an additional go routine to the main() function.. it does not throw any error and keeps waiting on the channel.
go func() {
for {
time.Sleep(2 * time.Millisecond)
}
}()
So does this mean.. the go runtime is just looking for presence of a running go routine that could potentially send data into the channel and hence not throwing the deadlock error ?
If you want some more insight into how Go implements the deadlock detection, have a look at the place in the code that throws the "all goroutines are asleep - deadlock!": https://github.com/golang/go/blob/master/src/runtime/proc.go#L3751
It looks like the Go runtime keeps some fairly simple accounting on how many goroutines there are, how many are idle, and how many are sleeping for locks (not sure which one sleep on channel I/O will increment). At any given time (serialized with the rest of the runtime), it just does some arithmetic and checks if all - idle - locked > 0... if so, then the program could still make progress... if it's 0, then you're definitely deadlocked.
It's possible you could introduce a livelock by preventing a goroutine from sleeping via an infinite loop (like what you did in your experiment, and apparently sleep for timers isn't treated the same by the runtime). The runtime wouldn't be able to detect a deadlock in that case, and run forever.
Furthermore, I'm not sure when exactly the runtime checks for deadlocks- further inspection of who calls that checkdead() may yield some insight there, if you're interested.
DISCLAIMER- I'm not a Go core developer, I just play one on TV :-)
The runtime panics with the "all go routines are a sleep - deadlock error" error when all goroutines are blocked on channel and mutex operations.
The sleeping goroutine does not block on one of these operations. There is no deadlock and therefore no panic.