I was just experimenting with Go channels on my Ubuntu 64 bit environment and got confused with the output the following program produced.
I got the output:
0
1
2
3
Exit
The output when I uncommented the two commented lines:
0
1
2
3
4
Exit
Please explain the behavior.
TIA.
package main
import (
"fmt"
//"time"
)
func main() {
ch := make(chan int)
done := make(chan bool)
go func() {
for i := 0; i < 5; i++ {
ch <- i
}
//time.Sleep(1 * time.Second)
done <- false
}()
go func() {
for {
select {
case message := <-ch:
fmt.Println(message)
case <-done:
return
}
}
}()
<-done
fmt.Println("Exit")
}
Your main thread is waiting on done, then exiting. Meanwhile your first go function pipes 5 values into ch, then sends to done.
The value in done then gets read from the main thread and happens to occur before the second go function reads the last value from ch. When it does so, it exits the program.
Note that if your second thread did happen to both read from ch and done, then your program would deadlock since the main thread would never receive on done and all running go threads would be blocked waiting to receive on channels.
You're not waiting for both goroutines, and only sending a single value over done to 2 receivers, which will deadlock if the second receiver happens to be main.
Using a WaitGroup simplifies the code, and allows you to easily wait for as many goroutines as needed. https://play.golang.org/p/MWknv_9AFKp
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
defer close(ch)
for i := 0; i < 5; i++ {
ch <- i
}
}()
wg.Add(1)
go func() {
defer wg.Done()
for message := range ch {
fmt.Println(message)
}
}()
wg.Wait()
fmt.Println("Exit")
You have two go routines running in parallel. One inserts 5 numbers into the channel and then signals the main thread to exit, and another one reads the numbers from the channel.
Note that once the go routine that is responsible for enqueuing the numbers into the channel finishes, it signals the main thread to exit, regardless of whether the go routine that reads the numbers finished or not. So you could end up with a case where the enqueueing go routine finished before the dequeuing finished, and the main thread exited.
By adding the sleep, you make the enqueueing go routine live a bit longer and give a chance to the dequeueing go routine to read and print all the numbers before the enqueueing go routine signals the main thread to exit.
To solve that, you could just run the dequeuing code in the main thread. No need to run it in a go routine in this case.
Related
I am trying to implement a simple logic where a Producer sends data to a channel ch with an forever for loop and a Consumer reads from the channel ch.
The Producer stops producing and exit the forever loop when it receives a signal on the channel quit.
The code is this (see also this playground)
func main() {
ch := make(chan int)
quit := make(chan bool)
var wg sync.WaitGroup
wg.Add(1)
go produce(ch, quit, &wg)
go consume(ch)
time.Sleep(1 * time.Millisecond)
fmt.Println("CLOSE")
close(quit)
wg.Wait()
}
func produce(ch chan int, quit chan bool, wg *sync.WaitGroup) {
for i := 0; ; i++ {
select {
case <-quit:
close(ch)
fmt.Println("exit")
wg.Done()
return //we exit
default:
ch <- i
fmt.Println("Producer sends", i)
}
}
}
func consume(ch chan int) {
for {
runtime.Gosched() // give the opportunity to the main goroutine to close the "quit" channel
select {
case i, more := <-ch:
if !more {
fmt.Println("exit consumer")
return
}
fmt.Println("Consumer receives", i)
}
}
}
If I run this piece of code on my machine (a Mac with 4 cores) everything works fine. If I try the same code on the Go Playgroud it always times out. I guess that this because the Go Playground is a single core and so the infinite loop does not give the chance to other goroutines to run, but then I do not understand why the instruction runtime.Gosched() does not have any effect.
Just to complete the picture I have seen that, if I set GOMAXPROCS=1 on my Mac, the program still works fine and exits as expected. If I set GOMAXPROCS=1 on my Mac and remove the runtime.Gosched() instruction, the behavior gets brittle: sometimes the program terminates as expected, some other times it seems to never exit the infinite loop.
You created a pathological situation that shouldn't happen in a real program, so the scheduler is not optimized to handle this. Combined with the fake time implementation in the playground, and you get far too many cycles of the producer and consumer before hitting a timeout.
The producer goroutine is creating values as fast as possible, while the consumer is always ready to receive them. With GOMAPXPROCS=1, the scheduler spends all its time bouncing between the two before it is forced to preempt the available work to check on the main goroutine, which takes longer than the playground will allow.
If we add something for the producer-consumer pair to do, we can limit the amount of time they have to monopolize the scheduler. For example, adding a time.Sleep(time.Microsecond) to the consumer will cause the playground to print 1000 values. This also goes to show how "accurate" the simulated time is in the playground, since that would not be possible with normal hardware which takes a non-zero amount time to process each message.
While an interesting case, this has little bearing on real programs.
A few notes, you can range over a channel to receive all values, you should always defer wg.Done at the start of the goroutine when possible, you can send values in the select case which allows you to actually cancel the for-select loop when the send isn't ready, and if you want the "exit consumer" message you need to send the WaitGroup to the consumer as well.
https://play.golang.org/p/WyPmpY9pFl7
func main() {
ch := make(chan int)
quit := make(chan bool)
var wg sync.WaitGroup
wg.Add(2)
go produce(ch, quit, &wg)
go consume(ch, &wg)
time.Sleep(50 * time.Microsecond)
fmt.Println("CLOSE")
close(quit)
wg.Wait()
}
func produce(ch chan int, quit chan bool, wg *sync.WaitGroup) {
defer wg.Done()
for i := 0; ; i++ {
select {
case <-quit:
close(ch)
fmt.Println("exit")
return
case ch <- i:
fmt.Println("Producer sends", i)
}
}
}
func consume(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
for i := range ch {
fmt.Println("Consumer receives", i)
time.Sleep(time.Microsecond)
}
fmt.Println("exit consumer")
return
}
TL;DR: A typical case of all goroutines are asleep, deadlock! but can't figure it out
I'm parsing the Wiktionary XML dump to build a DB of words. I defer the parsing of each article's text to a goroutine hoping that it will speed up the process.
It's 7GB and is processed in under 2 minutes in my machine when doing it serially, but if I can take advantage of all cores, why not.
I'm new to threading in general, I'm getting a all goroutines are asleep, deadlock! error.
What's wrong here?
This may not be performant at all, as it uses an unbuffered channel, so all goroutines effectively end up executing serially, but my idea is to learn and understand threading and to benchmark how long it takes with different alternatives:
unbuffered channel
different sized buffered channel
only calling as many goroutines at a time as there are runtime.NumCPU()
The summary of my code in pseudocode:
while tag := xml.getNextTag() {
wg.Add(1)
go parseTagText(chan, wg, tag.text)
// consume a channel message if available
select {
case msg := <-chan:
// do something with msg
default:
}
}
// reading tags finished, wait for running goroutines, consume what's left on the channel
for msg := range chan {
// do something with msg
}
// Sometimes this point is never reached, I get a deadlock
wg.Wait()
----
func parseTagText(chan, wg, tag.text) {
defer wg.Done()
// parse tag.text
chan <- whatever // just inform that the text has been parsed
}
Complete code:
https://play.golang.org/p/0t2EqptJBXE
In your complete example on the Go Playground, you:
Create a channel (line 39, results := make(chan langs)) and a wait-group (line 40, var wait sync.WaitGroup). So far so good.
Loop: in the loop, sometimes spin off a task:
if ...various conditions... {
wait.Add(1)
go parseTerm(results, &wait, text)
}
In the loop, sometimes do a non-blocking read from the channel (as shown in your question). No problem here either. But...
At the end of the loop, use:
for res := range results {
...
}
without ever calling close(results) in exactly one place, after all writers finish. This loop uses a blocking read from the channel. As long as some writer goroutine is still running, the blocking read can block without having the whole system stop, but when the last writer finishes writing and exits, there are no remaining writer goroutines. Any other remaining goroutines might rescue you, but there are none.
Since you use the var wait correctly (adding 1 in the right place, and calling Done() in the right place in the writer), the solution is to add one more goroutine, which will be the one to rescue you:
go func() {
wait.Wait()
close(results)
}()
You should spin off this rescuer goroutine just before entering the for res := range results loop. (If you spin it off any earlier, it might see the wait variable count down to zero too soon, just before it gets counted up again by spinning off another parseTerm.)
This anonymous function will block in the wait variable's Wait() function until the last writer goroutine has called the final wait.Done(), which will unblock this goroutine. Then this goroutine will call close(results), which will arrange for the for loop in your main goroutine to finish, unblocking that goroutine. When this goroutine (the rescuer) returns and thus terminates, there are no more rescuers, but we no longer need any.
(This main code then calls wait.Wait() unnecessarily: Since the for didn't terminate until the wait.Wait() in the new goroutine already unblocked, we know that this next wait.Wait() will return immediately. So we can drop this second call, although leaving it in is harmless.)
The problem is that nothing is closing the results channel, yet the range loop only exits when it closes. I've simplified your code to illustrate this and propsed a solution - basically consume the data in a goroutine:
// This is our producer
func foo(i int, ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
ch <- i
fmt.Println(i, "done")
}
// This is our consumer - it uses a different WG to signal it's done
func consumeData(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
for x := range ch {
fmt.Println(x)
}
fmt.Println("ALL DONE")
}
func main() {
ch := make(chan int)
wg := sync.WaitGroup{}
// create the producers
for i := 0; i < 10; i++ {
wg.Add(1)
go foo(i, ch, &wg)
}
// create the consumer on a different goroutine, and sync using another WG
consumeWg := sync.WaitGroup{}
consumeWg.Add(1)
go consumeData(ch,&consumeWg)
wg.Wait() // <<<< means that the producers are done
close(ch) // << Signal the consumer to exit
consumeWg.Wait() // << Wait for the consumer to exit
}
Here is my program which is producing deadlock, how do I avoid it and what is the recommended pattern to handle this kind of situation.
The problem is after timeout how do I detect that there is no reader on my channel ?
var wg sync.WaitGroup
func main() {
wg.Add(1)
c := make(chan int)
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
c <- 10
wg.Wait()
}
func readFromChannel(c chan int, ti <-chan time.Time) {
select {
case x := <-c:
fmt.Println("Read", x)
case <-ti:
fmt.Println("TIMED OUT")
}
wg.Done()
}
So, lets look at what's really going on in your source. You have two goroutines (there's more than two, but we're going to focus on the explicit ones), main and readFromChannel.
Lets look at what readFromChannel does:
if channel `c` is not empty before `ti` has expired, print its contents and return, after signalling its completion to wait group.
if `ti` has expired before `c` is not empty, print "TIMED OUT" and return, after signalling its completion to wait group.
now Main:
adds to waitgroup
make a channel `c`
start a goroutine `readFromChannel`
sleep for 5 seconds
send 10 to channel `c`
call wait for waitgroup
Now, lets go through the flow of execution for your code, concurrently (your code may/ may not execute in this order every time, keep that in mind)
1) wg.Add(1)
2) c := make(chan int)
3) go readFromChannel(c, time.After(time.Duration(2)*time.Second))
#timer ti starts#
4) time.Sleep(time.Duration(5) * time.Second)
#MAIN Goroutine begins sleep
#timer ti expires#
5) case <-ti:
6) fmt.Println("TIMED OUT")
7) wg.Done()
# readFromChannel Goroutine returns #
#MAIN Goroutine exits sleep#
8) c<-10
9) ......#DEADLOCK#
Now you can guess why you got a deadlock. In go, non buffered channels will block until something happens on the other end of the channel, regardless of whether you're sending or receiving. So c <- 10 will block until something reads from the other end of c, but the goroutine you had for that has dropped out of the picture 2 seconds ago. Therefore, c blocks forever, and since main is the last goroutine left, you get a Deadlock.
How to prevent it? When using channels, ensure that there's always a receive at the other end of the channel for every send.
Using a buffered channel in this scenario can serve as a quick fix, but may fuel potential gotchas in larger repositories. For example, assuming you wrote more data to c afterward and ran go readFromChannel(c, time.After(time.Duration(2)*time.Second)) a second time. You might see:
Read D1
Read D2
or
TIMED OUT
Read D1
solely based on chance. That's probably not the behavior you'd want.
Here's how I'd resolve the deadlock:
func main() {
wg.Add(1)
c := make(chan int)
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
c <- 10
wg.Wait()
}
func readFromChannel(c chan int, ti <-chan time.Time) {
// the forloop will run forever
loop: // **
for {
select {
case x := <-c:
fmt.Println("Read", x)
break loop // breaks out of the for loop and the select **
case <-ti:
fmt.Println("TIMED OUT")
}
}
wg.Done()
}
** see this answer for details
You have an unbuffered channel. According to the docs:
If the channel is unbuffered, the sender blocks until the receiver has
received the value. If the channel has a buffer, the sender blocks
only until the value has been copied to the buffer
By changing the channel to being buffered, we can avoid deadlock.
c := make(chan int, 10) // holds 10 ints
I also suggest reading https://golang.org/doc/effective_go.html#channels, it's got some good stuff in there related to channels.
Your problem is that you are using select statement but you are not using inside a goroutine.
go func() {
for {
select {
case x := <-c:
fmt.Println("Read", x)
case <-ti:
fmt.Println("TIMED OUT")
}
}
}()
Getting the values out of different concurrently executing goroutines can be accomplished with the select keyword, which closely resembles the switch control statement and is sometimes called the communications switch.
Using a send operation in a select statement with a default case guarantees that the send will be non-blocking! If there are no cases, the select blocks execution forever.
https://play.golang.org/p/Ai1ggveb4s
This is an older question, but i'm diving deep into learning channels myself and found this here.
I think you just need to close the channel after your done sending on it?
Code:
func main() {
wg.Add(1)
c := make(chan int)
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
c <- 10
close(c) // <- CLOSE IT HERE
wg.Wait()
}
To avoid deadlock here is some general advice from Deadlocks: the dark side of concurrency
Channel deadlock, Don’t send and receive to the same channel in the same goroutine
Read lock, Don’t take a read lock twice in the same goroutine
Release the lock as soon as possible
Objects which contain locks that call each other methods are a deadlock waiting to happen
Care total lock ordering
Access only read variables outside the lock
You can add a buffer to the c chan, like this:
var wg sync.WaitGroup
func main() {
wg.Add(1)
c := make(chan int, 1)
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
c <- 10 // never blocks, because of the buffer
wg.Wait()
}
func readFromChannel(c chan int, ti <-chan time.Time) {
select {
case x := <-c:
fmt.Println("Read", x)
case <-ti:
fmt.Println("TIMED OUT")
}
wg.Done()
}
a write operation(c <- 10) to an unbuffered channel will be blocked until it there is a read operation(case x := <-c:).
Your timeout(2s) is smaller than sleep duration(5s) and because of this channel c will never be read in the select block.
I would recommend you to read chapter 8 & 9 of this book
func main() {
wg.Add(1)
c := make(chan int)
go func() {
c <- 10
}()
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
wg.Wait()
}//This will read without deadlock
func main() {
wg.Add(1)
c := make(chan int)
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
go func() {
c <- 10
}()
wg.Wait()
}//Time out without deadlock
I am trying to repro an issue and came to a minimum use case with the following code. If I close all the channels (bypassing the i == 0 test), things are working as expected. Wg state decrements and done is triggered, main exits fine. When I skip closing one of these channel (on purpose), I expect the main routine to wait while the waitgroup semaphore will block indefinitely in this case. Instead, I am getting an error: "fatal error: all goroutines are asleep - deadlock!". Why is that? I must have missed something fundamental or this the runtime being overzealous?
package main
import (
"fmt"
"sync"
)
const N int = 4
func main() {
done := make(chan struct{})
defer close(done)
fmt.Println("Beginning...")
chans := make([]chan int, N)
var wg sync.WaitGroup
for i := 0; i < N; i++ {
wg.Add(1)
chans[i] = make(chan int)
go func(i int) { // p0
defer wg.Done()
for m := range chans[i] {
fmt.Println("Received ", m)
}
fmt.Println("Ending p", i)
}(i)
}
go func() {
wg.Wait()
done <- struct{}{} // signal main that we are done
}()
for i := 0; i < N; i++ {
fmt.Println("Closing c", i)
if i != 0 { // Skip #0 so wg doesn't reach '0'
close(chans[i])
}
}
<-done // wait to receive signal from anonymous join function
fmt.Println("Ending.")
}
UPDATE: I edited the code to avoid the race condition. Still getting this error.
The if i != 0 is there because it's intentional. I want the wg.Wait to block forever (with its semaphore never reaching 0.) Why can't I do that? It seems the same as if I were using <-done without a matching done <- struct{}{} somewhere else. Would the compiler complain too in that case?
Here's what's going on:
The first go func(i int) { goroutine does not exit because chans[0] is not closed.
Because the goroutine does not exit, wg.Done is not called.
The call to wg.Wait() blocks forever because of the previous point.
Main blocks forever because the signal is not sent to done.
You can fix the deadlock by removing the if i != 0 {, but there is another issue. There is a race on the wait group. It's possible that wg.Done() is called before wg.Add(1) is called. Call wg.Add() before starting the goroutine to avoid the race.
The if statement in your for loop doesn't let the last channel close, so your goroutine is left waiting on something to happen to chans[i] which will block the defer wg.Done() from ever happening which in turn will never let wg.Wait() finish WHICH THENNNNN will never let done <- struct{}{} get signalled
So in short, your if statement in your loop is not closing the last channel and causing a deadlock because nobody can do nothing.
As #CodingPickle did point out, move your wg.Add(1) to the beginning of your for loop to prevent any race conditions
http://play.golang.org/p/j1D5LZGUhd
I'm new to Go and am stumped on what appears to be a somewhat-rare race condition with a very small block of code running on Linux with Go version 1.2.
Basically, I create a channel for an int, start a go routine to read from the channel, and then write a single int to the channel.
package main
import "fmt"
func main() {
channel := make(chan int)
go func() {
number := <- channel
fmt.Printf("GOT IT: %d\n", number)
}()
fmt.Println("[+] putting num on channel")
channel <- 42
fmt.Println("[-] putting num on channel")
}
The output about 90% of the time is as expected:
$ go run test.go
[+] putting num on channel
GOT IT: 42
[-] putting num on channel
However, about 10% of the time, the go routine simply does not read the number from the channel and prints nothing:
$ go run test.go
[+] putting num on channel
[-] putting num on channel
I'm puzzled because this code is very similar to the example at https://gobyexample.com/channels, (which I do not have this problem with) except that I'm reading from the channel in my go routine instead of writing to the channel.
Do I have a fundamental misunderstanding of how channels work or is there something else at play here?
You should wait until your goroutine executes, and then your, for example, you can do it with sync.WaitGroup:
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
channel := make(chan int)
wg.Add(1)
go func() {
number := <-channel
fmt.Printf("GOT IT: %d\n", number)
wg.Done()
}()
fmt.Println("[+] putting num on channel")
channel <- 42
wg.Wait()
fmt.Println("[-] putting num on channel")
}
(goplay: http://play.golang.org/p/VycxTw_4vu)
Also you can do it with a "notification channel", that indicates that job is done:
package main
import "fmt"
func main() {
channel := make(chan int)
done := make(chan bool)
go func() {
number := <-channel
fmt.Printf("GOT IT: %d\n", number)
done <- true
}()
fmt.Println("[+] putting num on channel")
channel <- 42
<-done
fmt.Println("[-] putting num on channel")
}
(goplay: http://play.golang.org/p/fApWQgtr4D)
You seem to be expecting the receiving goroutine to run to completion before the second fmt.Println executes. This is not guaranteed to be the case. If the program terminates, goroutines are not guaranteed to reach the end of their functions.
When you see the output that doesn't display the "GOT IT" message, the channel delivered its message, but the main function completed before the goroutine did. The program terminated, and the goroutine never gets the chance to call fmt.Printf
In the example you cited, the main function ends with this:
go func() { messages <- "ping" }()
msg := <-messages
fmt.Println(msg)
Since the main function blocks until it receives a message, the goroutine always runs to completion in this example. In your code, your goroutine executes a step after it receives from the channel, and it's undefined whether the goroutine or the main function will execute the next line after the receive.
You have two goroutines, one in main() (which is implicitly a goroutine), and the anonymous one.
They communicate over a synchronous channel, so after the channel communication, it's guaranteed that they're synchronised.
At this point, the code left in the main() goroutine looks like this:
fmt.Println("[-] putting num on a channel")
and the code left in the anonymous goroutine looks like this:
fmt.Println("GOT IT: %d\n", number)
Now you're racing: the output from these Printlns may appear in either order, or even intermingled. When the Println() from main finishes, the next thing that will happen on that goroutine is that your program will be stopped. This may prevent some or all of the Println from the anonymous goroutine from appearing.