i am new to the Go language, and looking for some help for the concurrency model.
say I want to make 2 http calls concurrently, and wait for both of them to finish then process/merge the response data.
here is the code i have
func main() {
var wg sync.WaitGroup
wg.Add(2)
c1 := make(chan string)
c2 := make(chan string)
go foo(c1, &wg)
go bar(c2, &wg)
wg.Wait()
foo := <-c1
bar := <-c2
fmt.Println("foo: ", foo)
fmt.Println("bar: ", bar)
}
func foo(c chan string, wg *sync.WaitGroup) {
defer wg.Done()
c <- "foo"
}
func bar(c chan string, wg *sync.WaitGroup) {
defer wg.Done()
c <- "bar"
}
however when i run it, it gives the error fatal error: all goroutines are asleep - deadlock!
I can get it working without the WaitGroup, but just curious why this gets to the deadlock, and whats the best way of doing it
The goroutines will block waiting to write to the channel, because the channel read happens after goroutines end (after wg.Done), thus the deadlock.
Simples solution here is to get rid of the waitgroup. The channel read operations will block until the goroutines write to the channel, so after both channels are read, there is no need to wait.
Indeed the waitgroup is not really needed here. But if you want to continue experimenting with goroutines and channels, then you can also try and make the channels buffered, like:
c1 := make(chan string, 1)
c2 := make(chan string, 1)
What happens then is that you can write a single entry to each channel without blocking.
Reading/writing to an unbuffered channel is a blocking call, meaning that these lines:
c <- "foo"
c <- "bar"
will hang until you reach calls that pull the values from the channels, i.e., these lines:
foo := <-c1
bar := <-c2
The cause of your deadlock is calling wg.Wait() before these two lines. In order to move past wg.Wait(), all waitgroups must complete, but your waitgroups cannot because wg.Done() is deferred until c <- "foo"/c <- "bar" unblock. As mentioned before, these cannot unblock until you reach foo := <-c1/bar := <-c2, and these cannot be reached because of wg.Wait(). You see the how no progress is possible, hence the deadlock.
The best practice is to avoid using waitgroups and mutex until absolutely necessary. Often, (like in this case) a solution is possible in pure Go and these packages only complicate the code.
Related
TL;DR: A typical case of all goroutines are asleep, deadlock! but can't figure it out
I'm parsing the Wiktionary XML dump to build a DB of words. I defer the parsing of each article's text to a goroutine hoping that it will speed up the process.
It's 7GB and is processed in under 2 minutes in my machine when doing it serially, but if I can take advantage of all cores, why not.
I'm new to threading in general, I'm getting a all goroutines are asleep, deadlock! error.
What's wrong here?
This may not be performant at all, as it uses an unbuffered channel, so all goroutines effectively end up executing serially, but my idea is to learn and understand threading and to benchmark how long it takes with different alternatives:
unbuffered channel
different sized buffered channel
only calling as many goroutines at a time as there are runtime.NumCPU()
The summary of my code in pseudocode:
while tag := xml.getNextTag() {
wg.Add(1)
go parseTagText(chan, wg, tag.text)
// consume a channel message if available
select {
case msg := <-chan:
// do something with msg
default:
}
}
// reading tags finished, wait for running goroutines, consume what's left on the channel
for msg := range chan {
// do something with msg
}
// Sometimes this point is never reached, I get a deadlock
wg.Wait()
----
func parseTagText(chan, wg, tag.text) {
defer wg.Done()
// parse tag.text
chan <- whatever // just inform that the text has been parsed
}
Complete code:
https://play.golang.org/p/0t2EqptJBXE
In your complete example on the Go Playground, you:
Create a channel (line 39, results := make(chan langs)) and a wait-group (line 40, var wait sync.WaitGroup). So far so good.
Loop: in the loop, sometimes spin off a task:
if ...various conditions... {
wait.Add(1)
go parseTerm(results, &wait, text)
}
In the loop, sometimes do a non-blocking read from the channel (as shown in your question). No problem here either. But...
At the end of the loop, use:
for res := range results {
...
}
without ever calling close(results) in exactly one place, after all writers finish. This loop uses a blocking read from the channel. As long as some writer goroutine is still running, the blocking read can block without having the whole system stop, but when the last writer finishes writing and exits, there are no remaining writer goroutines. Any other remaining goroutines might rescue you, but there are none.
Since you use the var wait correctly (adding 1 in the right place, and calling Done() in the right place in the writer), the solution is to add one more goroutine, which will be the one to rescue you:
go func() {
wait.Wait()
close(results)
}()
You should spin off this rescuer goroutine just before entering the for res := range results loop. (If you spin it off any earlier, it might see the wait variable count down to zero too soon, just before it gets counted up again by spinning off another parseTerm.)
This anonymous function will block in the wait variable's Wait() function until the last writer goroutine has called the final wait.Done(), which will unblock this goroutine. Then this goroutine will call close(results), which will arrange for the for loop in your main goroutine to finish, unblocking that goroutine. When this goroutine (the rescuer) returns and thus terminates, there are no more rescuers, but we no longer need any.
(This main code then calls wait.Wait() unnecessarily: Since the for didn't terminate until the wait.Wait() in the new goroutine already unblocked, we know that this next wait.Wait() will return immediately. So we can drop this second call, although leaving it in is harmless.)
The problem is that nothing is closing the results channel, yet the range loop only exits when it closes. I've simplified your code to illustrate this and propsed a solution - basically consume the data in a goroutine:
// This is our producer
func foo(i int, ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
ch <- i
fmt.Println(i, "done")
}
// This is our consumer - it uses a different WG to signal it's done
func consumeData(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
for x := range ch {
fmt.Println(x)
}
fmt.Println("ALL DONE")
}
func main() {
ch := make(chan int)
wg := sync.WaitGroup{}
// create the producers
for i := 0; i < 10; i++ {
wg.Add(1)
go foo(i, ch, &wg)
}
// create the consumer on a different goroutine, and sync using another WG
consumeWg := sync.WaitGroup{}
consumeWg.Add(1)
go consumeData(ch,&consumeWg)
wg.Wait() // <<<< means that the producers are done
close(ch) // << Signal the consumer to exit
consumeWg.Wait() // << Wait for the consumer to exit
}
I have issue when using Go routine with channel. The code looks like this:
func main() {
c := make(chan int)
var wg sync.WaitGroup
wg.Add(1)
go func (c chan int, x int) {
c <- x
fmt.Println(x)
close(c)
defer wg.Done()
}(c,10)
wg.Wait()
}
When run the code I got this error:
fatal error: all goroutines are asleep - deadlock!
I cannot understand why this issue happens. Please help me to understand
You have 2 goroutines in your example: the main goroutine running the main() function, and the other you launch inside it. The main goroutine waits for the other to complete (to call wg.Done()), and the other goroutine blocks in the line where it attempts to send a value on channel c. Since nobody is receiving from that channel, and because that channel is unbuffered, this goroutine will never advance, so all your 2 goroutines will block forever.
Note that defer wg.Done() should be the first statement in the goroutine. If it's the last, defer won't make any difference.
If the channel would have a buffer of at least one, the send operation could proceed:
c := make(chan int, 1)
And output will be (try it on the Go Playground):
10
If we leave the channel unbuffered, there must be another goroutine that receives from the channel, e.g.:
wg.Add(1)
go func() {
defer wg.Done()
x := <-c
fmt.Println("Received:", x)
}()
Then output will be (try it on the Go Playground):
10
Received: 10
iam currently playing around with go routines, channels and sync.WaitGroup. I know waitgroup is used to wait for all go routines to complete based on weather wg.Done() has been called enough times to derement the value set in wg.Add().
i wrote a small bit of code to try to test this in golang playground. show below
var channel chan int
var wg sync.WaitGroup
func main() {
channel := make(chan int)
mynums := []int{1,2,3,4,5,6,7,8,9}
wg.Add(1)
go addStuff(mynums)
wg.Wait()
close(channel)
recieveStuff(channel)
}
func addStuff(mynums []int) {
for _, val := range mynums {
channel <- val
}
wg.Done()
}
func recieveStuff(channel chan int) {
for val := range channel{
fmt.Println(val)
}
}
Iam getting a deadlock error. iam trying to wait on the routing to return with wg.Wait()? then, close the channel. Afterwards, send the channel to the recievestuff method to output the values in the slice? but it doesnt work. Ive tried moving the close() method inside the go routine after the loop also as i thought i may of been trying to close on the wrong routine in main(). ive found this stuff relatively confusing so far coming from java and c#. Any help is appreciated.
The call to wg.Wait() wouldn't return until wg.Done() has been called once.
In addStuff(), you're writing values to a channel when there's no other goroutine to drain those values. Since the channel is unbuffered, the first call to channel <- val would block forever, resulting in a deadlock.
Moreover, the channel in addStuff() remains nil since you're creating a new variable binding inside main, instead of assigning to the package-level variable. Writing to a nil channel blocks forever:
channel := make(chan int) //doesn't affect the package-level variable
Here's a modified example that runs to completion by consuming all values from the channel:
https://play.golang.org/p/6gcyDWxov7
Here is my program which is producing deadlock, how do I avoid it and what is the recommended pattern to handle this kind of situation.
The problem is after timeout how do I detect that there is no reader on my channel ?
var wg sync.WaitGroup
func main() {
wg.Add(1)
c := make(chan int)
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
c <- 10
wg.Wait()
}
func readFromChannel(c chan int, ti <-chan time.Time) {
select {
case x := <-c:
fmt.Println("Read", x)
case <-ti:
fmt.Println("TIMED OUT")
}
wg.Done()
}
So, lets look at what's really going on in your source. You have two goroutines (there's more than two, but we're going to focus on the explicit ones), main and readFromChannel.
Lets look at what readFromChannel does:
if channel `c` is not empty before `ti` has expired, print its contents and return, after signalling its completion to wait group.
if `ti` has expired before `c` is not empty, print "TIMED OUT" and return, after signalling its completion to wait group.
now Main:
adds to waitgroup
make a channel `c`
start a goroutine `readFromChannel`
sleep for 5 seconds
send 10 to channel `c`
call wait for waitgroup
Now, lets go through the flow of execution for your code, concurrently (your code may/ may not execute in this order every time, keep that in mind)
1) wg.Add(1)
2) c := make(chan int)
3) go readFromChannel(c, time.After(time.Duration(2)*time.Second))
#timer ti starts#
4) time.Sleep(time.Duration(5) * time.Second)
#MAIN Goroutine begins sleep
#timer ti expires#
5) case <-ti:
6) fmt.Println("TIMED OUT")
7) wg.Done()
# readFromChannel Goroutine returns #
#MAIN Goroutine exits sleep#
8) c<-10
9) ......#DEADLOCK#
Now you can guess why you got a deadlock. In go, non buffered channels will block until something happens on the other end of the channel, regardless of whether you're sending or receiving. So c <- 10 will block until something reads from the other end of c, but the goroutine you had for that has dropped out of the picture 2 seconds ago. Therefore, c blocks forever, and since main is the last goroutine left, you get a Deadlock.
How to prevent it? When using channels, ensure that there's always a receive at the other end of the channel for every send.
Using a buffered channel in this scenario can serve as a quick fix, but may fuel potential gotchas in larger repositories. For example, assuming you wrote more data to c afterward and ran go readFromChannel(c, time.After(time.Duration(2)*time.Second)) a second time. You might see:
Read D1
Read D2
or
TIMED OUT
Read D1
solely based on chance. That's probably not the behavior you'd want.
Here's how I'd resolve the deadlock:
func main() {
wg.Add(1)
c := make(chan int)
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
c <- 10
wg.Wait()
}
func readFromChannel(c chan int, ti <-chan time.Time) {
// the forloop will run forever
loop: // **
for {
select {
case x := <-c:
fmt.Println("Read", x)
break loop // breaks out of the for loop and the select **
case <-ti:
fmt.Println("TIMED OUT")
}
}
wg.Done()
}
** see this answer for details
You have an unbuffered channel. According to the docs:
If the channel is unbuffered, the sender blocks until the receiver has
received the value. If the channel has a buffer, the sender blocks
only until the value has been copied to the buffer
By changing the channel to being buffered, we can avoid deadlock.
c := make(chan int, 10) // holds 10 ints
I also suggest reading https://golang.org/doc/effective_go.html#channels, it's got some good stuff in there related to channels.
Your problem is that you are using select statement but you are not using inside a goroutine.
go func() {
for {
select {
case x := <-c:
fmt.Println("Read", x)
case <-ti:
fmt.Println("TIMED OUT")
}
}
}()
Getting the values out of different concurrently executing goroutines can be accomplished with the select keyword, which closely resembles the switch control statement and is sometimes called the communications switch.
Using a send operation in a select statement with a default case guarantees that the send will be non-blocking! If there are no cases, the select blocks execution forever.
https://play.golang.org/p/Ai1ggveb4s
This is an older question, but i'm diving deep into learning channels myself and found this here.
I think you just need to close the channel after your done sending on it?
Code:
func main() {
wg.Add(1)
c := make(chan int)
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
c <- 10
close(c) // <- CLOSE IT HERE
wg.Wait()
}
To avoid deadlock here is some general advice from Deadlocks: the dark side of concurrency
Channel deadlock, Don’t send and receive to the same channel in the same goroutine
Read lock, Don’t take a read lock twice in the same goroutine
Release the lock as soon as possible
Objects which contain locks that call each other methods are a deadlock waiting to happen
Care total lock ordering
Access only read variables outside the lock
You can add a buffer to the c chan, like this:
var wg sync.WaitGroup
func main() {
wg.Add(1)
c := make(chan int, 1)
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
c <- 10 // never blocks, because of the buffer
wg.Wait()
}
func readFromChannel(c chan int, ti <-chan time.Time) {
select {
case x := <-c:
fmt.Println("Read", x)
case <-ti:
fmt.Println("TIMED OUT")
}
wg.Done()
}
a write operation(c <- 10) to an unbuffered channel will be blocked until it there is a read operation(case x := <-c:).
Your timeout(2s) is smaller than sleep duration(5s) and because of this channel c will never be read in the select block.
I would recommend you to read chapter 8 & 9 of this book
func main() {
wg.Add(1)
c := make(chan int)
go func() {
c <- 10
}()
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
wg.Wait()
}//This will read without deadlock
func main() {
wg.Add(1)
c := make(chan int)
go readFromChannel(c, time.After(time.Duration(2)*time.Second))
time.Sleep(time.Duration(5) * time.Second)
go func() {
c <- 10
}()
wg.Wait()
}//Time out without deadlock
I'm working through the Concurrency section of A Tour of Go, and I'm curious about proper Go convention for consuming finite channels. In this exercise, I need to read values from two channels and determine if the values are the same and in the same order. If not, I can immediately return false from my method. However, if I do that, will Go clean up my channels for me automatically, or will my goroutines be left hanging forever and consuming resources?
The best way to handle this would be to pass a cancel channel into my goroutines, but since the goroutines read a finite amount of data, it seems fine to just consume all the data. What is the best way to handle this case in real life?
Andrew Gerrand's talk at Gophercon covers this exact question on slide 37.
Create a quit channel and pass it to each walker. By closing quit
when the Same exits, any running walkers are terminated.
func Same(t1, t2 *tree.Tree) bool {
quit := make(chan struct{})
defer close(quit)
w1, w2 := Walk(t1, quit), Walk(t2, quit)
for {
v1, ok1 := <-w1
v2, ok2 := <-w2
if v1 != v2 || ok1 != ok2 {
return false
}
if !ok1 {
return true
}
}
}
Using quit channels,
as discussed in the
Go Concurrency Patterns: Pipelines and cancellation
blog article
and Heath Borders' answer,
is often a good idea.
There is also the
golang.org/x/net/context package
discussed in the
Go Concurrency Patterns: Context
blog article which adds timeouts, deadlines, and other features.
However, to directly address:
will Go clean up my channels for me automatically
It depends on the channel buffering and how the channels are written to.
E.g.
func writeValues(n int, c chan int) {
for i := 0; i < n; i++ {
c <- i
}
log.Println("writeValues", n, "done")
}
func main() {
ch1 := make(chan int, 12)
ch2 := make(chan int, 6)
ch3 := make(chan int)
go writeValues(10, ch1)
go writeValues(11, ch2)
go writeValues(12, ch3)
time.Sleep(time.Second) // XXX
}
Playground
Here the first goroutine will complete and ch1 (and anything buffered in it) would be garbage collected and cleaned up.
However, the later two goroutines will block waiting until they can write all their values. The garbage collector would never touch ch2 and ch3 since the blocked goroutines keep a reference to them.
Note ch2 would get cleaned up if as few as five items where read from the channel.
Usually, you only rely on this when doing something like:
errc := make(chan error, 1)
go func() { errc <- someErrorReturningFunc() }()
If the function being called has no way to cancel it, then this is a common idiom.
You can do this and abort/return early without reading from errc and know that the goroutine and channel will be cleaned up when the function eventually returns.
The buffer size of the errc channel is important here.