I wonder why we need to run the wg.Wait() in goroutine
// This one works as expected...
func main() {
var wg sync.WaitGroup
for i:=0; i<5; i++ {
wg.Add(1)
go func() {
defer wg.Done()
}()
time.Sleep(time.Second)
}
go func() {
wg.Wait()
}()
}
But this one never ends waiting forever
func main() {
var wg sync.WaitGroup
for i:=0; i<5; i++ {
wg.Add(1)
go func() {
defer wg.Done()
}()
time.Sleep(time.Second)
}
wg.Wait()
}
Can anybody explain why I need to wait in another goroutine?
Thanks!
why we need to run the wg.Wait() in goroutine?
In the example you mention (coop/blob/master/coop.go#L85), the wait is in a goroutine in order to return immediately a channel that will indicate when all the other goroutines have completed.
Those are the goroutines to be started:
for _, fn := range fns {
go func(f func()) {
f()
wg.Done()
}(fn)
}
They mention the completion through the var wg sync.WaitGroup.
That WaitGroup is set to wait for the right number of goroutines to finish:
wg.Add(len(fns))
The wait is done in a goroutine because it will in turn signal the global completion to a channel:
go func() {
wg.Wait()
doneSig(ch, true)
}()
But the channel is returned immediately.
ch := make(chan bool, 1)
...
return ch
That is why the Wait is done asynchronously: you don't want to wait in this function. You just want the channel.
Related
I'm having trouble wrangling go routines and getting them to communicate back to a channel on the main go routine. To simplify, my code looks something like this:
func main() {
channel := make(chan string)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go performTest(channel, &wg, i)
}
wg.Wait()
close(channel)
for line := range channel {
fmt.Print(line)
}
}
func performTest(channel chan string, wg *sync.WaitGroup, i int) {
defer wg.Done()
// perform some work here
result := fmt.sprintf("Pretend result %d", i)
channel <- result
}
This seems to enter into some kind of a deadlock, but I don't understand why. It gets stuck on wg.Wait(), even though I would expect it to continue once all the goroutines have called Done on the wait group. What am I missing here? I'd like to wait for the goroutines, and then iterate over all results in the channel.
You can wait for the group and close the channel in a separate go routine. If the channel is closed, your range over the channel will end after the last sent value has been received.
If you just wait, nothing will receive from the channel. Since the channel is unbuffered, the performTest goroutines won't be able to send. For an unbuffered channel, the send operation will block until it has been received. Therefore, the deferred wg.Done call would never happen, and your program is deadlocked. Since Done is only called after the forever-blocking send has been performed.
func main() {
channel := make(chan string)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go performTest(channel, &wg, i)
}
// this is the trick
go func() {
wg.Wait()
close(channel)
}()
for line := range channel {
fmt.Print(line)
}
}
func performTest(channel chan string, wg *sync.WaitGroup, i int) {
defer wg.Done()
// perform some work here
result := fmt.Sprintf("Pretend result %d\n", i)
channel <- result
}
https://play.golang.com/p/5pACJzwL4Hi
I need several goroutines to write in the same channel. Then all the data is read in one place until all the goroutines complete the process. But I'm not sure how best to close this channel.
this is my example implementation:
func main() {
ch := make(chan data)
wg := &sync.WaitGroup{}
for instance := range dataSet {
wg.Add(1)
go doStuff(ch, instance)
}
go func() {
wg.Wait()
close(ch)
}()
for v := range ch { //range until it closes
//proceed v
}
}
func doStuff(ch chan data, instance data) {
//do some stuff with instance...
ch <- instance
}
but I'm not sure that it is idiomatic.
As you are using WaitGroup and increase the counter at the time of starting new goroutine, you have to notify WaitGroup when a goroutine is finished by calling the Done() method. Also you have to pass the same WaitGroup to the goroutine. You can do it by passing the address of WaitGroup. Otherwise each goroutine will use it's own WaitGroup which will be on different scope.
func main() {
ch := make(chan data)
wg := &sync.WaitGroup{}
for _, instance := range dataSet {
wg.Add(1)
go doStuff(ch, instance, wg)
}
go func() {
wg.Wait()
close(ch)
}()
for v := range ch { //range until it closes
//proceed v
}
}
func doStuff(ch chan data, instance data, wg *sync.WaitGroup) {
//do some stuff with instance...
ch <- instance
// call done method to decrease the counter of WaitGroup
wg.Done()
}
There's 2 ways to write the repro:
The 1st way, the program exits clean.
func recurse(depth int, wg *sync.WaitGroup) {
defer wg.Done()
if depth == 0 {
return
}
wg.Add(1)
go recurse(depth - 1, wg)
}
func main() {
wg := sync.WaitGroup{}
wg.Add(1)
go recurse(3, &wg)
wg.Wait()
}
The 2nd way, the program gives "fatal error: all goroutines are asleep - deadlock!"
func recurse(depth int, wg sync.WaitGroup) {
defer wg.Done()
if depth == 0 {
return
}
wg.Add(1)
go recurse(depth - 1, wg)
}
func main() {
var wg sync.WaitGroup
wg.Add(1)
go recurse(3, wg)
wg.Wait()
}
Would anyone kindly explain the intricate way in which the 2nd way differs from the 1st way so as to cause "Deadlock"?
WaitGroup under the hood is just a struct containing a counter guarded by mutex. Go provide arguments to functions copy_by_value way. So recurse( depth, wg) function receive only a copy of counter when passed by value. Like this:
counter := 5
func(counter){
counter--
fmt.Println(counter) //will print "4"
}(counter)
fmt.Println(counter) //will be "5" again
I have some issues with the following code:
package main
import (
"fmt"
"sync"
)
// This program should go to 11, but sometimes it only prints 1 to 10.
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, wg) //
go func(){
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
// Print prints all numbers sent on the channel.
// The function returns when the channel is closed.
func Print(ch <-chan int, wg sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
I get a deadlock at the specified place. I have tried setting wg.Add(1) instead of 2 and it solves my problem. My belief is that I'm not successfully sending the channel as an argument to the Printer function. Is there a way to do that? Otherwise, a solution to my problem is replacing the go Print(ch, wg)line with:
go func() {
Print(ch)
defer wg.Done()
}
and changing the Printer function to:
func Print(ch <-chan int) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
}
What is the best solution?
Well, first your actual error is that you're giving the Print method a copy of the sync.WaitGroup, so it doesn't call the Done() method on the one you're Wait()ing on.
Try this instead:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
Now, changing your Print method to remove the WaitGroup of it is a generally good idea: the method doesn't need to know something is waiting for it to finish its job.
I agree with #Elwinar's solution, that the main problem in your code caused by passing a copy of your Waitgroup to the Print function.
This means the wg.Done() is operated on a copy of wg you defined in the main. Therefore, wg in the main could not get decreased, and thus a deadlock happens when you wg.Wait() in main.
Since you are also asking about the best practice, I could give you some suggestions of my own:
Don't remove defer wg.Done() in Print. Since your goroutine in main is a sender, and print is a receiver, removing wg.Done() in receiver routine will cause an unfinished receiver. This is because only your sender is synced with your main, so after your sender is done, your main is done, but it's possible that the receiver is still working. My point is: don't leave some dangling goroutines around after your main routine is finished. Close them or wait for them.
Remember to do panic recovery everywhere, especially anonymous goroutine. I have seen a lot of golang programmers forgetting to put panic recovery in goroutines, even if they remember to put recover in normal functions. It's critical when you want your code to behave correctly or at least gracefully when something unexpected happened.
Use defer before every critical calls, like sync related calls, at the beginning since you don't know where the code could break. Let's say you removed defer before wg.Done(), and a panic occurrs in your anonymous goroutine in your example. If you don't have panic recover, it will panic. But what happens if you have a panic recover? Everything's fine now? No. You will get deadlock at wg.Wait() since your wg.Done() gets skipped because of panic! However, by using defer, this wg.Done() will be executed at the end, even if panic happened. Also, defer before close is important too, since its result also affects the communication.
So here is the code modified according to the points I mentioned above:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer func() {
wg.Done()
}()
for i := 1; i <= 11; i++ {
ch <- i
if i == 7 {
panic("ahaha")
}
}
println("sender done")
close(ch)
}()
wg.Wait()
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer wg.Done()
for n := range ch {
fmt.Println(n)
}
println("print done")
}
Hope it helps :)
I looked at some code that I've wrote a long time ago, when go1.3 was released(I might be wrong). CODE HERE
The below code used to work as expected, but now since I've update go to the current master version(go version devel +bd1efd5 Fri Jul 31 16:11:21 2015 +0000 darwin/amd64), the last output message c <- "FUNC 1 DONE" is not printed, the code works as it should on play.golang.org. Did I do something wrong, or this is a bug?
package main
import ("fmt";"sync";"time")
func test(c chan string, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("EXEC FUNC 1")
time.Sleep(3 * time.Second)
c <- "FUNC 1 DONE"
}
func test1(c chan string, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("EXEC FUNC 2")
time.Sleep(2 * time.Second)
c <- "FUNC 2 DONE"
}
func main() {
ch := make(chan string)
var wg sync.WaitGroup
wg.Add(2)
go test(ch, &wg)
go test1(ch, &wg)
go func(c chan string) {
for txt := range c {
fmt.Println(txt)
}
}(ch)
wg.Wait()
}
UPDATE:
I'm not saying that, the above is the best way of doing those types of work, but I don't see anything wrong with it.
Also running it in go version go1.4.2 darwin/amd64 will return the expected output.
Your code has always had this bug. It was only by chance that your program managed to print all messages before main exited.
To make this work correctly, I would invert where you have the wg.Wait() and the channel receives, so you can asynchronously close the channel. This way the receive operations are what is blocking main, and the channel is closed as soon as all send operations are done.
func main() {
ch := make(chan string)
var wg sync.WaitGroup
wg.Add(2)
go test(ch, &wg)
go test1(ch, &wg)
go func() {
wg.Wait()
close(ch)
}()
for txt := range ch {
fmt.Println(txt)
}
}