WaitGroup goroutines with channel - go

I am learning WaitGroup from the blog https://nathanleclaire.com/blog/2014/02/15/how-to-wait-for-all-goroutines-to-finish-executing-before-continuing/
the code:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
messages := make(chan int)
var wg sync.WaitGroup
// you can also add these one at
// a time if you need to
wg.Add(3)
go func() {
defer wg.Done()
time.Sleep(time.Second * 3)
messages <- 1
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 2)
messages <- 2
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 1)
messages <- 3
}()
go func() {
for i := range messages {
fmt.Println(i)
}
}()
wg.Wait()
}
I think it should print 3, 2 and 1 in order. But it only prints 3, 2 but 1 is missing, what's the problem?
You can tree it on https://play.golang.org/p/kZCvDhykYM

Right after the latest messages <- 1, the deferred wg.Done() is invoked which releases wg.Wait() in the end of the program and the program quits. When a program quits all the goroutines get killed, so the printing goroutine does not have a chance to print the latest value.
If you put something like time.Sleep(time.Second * 1) right after wg.Done() you would be able to see all the output lines.

package main
import (
"fmt"
"sync"
"time"
)
func main() {
messages := make(chan int)
var wg sync.WaitGroup
wg.Add(3)
// created this goroutine to wait for other
// goroutines to complete and to close the channel
go func() {
wg.Wait()
close(messages)
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 3)
messages <- 1
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 2)
messages <- 2
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 1)
messages <- 3
}()
// this for loop is blocked in main goroutine.
// till it reads all messages and channel is closed.
for v := range messages {
fmt.Print(v)
}
}

The mentioned blog starts with following comment:
EDIT: As pointed out by effenn in this Reddit comment, a lot of information in this article is “dangerously inaccurate”. OOPS! I’ve written a followup/correction article here for your viewing pleasure, but I’m leaving this article up for “historical purposes”.
The Reddit comment and the followup article both describe the problem and give a solution to your problem. (Adding time.Sleep(...) to make the program work the way you expect is really hacky...)

package main
import (
"fmt"
"sync"
"time"
)
func main() {
messages := make(chan int)
var wg sync.WaitGroup
// you can also add these one at
// a time if you need to
wg.Add(3)
go func() {
defer wg.Done()
time.Sleep(time.Second * 3)
messages <- 1
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 2)
messages <- 2
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 1)
messages <- 3
}()
exit:
for {
select {
case i, ok := <-messages:
if !ok {
break exit
}
fmt.Println(i)
default:
time.Sleep(time.Second)
}
}
wg.Wait()
}

Related

How to make this golang for-select code work?

Question
How to make the below code print "QUIT" after 3 seconds?
Code
package main
import (
"fmt"
"time"
)
func main() {
quit := make(chan struct{})
tasks := make(chan struct{})
go func(){
time.Sleep(1 * time.Second)
tasks <- struct{}{}
}()
go func(){
time.Sleep(3 * time.Second)
quit <- struct{}{}
}()
for {
select {
case <-quit:
fmt.Println("QUIT")
return
case <-tasks:
fmt.Println("Doing")
// do some long time jobs more than 10 seconds
time.Sleep(10 * time.Second)
}
}
}
Observations
The above code prints "Doing". and sleep 10 seconds, then print "QUIT".
How to interrupt this sleep, let it receive quit channel after 3 seconds and print "QUIT"?
It seems select is blocked by case tasks, and it will not receive from the quit channel after 3 seconds.
To signal end of an asynchronous task it is a best practice to close the channel, this is rather important to prevent many missuse that leads to various deadlocks.
In your original code I would have written close(quite) rather than quit <- struct{}{}
Remember, read on a closed does not block and always return the zero value, that is the trick.
Anyways, appart from this, an elegant way to solve your problem is to use a combination of both context.Context and time.After.
time.After will help you block on a selectable task set.
context.Context is better suited to handle this kind of signals.
https://play.golang.org/p/ZVsZw3P-YHd
package main
import (
"context"
"log"
"time"
)
func main() {
log.Println("start")
defer log.Println("end")
ctx, cancel := context.WithCancel(context.Background())
tasks := make(chan struct{})
go func() {
time.Sleep(1 * time.Second) // a job
tasks <- struct{}{}
}()
go func() {
time.Sleep(3 * time.Second) // a job
cancel()
}()
for {
select {
case <-ctx.Done():
log.Println("QUIT")
return
case <-tasks:
log.Println("Doing")
// do some long time jobs more than 10 seconds
select {
case <-ctx.Done():
return
case <-time.After(time.Second * 10):
}
}
}
}
The second job is running for 10 seconds, so it will block the loop for that time, and the signal you sent into the quit channel will not be received until this job is complete.
To interrupt the time-consuming job, maybe you can split it into another goroutine. Something like this would do:
go func() {
for {
select {
case <-tasks:
fmt.Println("Doing")
// do some long time jobs more than 10 seconds
time.Sleep(10 * time.Second)
}
}
}()
<-quit
fmt.Println("QUIT")
It would be easier to use [sync.WaitGroup.
package main
import (
"fmt"
"sync"
"time"
)
func worker(msg string, duration time.Duration, doing bool, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(duration)
fmt.Println(msg)
if doing {
time.Sleep(10 * time.Second)
}
}
func main() {
var wg sync.WaitGroup
msgs := [2]string{"QUIT", "Doing"}
durations := [2]time.Duration{3 * time.Second, 1 * time.Second}
doing := [2]bool{false, true}
for i, msg := range msgs {
wg.Add(1)
go worker(msg, durations[i], doing[i], &wg)
}
wg.Wait()
}

How to detect goroutine leaks?

In the below code:
package main
import (
"context"
"fmt"
"time"
)
func cancellation() {
duration := 150 * time.Millisecond
ctx, cancel := context.WithTimeout(context.Background(), duration)
defer cancel()
ch := make(chan string)
go func() {
time.Sleep(time.Duration(500) * time.Millisecond)
ch <- "paper"
}()
select {
case d := <-ch:
fmt.Println("work complete", d)
case <-ctx.Done():
fmt.Println("work cancelled")
}
time.Sleep(time.Second)
fmt.Println("--------------------------------------")
}
func main() {
cancellation()
}
Because of unbuffered channel(ch := make(chan string)), go-routine leaks due to block on send(ch <- "paper"), if main goroutine is not ready to receive.
Using buffered channel ch := make(chan string, 1) does not block send(ch <- "paper")
How to detect such go-routine leaks?
There are some packages that let you do that. Two that I've used in the past:
https://github.com/fortytw2/leaktest
https://github.com/uber-go/goleak
Generally, they use functionality from the runtime package to examine the stack before and after your code runs and report suspected leaks. It's recommended to use them in tests. I found this works well in practice and used it in a couple of projects.

How to handle multiple go-routines closing the same channel?

I am having 2 go-routines reading from a single channel. After 4 seconds, I cancel the context and terminate the select loop. Before terminating the loop I call close on the channel, since there are 2 go-routines the close gets called twice and causes a panic because one of the go-routines would have already closed the channel. Currently I am using a recover to recover from the panic, is there a better way of doing this?
package main
import (
"context"
"fmt"
"sync"
"time"
)
func numberGen(ctx context.Context, numChan chan int) {
num := 0
doneCh := ctx.Done()
defer func() {
if r := recover(); r != nil {
fmt.Println("recovered from ", r)
}
}()
for {
select {
case <-doneCh:
fmt.Println("done generating...")
close(numChan)
return
default:
num++
numChan <- num
}
}
}
func main() {
ctx, cancelFn := context.WithCancel(context.Background())
numChan := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go numberGen(ctx, numChan)
go numberGen(ctx, numChan)
go func(cfn context.CancelFunc) {
time.Sleep(10 * time.Millisecond)
cfn()
}(cancelFn)
for n := range numChan {
fmt.Println("received value ", n)
}
time.Sleep(2 * time.Second)
}
Close the channel after the goroutines are done sending values.
var wg sync.WaitGroup
wg.Add(2)
go numberGen(ctx, numChan, &wg)
go numberGen(ctx, numChan, &wg)
go func() {
wg.Wait()
close(numChan)
}()
Update numberGen to call Done() on the wait group. Also, remove the call to close.
func numberGen(ctx context.Context, numChan chan int, wg *sync.WaitGroup) {
defer wg.Done()
...

Best way of using sync.WaitGroup with external function

I have some issues with the following code:
package main
import (
"fmt"
"sync"
)
// This program should go to 11, but sometimes it only prints 1 to 10.
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, wg) //
go func(){
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
// Print prints all numbers sent on the channel.
// The function returns when the channel is closed.
func Print(ch <-chan int, wg sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
I get a deadlock at the specified place. I have tried setting wg.Add(1) instead of 2 and it solves my problem. My belief is that I'm not successfully sending the channel as an argument to the Printer function. Is there a way to do that? Otherwise, a solution to my problem is replacing the go Print(ch, wg)line with:
go func() {
Print(ch)
defer wg.Done()
}
and changing the Printer function to:
func Print(ch <-chan int) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
}
What is the best solution?
Well, first your actual error is that you're giving the Print method a copy of the sync.WaitGroup, so it doesn't call the Done() method on the one you're Wait()ing on.
Try this instead:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
Now, changing your Print method to remove the WaitGroup of it is a generally good idea: the method doesn't need to know something is waiting for it to finish its job.
I agree with #Elwinar's solution, that the main problem in your code caused by passing a copy of your Waitgroup to the Print function.
This means the wg.Done() is operated on a copy of wg you defined in the main. Therefore, wg in the main could not get decreased, and thus a deadlock happens when you wg.Wait() in main.
Since you are also asking about the best practice, I could give you some suggestions of my own:
Don't remove defer wg.Done() in Print. Since your goroutine in main is a sender, and print is a receiver, removing wg.Done() in receiver routine will cause an unfinished receiver. This is because only your sender is synced with your main, so after your sender is done, your main is done, but it's possible that the receiver is still working. My point is: don't leave some dangling goroutines around after your main routine is finished. Close them or wait for them.
Remember to do panic recovery everywhere, especially anonymous goroutine. I have seen a lot of golang programmers forgetting to put panic recovery in goroutines, even if they remember to put recover in normal functions. It's critical when you want your code to behave correctly or at least gracefully when something unexpected happened.
Use defer before every critical calls, like sync related calls, at the beginning since you don't know where the code could break. Let's say you removed defer before wg.Done(), and a panic occurrs in your anonymous goroutine in your example. If you don't have panic recover, it will panic. But what happens if you have a panic recover? Everything's fine now? No. You will get deadlock at wg.Wait() since your wg.Done() gets skipped because of panic! However, by using defer, this wg.Done() will be executed at the end, even if panic happened. Also, defer before close is important too, since its result also affects the communication.
So here is the code modified according to the points I mentioned above:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer func() {
wg.Done()
}()
for i := 1; i <= 11; i++ {
ch <- i
if i == 7 {
panic("ahaha")
}
}
println("sender done")
close(ch)
}()
wg.Wait()
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer wg.Done()
for n := range ch {
fmt.Println(n)
}
println("print done")
}
Hope it helps :)

GoRoutines, Channels with WaitGroup unexpected output

I looked at some code that I've wrote a long time ago, when go1.3 was released(I might be wrong). CODE HERE
The below code used to work as expected, but now since I've update go to the current master version(go version devel +bd1efd5 Fri Jul 31 16:11:21 2015 +0000 darwin/amd64), the last output message c <- "FUNC 1 DONE" is not printed, the code works as it should on play.golang.org. Did I do something wrong, or this is a bug?
package main
import ("fmt";"sync";"time")
func test(c chan string, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("EXEC FUNC 1")
time.Sleep(3 * time.Second)
c <- "FUNC 1 DONE"
}
func test1(c chan string, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("EXEC FUNC 2")
time.Sleep(2 * time.Second)
c <- "FUNC 2 DONE"
}
func main() {
ch := make(chan string)
var wg sync.WaitGroup
wg.Add(2)
go test(ch, &wg)
go test1(ch, &wg)
go func(c chan string) {
for txt := range c {
fmt.Println(txt)
}
}(ch)
wg.Wait()
}
UPDATE:
I'm not saying that, the above is the best way of doing those types of work, but I don't see anything wrong with it.
Also running it in go version go1.4.2 darwin/amd64 will return the expected output.
Your code has always had this bug. It was only by chance that your program managed to print all messages before main exited.
To make this work correctly, I would invert where you have the wg.Wait() and the channel receives, so you can asynchronously close the channel. This way the receive operations are what is blocking main, and the channel is closed as soon as all send operations are done.
func main() {
ch := make(chan string)
var wg sync.WaitGroup
wg.Add(2)
go test(ch, &wg)
go test1(ch, &wg)
go func() {
wg.Wait()
close(ch)
}()
for txt := range ch {
fmt.Println(txt)
}
}

Resources