I am having 2 go-routines reading from a single channel. After 4 seconds, I cancel the context and terminate the select loop. Before terminating the loop I call close on the channel, since there are 2 go-routines the close gets called twice and causes a panic because one of the go-routines would have already closed the channel. Currently I am using a recover to recover from the panic, is there a better way of doing this?
package main
import (
"context"
"fmt"
"sync"
"time"
)
func numberGen(ctx context.Context, numChan chan int) {
num := 0
doneCh := ctx.Done()
defer func() {
if r := recover(); r != nil {
fmt.Println("recovered from ", r)
}
}()
for {
select {
case <-doneCh:
fmt.Println("done generating...")
close(numChan)
return
default:
num++
numChan <- num
}
}
}
func main() {
ctx, cancelFn := context.WithCancel(context.Background())
numChan := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go numberGen(ctx, numChan)
go numberGen(ctx, numChan)
go func(cfn context.CancelFunc) {
time.Sleep(10 * time.Millisecond)
cfn()
}(cancelFn)
for n := range numChan {
fmt.Println("received value ", n)
}
time.Sleep(2 * time.Second)
}
Close the channel after the goroutines are done sending values.
var wg sync.WaitGroup
wg.Add(2)
go numberGen(ctx, numChan, &wg)
go numberGen(ctx, numChan, &wg)
go func() {
wg.Wait()
close(numChan)
}()
Update numberGen to call Done() on the wait group. Also, remove the call to close.
func numberGen(ctx context.Context, numChan chan int, wg *sync.WaitGroup) {
defer wg.Done()
...
Related
Run the below program and run CTRL + C, the handle routine gets blocked as it is trying to send to a channel but the process routine has shutdown. What is a better concurrency design to solve this?
Edited the program to describe the problem applying the rules suggested here https://stackoverflow.com/a/66708290/4106031
package main
import (
"context"
"fmt"
"os"
"os/signal"
"sync"
"syscall"
"time"
)
func process(ctx context.Context, c chan string) {
fmt.Println("process: processing (select)")
for {
select {
case <-ctx.Done():
fmt.Printf("process: ctx done bye\n")
return
case i := <-c:
fmt.Printf("process: received i: %v\n", i)
}
}
}
func handle(ctx context.Context, readChan <-chan string) {
c := make(chan string, 1)
wg := &sync.WaitGroup{}
wg.Add(1)
go func() {
process(ctx, c)
wg.Done()
}()
defer wg.Wait()
for i := 0; ; i++ {
select {
case <-ctx.Done():
fmt.Printf("handle: ctx done bye\n")
return
case i := <-readChan:
fmt.Printf("handle: received: %v\n", i)
fmt.Printf("handle: sending for processing: %v\n", i)
// suppose huge time passes here
// to cause the issue we want to happen
// we want the process() to exit due to ctx
// cancellation before send to it happens, this creates deadlock
time.Sleep(5 * time.Second)
// deadlock
c <- i
}
}
}
func main() {
wg := &sync.WaitGroup{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
readChan := make(chan string, 10)
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; ; i++ {
select {
case <-ctx.Done():
fmt.Printf("read: ctx done bye\n")
return
case readChan <- fmt.Sprintf("%d", i):
fmt.Printf("read: sent msg: %v\n", i)
}
}
}()
wg.Add(1)
go func() {
handle(ctx, readChan)
wg.Done()
}()
go func() {
sigterm := make(chan os.Signal, 1)
signal.Notify(sigterm, syscall.SIGINT, syscall.SIGTERM)
select {
case <-sigterm:
fmt.Printf("SIGTERM signal received\n")
cancel()
}
}()
wg.Wait()
}
Output
$ go run chan-shared.go
read: sent msg: 0
read: sent msg: 1
read: sent msg: 2
read: sent msg: 3
process: processing (select)
read: sent msg: 4
read: sent msg: 5
read: sent msg: 6
handle: received: 0
handle: sending for processing: 0
read: sent msg: 7
read: sent msg: 8
read: sent msg: 9
read: sent msg: 10
handle: received: 1
handle: sending for processing: 1
read: sent msg: 11
process: received i: 0
process: received i: 1
read: sent msg: 12
handle: received: 2
handle: sending for processing: 2
^CSIGTERM signal received
process: ctx done bye
read: ctx done bye
handle: received: 3
handle: sending for processing: 3
Killed: 9
the step by step review
Always cancel context, whatever you think.
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
Dont wd.Add after starting a routine
wg.Add(1)
go handle(ctx, wg)
Dont sparsely consume waitgroups
wg.Add(1)
go func() {
handle(ctx)
wg.Done()
}()
dont for loop on a channel with a default case. Just read from it and let it unblocks
<-sigterm
fmt.Printf("SIGTERM signal received\n")
main never block on signals, main blocks on the processing routines. Signaling should just do signaling, ie cancel the context.
go func() {
sigterm := make(chan os.Signal, 1)
signal.Notify(sigterm, syscall.SIGINT, syscall.SIGTERM)
<-sigterm
fmt.Printf("SIGTERM signal received\n")
cancel()
}()
It is possible to check for context cancellation on channel writes.
select {
case <-ctx.Done():
fmt.Printf("process: ctx done bye\n")
return
case c <- fmt.Sprintf("%d", i):
fmt.Printf("handled: sent to channel: %v\n", i)
}
Dont time.Sleep, you can t test for context cancellation with it.
select {
case <-ctx.Done():
fmt.Printf("process: ctx done bye\n")
return
case <-time.After(time.Second * 5):
}
So a complete revised version of the code with those various rules applied gives us
package main
import (
"context"
"fmt"
"os"
"os/signal"
"sync"
"syscall"
"time"
)
func process(ctx context.Context, c chan string) {
fmt.Println("process: processing (select)")
for {
select {
case <-ctx.Done():
fmt.Printf("process: ctx done bye\n")
return
case msg := <-c:
fmt.Printf("process: got msg: %v\n", msg)
}
}
}
func handle(ctx context.Context) {
c := make(chan string, 3)
wg := &sync.WaitGroup{}
wg.Add(1)
go func() {
process(ctx, c)
wg.Done()
}()
defer wg.Wait()
for i := 0; ; i++ {
select {
case <-ctx.Done():
fmt.Printf("process: ctx done bye\n")
return
case <-time.After(time.Second * 5):
}
select {
case <-ctx.Done():
fmt.Printf("process: ctx done bye\n")
return
case c <- fmt.Sprintf("%d", i):
fmt.Printf("handled: sent to channel: %v\n", i)
}
}
}
func main() {
wg := &sync.WaitGroup{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
wg.Add(1)
go func() {
handle(ctx)
wg.Done()
}()
go func() {
sigterm := make(chan os.Signal, 1)
signal.Notify(sigterm, syscall.SIGINT, syscall.SIGTERM)
<-sigterm
fmt.Printf("SIGTERM signal received\n")
cancel()
}()
wg.Wait()
}
There is more to tell about exit conditions, but this is dependent on the requirements.
As mentioned https://stackoverflow.com/a/66708290/4106031 this change has fixed the issue for me. Thanks mh-cbon for the rules too!
I am prototyping a series of go routines for a pipeline that each perform a transformation. The routines are terminating before all the data has passed through.
I have checked Donavan and Kernighan book and Googled for solutions.
Here is my code:
package main
import (
"fmt"
"sync"
)
func main() {
a1 := []string{"apple", "apricot"}
chan1 := make(chan string)
chan2 := make(chan string)
chan3 := make(chan string)
var wg sync.WaitGroup
go Pipe1(chan2, chan1, &wg)
go Pipe2(chan3, chan2, &wg)
go Pipe3(chan3, &wg)
func (data []string) {
defer wg.Done()
for _, s := range data {
wg.Add(1)
chan1 <- s
}
go func() {
wg.Wait()
close(chan1)
}()
}(a1)
}
func Pipe1(out chan<- string, in <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for s := range in {
wg.Add(1)
out <- s + "s are"
}
}
func Pipe2(out chan<- string, in <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for s := range in {
wg.Add(1)
out <- s + " good for you"
}
}
func Pipe3(in <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for s := range in {
wg.Add(1)
fmt.Println(s)
}
}
My expected output is:
apples are good for you
apricots are good for you
The results of running main are inconsistent. Sometimes I get both lines. Sometimes I just get the apples. Sometimes nothing is output.
As Adrian already pointed out, your WaitGroup.Add and WaitGroup.Done calls are mismatched. However, in cases like this the "I am done" signal is typically given by closing the output channel. WaitGroups are only necessary if work is shared between several goroutines (i.e. several goroutines consume the same channel), which isn't the case here.
package main
import (
"fmt"
)
func main() {
a1 := []string{"apple", "apricot"}
chan1 := make(chan string)
chan2 := make(chan string)
chan3 := make(chan string)
go func() {
for _, s := range a1 {
chan1 <- s
}
close(chan1)
}()
go Pipe1(chan2, chan1)
go Pipe2(chan3, chan2)
// This range loop terminates when chan3 is closed, which Pipe2 does after
// chan2 is closed, which Pipe1 does after chan1 is closed, which the
// anonymous goroutine above does after it sent all values.
for s := range chan3 {
fmt.Println(s)
}
}
func Pipe1(out chan<- string, in <-chan string) {
for s := range in {
out <- s + "s are"
}
close(out) // let caller know that we're done
}
func Pipe2(out chan<- string, in <-chan string) {
for s := range in {
out <- s + " good for you"
}
close(out) // let caller know that we're done
}
Try it on the playground: https://play.golang.org/p/d2J4APjs_lL
You're calling wg.Wait in a goroutine, so main is allowed to return (and therefore your program exits) before the other routines have finished. This would cause the behavior your see, but taking out of a goroutine alone isn't enough.
You're also misusing the WaitGroup in general; your Add and Done calls don't relate to one another, and you don't have as many Dones as you have Adds, so the WaitGroup will never finish. If you're calling Add in a loop, then every loop iteration must also result in a Done call; as you have it now, you defer wg.Done() before each of your loops, then call Add inside the loop, resulting in one Done and many Adds. This code would need to be significantly revised to work as intended.
I am learning WaitGroup from the blog https://nathanleclaire.com/blog/2014/02/15/how-to-wait-for-all-goroutines-to-finish-executing-before-continuing/
the code:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
messages := make(chan int)
var wg sync.WaitGroup
// you can also add these one at
// a time if you need to
wg.Add(3)
go func() {
defer wg.Done()
time.Sleep(time.Second * 3)
messages <- 1
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 2)
messages <- 2
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 1)
messages <- 3
}()
go func() {
for i := range messages {
fmt.Println(i)
}
}()
wg.Wait()
}
I think it should print 3, 2 and 1 in order. But it only prints 3, 2 but 1 is missing, what's the problem?
You can tree it on https://play.golang.org/p/kZCvDhykYM
Right after the latest messages <- 1, the deferred wg.Done() is invoked which releases wg.Wait() in the end of the program and the program quits. When a program quits all the goroutines get killed, so the printing goroutine does not have a chance to print the latest value.
If you put something like time.Sleep(time.Second * 1) right after wg.Done() you would be able to see all the output lines.
package main
import (
"fmt"
"sync"
"time"
)
func main() {
messages := make(chan int)
var wg sync.WaitGroup
wg.Add(3)
// created this goroutine to wait for other
// goroutines to complete and to close the channel
go func() {
wg.Wait()
close(messages)
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 3)
messages <- 1
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 2)
messages <- 2
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 1)
messages <- 3
}()
// this for loop is blocked in main goroutine.
// till it reads all messages and channel is closed.
for v := range messages {
fmt.Print(v)
}
}
The mentioned blog starts with following comment:
EDIT: As pointed out by effenn in this Reddit comment, a lot of information in this article is “dangerously inaccurate”. OOPS! I’ve written a followup/correction article here for your viewing pleasure, but I’m leaving this article up for “historical purposes”.
The Reddit comment and the followup article both describe the problem and give a solution to your problem. (Adding time.Sleep(...) to make the program work the way you expect is really hacky...)
package main
import (
"fmt"
"sync"
"time"
)
func main() {
messages := make(chan int)
var wg sync.WaitGroup
// you can also add these one at
// a time if you need to
wg.Add(3)
go func() {
defer wg.Done()
time.Sleep(time.Second * 3)
messages <- 1
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 2)
messages <- 2
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 1)
messages <- 3
}()
exit:
for {
select {
case i, ok := <-messages:
if !ok {
break exit
}
fmt.Println(i)
default:
time.Sleep(time.Second)
}
}
wg.Wait()
}
I have some issues with the following code:
package main
import (
"fmt"
"sync"
)
// This program should go to 11, but sometimes it only prints 1 to 10.
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, wg) //
go func(){
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
// Print prints all numbers sent on the channel.
// The function returns when the channel is closed.
func Print(ch <-chan int, wg sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
I get a deadlock at the specified place. I have tried setting wg.Add(1) instead of 2 and it solves my problem. My belief is that I'm not successfully sending the channel as an argument to the Printer function. Is there a way to do that? Otherwise, a solution to my problem is replacing the go Print(ch, wg)line with:
go func() {
Print(ch)
defer wg.Done()
}
and changing the Printer function to:
func Print(ch <-chan int) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
}
What is the best solution?
Well, first your actual error is that you're giving the Print method a copy of the sync.WaitGroup, so it doesn't call the Done() method on the one you're Wait()ing on.
Try this instead:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
Now, changing your Print method to remove the WaitGroup of it is a generally good idea: the method doesn't need to know something is waiting for it to finish its job.
I agree with #Elwinar's solution, that the main problem in your code caused by passing a copy of your Waitgroup to the Print function.
This means the wg.Done() is operated on a copy of wg you defined in the main. Therefore, wg in the main could not get decreased, and thus a deadlock happens when you wg.Wait() in main.
Since you are also asking about the best practice, I could give you some suggestions of my own:
Don't remove defer wg.Done() in Print. Since your goroutine in main is a sender, and print is a receiver, removing wg.Done() in receiver routine will cause an unfinished receiver. This is because only your sender is synced with your main, so after your sender is done, your main is done, but it's possible that the receiver is still working. My point is: don't leave some dangling goroutines around after your main routine is finished. Close them or wait for them.
Remember to do panic recovery everywhere, especially anonymous goroutine. I have seen a lot of golang programmers forgetting to put panic recovery in goroutines, even if they remember to put recover in normal functions. It's critical when you want your code to behave correctly or at least gracefully when something unexpected happened.
Use defer before every critical calls, like sync related calls, at the beginning since you don't know where the code could break. Let's say you removed defer before wg.Done(), and a panic occurrs in your anonymous goroutine in your example. If you don't have panic recover, it will panic. But what happens if you have a panic recover? Everything's fine now? No. You will get deadlock at wg.Wait() since your wg.Done() gets skipped because of panic! However, by using defer, this wg.Done() will be executed at the end, even if panic happened. Also, defer before close is important too, since its result also affects the communication.
So here is the code modified according to the points I mentioned above:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer func() {
wg.Done()
}()
for i := 1; i <= 11; i++ {
ch <- i
if i == 7 {
panic("ahaha")
}
}
println("sender done")
close(ch)
}()
wg.Wait()
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer wg.Done()
for n := range ch {
fmt.Println(n)
}
println("print done")
}
Hope it helps :)
FATAL Error All go routines are asleep. Deadlock.
This is what I tried. I am calling wg.Done(). What is missing?
package main
import (
"fmt"
"strconv"
"sync"
)
func sender(wg *sync.WaitGroup, cs chan int) {
defer wg.Done()
for i := 0; i < 2; i++ {
fmt.Println(i)
cs <- i
}
}
func reciever(wg *sync.WaitGroup, cs chan int) {
x, ok := <-cs
for ok {
fmt.Println("Retrieved" + strconv.Itoa(x))
x, ok = <-cs
if !ok {
wg.Done()
break
}
}
}
func main() {
wg := &sync.WaitGroup{}
cs := make(chan int, 1000)
wg.Add(1)
go sender(wg, cs)
for i := 1; i < 30; i++ {
wg.Add(1)
go reciever(wg, cs)
}
wg.Wait()
close(cs)
}
You should to close channel before wg.Wait.
All your receivers are waiting for data from channel. That's why you have deadlock.
You can close channel at defer statement of sender function.
Also you need to wg.Done() if the first attempt of receiving from channel was unsuccessful (because channel already closed)
http://play.golang.org/p/qdEIEfY-kl
There are couple of things:
You need to close the channel once sender is completed.
In receiver, range over channel
Don't need to add 1 to wait group and call Done in sender
Please refer to http://play.golang.org/p/vz39RY6WA7