I looked at some code that I've wrote a long time ago, when go1.3 was released(I might be wrong). CODE HERE
The below code used to work as expected, but now since I've update go to the current master version(go version devel +bd1efd5 Fri Jul 31 16:11:21 2015 +0000 darwin/amd64), the last output message c <- "FUNC 1 DONE" is not printed, the code works as it should on play.golang.org. Did I do something wrong, or this is a bug?
package main
import ("fmt";"sync";"time")
func test(c chan string, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("EXEC FUNC 1")
time.Sleep(3 * time.Second)
c <- "FUNC 1 DONE"
}
func test1(c chan string, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("EXEC FUNC 2")
time.Sleep(2 * time.Second)
c <- "FUNC 2 DONE"
}
func main() {
ch := make(chan string)
var wg sync.WaitGroup
wg.Add(2)
go test(ch, &wg)
go test1(ch, &wg)
go func(c chan string) {
for txt := range c {
fmt.Println(txt)
}
}(ch)
wg.Wait()
}
UPDATE:
I'm not saying that, the above is the best way of doing those types of work, but I don't see anything wrong with it.
Also running it in go version go1.4.2 darwin/amd64 will return the expected output.
Your code has always had this bug. It was only by chance that your program managed to print all messages before main exited.
To make this work correctly, I would invert where you have the wg.Wait() and the channel receives, so you can asynchronously close the channel. This way the receive operations are what is blocking main, and the channel is closed as soon as all send operations are done.
func main() {
ch := make(chan string)
var wg sync.WaitGroup
wg.Add(2)
go test(ch, &wg)
go test1(ch, &wg)
go func() {
wg.Wait()
close(ch)
}()
for txt := range ch {
fmt.Println(txt)
}
}
Related
I am prototyping a series of go routines for a pipeline that each perform a transformation. The routines are terminating before all the data has passed through.
I have checked Donavan and Kernighan book and Googled for solutions.
Here is my code:
package main
import (
"fmt"
"sync"
)
func main() {
a1 := []string{"apple", "apricot"}
chan1 := make(chan string)
chan2 := make(chan string)
chan3 := make(chan string)
var wg sync.WaitGroup
go Pipe1(chan2, chan1, &wg)
go Pipe2(chan3, chan2, &wg)
go Pipe3(chan3, &wg)
func (data []string) {
defer wg.Done()
for _, s := range data {
wg.Add(1)
chan1 <- s
}
go func() {
wg.Wait()
close(chan1)
}()
}(a1)
}
func Pipe1(out chan<- string, in <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for s := range in {
wg.Add(1)
out <- s + "s are"
}
}
func Pipe2(out chan<- string, in <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for s := range in {
wg.Add(1)
out <- s + " good for you"
}
}
func Pipe3(in <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for s := range in {
wg.Add(1)
fmt.Println(s)
}
}
My expected output is:
apples are good for you
apricots are good for you
The results of running main are inconsistent. Sometimes I get both lines. Sometimes I just get the apples. Sometimes nothing is output.
As Adrian already pointed out, your WaitGroup.Add and WaitGroup.Done calls are mismatched. However, in cases like this the "I am done" signal is typically given by closing the output channel. WaitGroups are only necessary if work is shared between several goroutines (i.e. several goroutines consume the same channel), which isn't the case here.
package main
import (
"fmt"
)
func main() {
a1 := []string{"apple", "apricot"}
chan1 := make(chan string)
chan2 := make(chan string)
chan3 := make(chan string)
go func() {
for _, s := range a1 {
chan1 <- s
}
close(chan1)
}()
go Pipe1(chan2, chan1)
go Pipe2(chan3, chan2)
// This range loop terminates when chan3 is closed, which Pipe2 does after
// chan2 is closed, which Pipe1 does after chan1 is closed, which the
// anonymous goroutine above does after it sent all values.
for s := range chan3 {
fmt.Println(s)
}
}
func Pipe1(out chan<- string, in <-chan string) {
for s := range in {
out <- s + "s are"
}
close(out) // let caller know that we're done
}
func Pipe2(out chan<- string, in <-chan string) {
for s := range in {
out <- s + " good for you"
}
close(out) // let caller know that we're done
}
Try it on the playground: https://play.golang.org/p/d2J4APjs_lL
You're calling wg.Wait in a goroutine, so main is allowed to return (and therefore your program exits) before the other routines have finished. This would cause the behavior your see, but taking out of a goroutine alone isn't enough.
You're also misusing the WaitGroup in general; your Add and Done calls don't relate to one another, and you don't have as many Dones as you have Adds, so the WaitGroup will never finish. If you're calling Add in a loop, then every loop iteration must also result in a Done call; as you have it now, you defer wg.Done() before each of your loops, then call Add inside the loop, resulting in one Done and many Adds. This code would need to be significantly revised to work as intended.
I'm spawning some goroutines and want to hand them a channel to send back errors. In the parent goroutine, I select the first error and return that, or the wg.Done() condition, which is synchronized with closing a done channel.
The closing of errc is deferred to avoid a goroutine leak; but it causes a race condition.
package main
import (
"log"
"sync"
"time"
)
func f(ch chan<- bool, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(1 * time.Second)
log.Println("f sending a value")
ch <- true
log.Println("f sent a value")
}
func g(ch chan<- bool, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(2 * time.Second)
log.Println("g sending a value")
ch <- true
log.Println("g sent a value")
}
func main() {
var wg sync.WaitGroup
ch := make(chan bool)
bufc := make(chan bool, 2)
defer func() {
log.Println("Closing bufc")
close(bufc)
log.Println("Closed bufc")
time.Sleep(5 * time.Second)
}()
wg.Add(2)
go f(bufc, &wg)
go g(bufc, &wg)
go func() {
wg.Wait()
close(ch)
}()
select {
case done, ok := <-bufc:
log.Printf("bufc closed: %v %v", done, ok)
case <-ch:
log.Println("ch was closed")
}
}
Result:
❗ ~/c/scrap
(i) go run test.go
2018/05/01 20:28:03 f sending a value
2018/05/01 20:28:03 f sent a value
2018/05/01 20:28:03 bufc closed: true true
2018/05/01 20:28:03 Closing bufc
2018/05/01 20:28:03 Closed bufc
2018/05/01 20:28:04 g sending a value
panic: send on closed channel
goroutine 19 [running]:
main.g(0xc42009c000, 0xc42008a010)
/Users/yangmillstheory/code/scrap/test.go:23 +0xb2
created by main.main
/Users/yangmillstheory/code/scrap/test.go:42 +0x11e
exit status 2
Is there any way to do proper cleanup of the errc channel without causing a panic? Do I even need to close errc? Given that it's buffered, senders on that channel wouldn't block, so I would guess the answer is no?
Your error is clear enough--the channel bufc (which I assume you refer to as errc) is closed before g can send the value to it because the select statement receives only once from bufc and it's closed by defer. Instead of deferring the closing of bufc you'd have to make some synchronization, possibly using a sync.WaitGroup to make sure all the values are sent before closing it, for example by just moving close(bufc) to after wg.Wait():
go func() {
wg.Wait()
close(ch)
close(bufc)
}()
In your case since bufc is buffered you don't have to close it because it's not blocking on the receiving end, but once you have more than two goroutines sending you'll still need to close it to signal properly.
I ended up with the following implementation, which drains the buffered channel bufc and works correctly in all cases.
var (
err error
wg sync.WaitGroup
)
ch := make(chan bool)
bufc := make(chan error, 2)
wg.Add(2)
go f(bufc, &wg)
go g(bufc, &wg)
go func() {
wg.Wait()
close(ch)
close(bufc)
}()
<-ch
log.Println("Goroutines are done")
for err = range bufc {
log.Printf("Got an error: %v", err)
}
log.Println("Returning.")
return err
I have an issue. Here is example: https://play.golang.org/p/QSWY2INQuSE
func Avg(c chan string, wg *sync.WaitGroup) {
defer wg.Done()
c <- "test"
}
func main() {
var wg sync.WaitGroup
c := make(chan string)
timer1 := time.NewTicker(5 * time.Second)
for {
select {
case <-timer1.C:
wg.Add(1)
go Avg(c, &wg)
wg.Wait()
}
}
fmt.Println(<-c)
}
Why data does not reach fmt.Println(<-c)
Thank you!
Because you have an endless for, so the last fmt.Println() statement is never reached.
You have to break out of the loop if you want the last fmt.Println() statement to ever execute, for example:
loop:
for {
select {
case <-timer1.C:
wg.Add(1)
go Avg(c, &wg)
wg.Wait()
break loop
}
}
fmt.Println(<-c)
Note that you have to use a label, else the break would only break out of the select statement (and not from the for loop).
Also note that this alone won't work, as the channel is unbuffered, and thus Avg() will be blocked forever, trying to send a value on c while noone is ever trying to receive from it.
This simple example can be made working if you create the channel to be buffered:
c := make(chan string, 1) // Buffer for 1 value
Now it works and prints (try it on the Go Playground):
test
I am learning WaitGroup from the blog https://nathanleclaire.com/blog/2014/02/15/how-to-wait-for-all-goroutines-to-finish-executing-before-continuing/
the code:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
messages := make(chan int)
var wg sync.WaitGroup
// you can also add these one at
// a time if you need to
wg.Add(3)
go func() {
defer wg.Done()
time.Sleep(time.Second * 3)
messages <- 1
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 2)
messages <- 2
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 1)
messages <- 3
}()
go func() {
for i := range messages {
fmt.Println(i)
}
}()
wg.Wait()
}
I think it should print 3, 2 and 1 in order. But it only prints 3, 2 but 1 is missing, what's the problem?
You can tree it on https://play.golang.org/p/kZCvDhykYM
Right after the latest messages <- 1, the deferred wg.Done() is invoked which releases wg.Wait() in the end of the program and the program quits. When a program quits all the goroutines get killed, so the printing goroutine does not have a chance to print the latest value.
If you put something like time.Sleep(time.Second * 1) right after wg.Done() you would be able to see all the output lines.
package main
import (
"fmt"
"sync"
"time"
)
func main() {
messages := make(chan int)
var wg sync.WaitGroup
wg.Add(3)
// created this goroutine to wait for other
// goroutines to complete and to close the channel
go func() {
wg.Wait()
close(messages)
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 3)
messages <- 1
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 2)
messages <- 2
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 1)
messages <- 3
}()
// this for loop is blocked in main goroutine.
// till it reads all messages and channel is closed.
for v := range messages {
fmt.Print(v)
}
}
The mentioned blog starts with following comment:
EDIT: As pointed out by effenn in this Reddit comment, a lot of information in this article is “dangerously inaccurate”. OOPS! I’ve written a followup/correction article here for your viewing pleasure, but I’m leaving this article up for “historical purposes”.
The Reddit comment and the followup article both describe the problem and give a solution to your problem. (Adding time.Sleep(...) to make the program work the way you expect is really hacky...)
package main
import (
"fmt"
"sync"
"time"
)
func main() {
messages := make(chan int)
var wg sync.WaitGroup
// you can also add these one at
// a time if you need to
wg.Add(3)
go func() {
defer wg.Done()
time.Sleep(time.Second * 3)
messages <- 1
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 2)
messages <- 2
}()
go func() {
defer wg.Done()
time.Sleep(time.Second * 1)
messages <- 3
}()
exit:
for {
select {
case i, ok := <-messages:
if !ok {
break exit
}
fmt.Println(i)
default:
time.Sleep(time.Second)
}
}
wg.Wait()
}
I have some issues with the following code:
package main
import (
"fmt"
"sync"
)
// This program should go to 11, but sometimes it only prints 1 to 10.
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, wg) //
go func(){
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
// Print prints all numbers sent on the channel.
// The function returns when the channel is closed.
func Print(ch <-chan int, wg sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
I get a deadlock at the specified place. I have tried setting wg.Add(1) instead of 2 and it solves my problem. My belief is that I'm not successfully sending the channel as an argument to the Printer function. Is there a way to do that? Otherwise, a solution to my problem is replacing the go Print(ch, wg)line with:
go func() {
Print(ch)
defer wg.Done()
}
and changing the Printer function to:
func Print(ch <-chan int) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
}
What is the best solution?
Well, first your actual error is that you're giving the Print method a copy of the sync.WaitGroup, so it doesn't call the Done() method on the one you're Wait()ing on.
Try this instead:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
Now, changing your Print method to remove the WaitGroup of it is a generally good idea: the method doesn't need to know something is waiting for it to finish its job.
I agree with #Elwinar's solution, that the main problem in your code caused by passing a copy of your Waitgroup to the Print function.
This means the wg.Done() is operated on a copy of wg you defined in the main. Therefore, wg in the main could not get decreased, and thus a deadlock happens when you wg.Wait() in main.
Since you are also asking about the best practice, I could give you some suggestions of my own:
Don't remove defer wg.Done() in Print. Since your goroutine in main is a sender, and print is a receiver, removing wg.Done() in receiver routine will cause an unfinished receiver. This is because only your sender is synced with your main, so after your sender is done, your main is done, but it's possible that the receiver is still working. My point is: don't leave some dangling goroutines around after your main routine is finished. Close them or wait for them.
Remember to do panic recovery everywhere, especially anonymous goroutine. I have seen a lot of golang programmers forgetting to put panic recovery in goroutines, even if they remember to put recover in normal functions. It's critical when you want your code to behave correctly or at least gracefully when something unexpected happened.
Use defer before every critical calls, like sync related calls, at the beginning since you don't know where the code could break. Let's say you removed defer before wg.Done(), and a panic occurrs in your anonymous goroutine in your example. If you don't have panic recover, it will panic. But what happens if you have a panic recover? Everything's fine now? No. You will get deadlock at wg.Wait() since your wg.Done() gets skipped because of panic! However, by using defer, this wg.Done() will be executed at the end, even if panic happened. Also, defer before close is important too, since its result also affects the communication.
So here is the code modified according to the points I mentioned above:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer func() {
wg.Done()
}()
for i := 1; i <= 11; i++ {
ch <- i
if i == 7 {
panic("ahaha")
}
}
println("sender done")
close(ch)
}()
wg.Wait()
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer wg.Done()
for n := range ch {
fmt.Println(n)
}
println("print done")
}
Hope it helps :)