How can one close a channel in a defer block safely? - go

Consider the following example:
package main
import (
"fmt"
"time"
)
func main() {
ticker := time.NewTicker(2 * time.Second)
done := make(chan bool)
defer func() {
fmt.Println("exiting..")
done <- true
close(done)
}()
go func(ticker *time.Ticker, done chan bool) {
for {
select {
case <-done:
fmt.Println("DONE!")
break
case <-ticker.C:
fmt.Println("TICK!...")
}
}
}(ticker, done)
time.Sleep(7 * time.Second)
}
The goroutine waiting to receive from done never receives as (I am guessing) the main goroutine finished beforehand. However if I change the sleep time of the main goroutine to 8 seconds it receives a message; Why is there this dependency on the sleep time?
Is it because there is that second difference that keeps the goroutine alive and the there isn't enough time to kill it?
How would I than kill the goroutine gracefully?

You need to ensure that main does not return before the goroutine finishes.
The simplest way to do this is using a WaitGroup:
var wg sync.WaitGroup
defer wg.Wait()
wg.Add(1)
go func() {
defer wg.Done()
// …
Note that defers run in reverse order, so you must put defer wg.Wait() before defer close(done), otherwise it will deadlock.

Related

How to make this golang for-select code work?

Question
How to make the below code print "QUIT" after 3 seconds?
Code
package main
import (
"fmt"
"time"
)
func main() {
quit := make(chan struct{})
tasks := make(chan struct{})
go func(){
time.Sleep(1 * time.Second)
tasks <- struct{}{}
}()
go func(){
time.Sleep(3 * time.Second)
quit <- struct{}{}
}()
for {
select {
case <-quit:
fmt.Println("QUIT")
return
case <-tasks:
fmt.Println("Doing")
// do some long time jobs more than 10 seconds
time.Sleep(10 * time.Second)
}
}
}
Observations
The above code prints "Doing". and sleep 10 seconds, then print "QUIT".
How to interrupt this sleep, let it receive quit channel after 3 seconds and print "QUIT"?
It seems select is blocked by case tasks, and it will not receive from the quit channel after 3 seconds.
To signal end of an asynchronous task it is a best practice to close the channel, this is rather important to prevent many missuse that leads to various deadlocks.
In your original code I would have written close(quite) rather than quit <- struct{}{}
Remember, read on a closed does not block and always return the zero value, that is the trick.
Anyways, appart from this, an elegant way to solve your problem is to use a combination of both context.Context and time.After.
time.After will help you block on a selectable task set.
context.Context is better suited to handle this kind of signals.
https://play.golang.org/p/ZVsZw3P-YHd
package main
import (
"context"
"log"
"time"
)
func main() {
log.Println("start")
defer log.Println("end")
ctx, cancel := context.WithCancel(context.Background())
tasks := make(chan struct{})
go func() {
time.Sleep(1 * time.Second) // a job
tasks <- struct{}{}
}()
go func() {
time.Sleep(3 * time.Second) // a job
cancel()
}()
for {
select {
case <-ctx.Done():
log.Println("QUIT")
return
case <-tasks:
log.Println("Doing")
// do some long time jobs more than 10 seconds
select {
case <-ctx.Done():
return
case <-time.After(time.Second * 10):
}
}
}
}
The second job is running for 10 seconds, so it will block the loop for that time, and the signal you sent into the quit channel will not be received until this job is complete.
To interrupt the time-consuming job, maybe you can split it into another goroutine. Something like this would do:
go func() {
for {
select {
case <-tasks:
fmt.Println("Doing")
// do some long time jobs more than 10 seconds
time.Sleep(10 * time.Second)
}
}
}()
<-quit
fmt.Println("QUIT")
It would be easier to use [sync.WaitGroup.
package main
import (
"fmt"
"sync"
"time"
)
func worker(msg string, duration time.Duration, doing bool, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(duration)
fmt.Println(msg)
if doing {
time.Sleep(10 * time.Second)
}
}
func main() {
var wg sync.WaitGroup
msgs := [2]string{"QUIT", "Doing"}
durations := [2]time.Duration{3 * time.Second, 1 * time.Second}
doing := [2]bool{false, true}
for i, msg := range msgs {
wg.Add(1)
go worker(msg, durations[i], doing[i], &wg)
}
wg.Wait()
}

A question about main routine and a child routine listening on the same channel simultaneously

func main() {
c := make(chan os.Signal, 1)
signal.Notify(c)
ticker := time.NewTicker(time.Second)
stop := make(chan bool)
go func() {
defer func() { stop <- true }()
for {
select {
case <-ticker.C:
fmt.Println("Tick")
case <-stop:
fmt.Println("Goroutine closing")
return
}
}
}()
<-c
ticker.Stop()
stop <- true
<-stop
fmt.Println("Application stopped")
}
No matter how many times I run the code above, I got the same result. That is, "Goroutine closing" is always printed before "Application stopped" after I press Ctrl+C.
I think, theoretically, there is a chance that "Goroutine closing" won't be printed at all. Am I right? Unfortunately, I never get this theoretical result.
BTW: I know reading and writing a channel in one routine should be avoided. Ignore that temporarily.
In your case, Goroutine closing will always be executed and it will always be printed before Application stopped, because your stop channel is not buffered. This means that the sending will block until the result is received.
In your code, the stop <- true in your main will block until the goroutine has received the value, causing the channel to be empty again. Then the <-stop in your main will block until another value is sent to the channel, which happens when your goroutine returns after printing Goroutine closing.
If you would initialize your channel in a buffered fashion
stop := make(chan bool, 1)
then Goroutine closing might not be executed. To see this, you can add a time.Sleep right after printing Tick, as this makes this case more likely (it will occur everytime you press Ctrl+C during the sleep).
Using a sync.WaitGroup to wait for goroutines to finish is a good alternative, especially if you have to wait for more than one goroutine. You can also use a context.Context to stop goroutines. Reworking your code to use these two methods could look something like this:
func main() {
c := make(chan os.Signal, 1)
signal.Notify(c)
ticker := time.NewTicker(time.Second)
ctx, cancel := context.WithCancel(context.Background())
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer func() { wg.Done() }()
for {
select {
case <-ctx.Done():
fmt.Println("Goroutine closing")
return
case <-ticker.C:
fmt.Println("Tick")
time.Sleep(time.Second)
}
}
}()
<-c
ticker.Stop()
cancel()
wg.Wait()
fmt.Println("Application stopped")
}

Panic while trying to avoid goroutine leak

I'm spawning some goroutines and want to hand them a channel to send back errors. In the parent goroutine, I select the first error and return that, or the wg.Done() condition, which is synchronized with closing a done channel.
The closing of errc is deferred to avoid a goroutine leak; but it causes a race condition.
package main
import (
"log"
"sync"
"time"
)
func f(ch chan<- bool, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(1 * time.Second)
log.Println("f sending a value")
ch <- true
log.Println("f sent a value")
}
func g(ch chan<- bool, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(2 * time.Second)
log.Println("g sending a value")
ch <- true
log.Println("g sent a value")
}
func main() {
var wg sync.WaitGroup
ch := make(chan bool)
bufc := make(chan bool, 2)
defer func() {
log.Println("Closing bufc")
close(bufc)
log.Println("Closed bufc")
time.Sleep(5 * time.Second)
}()
wg.Add(2)
go f(bufc, &wg)
go g(bufc, &wg)
go func() {
wg.Wait()
close(ch)
}()
select {
case done, ok := <-bufc:
log.Printf("bufc closed: %v %v", done, ok)
case <-ch:
log.Println("ch was closed")
}
}
Result:
❗ ~/c/scrap
(i) go run test.go
2018/05/01 20:28:03 f sending a value
2018/05/01 20:28:03 f sent a value
2018/05/01 20:28:03 bufc closed: true true
2018/05/01 20:28:03 Closing bufc
2018/05/01 20:28:03 Closed bufc
2018/05/01 20:28:04 g sending a value
panic: send on closed channel
goroutine 19 [running]:
main.g(0xc42009c000, 0xc42008a010)
/Users/yangmillstheory/code/scrap/test.go:23 +0xb2
created by main.main
/Users/yangmillstheory/code/scrap/test.go:42 +0x11e
exit status 2
Is there any way to do proper cleanup of the errc channel without causing a panic? Do I even need to close errc? Given that it's buffered, senders on that channel wouldn't block, so I would guess the answer is no?
Your error is clear enough--the channel bufc (which I assume you refer to as errc) is closed before g can send the value to it because the select statement receives only once from bufc and it's closed by defer. Instead of deferring the closing of bufc you'd have to make some synchronization, possibly using a sync.WaitGroup to make sure all the values are sent before closing it, for example by just moving close(bufc) to after wg.Wait():
go func() {
wg.Wait()
close(ch)
close(bufc)
}()
In your case since bufc is buffered you don't have to close it because it's not blocking on the receiving end, but once you have more than two goroutines sending you'll still need to close it to signal properly.
I ended up with the following implementation, which drains the buffered channel bufc and works correctly in all cases.
var (
err error
wg sync.WaitGroup
)
ch := make(chan bool)
bufc := make(chan error, 2)
wg.Add(2)
go f(bufc, &wg)
go g(bufc, &wg)
go func() {
wg.Wait()
close(ch)
close(bufc)
}()
<-ch
log.Println("Goroutines are done")
for err = range bufc {
log.Printf("Got an error: %v", err)
}
log.Println("Returning.")
return err

How to correctly use sync.Cond?

I'm having trouble figuring out how to correctly use sync.Cond. From what I can tell, a race condition exists between locking the Locker and invoking the condition's Wait method. This example adds an artificial delay between the two lines in the main goroutine to simulate the race condition:
package main
import (
"sync"
"time"
)
func main() {
m := sync.Mutex{}
c := sync.NewCond(&m)
go func() {
time.Sleep(1 * time.Second)
c.Broadcast()
}()
m.Lock()
time.Sleep(2 * time.Second)
c.Wait()
}
[Run on the Go Playground]
This causes an immediate panic:
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_Syncsemacquire(0x10330208, 0x1)
/usr/local/go/src/runtime/sema.go:241 +0x2e0
sync.(*Cond).Wait(0x10330200, 0x0)
/usr/local/go/src/sync/cond.go:63 +0xe0
main.main()
/tmp/sandbox301865429/main.go:17 +0x1a0
What am I doing wrong? How do I avoid this apparent race condition? Is there a better synchronization construct I should be using?
Edit: I realize I should have better explained the problem I'm trying to solve here. I have a long-running goroutine that downloads a large file and a number of other goroutines that need access to the HTTP headers when they are available. This problem is harder than it sounds.
I can't use channels since only one goroutine would then receive the value. And some of the other goroutines would be trying to retrieve the headers long after they are already available.
The downloader goroutine could simply store the HTTP headers in a variable and use a mutex to safeguard access to them. However, this doesn't provide a way for the other goroutines to "wait" for them to become available.
I had thought that both a sync.Mutex and sync.Cond together could accomplish this goal but it appears that this is not possible.
OP answered his own, but did not directly answer the original question, I am going to post how to correctly use sync.Cond.
You do not really need sync.Cond if you have one goroutine for each write and read - a single sync.Mutex would suffice to communicate between them. sync.Cond could useful in situations where multiple readers wait for the shared resources to be available.
var sharedRsc = make(map[string]interface{})
func main() {
var wg sync.WaitGroup
wg.Add(2)
m := sync.Mutex{}
c := sync.NewCond(&m)
go func() {
// this go routine wait for changes to the sharedRsc
c.L.Lock()
for len(sharedRsc) == 0 {
c.Wait()
}
fmt.Println(sharedRsc["rsc1"])
c.L.Unlock()
wg.Done()
}()
go func() {
// this go routine wait for changes to the sharedRsc
c.L.Lock()
for len(sharedRsc) == 0 {
c.Wait()
}
fmt.Println(sharedRsc["rsc2"])
c.L.Unlock()
wg.Done()
}()
// this one writes changes to sharedRsc
c.L.Lock()
sharedRsc["rsc1"] = "foo"
sharedRsc["rsc2"] = "bar"
c.Broadcast()
c.L.Unlock()
wg.Wait()
}
Playground
Having said that, using channels is still the recommended way to pass data around if the situation permitting.
Note: sync.WaitGroup here is only used to wait for the goroutines to complete their executions.
You need to make sure that c.Broadcast is called after your call to c.Wait. The correct version of your program would be:
package main
import (
"fmt"
"sync"
)
func main() {
m := &sync.Mutex{}
c := sync.NewCond(m)
m.Lock()
go func() {
m.Lock() // Wait for c.Wait()
c.Broadcast()
m.Unlock()
}()
c.Wait() // Unlocks m, waits, then locks m again
m.Unlock()
}
https://play.golang.org/p/O1r8v8yW6h
package main
import (
"fmt"
"sync"
"time"
)
func main() {
m := sync.Mutex{}
m.Lock() // main gouroutine is owner of lock
c := sync.NewCond(&m)
go func() {
m.Lock() // obtain a lock
defer m.Unlock()
fmt.Println("3. goroutine is owner of lock")
time.Sleep(2 * time.Second) // long computing - because you are the owner, you can change state variable(s)
c.Broadcast() // State has been changed, publish it to waiting goroutines
fmt.Println("4. goroutine will release lock soon (deffered Unlock")
}()
fmt.Println("1. main goroutine is owner of lock")
time.Sleep(1 * time.Second) // initialization
fmt.Println("2. main goroutine is still lockek")
c.Wait() // Wait temporarily release a mutex during wating and give opportunity to other goroutines to change the state.
// Because you don't know, whether this is state, that you are waiting for, is usually called in loop.
m.Unlock()
fmt.Println("Done")
}
http://play.golang.org/p/fBBwoL7_pm
Looks like you c.Wait for Broadcast which would never happens with your time intervals.
With
time.Sleep(3 * time.Second) //Broadcast after any Wait for it
c.Broadcast()
your snippet seems to work http://play.golang.org/p/OE8aP4i6gY .Or am I missing something that you try to achive?
I finally discovered a way to do this and it doesn't involve sync.Cond at all - just the mutex.
type Task struct {
m sync.Mutex
headers http.Header
}
func NewTask() *Task {
t := &Task{}
t.m.Lock()
go func() {
defer t.m.Unlock()
// ...do stuff...
}()
return t
}
func (t *Task) WaitFor() http.Header {
t.m.Lock()
defer t.m.Unlock()
return t.headers
}
How does this work?
The mutex is locked at the beginning of the task, ensuring that anything calling WaitFor() will block. Once the headers are available and the mutex unlocked by the goroutine, each call to WaitFor() will execute one at a time. All future calls (even after the goroutine ends) will have no problem locking the mutex, since it will always be left unlocked.
Yes you can use one channel to pass Header to multiple Go routines.
headerChan := make(chan http.Header)
go func() { // This routine can be started many times
header := <-headerChan // Wait for header
// Do things with the header
}()
// Feed the header to all waiting go routines
for more := true; more; {
select {
case headerChan <- r.Header:
default: more = false
}
}
This can be done with channels pretty easily and the code will be clean. Below is the example. Hope this helps!
package main
import (
"fmt"
"net/http"
"sync"
)
func main() {
done := make(chan struct{})
var wg sync.WaitGroup
// fork required number of goroutines
for i := 0; i < 5; i++ {
wg.Add(1)
go func() {
defer wg.Done()
<-done
fmt.Println("read the http headers from here")
}()
}
time.Sleep(1) //download your large file here
fmt.Println("Unblocking goroutines...")
close(done) // this will unblock all the goroutines
wg.Wait()
}
In the excellent book "Concurrency in Go" they provide the following easy solution while leveraging the fact that a channel that is closed will release all waiting clients.
package main
import (
"fmt"
"time"
)
func main() {
httpHeaders := []string{}
headerChan := make(chan interface{})
var consumerFunc= func(id int, stream <-chan interface{}, funcHeaders *[]string)
{
<-stream
fmt.Println("Consumer ",id," got headers:", funcHeaders )
}
for i:=0;i<3;i++ {
go consumerFunc(i, headerChan, &httpHeaders)
}
fmt.Println("Getting headers...")
time.Sleep(2*time.Second)
httpHeaders=append(httpHeaders, "test1");
fmt.Println("Publishing headers...")
close(headerChan )
time.Sleep(5*time.Second)
}
https://play.golang.org/p/cE3SiKWNRIt

Best way of using sync.WaitGroup with external function

I have some issues with the following code:
package main
import (
"fmt"
"sync"
)
// This program should go to 11, but sometimes it only prints 1 to 10.
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, wg) //
go func(){
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
// Print prints all numbers sent on the channel.
// The function returns when the channel is closed.
func Print(ch <-chan int, wg sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
I get a deadlock at the specified place. I have tried setting wg.Add(1) instead of 2 and it solves my problem. My belief is that I'm not successfully sending the channel as an argument to the Printer function. Is there a way to do that? Otherwise, a solution to my problem is replacing the go Print(ch, wg)line with:
go func() {
Print(ch)
defer wg.Done()
}
and changing the Printer function to:
func Print(ch <-chan int) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
}
What is the best solution?
Well, first your actual error is that you're giving the Print method a copy of the sync.WaitGroup, so it doesn't call the Done() method on the one you're Wait()ing on.
Try this instead:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
Now, changing your Print method to remove the WaitGroup of it is a generally good idea: the method doesn't need to know something is waiting for it to finish its job.
I agree with #Elwinar's solution, that the main problem in your code caused by passing a copy of your Waitgroup to the Print function.
This means the wg.Done() is operated on a copy of wg you defined in the main. Therefore, wg in the main could not get decreased, and thus a deadlock happens when you wg.Wait() in main.
Since you are also asking about the best practice, I could give you some suggestions of my own:
Don't remove defer wg.Done() in Print. Since your goroutine in main is a sender, and print is a receiver, removing wg.Done() in receiver routine will cause an unfinished receiver. This is because only your sender is synced with your main, so after your sender is done, your main is done, but it's possible that the receiver is still working. My point is: don't leave some dangling goroutines around after your main routine is finished. Close them or wait for them.
Remember to do panic recovery everywhere, especially anonymous goroutine. I have seen a lot of golang programmers forgetting to put panic recovery in goroutines, even if they remember to put recover in normal functions. It's critical when you want your code to behave correctly or at least gracefully when something unexpected happened.
Use defer before every critical calls, like sync related calls, at the beginning since you don't know where the code could break. Let's say you removed defer before wg.Done(), and a panic occurrs in your anonymous goroutine in your example. If you don't have panic recover, it will panic. But what happens if you have a panic recover? Everything's fine now? No. You will get deadlock at wg.Wait() since your wg.Done() gets skipped because of panic! However, by using defer, this wg.Done() will be executed at the end, even if panic happened. Also, defer before close is important too, since its result also affects the communication.
So here is the code modified according to the points I mentioned above:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer func() {
wg.Done()
}()
for i := 1; i <= 11; i++ {
ch <- i
if i == 7 {
panic("ahaha")
}
}
println("sender done")
close(ch)
}()
wg.Wait()
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer wg.Done()
for n := range ch {
fmt.Println(n)
}
println("print done")
}
Hope it helps :)

Resources