I'm having trouble figuring out how to correctly use sync.Cond. From what I can tell, a race condition exists between locking the Locker and invoking the condition's Wait method. This example adds an artificial delay between the two lines in the main goroutine to simulate the race condition:
package main
import (
"sync"
"time"
)
func main() {
m := sync.Mutex{}
c := sync.NewCond(&m)
go func() {
time.Sleep(1 * time.Second)
c.Broadcast()
}()
m.Lock()
time.Sleep(2 * time.Second)
c.Wait()
}
[Run on the Go Playground]
This causes an immediate panic:
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_Syncsemacquire(0x10330208, 0x1)
/usr/local/go/src/runtime/sema.go:241 +0x2e0
sync.(*Cond).Wait(0x10330200, 0x0)
/usr/local/go/src/sync/cond.go:63 +0xe0
main.main()
/tmp/sandbox301865429/main.go:17 +0x1a0
What am I doing wrong? How do I avoid this apparent race condition? Is there a better synchronization construct I should be using?
Edit: I realize I should have better explained the problem I'm trying to solve here. I have a long-running goroutine that downloads a large file and a number of other goroutines that need access to the HTTP headers when they are available. This problem is harder than it sounds.
I can't use channels since only one goroutine would then receive the value. And some of the other goroutines would be trying to retrieve the headers long after they are already available.
The downloader goroutine could simply store the HTTP headers in a variable and use a mutex to safeguard access to them. However, this doesn't provide a way for the other goroutines to "wait" for them to become available.
I had thought that both a sync.Mutex and sync.Cond together could accomplish this goal but it appears that this is not possible.
OP answered his own, but did not directly answer the original question, I am going to post how to correctly use sync.Cond.
You do not really need sync.Cond if you have one goroutine for each write and read - a single sync.Mutex would suffice to communicate between them. sync.Cond could useful in situations where multiple readers wait for the shared resources to be available.
var sharedRsc = make(map[string]interface{})
func main() {
var wg sync.WaitGroup
wg.Add(2)
m := sync.Mutex{}
c := sync.NewCond(&m)
go func() {
// this go routine wait for changes to the sharedRsc
c.L.Lock()
for len(sharedRsc) == 0 {
c.Wait()
}
fmt.Println(sharedRsc["rsc1"])
c.L.Unlock()
wg.Done()
}()
go func() {
// this go routine wait for changes to the sharedRsc
c.L.Lock()
for len(sharedRsc) == 0 {
c.Wait()
}
fmt.Println(sharedRsc["rsc2"])
c.L.Unlock()
wg.Done()
}()
// this one writes changes to sharedRsc
c.L.Lock()
sharedRsc["rsc1"] = "foo"
sharedRsc["rsc2"] = "bar"
c.Broadcast()
c.L.Unlock()
wg.Wait()
}
Playground
Having said that, using channels is still the recommended way to pass data around if the situation permitting.
Note: sync.WaitGroup here is only used to wait for the goroutines to complete their executions.
You need to make sure that c.Broadcast is called after your call to c.Wait. The correct version of your program would be:
package main
import (
"fmt"
"sync"
)
func main() {
m := &sync.Mutex{}
c := sync.NewCond(m)
m.Lock()
go func() {
m.Lock() // Wait for c.Wait()
c.Broadcast()
m.Unlock()
}()
c.Wait() // Unlocks m, waits, then locks m again
m.Unlock()
}
https://play.golang.org/p/O1r8v8yW6h
package main
import (
"fmt"
"sync"
"time"
)
func main() {
m := sync.Mutex{}
m.Lock() // main gouroutine is owner of lock
c := sync.NewCond(&m)
go func() {
m.Lock() // obtain a lock
defer m.Unlock()
fmt.Println("3. goroutine is owner of lock")
time.Sleep(2 * time.Second) // long computing - because you are the owner, you can change state variable(s)
c.Broadcast() // State has been changed, publish it to waiting goroutines
fmt.Println("4. goroutine will release lock soon (deffered Unlock")
}()
fmt.Println("1. main goroutine is owner of lock")
time.Sleep(1 * time.Second) // initialization
fmt.Println("2. main goroutine is still lockek")
c.Wait() // Wait temporarily release a mutex during wating and give opportunity to other goroutines to change the state.
// Because you don't know, whether this is state, that you are waiting for, is usually called in loop.
m.Unlock()
fmt.Println("Done")
}
http://play.golang.org/p/fBBwoL7_pm
Looks like you c.Wait for Broadcast which would never happens with your time intervals.
With
time.Sleep(3 * time.Second) //Broadcast after any Wait for it
c.Broadcast()
your snippet seems to work http://play.golang.org/p/OE8aP4i6gY .Or am I missing something that you try to achive?
I finally discovered a way to do this and it doesn't involve sync.Cond at all - just the mutex.
type Task struct {
m sync.Mutex
headers http.Header
}
func NewTask() *Task {
t := &Task{}
t.m.Lock()
go func() {
defer t.m.Unlock()
// ...do stuff...
}()
return t
}
func (t *Task) WaitFor() http.Header {
t.m.Lock()
defer t.m.Unlock()
return t.headers
}
How does this work?
The mutex is locked at the beginning of the task, ensuring that anything calling WaitFor() will block. Once the headers are available and the mutex unlocked by the goroutine, each call to WaitFor() will execute one at a time. All future calls (even after the goroutine ends) will have no problem locking the mutex, since it will always be left unlocked.
Yes you can use one channel to pass Header to multiple Go routines.
headerChan := make(chan http.Header)
go func() { // This routine can be started many times
header := <-headerChan // Wait for header
// Do things with the header
}()
// Feed the header to all waiting go routines
for more := true; more; {
select {
case headerChan <- r.Header:
default: more = false
}
}
This can be done with channels pretty easily and the code will be clean. Below is the example. Hope this helps!
package main
import (
"fmt"
"net/http"
"sync"
)
func main() {
done := make(chan struct{})
var wg sync.WaitGroup
// fork required number of goroutines
for i := 0; i < 5; i++ {
wg.Add(1)
go func() {
defer wg.Done()
<-done
fmt.Println("read the http headers from here")
}()
}
time.Sleep(1) //download your large file here
fmt.Println("Unblocking goroutines...")
close(done) // this will unblock all the goroutines
wg.Wait()
}
In the excellent book "Concurrency in Go" they provide the following easy solution while leveraging the fact that a channel that is closed will release all waiting clients.
package main
import (
"fmt"
"time"
)
func main() {
httpHeaders := []string{}
headerChan := make(chan interface{})
var consumerFunc= func(id int, stream <-chan interface{}, funcHeaders *[]string)
{
<-stream
fmt.Println("Consumer ",id," got headers:", funcHeaders )
}
for i:=0;i<3;i++ {
go consumerFunc(i, headerChan, &httpHeaders)
}
fmt.Println("Getting headers...")
time.Sleep(2*time.Second)
httpHeaders=append(httpHeaders, "test1");
fmt.Println("Publishing headers...")
close(headerChan )
time.Sleep(5*time.Second)
}
https://play.golang.org/p/cE3SiKWNRIt
Related
Consider the following example:
package main
import (
"fmt"
"time"
)
func main() {
ticker := time.NewTicker(2 * time.Second)
done := make(chan bool)
defer func() {
fmt.Println("exiting..")
done <- true
close(done)
}()
go func(ticker *time.Ticker, done chan bool) {
for {
select {
case <-done:
fmt.Println("DONE!")
break
case <-ticker.C:
fmt.Println("TICK!...")
}
}
}(ticker, done)
time.Sleep(7 * time.Second)
}
The goroutine waiting to receive from done never receives as (I am guessing) the main goroutine finished beforehand. However if I change the sleep time of the main goroutine to 8 seconds it receives a message; Why is there this dependency on the sleep time?
Is it because there is that second difference that keeps the goroutine alive and the there isn't enough time to kill it?
How would I than kill the goroutine gracefully?
You need to ensure that main does not return before the goroutine finishes.
The simplest way to do this is using a WaitGroup:
var wg sync.WaitGroup
defer wg.Wait()
wg.Add(1)
go func() {
defer wg.Done()
// …
Note that defers run in reverse order, so you must put defer wg.Wait() before defer close(done), otherwise it will deadlock.
I am trying to understand context in golang. I copied an example from https://golang.org/pkg/context/#example_WithCancel and changed it a bit:
Playgroud: https://play.golang.org/p/Aczc2CqcVZR
package main
import (
"context"
"fmt"
"time"
)
func main() {
// gen generates integers in a separate goroutine and
// sends them to the returned channel.
// The callers of gen need to cancel the context once
// they are done consuming generated integers not to leak
// the internal goroutine started by gen.
gen := func(ctx context.Context) <-chan int {
dst := make(chan int)
n := 1
go func() {
for {
select {
case <-ctx.Done():
fmt.Println("DONE")
return // returning not to leak the goroutine
case dst <- n:
n++
}
}
fmt.Println("END")
}()
return dst
}
ctx, cancel := context.WithCancel(context.Background())
defer time.Sleep(1 * time.Second)
defer fmt.Println("Before cancel")
defer cancel() // cancel when we are finished consuming integers
defer fmt.Println("After cancel")
channel := gen(ctx)
for n := range channel {
fmt.Println(n)
if n == 5 {
break
}
}
fmt.Println( <-channel)
}
When commenting out the
defer time.Sleep(1 * time.Second)
the "DONE" never gets printed. Playgroud: (https://play.golang.org/p/K0OcyZaj_xK)
I would expect the go routine which was started in the anonymous function still to be active. Once cancel() is called due to being deferred, the select should no longer block as
case <-ctx.Done():
should be available. However it seems to just end, unless I wait for 1 second and give it time. This behavior seems very wrong.
This behavior seems very wrong.
It's not. That's how program execution is specified. After main and its deferred functions return, the program exits.
Program execution begins by initializing the main package and then invoking the function main. When that function invocation returns, the program exits. It does not wait for other (non-main) goroutines to complete.
https://golang.org/ref/spec#Program_execution
in the next example, I don't understand why end value not printed when received
package main
import "fmt"
func main() {
start := make(chan int)
end := make(chan int)
go func() {
fmt.Println("Start")
fmt.Println(<-start)
}()
go func() {
fmt.Println("End")
fmt.Println(<-end)
}()
start <- 1
end <- 2
}
I know sync.WaitGroup can solve this problem.
Because the program exits when it reaches the end of func main, regardless of whether any other goroutines are running. As soon as the second function receives from the end channel, main's send on that channel is unblocked and the program finishes, before the received value gets a chance to be passed to Println.
The end value is not printed because as soon as the main goroutine (the main function is actually a goroutine) is finished (in other terms get unblocked) the other non-main goroutines does not have the chance to get completed.
When the function main() returns, the program exits. Moreover goroutines are independent units of execution and when a number of them starts one after the other you cannot depend on when a goroutine will actually be started. The logic of your code must be independent of the order in which goroutines are invoked.
One way to solve your problem (and the easiest one in your case) is to put a time.Sleep at the end of your main() function.
time.Sleep(1e9)
This will guarantee that the main goroutine will not unblock and the other goroutines will have a change to get executed.
package main
import (
"fmt"
"time"
)
func main() {
start := make(chan int)
end := make(chan int)
go func() {
fmt.Println("Start")
fmt.Println(<-start)
}()
go func() {
fmt.Println("End")
fmt.Println(<-end)
}()
start <- 1
end <- 2
time.Sleep(1e9)
}
Another solution as you mentioned is to use waitgroup.
Apart from sleep where you have to specify the time, you can use waitgroup to make you program wait until the goroutine completes execution.
package main
import "fmt"
import "sync"
var wg sync.WaitGroup
func main() {
start := make(chan int)
end := make(chan int)
wg.Add(2)
go func() {
defer wg.Done()
fmt.Println("Start")
fmt.Println(<-start)
}()
go func() {
defer wg.Done()
fmt.Println("End")
fmt.Println(<-end)
}()
start <- 1
end <- 2
wg.Wait()
}
I wish to know what is the proper way of waiting for a go routine to finish before exiting the program. Reading some other answers it seems that a bool chan will do the trick, as in Playground link
func do_stuff(done chan bool) {
fmt.Println("Doing stuff")
done <- true
}
func main() {
fmt.Println("Main")
done := make(chan bool)
go do_stuff(done)
<-done
//<-done
}
I have two questions here:
why the <- done works at all?
what happens if I uncomment the last line? I have a deadlock error. Is this because the channel is empty and there is no other function sending values to it?
Listening to channel <- done, is a blocking operation, so your program won't continue until true or false is sent i.e. done <- true.
Your question can have a few different answers depending on the circumstance.
For instance, suppose you wanted to parallelize a series of function calls that take a long time.
I would use the sync package for this
package main
import (
"fmt"
"sync"
"time"
)
func main() {
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func() {
longOp()
wg.Done()
}()
}
// will wait until wg.Done is called 10 times
// since we made wg.Add(1) call 10 times
wg.Wait()
}
func longOp() {
time.Sleep(time.Second * 2)
fmt.Println("long op done")
}
Why the <- done works at all?
It works because the runtime detects that you're writing something to the channel somewhere else.
what happens if I uncomment the last line?
The runtime is smart enough to know that there's nothing else being written and it deadlocks.
Bonus, if you're extremely limited on memory, you can use done := make(chan struct{}) and done <- struct{}{}, struct{} is guaranteed to use 0 memory.
package main
import(
"fmt"
"time"
)
func main(){
fmt.Println("1")
defer fmt.Println("-1")
go func() {
fmt.Println("2")
defer fmt.Println("-2")
time.Sleep(9 * time.Second)
}()
time.Sleep(1 * time.Second)
fmt.Println("3")
}
Produces the output: 1 2 3 -1
But I would've thought the goroutine's defer would have been called to produce: 1 2 3 -2 -1
In my actual code my goroutine is blocked on a websocket... I suppose I could send a shutdown signal but I haven't yet figured out how to do a wait-on-multiple-objects kind of thing (if it can actually be done in go). I'm currently solving my problem by hoisting the deferred -2 into the main scope.
Is there some trick to defer placement I'm not doing right?
http://play.golang.org/p/qv8UEuF2Rb
(I don't have enough reps to comment?)
Yes, the issue is with goroutine scheduling and is platform dependent. Use runtime.Gosched at the end of the function with the goroutine to ensure it gets some cputime. Presently on x86_64 linux, the code below will produce "3" but without Gosched it produces "1":
No. That's not the issue, and that only solves it by co-incidence(via an implementation detail). As has already been hinted at in goroutine's defer not called when main scope ends . Your code is racy and you must therefore explicitly synchronize the goroutines. The simplest way is to to use a channel or a waitgroup http://golang.org/pkg/sync/#WaitGroup . Which option is more appropriate depends on what you're doing. If all you want to do is wait for some result, then have the gorutine send its result on the channel and you simply wait for it. If you just want to wait for a number of goroutines, then use a waitgroup.
The process is ending, so the internal function is terminated. Try sleeping at the end of main:
func main(){
fmt.Println("1")
defer fmt.Println("-1")
go func() {
fmt.Println("2")
defer fmt.Println("-2")
time.Sleep(9 * time.Second)
}()
time.Sleep(1 * time.Second)
fmt.Println("3")
time.Sleep(9 * time.Second)
}
http://play.golang.org/p/Qrtz_0xiRs
EDIT: as mentioned in the comments, this is not the right approach in general. You should explicitly wait for the goroutine.
Yes, the issue is with goroutine scheduling and is platform dependent. Use runtime.Gosched at the end of the function with the goroutine to ensure it gets some cputime. Presently on x86_64 linux, the code below will produce "3" but without Gosched it produces "1":
q = 1
go func() {
q = 2
defer func() { q = 3 }()
}()
runtime.Gosched()
fmt.Println(q)
http://play.golang.org/p/dNFQv9IxXv
WaitGroup can help you resolving the issue:
package main
import(
"fmt"
"sync"
"time"
)
func main(){
fmt.Println("-start main-")
defer fmt.Println("-defer main-")
wg := sync.WaitGroup{}
wg.Add(1)
go func() {
fmt.Println("-start goroutine-")
defer fmt.Println("-defer goroutine-")
time.Sleep(2 * time.Second)
wg.Done()
}()
fmt.Println("-end main-")
wg.Wait()
}
Prints:
-start main-
-end main-
-start goroutine-
-defer goroutine-
-defer main-
https://play.golang.org/p/ycmPJJ6pvC