I have the following piece of code:
func sendRegularHeartbeats(ctx context.Context) {
for {
select {
case <-ctx.Done():
return
case <-time.After(1 * time.Second):
sendHeartbeat()
}
}
}
This function is executed in a dedicated go-routine and sends a heartbeat-message every second. The whole process should stop immediately when the context is canceled.
Now consider the following scenario:
ctx, cancel := context.WithCancel(context.Background())
cancel()
go sendRegularHeartbeats(ctx)
This starts the heartbeat-routine with a closed context. In such a case, I don't want any heartbeats to be transmitted. So the first case block in the select should be entered immediately.
However, it seems that the order in which case blocks are evaluated is not guaranteed, and that the code sometimes sends a heartbeat message, even though the context is already canceled.
What is the correct way to implement such a behaviour?
I could add a "isContextclosed"-check in the second case, but that looks more like an ugly workaround for the problem.
The accepted answer has a wrong suggestion:
func sendRegularHeartbeats(ctx context.Context) {
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
for {
//first select
select {
case <-ctx.Done():
return
default:
}
//second select
select {
case <-ctx.Done():
return
case <-ticker.C:
sendHeartbeat()
}
}
}
This doesn't help, because of the following scenario:
both channels are empty
first select runs
both channels get a message concurrently
you are in the same probability game as if you haven't done anything in the first select
An alternative but still imperfect way is to guard against concurrent Done() events (the "wrong select") after consuming the ticker event i.e.
func sendRegularHeartbeats(ctx context.Context) {
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
for {
//select as usual
select {
case <-ctx.Done():
return
case <-ticker.C:
//give priority to a possible concurrent Done() event non-blocking way
select {
case <-ctx.Done():
return
default:
}
sendHeartbeat()
}
}
}
Caveat: the problem with this one is that it allows for "close enough" events to be confused - e.g. even though a ticker event arrived earlier, the Done event came soon enough to preempt the heartbeat. There is no perfect solution as of now.
Note beforehand:
Your example will work as you intend it to, as if the context is already cancelled when sendRegularHeartbeats() is called, the case <-ctx.Done() communication will be the only one ready to proceed and therefore chosen. The other case <-time.After(1 * time.Second) will only be ready to proceed after 1 second, so it will not be chosen at first. But to explicitly handle priorities when multiple cases might be ready, read on.
Unlike the case branches of a switch statement (where the evaluation order is the order they are listed), there is no priority or any order guaranteed in the case branches of a select statement.
Quoting from Spec: Select statements:
If one or more of the communications can proceed, a single one that can proceed is chosen via a uniform pseudo-random selection. Otherwise, if there is a default case, that case is chosen. If there is no default case, the "select" statement blocks until at least one of the communications can proceed.
If more communications can proceed, one is chosen randomly. Period.
If you want to maintain priority, you have to do that yourself (manually). You may do it using multiple select statements (subsequent, not nested), listing ones with higher priority in an earlier select, also be sure to add a default branch to avoid blocking if those are not ready to proceed. Your example requires 2 select statements, first one checking <-ctx.Done() as that is the one you want higher priority for.
I also recommend using a single time.Ticker instead of calling time.After() in each iteration (time.After() also uses a time.Ticker under the hood, but it doesn't reuse it just "throws it away" and creates a new one on the next call).
Here's an example implementation:
func sendRegularHeartbeats(ctx context.Context) {
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
default:
}
select {
case <-ctx.Done():
return
case <-ticker.C:
sendHeartbeat()
}
}
}
This will send no heartbeat if the context is already cancelled when sendRegularHeartbeats() is called, as you can check / verify it on the Go Playground.
If you delay the cancel() call for 2.5 seconds, then exactly 2 heartbeats will be sent:
ctx, cancel := context.WithCancel(context.Background())
go sendRegularHeartbeats(ctx)
time.Sleep(time.Millisecond * 2500)
cancel()
time.Sleep(time.Second * 2)
Try this one on the Go Playground.
If it is absolutely critical to maintain that priority of operations, you can:
Consume from each channel in a separate goroutine
Have each of those goroutines write a message to a shared third channel indicating its type
Have a third goroutine consume from that channel, reading the messages it receives to determine if it is a tick and should sendHeartbeat or if it is a cancel and it should exit
This way, messages received on the other channels will (probably - you can't guarantee order of execution of concurrent routines) come in on the third channel in the order they're triggered, allowing you to handle them appropriately.
However, it's worth noting that this is probably not necessary. A select does not guarantee which case will execute if multiple cases succeed simultaneously. That is probably a rare event; the cancel and ticker would both have to fire before either was handled by the select. The vast majority of the time, only one or the other will fire at any given loop iteration, so it will behave exactly as expected. If you can tolerate rare occurrences of firing one additional heartbeat after a cancellation, you're better off keeping the code you have, as it is more efficient and more readable.
Related
I have a scenario in which I'm processing events on a channel, and one of those events is a heartbeat which needs to occur within a certain timeframe. Events which are not heartbeats will continue consuming the timer, however whenever the heartbeat is received I want to reset the timer. The obvious way to do this would be by using a time.NewTimer.
For example:
func main() {
to := time.NewTimer(3200 * time.Millisecond)
for {
select {
case event, ok := <-c:
if !ok {
return
} else if event.Msg == "heartbeat" {
to.Reset(3200 * time.Millisecond)
}
case remediate := <-to.C:
fmt.Println("do some stuff ...")
return
}
}
}
Note that a time.Ticker won't work here as the remediation should only be triggered if the heartbeat hasn't been received, not every time.
The above solution works in the handful of low volume tests I've tried it on, however I came across a Github issue indicating that resetting a Timer which has not fired is a no-no. Additionally the documentation states:
Reset should be invoked only on stopped or expired timers with drained channels. If a program has already received a value from t.C, the timer is known to have expired and the channel drained, so t.Reset can be used directly. If a program has not yet received a value from t.C, however, the timer must be stopped and—if Stop reports that the timer expired before being stopped—the channel explicitly drained:
if !t.Stop() {
<-t.C
}
t.Reset(d)
This gives me pause, as it seems to describe exactly what I'm attempting to do. I'm resetting the Timer whenever the heartbeat is received, prior to it having fired. I'm not experienced enough with Go yet to digest the whole post, but it certainly seems like I may be headed down a dangerous path.
One other solution I thought of is to simply replace the Timer with a new one whenever the heartbeat occurs, e.g:
else if event.Msg == "heartbeat" {
to = time.NewTimer(3200 * time.Millisecond)
}
At first I was worried that the rebinding to = time.NewTimer(3200 * time.Millisecond) wouldn't be visible within the select:
For all the cases in the statement, the channel operands of receive operations and the channel and right-hand-side expressions of send statements are evaluated exactly once, in source order, upon entering the "select" statement. The result is a set of channels to receive from or send to, and the corresponding values to send.
But in this particular case since we are inside a loop, I would expect that upon each iteration we re-enter select and therefore the new binding should be visible. Is that a fair assumption?
I realize there are similar questions out there, and I've tried to read the relevant posts/documentation, but I am new to Go just want to be sure I'm understanding things correctly here.
So my questions are:
Is my use of timer.Reset() unsafe, or are the cases mentioned in the Github issue highlighting other problems which are not applicable here? Is the explanation in the docs cryptic or do I just need more experience with Go?
If it is unsafe, is my second proposed solution acceptable (rebinding the timer on each iteration).
ADDENDUM
Upon further reading, most of the pitfalls outlined in the issues are describing scenarios in which the timer has already fired (placing a result on the channel), and subsequent to that firing some other process attempts to Reset it. For this narrow case, I understand the need to test with !t.Stop() since a false return of Stop would indicate the timer has already fired, and as such must be drained prior to calling Reset.
What I still do not understand, is why it is necessary to call t.Stop() prior to t.Reset(), when the Timer has yet to fire. None of the examples go into that as far as I can tell.
What I still do not understand, is why it is necessary to call t.Stop() prior to t.Reset(), when the Timer has yet to fire.
The "when the Timer has yet to fire" bit is critical here. The timer fires within a separate go routine (part of the runtime) and this can happen at any time. You have no way of knowing whether the timer has fired at the time you call to.Reset(3200 * time.Millisecond) (it may even fire while that function is running!).
Here is an example that demonstrates this and is somewhat similar to what you are attempting (based on this):
func main() {
eventC := make(chan struct{}, 1)
go keepaliveLoop(eventC )
// Reset the timer 1000 (approx) times; once every millisecond (approx)
// This should prevent the timer from firing (because that only happens after 2 ms)
for i := 0; i < 1000; i++ {
time.Sleep(time.Millisecond)
// Don't block if there is already a reset request
select {
case eventC <- struct{}{}:
default:
}
}
}
func keepaliveLoop(eventC chan struct{}) {
to := time.NewTimer(2 * time.Millisecond)
for {
select {
case <-eventC:
//if event.Msg == "heartbeat"...
time.Sleep(3 * time.Millisecond) // Simulate reset work (delay could be partly dur to whatever is triggering the
to.Reset(2 * time.Millisecond)
case <-to.C:
panic("this should never happen")
}
}
}
Try it in the playground.
This may appear contrived due to the time.Sleep(3 * time.Millisecond) but that is just included to consistently demonstrate the issue. Your code may work 99.9% of the time but there is always the possibility that both the event and timer channels will fire before the select is run (in which a random case will run) or while the code in the case event, ok := <-c: block is running (including while Reset() is in progress). The result of this happening would be unexpected calls of the remediate code (which may not be a big issue).
Fortunately solving the issue is relatively easy (following the advice in the documentation):
time.Sleep(3 * time.Millisecond) // Simulate reset work (delay could be partly dur to whatever is triggering the
if !to.Stop() {
<-to.C
}
to.Reset(2 * time.Millisecond)
Try this in the playground.
This works because to.Stop returns "true if the call stops the timer, false if the timer has already expired or been stopped". Note that things get a more complicated if the timer is used in multiple go-routines "This cannot be done concurrent to other receives from the Timer's channel or other calls to the Timer's Stop method" but this is not the case in your use-case.
Is my use of timer.Reset() unsafe, or are the cases mentioned in the Github issue highlighting other problems which are not applicable here?
Yes - it is unsafe. However the impact is fairly low. The event arriving and timer triggering would need to happen almost concurrently and, in that case, running the remediate code might not be a big issue. Note that the fix is fairly simple (as per the docs)
If it is unsafe, is my second proposed solution acceptable (rebinding the timer on each iteration).
Your second proposed solution also works (but note that the garbage collector cannot free the timer until after it has fired, or been stopped, which may cause issues if you are creating timers rapidly).
Note: Re the suggestion from #JotaSantos
Another thing that could be done is to add a select when draining <-to.C (on the Stop "if") with a default clause. That would prevent the pause.
See this comment for details of why this may not be a good approach (it's also unnecessary in your situation).
I've faced a similar issue. After reading a lot of information, I came up with a solution that goes along these lines:
package main
import (
"fmt"
"time"
)
func main() {
const timeout = 2 * time.Second
// Prepare a timer that is stopped and ready to be reset.
// Stop will never return false, because an hour is too long
// for timer to fire. Thus there's no need to drain timer.C.
timer := time.NewTimer(timeout)
timer.Stop()
// Make sure to stop the timer when we return.
defer timer.Stop()
// This variable is needed because we need to track if we can safely reset the timer
// in a loop. Calling timer.Stop() will return false on every iteration, but we can only
// drain the timer.C once, otherwise it will deadlock.
var timerSet bool
c := make(chan time.Time)
// Simulate events that come in every second
// and every 5th event delays so that timer can fire.
go func() {
var i int
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
for t := range ticker.C {
i++
if i%5 == 0 {
fmt.Println("Sleeping")
time.Sleep(3 * time.Second)
}
c <- t
if i == 20 {
break
}
}
close(c)
}()
for {
select {
case t, ok := <-c:
if !ok {
fmt.Println("Closed channel")
return
}
fmt.Println("Got event", t, timerSet)
// We got an event, and timer was already set.
// We need to stop the timer and drain the channel if needed,
// so that we can safely reset it later.
if timerSet {
if !timer.Stop() {
<-timer.C
}
timerSet = false
}
// If timer was not set, or it was stopped before, it's safe to reset it.
if !timerSet {
timerSet = true
timer.Reset(timeout)
}
case remediate := <-timer.C:
fmt.Println("Timeout", remediate)
// It's important to store that timer is not set anymore.
timerSet = false
}
}
}
Link to playground: https://play.golang.org/p/0QlujZngEGg
In my program I have several go-routines who are essentially running endless processes. Why? you may ask, long story short it is the purpose of my entire application so it's out of question to change that. I would like to give users the ability to stop a single go-routine. I understand that I can use channel to signal the go-routines to stop, however there may be cases where I have, say, 10 go-routines running and I only want to stop 1. The issue is that the number of go-routines I want to run is dynamic and based on user input. What is the best way for me to add the ability to stop a go-routine dynamically and allow for singles to be stopped without the rest?
You need design a map to manage contexts.
Assume you've already known usage of context. It might look like:
ctx, cancel := context.WithCancel(ctx.TODO())
go func(ctx){
for {
select {
case <-ctx.Done():
return
default:
// job
}
}
}(ctx)
cancel()
Ok, now you can convert your question to another, it might called 'how to manage contexts of many goroutine'
type GoroutineManager struct{
m sync.Map
}
func (g *GoroutineManager) Add(cancel context.CancelFunc, key string)) {
g.m.Store(key, cancel)
}
func (g *GoroutineManager) KillGoroutine(key string) {
cancel, exist := g.m.Load(key)
if exist {
cancel()
}
}
Ok, Now you can manage your goroutine like :
ctx, cancel := context.WithCancel(ctx.TODO())
manager.Add(cancel, "routine-job-1")
go func(ctx){
for {
select {
case <-ctx.Done():
return
default:
// job
}
}
}(ctx)
// kill it as your wish
manager.KillGoroutine("routine-job-1")
I've been working with examples trying to get my first "go routine" running and while I got it running, it won't work as prescribed by the go documentation with the timer.Reset() function.
In my case I believe that the way I am doing it is just fine because I don't actually care what's in the chan buffer, if anything. All as this is meant to do is trigger case <-tmr.C: if anything happened on case _, ok := <-watcher.Events: and then all goes quiet for at least one second. The reason for this is that case _, ok := <-watcher.Events: can get from one to dozens of events microseconds apart and I only care once they are all done and things have settled down again.
However I'm concerned that doing it the way that the documentation says you "must do" doesn't work. If I knew go better I would say the documentation is flawed because it assumes there is something in the buffer when there may not be but I don't know go well enough to have confidence in making that determination so I'm hoping some experts out there can enlighten me.
Below is the code. I haven't put this up on playground because I would have to do some cleaning up (remove calls to other parts of the program) and I'm not sure how I would make it react to filesystem changes for showing it working.
I've clearly marked in the code which alternative works and which doesn't.
func (pm *PluginManager) LoadAndWatchPlugins() error {
// DOING OTHER STUFF HERE
fmt.Println(`m1`)
done := make(chan interface{})
terminated := make(chan interface{})
go pm.watchDir(done, terminated, nil)
fmt.Println(`m2.pre-10`)
time.Sleep(10 * time.Second)
fmt.Println(`m3-post-10`)
go pm.cancelWatchDir(done)
fmt.Println(`m4`)
<-terminated
fmt.Println(`m5`)
os.Exit(0) // Temporary for testing
return Err
}
func (pm *PluginManager) cancelWatchDir(done chan interface{}) {
fmt.Println(`t1`)
time.Sleep(5 * time.Second)
fmt.Println()
fmt.Println(`t2`)
close(done)
}
func (pm *PluginManager) watchDir(done <-chan interface{}, terminated chan interface{}, strings <-chan string) {
watcher, err := fsnotify.NewWatcher()
if err != nil {
Logger("watchDir::"+err.Error(), `plugins`, Error)
}
//err = watcher.Add(pm.pluginDir)
err = watcher.Add(`/srv/plugins/`)
if err != nil {
Logger("watchDir::"+err.Error(), `plugins`, Error)
}
var tmr = time.NewTimer(time.Second)
tmr.Stop()
defer close(terminated)
defer watcher.Close()
defer tmr.Stop()
for {
select {
case <-tmr.C:
fmt.Println(`UPDATE FIRED`)
tmr.Stop()
case _, ok := <-watcher.Events:
if !ok {
return
}
fmt.Println(`Ticker: STOP`)
/*
* START OF ALTERNATIVES
*
* THIS IS BY EXAMPLE AND STATED THAT IT "MUST BE" AT:
* https://golang.org/pkg/time/#Timer.Reset
*
* BUT DOESN'T WORK
*/
if !tmr.Stop() {
fmt.Println(`Ticker: CHAN DRAIN`)
<-tmr.C // STOPS HERE AND GOES NO FURTHER
}
/*
* BUT IF I JUST DO THIS IT WORKS
*/
tmr.Stop()
/*
* END OF ALTERNATIVES
*/
fmt.Println(`Ticker: RESET`)
tmr.Reset(time.Second)
case <-done:
fmt.Println(`DONE TRIGGERED`)
return
}
}
}
Besides what icza said (q.v.), note that the documentation says:
For example, assuming the program has not received from t.C already:
if !t.Stop() {
<-t.C
}
This cannot be done concurrent to other receives from the Timer's channel.
One could argue that this is not a great example since it assumes that the timer was running at the time you called t.Stop. But it does go on to mention that this is a bad idea if there's already some existing goroutine that is or may be reading from t.C.
(The Reset documentation repeats all of this, and kind of in the wrong order because Reset sorts before Stop.)
Essentially, the whole area is a bit fraught. There's no good general answer, because there are at least three possible situations during the return from t.Stop back to your call:
No one is listening to the channel, and no timer-tick is in the channel now. This is often the case if the timer was already stopped before the call to t.Stop. If the timer was already stopped, t.Stop always returns false.
No one is listening to the channel, and a timer-tick is in the channel now. This is always the case when the timer was running but t.Stop was unable to stop it from firing. In this case, t.Stop returns false. It's also the case when the timer was running but fired before you even called t.Stop, and had therefore stopped on its own, so that t.Stop was not able to stop it and returned false.
Someone else is listening to the channel.
In the last situation, you should do nothing. In the first situation, you should do nothing. In the second situation, you probably want to receive from the channel so as to clear it out. That's what their example is for.
One could argue that:
if !t.Stop() {
select {
case <-t.C:
default:
}
}
is a better example. It does one non-blocking attempt that will consume the timer-tick if present, and does nothing if there is no timer-tick. This works whether or not the timer was not actually running when you called t.Stop. Indeed, it even works if t.Stop returns true, though in that case, t.Stop stopped the timer, so the timer never managed to put a timer-tick into the channel. (Thus, if there is a datum in the channel, it must necessarily be left over from a previous failure to clear the channel. If there are no such bugs, the attempt to receive was in turn unnecessary.)
But, if someone else—some other goroutine—is or may be reading the channel, you should not do any of this at all. There is no way to know who (you or them) will get any timer tick that might be in the channel despite the call to Stop.
Meanwhile, if you're not going to use the timer any further, it's relatively harmless just to leave a timer-tick, if there is one, in the channel. It will be garbage-collected when the channel itself is garbage-collected. Of course, whether this is sensible depends on what you are doing with the timer, but in these cases it suffices to just call t.Stop and ignore its return value.
You create a timer and you stop it immediately:
var tmr = time.NewTimer(time.Second)
tmr.Stop()
This doesn't make any sense, I assume this is just an "accident" from your part.
But going further, inside your loop:
case _, ok := <-watcher.Events:
When this happens, you claim this doesn't work:
if !tmr.Stop() {
fmt.Println(`Ticker: CHAN DRAIN`)
<-tmr.C // STOPS HERE AND GOES NO FURTHER
}
Timer.Stop() documents that it returns true if this call stops the timer, and false if the timer has already been stopped (or expired). But your timer was already stopped, right after its creation, so tmr.Stop() returns false properly, so you go inside the if and try to receive from tmr.C, but since the timer was "long" stopped, nothing will be sent on its channel, so this is a blocking (forever) operation.
If you're the one stopping the timer explicitly with timer.Stop(), the recommended "pattern" to drain its channel doesn't make any sense and doesn't work for the 2nd Timer.Stop() call.
I'm learning Golang, and after reading this post on Go's blog, I have the following question.
I start with the following code (from the post):
select {
case <-ch:
// a read from ch has occurred
case <-timeout:
// the read from ch has timed out
}
And based on what A Tour of Go states:
... It chooses one at random if multiple are ready.
As I understand, it is possible to have my result ready and have a timeout at the same time. My question is whether it is worth it (or correct) to double-check for this inside the default case.
Something like the following:
select {
case <-ch:
// a read from ch has occurred
case <-timeout:
// the read from ch has timed out
// So check ch one last time
select {
case <-ch:
// a read from ch has occurred at same time of a timeout,
// so ignore the timeout
default:
// definitive timeout
}
}
If one of the channels is a timeout, odds of your work being done and the timeout firing at exactly the same time are so small they make no sense to consider.
The statement "... It chooses one at random if multiple are ready." is applicable when you actually have a viable reason for this to happen - when you have a select case on multiple job channels that you're processing with a single goroutine, for instance.
I am a Go rookie.
I'm looking at this construct:
for {
select {
case <-resyncCh:
case <-stopCh:
return
case <-cancelCh:
return
}
if r.ShouldResync == nil || r.ShouldResync() {
// do stuff
}
resyncCh = r.resyncChan()
}
I understand that the for loop runs forever.
I understand that break is implicit in Go.
I understand that channel operations in a select statement are blocking if there's no default clause (which there isn't here).
Suppose resyncCh does not have a message on it.
Are all the cases evaluated (blocked on) in parallel? Or is there another path through this I'm not seeing?
I read this as:
Block on the resyncCh, the stopCh and the cancelCh chans in parallel waiting for messages
If a message is received on resyncCh, we effectively fall through to the r.ShouldResync stuff, but the other blocks on the other chans remain.
If a message is received at any point on either the stopCh or the cancelCh chan, return, effectively "disconnecting" from all chans here.
Is that correct?
In direct answer to your questions:
Block on the resyncCh, the stopCh and the cancelCh chans in parallel waiting for messages. YES.
If a message is received on resyncCh, we effectively fall through to the r.ShouldResync stuff, but the other blocks on the other chans remain. No, they don't remain, you are past the select However, since this loops, you will block again. You could also use the fallthrough keyword to make them block after passing the initial one.
If a message is received at any point on either the stopCh or the cancelCh chan, return, effectively "disconnecting" from all chans here. Correct - they would return from this function.
Also, bear in mind what you can do with a default --> https://gobyexample.com/non-blocking-channel-operations
for {
select {
case <-resyncCh:
case <-stopCh:
return
case <-cancelCh:
return
default:
fmt.Printf("will keep printing\n")
}
if r.ShouldResync == nil || r.ShouldResync() {
// do stuff
}
resyncCh = r.resyncChan()
}
update: Another useful pattern, I'm using right now, which takes advantage of this:
select {
case m := <-c:
handle(m)
case <-time.After(5 * time.Minute):
fmt.Println("timed out")
}
Here you can wait, blocking, on a channel, but eventually timeout, just using the golang time package. Very succinct and easy to read. Compare that to poll() with timespec values.
https://golang.org/pkg/time/#After
select takes first not blocked action and goes to next operation.
Block on the resyncCh, the stopCh and the cancelCh chans in parallel waiting for messages
Yes, waiting for first of them.
If a message is received on resyncCh, we effectively fall through to the r.ShouldResync stuff, but the other blocks on the other chans remain.
Unlike in some other languages fallthrough is explicit in go - you should state it.
If a message is received at any point on either the stopCh or the cancelCh chan, return, effectively "disconnecting" from all chans here.
Exit from the function where the code located. Yes, we do not wait for new messages more.