Related
I have a scenario in which I'm processing events on a channel, and one of those events is a heartbeat which needs to occur within a certain timeframe. Events which are not heartbeats will continue consuming the timer, however whenever the heartbeat is received I want to reset the timer. The obvious way to do this would be by using a time.NewTimer.
For example:
func main() {
to := time.NewTimer(3200 * time.Millisecond)
for {
select {
case event, ok := <-c:
if !ok {
return
} else if event.Msg == "heartbeat" {
to.Reset(3200 * time.Millisecond)
}
case remediate := <-to.C:
fmt.Println("do some stuff ...")
return
}
}
}
Note that a time.Ticker won't work here as the remediation should only be triggered if the heartbeat hasn't been received, not every time.
The above solution works in the handful of low volume tests I've tried it on, however I came across a Github issue indicating that resetting a Timer which has not fired is a no-no. Additionally the documentation states:
Reset should be invoked only on stopped or expired timers with drained channels. If a program has already received a value from t.C, the timer is known to have expired and the channel drained, so t.Reset can be used directly. If a program has not yet received a value from t.C, however, the timer must be stopped and—if Stop reports that the timer expired before being stopped—the channel explicitly drained:
if !t.Stop() {
<-t.C
}
t.Reset(d)
This gives me pause, as it seems to describe exactly what I'm attempting to do. I'm resetting the Timer whenever the heartbeat is received, prior to it having fired. I'm not experienced enough with Go yet to digest the whole post, but it certainly seems like I may be headed down a dangerous path.
One other solution I thought of is to simply replace the Timer with a new one whenever the heartbeat occurs, e.g:
else if event.Msg == "heartbeat" {
to = time.NewTimer(3200 * time.Millisecond)
}
At first I was worried that the rebinding to = time.NewTimer(3200 * time.Millisecond) wouldn't be visible within the select:
For all the cases in the statement, the channel operands of receive operations and the channel and right-hand-side expressions of send statements are evaluated exactly once, in source order, upon entering the "select" statement. The result is a set of channels to receive from or send to, and the corresponding values to send.
But in this particular case since we are inside a loop, I would expect that upon each iteration we re-enter select and therefore the new binding should be visible. Is that a fair assumption?
I realize there are similar questions out there, and I've tried to read the relevant posts/documentation, but I am new to Go just want to be sure I'm understanding things correctly here.
So my questions are:
Is my use of timer.Reset() unsafe, or are the cases mentioned in the Github issue highlighting other problems which are not applicable here? Is the explanation in the docs cryptic or do I just need more experience with Go?
If it is unsafe, is my second proposed solution acceptable (rebinding the timer on each iteration).
ADDENDUM
Upon further reading, most of the pitfalls outlined in the issues are describing scenarios in which the timer has already fired (placing a result on the channel), and subsequent to that firing some other process attempts to Reset it. For this narrow case, I understand the need to test with !t.Stop() since a false return of Stop would indicate the timer has already fired, and as such must be drained prior to calling Reset.
What I still do not understand, is why it is necessary to call t.Stop() prior to t.Reset(), when the Timer has yet to fire. None of the examples go into that as far as I can tell.
What I still do not understand, is why it is necessary to call t.Stop() prior to t.Reset(), when the Timer has yet to fire.
The "when the Timer has yet to fire" bit is critical here. The timer fires within a separate go routine (part of the runtime) and this can happen at any time. You have no way of knowing whether the timer has fired at the time you call to.Reset(3200 * time.Millisecond) (it may even fire while that function is running!).
Here is an example that demonstrates this and is somewhat similar to what you are attempting (based on this):
func main() {
eventC := make(chan struct{}, 1)
go keepaliveLoop(eventC )
// Reset the timer 1000 (approx) times; once every millisecond (approx)
// This should prevent the timer from firing (because that only happens after 2 ms)
for i := 0; i < 1000; i++ {
time.Sleep(time.Millisecond)
// Don't block if there is already a reset request
select {
case eventC <- struct{}{}:
default:
}
}
}
func keepaliveLoop(eventC chan struct{}) {
to := time.NewTimer(2 * time.Millisecond)
for {
select {
case <-eventC:
//if event.Msg == "heartbeat"...
time.Sleep(3 * time.Millisecond) // Simulate reset work (delay could be partly dur to whatever is triggering the
to.Reset(2 * time.Millisecond)
case <-to.C:
panic("this should never happen")
}
}
}
Try it in the playground.
This may appear contrived due to the time.Sleep(3 * time.Millisecond) but that is just included to consistently demonstrate the issue. Your code may work 99.9% of the time but there is always the possibility that both the event and timer channels will fire before the select is run (in which a random case will run) or while the code in the case event, ok := <-c: block is running (including while Reset() is in progress). The result of this happening would be unexpected calls of the remediate code (which may not be a big issue).
Fortunately solving the issue is relatively easy (following the advice in the documentation):
time.Sleep(3 * time.Millisecond) // Simulate reset work (delay could be partly dur to whatever is triggering the
if !to.Stop() {
<-to.C
}
to.Reset(2 * time.Millisecond)
Try this in the playground.
This works because to.Stop returns "true if the call stops the timer, false if the timer has already expired or been stopped". Note that things get a more complicated if the timer is used in multiple go-routines "This cannot be done concurrent to other receives from the Timer's channel or other calls to the Timer's Stop method" but this is not the case in your use-case.
Is my use of timer.Reset() unsafe, or are the cases mentioned in the Github issue highlighting other problems which are not applicable here?
Yes - it is unsafe. However the impact is fairly low. The event arriving and timer triggering would need to happen almost concurrently and, in that case, running the remediate code might not be a big issue. Note that the fix is fairly simple (as per the docs)
If it is unsafe, is my second proposed solution acceptable (rebinding the timer on each iteration).
Your second proposed solution also works (but note that the garbage collector cannot free the timer until after it has fired, or been stopped, which may cause issues if you are creating timers rapidly).
Note: Re the suggestion from #JotaSantos
Another thing that could be done is to add a select when draining <-to.C (on the Stop "if") with a default clause. That would prevent the pause.
See this comment for details of why this may not be a good approach (it's also unnecessary in your situation).
I've faced a similar issue. After reading a lot of information, I came up with a solution that goes along these lines:
package main
import (
"fmt"
"time"
)
func main() {
const timeout = 2 * time.Second
// Prepare a timer that is stopped and ready to be reset.
// Stop will never return false, because an hour is too long
// for timer to fire. Thus there's no need to drain timer.C.
timer := time.NewTimer(timeout)
timer.Stop()
// Make sure to stop the timer when we return.
defer timer.Stop()
// This variable is needed because we need to track if we can safely reset the timer
// in a loop. Calling timer.Stop() will return false on every iteration, but we can only
// drain the timer.C once, otherwise it will deadlock.
var timerSet bool
c := make(chan time.Time)
// Simulate events that come in every second
// and every 5th event delays so that timer can fire.
go func() {
var i int
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
for t := range ticker.C {
i++
if i%5 == 0 {
fmt.Println("Sleeping")
time.Sleep(3 * time.Second)
}
c <- t
if i == 20 {
break
}
}
close(c)
}()
for {
select {
case t, ok := <-c:
if !ok {
fmt.Println("Closed channel")
return
}
fmt.Println("Got event", t, timerSet)
// We got an event, and timer was already set.
// We need to stop the timer and drain the channel if needed,
// so that we can safely reset it later.
if timerSet {
if !timer.Stop() {
<-timer.C
}
timerSet = false
}
// If timer was not set, or it was stopped before, it's safe to reset it.
if !timerSet {
timerSet = true
timer.Reset(timeout)
}
case remediate := <-timer.C:
fmt.Println("Timeout", remediate)
// It's important to store that timer is not set anymore.
timerSet = false
}
}
}
Link to playground: https://play.golang.org/p/0QlujZngEGg
I've been working with examples trying to get my first "go routine" running and while I got it running, it won't work as prescribed by the go documentation with the timer.Reset() function.
In my case I believe that the way I am doing it is just fine because I don't actually care what's in the chan buffer, if anything. All as this is meant to do is trigger case <-tmr.C: if anything happened on case _, ok := <-watcher.Events: and then all goes quiet for at least one second. The reason for this is that case _, ok := <-watcher.Events: can get from one to dozens of events microseconds apart and I only care once they are all done and things have settled down again.
However I'm concerned that doing it the way that the documentation says you "must do" doesn't work. If I knew go better I would say the documentation is flawed because it assumes there is something in the buffer when there may not be but I don't know go well enough to have confidence in making that determination so I'm hoping some experts out there can enlighten me.
Below is the code. I haven't put this up on playground because I would have to do some cleaning up (remove calls to other parts of the program) and I'm not sure how I would make it react to filesystem changes for showing it working.
I've clearly marked in the code which alternative works and which doesn't.
func (pm *PluginManager) LoadAndWatchPlugins() error {
// DOING OTHER STUFF HERE
fmt.Println(`m1`)
done := make(chan interface{})
terminated := make(chan interface{})
go pm.watchDir(done, terminated, nil)
fmt.Println(`m2.pre-10`)
time.Sleep(10 * time.Second)
fmt.Println(`m3-post-10`)
go pm.cancelWatchDir(done)
fmt.Println(`m4`)
<-terminated
fmt.Println(`m5`)
os.Exit(0) // Temporary for testing
return Err
}
func (pm *PluginManager) cancelWatchDir(done chan interface{}) {
fmt.Println(`t1`)
time.Sleep(5 * time.Second)
fmt.Println()
fmt.Println(`t2`)
close(done)
}
func (pm *PluginManager) watchDir(done <-chan interface{}, terminated chan interface{}, strings <-chan string) {
watcher, err := fsnotify.NewWatcher()
if err != nil {
Logger("watchDir::"+err.Error(), `plugins`, Error)
}
//err = watcher.Add(pm.pluginDir)
err = watcher.Add(`/srv/plugins/`)
if err != nil {
Logger("watchDir::"+err.Error(), `plugins`, Error)
}
var tmr = time.NewTimer(time.Second)
tmr.Stop()
defer close(terminated)
defer watcher.Close()
defer tmr.Stop()
for {
select {
case <-tmr.C:
fmt.Println(`UPDATE FIRED`)
tmr.Stop()
case _, ok := <-watcher.Events:
if !ok {
return
}
fmt.Println(`Ticker: STOP`)
/*
* START OF ALTERNATIVES
*
* THIS IS BY EXAMPLE AND STATED THAT IT "MUST BE" AT:
* https://golang.org/pkg/time/#Timer.Reset
*
* BUT DOESN'T WORK
*/
if !tmr.Stop() {
fmt.Println(`Ticker: CHAN DRAIN`)
<-tmr.C // STOPS HERE AND GOES NO FURTHER
}
/*
* BUT IF I JUST DO THIS IT WORKS
*/
tmr.Stop()
/*
* END OF ALTERNATIVES
*/
fmt.Println(`Ticker: RESET`)
tmr.Reset(time.Second)
case <-done:
fmt.Println(`DONE TRIGGERED`)
return
}
}
}
Besides what icza said (q.v.), note that the documentation says:
For example, assuming the program has not received from t.C already:
if !t.Stop() {
<-t.C
}
This cannot be done concurrent to other receives from the Timer's channel.
One could argue that this is not a great example since it assumes that the timer was running at the time you called t.Stop. But it does go on to mention that this is a bad idea if there's already some existing goroutine that is or may be reading from t.C.
(The Reset documentation repeats all of this, and kind of in the wrong order because Reset sorts before Stop.)
Essentially, the whole area is a bit fraught. There's no good general answer, because there are at least three possible situations during the return from t.Stop back to your call:
No one is listening to the channel, and no timer-tick is in the channel now. This is often the case if the timer was already stopped before the call to t.Stop. If the timer was already stopped, t.Stop always returns false.
No one is listening to the channel, and a timer-tick is in the channel now. This is always the case when the timer was running but t.Stop was unable to stop it from firing. In this case, t.Stop returns false. It's also the case when the timer was running but fired before you even called t.Stop, and had therefore stopped on its own, so that t.Stop was not able to stop it and returned false.
Someone else is listening to the channel.
In the last situation, you should do nothing. In the first situation, you should do nothing. In the second situation, you probably want to receive from the channel so as to clear it out. That's what their example is for.
One could argue that:
if !t.Stop() {
select {
case <-t.C:
default:
}
}
is a better example. It does one non-blocking attempt that will consume the timer-tick if present, and does nothing if there is no timer-tick. This works whether or not the timer was not actually running when you called t.Stop. Indeed, it even works if t.Stop returns true, though in that case, t.Stop stopped the timer, so the timer never managed to put a timer-tick into the channel. (Thus, if there is a datum in the channel, it must necessarily be left over from a previous failure to clear the channel. If there are no such bugs, the attempt to receive was in turn unnecessary.)
But, if someone else—some other goroutine—is or may be reading the channel, you should not do any of this at all. There is no way to know who (you or them) will get any timer tick that might be in the channel despite the call to Stop.
Meanwhile, if you're not going to use the timer any further, it's relatively harmless just to leave a timer-tick, if there is one, in the channel. It will be garbage-collected when the channel itself is garbage-collected. Of course, whether this is sensible depends on what you are doing with the timer, but in these cases it suffices to just call t.Stop and ignore its return value.
You create a timer and you stop it immediately:
var tmr = time.NewTimer(time.Second)
tmr.Stop()
This doesn't make any sense, I assume this is just an "accident" from your part.
But going further, inside your loop:
case _, ok := <-watcher.Events:
When this happens, you claim this doesn't work:
if !tmr.Stop() {
fmt.Println(`Ticker: CHAN DRAIN`)
<-tmr.C // STOPS HERE AND GOES NO FURTHER
}
Timer.Stop() documents that it returns true if this call stops the timer, and false if the timer has already been stopped (or expired). But your timer was already stopped, right after its creation, so tmr.Stop() returns false properly, so you go inside the if and try to receive from tmr.C, but since the timer was "long" stopped, nothing will be sent on its channel, so this is a blocking (forever) operation.
If you're the one stopping the timer explicitly with timer.Stop(), the recommended "pattern" to drain its channel doesn't make any sense and doesn't work for the 2nd Timer.Stop() call.
I am trying to reuse timers by stopping and resetting them. I am following the pattern provided by the documentation. Here is a simple example which can be run in go playground that demonstrates the issue I am experiencing.
Is there a correct way to stop and reset a timer that doesn't involve deadlock or race conditions? I am aware that using a select with default involves a race condition on channel message delivery timing and cannot be depended on.
package main
import (
"fmt"
"time"
"sync"
)
func main() {
fmt.Println("Hello, playground")
timer := time.NewTimer(1 * time.Second)
wg := &sync.WaitGroup{}
wg.Add(1)
go func(_wg *sync.WaitGroup) {
<- timer.C
fmt.Println("Timer done")
_wg.Done()
}(wg)
wg.Wait()
fmt.Println("Checking timer")
if !timer.Stop() {
<- timer.C
}
fmt.Println("Done")
}
According to the timer.Stop docs, there is a caveat for draining the channel:
assuming the program has not received from t.C already ...
This cannot be done concurrent to other receives from the Timer's
channel.
Since the channel has already been drained - and will never fire again, the second <-timer.C will block forever.
The question asks, in the first place, why the timer hangs. That's a good question, because even in the absence of bugs in the user program, there is... at least some weird ambiguity in how this thing, called time.Timer, works in Go. The spec says, specifically this:
Stop prevents the Timer from firing. It returns true if the call stops
the timer, false if the timer has already expired or been stopped.
Stop does not close the channel, to prevent a read from the channel
succeeding incorrectly.
To ensure the channel is empty after a call to Stop, check the return
value and drain the channel. For example, assuming the program has not
received from t.C already:
if !t.Stop() {
<-t.C
}
This cannot be done concurrent to other receives from the Timer's
channel or other calls to the Timer's Stop method.
There are very short and precise words, but it may be not that easy to understand them (at least for me). I tried to use the Timer repeatedly in a piece of code, and reset it each time before the next use. Each time I do so, I may want to Stop() it - just for sure. The spec above implies how you should do that, and provides an example - and it may not work! It depends, it depends where you try to apply the Stop idiom. In case you do it after you already in a select-case on this very timer, then it will hang the program.
Specifically, I do not have any concurrent receivers, only a single goroutine. So let's make a simple test program, and try to experiment with it (https://play.golang.org/p/d7BlNReE9Jz):
package main
import (
"fmt"
"time"
)
func main() {
i := 0
d2s := time.Second * 1
i++; fmt.Println(i)
t := time.NewTimer(d2s)
<-t.C
i++; fmt.Println(i)
t.Reset(d2s)
<-t.C
i++; fmt.Println(i)
// if !t.Stop() { <-t.C }
// if !t.Stop() { select { case <-t.C: default: } }
t.Reset(d2s)
<-t.C
i++; fmt.Println(i)
}
This code WORKS. It prints 1,2,3,4, delayed by 1 sec, and that's what it is expected to print. So far so good.
Now, try to un-comment the first commented line. Now the thing: according to spec, it is 100% right (is it?), and must work, but it does not, and hangs. Why? Because, according to spec, it must hang! I already read the channel, and the timer is stopped, so the if fires, and the channel drain op hangs.
Is this a bug? No. Is the spec wrong? No, it's correct. But, it's contrary to what a typical timer user would want. (Maybe a subject for proposal to Go?). All we need, is something like:
t.SafeStopDrain()
Which would do this right, and never hang. But, sadly, it is non-existent.
Here's the life-hack, the workaround, to make this work, is the second commented line. Un-comment it, and that will both work, and do what you wanted - make sure the timer is stopped, channel drained, and the whole thing is fresh anew for re-use.
I'm learning Golang, and after reading this post on Go's blog, I have the following question.
I start with the following code (from the post):
select {
case <-ch:
// a read from ch has occurred
case <-timeout:
// the read from ch has timed out
}
And based on what A Tour of Go states:
... It chooses one at random if multiple are ready.
As I understand, it is possible to have my result ready and have a timeout at the same time. My question is whether it is worth it (or correct) to double-check for this inside the default case.
Something like the following:
select {
case <-ch:
// a read from ch has occurred
case <-timeout:
// the read from ch has timed out
// So check ch one last time
select {
case <-ch:
// a read from ch has occurred at same time of a timeout,
// so ignore the timeout
default:
// definitive timeout
}
}
If one of the channels is a timeout, odds of your work being done and the timeout firing at exactly the same time are so small they make no sense to consider.
The statement "... It chooses one at random if multiple are ready." is applicable when you actually have a viable reason for this to happen - when you have a select case on multiple job channels that you're processing with a single goroutine, for instance.
I'm trying to understand a bit more about what happens under the surface during various blocking/waiting types of operations in Go. Take the following example:
otherChan = make(chan int)
t = time.NewTicker(time.Second)
for {
doThings()
// OPTION A: Sleep
time.Sleep(time.Second)
// OPTION B: Blocking ticker
<- t.C
// OPTION C: Select multiple
select {
case <- otherChan:
case <- t.C:
}
}
From a low level view (system calls, cpu scheduling) what is the difference between these while waiting?
My understanding is that time.Sleep leaves the CPU free to perform other tasks until the specified time has elapsed. Does the blocking ticker <- t.C do the same? Is the processor polling the channel or is there an interrupt involved? Does having multiple channels in a select change anything?
In other words, assuming that otherChan never had anything put into it, would these three options execute in an identical way, or would one be less resource intensive than the others?
That's a very interesting question, so I did cd into my Go source to start looking.
time.Sleep
time.Sleep is defined like this:
// src/time/sleep.go
// Sleep pauses the current goroutine for at least the duration d.
// A negative or zero duration causes Sleep to return immediately.
func Sleep(d Duration)
No body, no definition in an OS-specific time_unix.go!?! A little search and the answer is because time.Sleep is actually defined in the runtime:
// src/runtime/time.go
// timeSleep puts the current goroutine to sleep for at least ns nanoseconds.
//go:linkname timeSleep time.Sleep
func timeSleep(ns int64) {
// ...
}
Which in retrospect makes a lot of sense, as it has to interact with the goroutine scheduler. It ends up calling goparkunlock, which "puts the goroutine into a waiting state". time.Sleep creates a runtime.timer with a callback function that is called when the timer expires - that callback function wakes up the goroutine by calling goready. See next section for more details on the runtime.timer.
time.NewTicker
time.NewTicker creates a *Ticker (and time.Tick is a helper function that does the same thing but directly returns *Ticker.C, the ticker's receive channel, instead of *Ticker, so you could've written your code with it instead) has similar hooks into the runtime: a ticker is a struct that holds a runtimeTimer and a channel on which to signal the ticks.
runtimeTimer is defined in the time package but it must be kept in sync with timer in src/runtime/time.go, so it is effectively a runtime.timer. Remember that in time.Sleep, the timer had a callback function to wake up the sleeping goroutine? In the case of *Ticker, the timer's callback function sends the current time on the ticker's channel.
Then, the real waiting/scheduling happens on the receive from the channel, which is essentially the same as the select statement unless otherChan sends something before the tick, so let's look at what happens on a blocking receive.
<- chan
Channels are implemented (now in Go!) in src/runtime/chan.go, by the hchan struct. Channel operations have matching functions, and a receive is implemented by chanrecv:
// chanrecv receives on channel c and writes the received data to ep.
// ep may be nil, in which case received data is ignored.
// If block == false and no elements are available, returns (false, false).
// Otherwise, if c is closed, zeros *ep and returns (true, false).
// Otherwise, fills in *ep with an element and returns (true, true).
func chanrecv(t *chantype, c *hchan, ep unsafe.Pointer, block bool) (selected, received bool) {
// ...
}
This part has a lot of different cases, but in your example, it is a blocking receive from an asynchronous channel (time.NewTicker creates a channel with a buffer of 1), but anyway it ends up calling... goparkunlock, again to allow other goroutines to proceed while this one is stuck waiting.
So...
In all cases, the goroutine ends up being parked (which is not really shocking - it can't make progress, so it has to leave its thread available for a different goroutine if there's any available). A glance at the code seems to suggest that the channel has a bit more overhead than a straight-up time.Sleep. However, it allows far more powerful patterns, such as the last one in your example: the goroutine can be woken up by another channel, whichever comes first.
To answer your other questions, regarding polling, the timers are managed by a goroutine that sleeps until the next timer in its queue, so it's working only when it knows a timer has to be triggered. When the next timer has expired, it wakes up the goroutine that called time.Sleep (or sends the value on the ticker's channel, it does whatever the callback function does).
There's no polling in channels, the receive is unlocked when a send is made on the channel, in chansend of the chan.go file:
// wake up a waiting receiver
sg := c.recvq.dequeue()
if sg != nil {
recvg := sg.g
unlock(&c.lock)
if sg.releasetime != 0 {
sg.releasetime = cputicks()
}
goready(recvg, 3)
} else {
unlock(&c.lock)
}
That was an interesting dive into Go's source code, very interesting question! Hope I answered at least part of it!