I have a strange bug - go

given are the following 2 functions.
func main() {
index := int(0)
for {
Loop(index)
index = (index + 1) % 86400 // Max interval: 1 day
time.Sleep(1 * time.Second)
}
}
func Loop(index int) {
if index%10 == 0 {
go doSomething...
}
}
I want to execute something every 10/60/3600 seconds. So I thought an incrementing index with modulo should do this.
But what I noticed (especially on high traffic servers) that it appears to skip some of that loops.
I looked in my logs and sometimes there is something every 10 seconds but sometimes there is a gap up to 1 minute.
Does anybody know why this is happening?

I'd recommend using a time.Ticker to perform some action every N seconds. That way, you use built-in timers and only wake the CPU when something needs to be done. Even if the CPU is not heavily used, time.Sleep and a for loop is not the most reliable way to schedule tasks. For example (from the link above):
package main
import (
"fmt"
"time"
)
func main() {
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
done := make(chan bool)
go func() {
time.Sleep(10 * time.Second)
done <- true
}()
for {
select {
case <-done:
fmt.Println("Done!")
return
case t := <-ticker.C:
fmt.Println("Current time: ", t)
}
}
}

Related

Run function every N seconds with context timeout

I have a basic question about scheduling "cancellable" goroutines.
I want to schedule a function execution, every 3 seconds.
The function can take up to 5 seconds.
In case it takes more than 2999ms I want to stop/terminate it, to avoid overlapping w/ the next one.
I'm doing it wrong:
func main() {
fmt.Println("startProcessing")
go startProcessing()
time.Sleep(time.Second * 60)
fmt.Println("endProcessing after 60s")
}
func startProcessing() {
ticker := time.NewTicker(3 * time.Second)
for _ = range ticker.C {
ctx, _ := context.WithTimeout(context.Background(), (time.Second*3)-time.Millisecond)
fmt.Println("start doSomething")
doSomething(ctx)
}
}
func doSomething(ctx context.Context) {
executionTime := time.Duration(rand.Intn(5)+1) * time.Second
for {
select {
case <-ctx.Done():
fmt.Printf("timed out after %s\n", executionTime)
return
default:
time.Sleep(executionTime)
fmt.Printf("did something in %s\n", executionTime)
return
}
}
}
This is my output now:
startProcessing
start doSomething
did something in 2s
start doSomething
did something in 3s
start doSomething
did something in 3s
start doSomething
did something in 5s
start doSomething
did something in 2s
...
I want to read timed out after 5s instead of did something in 5s.
You just need to put the time.Sleep(executionTime) outside the select and there is no need for the for loop. I think this is somehow what you want but beware that it's not good practice. So take a look at the warning below.
func doSomething(ctx context.Context) {
executionTime := time.Duration(rand.Intn(5)+1) * time.Second
processed := make(chan int)
go func() {
time.Sleep(executionTime)
processed <- 1
}()
select {
case <-ctx.Done():
fmt.Printf("timed out after %s\n", executionTime)
case <-processed:
fmt.Printf("did something in %s\n", executionTime)
}
}
Obs: I changed the original answer a bit. We can not interrupt a goroutine in the middle of its execution. We could delegate the long-running task to another goroutine and receive the result through a dedicated channel.
Warning: I wouldn't recommend that if you expect the processing time to exceed the deadline because now you will have a leaking goroutine.

Default case in select statement almost behaves as blocking

I noticed a weird behavior that is not entirely obvious when using select statement inside a loop, for example if I have:
package main
import (
"fmt"
"time"
)
func main() {
done := make(chan bool)
go func() {
fmt.Println("here we go")
for {
select {
case <-done:
fmt.Println("Bye")
return
default:
fmt.Println("this should continue to print")
}
fmt.Println("loop continues")
}
}()
time.Sleep(2 * time.Second)
done <- true
}
I would assume that the default case should only print for no longer than 2 seconds, however this is not the case, at least in my CPU it is lasting for about 10 seconds before seeing the Bye message.
However if I add time.Sleep(1 * time.Second) right after "loop continues" It is more or less in sync with the 2 second sleep before done <- true.
Given this my assumption is that print calls are getting stacked and the extended completion time is due to the call stack taking longer to get through.
Is is this a correct description of what's happening? Or am I missing something more obvious.
Edit: To put my hypothesis to test I did this and it now works as expected, it takes 2 seconds to finish the program:
go func() {
fmt.Println("here we go")
for {
select {
case <-done:
fmt.Println("Bye")
return
default:
}
}
}()

How to start a cron job at regular intervals starting from a time in the past

I would like to build a program which runs a number of cron jobs which start at a time which, in general, is in the past. Here is a simplified example using gocron:
package main
import (
"time"
"github.com/jasonlvhit/gocron"
"github.com/sirupsen/logrus"
)
// This slice would be obtained from persistent storage
var startTimes = []time.Time{
time.Now().Add(-4 * time.Second),
time.Now().Add(-3 * time.Second),
}
func format(t time.Time) string {
return t.Format("15:04:05")
}
func notify(startTime time.Time) {
logrus.WithField("time", format(time.Now())).Infof("I started at %s\n", format(startTime))
}
func main() {
for _, startTime := range startTimes {
gocron.Every(10).Seconds().From(&startTime).Do(notify, startTime)
}
logrus.Infof("Starting at %s...\n", format(time.Now()))
<-gocron.Start()
}
If I run this, I get the following output:
INFO[0000] Starting at 00:30:54...
INFO[0010] I started at 00:30:50 fields.time="00:31:04"
INFO[0010] I started at 00:30:51 fields.time="00:31:04"
What I observe is that all the events are happening at once, 10 seconds after I start the program.
However, since the startTimes are 4 and 3 seconds before the program starts, what I would like is for the events to occur 6 and 7 seconds after the program starts (and every 10 seconds after that), respectively.
Is there a way to do this with gocron or some other tool?
This seems like just simple math:
interval := 10 * time.Second
nextTime := time.Now().Add((time.Since(startTime) + interval) % interval)
gocron.Every(10).Seconds().From(&nextTime).Do(notify, nextTime)
https://play.golang.org/p/pwEZqy_LUuk
We can try to leverage a simple ticker, it is not a complete solution but should be easy enough to adapt.
package main
import (
"fmt"
"time"
)
func ticker(period time.Duration, length ...time.Duration) <-chan time.Time {
ticker := time.NewTicker(period)
if len(length) > 0 {
done := make(chan bool)
go func() {
time.Sleep(period + length[0])
done <- true
}()
go func() {
<-done
ticker.Stop()
}()
}
return ticker.C
}
func main() {
t1 := ticker(6*time.Second, 4*time.Second)
t2 := ticker(7*time.Second, 3*time.Second)
t3 := ticker(10 * time.Second)
for {
select {
case t1 := <-t1:
fmt.Println("t1: ", t1)
case t2 := <-t2:
fmt.Println("t2: ", t2)
case t3 := <-t3:
fmt.Println("t3: ", t3)
}
}
}

Wanted data-race or bad design?

I am implementing an app that integrates a third party API that has a limit of hits per second. I wrote my adapter and I was a happy man until I run my tests with the race condition detector.
The design is simple, there is a:
A struct that counts the requests it has made
A tick that resets this counter to 0 every second
A private function on this struct which is blocking until conditions are met to allow to do an extra call to the API.
Running this test case works very well until you give it the -race flag.
I believe the data-race is caused by the tick thread trying to reset the hit counter and the call requests who increments it...
Is my design bad or should I just live with a data-race alert ?
import (
"sync"
"testing"
"time"
)
var subject httpClientWrapper
func init() {
subject = httpClientWrapper{
hits: 0,
hitsSecond: 1,
}
// reset hits every second to 0
go func() {
tick := time.Tick(1 * time.Second)
for range tick {
subject.hits = 0
}
}()
}
type httpClientWrapper struct {
hits, hitsSecond int
}
var m sync.Mutex
func (c *httpClientWrapper) allowCall() {
m.Lock()
callAllowanceReached := c.hits >= c.hitsSecond
for callAllowanceReached {
// cool down for one second
time.Sleep(1 * time.Second)
callAllowanceReached = c.hits >= c.hitsSecond
}
c.hits = c.hits + 1
m.Unlock()
}
func TestItSleeps(t *testing.T) {
timeStart := time.Now()
var wg = sync.WaitGroup{}
for i := 0; i < 3; i++ {
wg.Add(1)
go func() {
subject.allowCall()
wg.Done()
}()
}
wg.Wait()
elapsedTime := time.Since(timeStart)
if elapsedTime < (1 * time.Second) {
t.Errorf("this test should not had been able to run in less than a second due to locks and cool down")
}
}
Any access to .hits should be behind the mutex, so
// reset hits every second to 0
go func() {
tick := time.Tick(1 * time.Second)
for range tick {
m.Lock()
subject.hits = 0
m.Unlock()
}
}()
Also any sleeps should not occur with the mutex locked, so
m.Lock()
...
{
m.Unlock()
// cool down for one second
time.Sleep(1 * time.Second)
m.Lock()
...
}
...
m.Unlock()

measure execution time and stop waiting if too long in golang

I am trying to measure execution time of funcWithUnpredictiveExecutionTime function.
func measureTime(expectedMs float64) (ok bool) {
t1 := time.Now()
funcWithUnpredictiveExecutionTime()
t2 := time.Now()
diff := t2.Sub(t1)
The measuring is fine when funcWithUnpredictiveExecutionTime works faster than I expected. But if it works slower than expectedMs the measuring will not stop right after expected amount of milliseconds passed.
Is it possible to stop time measuring when funcWithUnpredictiveExecutionTime works longer than expectedMs without waiting funcWithUnpredictiveExecutionTime to finish?
In other words, measureTime(200) should return in 200 ms anyway with a good or bad result.
I guess I should use channels and then somehow cancel waiting for a channel. But how to do it exactly?
Full code:
package main
import (
"fmt"
"math/rand"
"time"
)
// random number between min and max
func random(min, max int) int {
rand.Seed(time.Now().Unix())
return rand.Intn(max-min) + min
}
// sleeps for a random milliseconds amount between 200 and 1000
func funcWithUnpredictiveExecutionTime() {
millisToSleep := random(200, 1000)
fmt.Println(fmt.Sprintf("Sleeping for %d milliseconds", millisToSleep))
time.Sleep(time.Millisecond * time.Duration(millisToSleep))
}
// measures execution time of a function funcWithUnpredictiveExecutionTime
// if expectedMs < actual execution time, it's ok.
// if expectedMs milliseconds passed and funcWithUnpredictiveExecutionTime
// still did not finish execution it should return
// without waiting for funcWithUnpredictiveExecutionTime
func measureTime(expectedMs float64) (ok bool) {
t1 := time.Now()
funcWithUnpredictiveExecutionTime()
t2 := time.Now()
diff := t2.Sub(t1)
actualMs := diff.Seconds() * 1000
ok = actualMs < expectedMs
fmt.Println(actualMs)
return
}
// prints results: Ok or too late
func printTimeResults(ok bool) {
if ok {
fmt.Println("Ok")
} else {
fmt.Println("Too late")
}
}
func main() {
printTimeResults(measureTime(200)) // expect it to finish in 200 ms anyway
printTimeResults(measureTime(1000)) // expect it to finish in 1000 ms anyway
}
Output:
Sleeping for 422 milliseconds
424.11895200000004
Too late
Sleeping for 422 milliseconds
425.27274900000003
Ok
Playground
You can't cancel a goroutine, unless you design it to be canceled. You can short circuit your timing function, by using a channel to signal the completion of the function being timed:
func measureTime(expectedMs float64) (ok bool) {
done := make(chan struct{})
t1 := time.Now()
go func() {
funcWithUnpredictiveExecutionTime()
close(done)
}()
select {
case <-done:
ok = true
case <-time.After(time.Duration(expectedMs) * time.Millisecond):
}
fmt.Println(time.Since(t1))
return ok
}
Extending JimB's example a little with the design I've personally followed for async background workers. I would say in most cases it's unacceptable to launch a go routine without passing an abort channel... All of your async methods should accept one as an argument or have one defined on their receiving type so you can actually control execution. fyi there are libraries for this, here's a simple one an old colleague of mine made; https://github.com/lytics/squaredance
If your program does not have an abort path for every goroutine you're probably going to face significant quality issues sooner or later. Also, for applications that are doing any heavy lifting in a goroutine, you will likely not be able to gracefully stop and start your application.
func measureTime(expectedMs float64) (ok bool) {
done := make(chan struct{})
abort := make(chan struct{})
t1 := time.Now()
go func() {
funcWithUnpredictiveExecutionTime(abort)
close(done)
}()
select {
case <-done:
ok = true
case <-time.After(time.Duration(expectedMs) * time.Millisecond):
// after signals here when the duration is reached so I close abort
close(abort)
}
fmt.Println(time.Since(t1))
return ok
}
funcWithUnpredictiveExecutionTime(abort) {
for {
select {
// doing work in here
case abort:
// except here, we clean up and return
}
}
}

Resources