I have some code, there are 3 timers,
GracefulExecutionTimeout - total running time
WaitTimeout - time to initial wait for first message
IdleTimeout - time to wait for subsequent messages
If any of the timers are reached the app should cleanly exit. I have it working below
msgs := make(chan string)
go func() {
time.Sleep(time.Second)
msgs <- "test"
}()
// graceful max execution time
gracefulMaxTimeout := time.Second * time.Duration(10)
gracefulMaxTimer := time.NewTimer(gracefulMaxTimeout)
// idleTimeout
idleTimeout := time.Second * time.Duration(5)
idleTimer := time.NewTimer(idleTimeout)
// waitTimeout
waitTimeout := time.Second * time.Duration(2)
waitTimer := time.NewTimer(waitTimeout)
for {
select {
case <-gracefulMaxTimer.C:
fmt.Println("GracefulMaxExecutionTimeout Reached")
// graceful exit
os.Exit(0)
case <-idleTimer.C:
fmt.Println("IdleTimeout Reached")
// graceful exit
os.Exit(0)
case <-waitTimer.C:
fmt.Println("WaitTimeout Reached")
// graceful exit
os.Exit(0)
case msg := <-msgs:
// stop wait timer
waitTimer.Stop()
fmt.Println(msg)
// Reset idle timer
if !idleTimer.Stop() {
<-idleTimer.C
}
fmt.Println("IdleIimeout Reset")
idleTimer.Reset(idleTimeout)
}
}
Go Playground
I want to make the WaitTimeout optional but not sure how to approach it. If i surround the construction of the waitTimer with an if statement then it wont work as the waitTimer isnt defined for the select statement ... How can i make the WaitTimeout conditional ?
I can just .Stop() the timer after its created but that seems a little dirty ...
You may declare the wait timer and its channel outside the if statement, and only initialize them if wait timer is needed. If not, the channel may remain its zero value–which is nil–because receiving from a nil channel blocks forever, so this case will never be ready (for details, see How does a non initialized channel behave?).
useWaitTimer := true
var (
waitTimer *time.Timer
waitTimerC <-chan time.Time
)
if useWaitTimer {
waitTimeout := time.Millisecond * time.Duration(500)
waitTimer = time.NewTimer(waitTimeout)
waitTimerC = waitTimer.C
}
// ...
for {
select {
// ...
case <-waitTimerC:
fmt.Println("WaitTimeout Reached")
// graceful exit
os.Exit(0)
// ...
}
}
Then of course you can only reset the wait timer if it exists, this must also be checked (and don't forget to drain the channel if it returns false):
// stop wait timer if exists
if waitTimer != nil && !waitTimer.Stop() {
<-waitTimerC
}
Try it on the Go Playground.
Related
func main() {
// Start a long-running process, capture stdout and stderr
findCmd := cmd.NewCmd("find", "/", "--name", "needle")
statusChan := findCmd.Start() // non-blocking
ticker := time.NewTicker(2 * time.Second)
// Print last line of stdout every 2s
go func() { ------------ this keeps running in background, how do I stop this goroutine
for range ticker.C {
status := findCmd.Status()
n := len(status.Stdout)
if len > 10 {
findCmd.Stop() ------- at this point I want to stop this goroutine
}
fmt.Println(status.Stdout[n-1])
}
}()
// Stop command after 1 hour
go func() {
<-time.After(1 * time.Hour)
findCmd.Stop()
}()
// Check if command is done
select {
case finalStatus := <-statusChan:
// done
default:
// no, still running
}
// Block waiting for command to exit, be stopped, or be killed
finalStatus := <-statusChan
}
Goroutine is used to print last line of stdout every 2s, tried few things to stop the goroutine but none worked.
How can I stop that goroutine using channels
This is where select, case is used. You can do something like this:
ticker := time.NewTicker(2 * time.Second)
done := make(chan struct{})
// The go routine that you want to stop when needed
go func() {
select {
case <-ticker.C:
// do stuff
case <-done:
ticker.Stop() // this is optional based on your purpose
}
}()
// do stuff
// stop above go routine
close(done) // when there are many go routine you want to stop as reading from closed channel return the channel zero value
// OR
done <- struct{}{} // when there is only 1 go routine you want to stop
One of the options is to pass the context to such a function/feature. Whenever you pass a context you can catch the cancel signal and use it in the select statements.
func main() {
appCtx := context.Background()
// ...
ticker := time.NewTicker(2 * time.Second)
tickerCtx, tickerCancel := context.WithCancel(appCtx) // create context for ticker, derive from root context
// Print last line of stdout every 2s
go func(ctx context.Context) {
for {
select {
case <-ticker.C: // tick!
// ...
// if you need to perform any operation requiring the context pass `ctx`.
// If case the ctx is canceled all operations should be terminated as well.
case <-ctx.Done() // context has been cancelled
return
}
}
}(tickerCtx)
// Stop command after 1 hour
go func() {
<-time.After(1 * time.Hour)
tickerCancel()
}()
// ...
}
So, this way we are using context, and we have control over all operations where a context is needed (we can cancel them at any time).
In your case there could be another improvement, to terminate the ctx-based operation after some time (without the need to use another goroutine with a timer):
func main() {
appCtx := context.Background()
// ...
ticker := time.NewTicker(2 * time.Second)
tickerCtx, tickerCancel := context.WithTimeout(appCtx, time.Duration(1) * time.Hour) // create a context with a timeout
// Print last line of stdout every 2s
go func(ctx context.Context) {
for {
select {
case <-ticker.C: // tick!
// ...
case <-ctx.Done() // context has been cancelled
return
}
}
}(tickerCtx)
// We don't need to run a new goroutine terminate the ctx. The timeout we set will do the job.
// ...
}
I am implementing a very simple concurrent program in Go. There are 2 channels todo and done that are used for signaling which task is done. There are 5 routines that are executed and each one require its own time to complete. I would like to see every 100ms the status of what is happening.
However I tried but the polling branch case <-time.After(100 * time.Millisecond): seems that is never been called. It is called sometimes (not in a consisted way) if I reduce the time to something less than 100ms.
My understanding is that go func executes the method in a separate Go scheduler thread. I do not understand therefore why the case of the polling is never hit. I tried to move the specific case branch before/after the other but nothing changed.
Any suggestions?
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
func concurrent(id int, done chan int, todo chan int) {
for {
// doing a task
t := randInt(50, 100)
time.Sleep(time.Duration(t) * time.Millisecond)
done <- id
// redo again this task
t = randInt(50, 100)
time.Sleep(time.Duration(t) * time.Millisecond)
todo <- id
}
}
func randInt(min int, max int) int {
return (min + rand.Intn(max-min))
}
func seedRandom() {
rand.Seed(time.Now().UTC().UnixNano())
}
func main() {
seedRandom()
todo := make(chan int, 5)
done := make(chan int, 5)
for i := 0; i < 5; i++ {
todo <- i
}
timeout := make(chan bool)
go func() {
time.Sleep(1 * time.Second)
timeout <- true
}()
var mu sync.Mutex
var output []int
loop:
for {
select {
case <-time.After(100 * time.Millisecond):
//this branch is never hit?
fmt.Printf("\nPolling status: %v\n", output)
case <-timeout:
fmt.Printf("\nDing ding, time is up!\n")
break loop
case id := <-done:
mu.Lock()
output = append(output, id)
fmt.Printf(".")
mu.Unlock()
case id := <-todo:
go concurrent(id, done, todo)
}
}
}
Update After following the answers I created this version in Go Playgound: https://play.golang.org/p/f08t984BdPt. That works as expected
you are creating 5 goroutines (func concurrent) and in your select case using the todo channel and this channel is being used in concurrent function so you end up creating a lot of goroutines
func concurrent(id int, done chan int, todo chan int) {
for {
// doing a task
t := randInt(50, 100)
time.Sleep(time.Duration(t) * time.Millisecond)
done <- id
// redo again this task
t = randInt(50, 100)
time.Sleep(time.Duration(t) * time.Millisecond)
by doing this call you are re-crating the go-routime
todo <- id
}
}
when I ran your code I got "runtime.NumGoroutine()"
"number of goRoutines still running 347"
as you are implementing the time.After(100 * time.Millisecond) inside the for loop it gets reset every time some other case gets hit and in your case
case id := <-todo: && id := <-done: will always get hit within 100 Milliseconds that's why you didn't get the expected output (from how your code is now i would say that the number of go-routines would increase exponentially and each of em would be waiting to send value to done and few on todo channel so your loop wont get enough time(100 ms) to wait on time.After)
loop:
for {
select {
case <-time.After(100 * time.Millisecond): ->this will always get reset ( we can use time.Ticker as it will create a single object that will signal for each and every 100ms https://golang.org/pkg/time/#NewTicker
//this branch is never hit?
fmt.Printf("\nPolling status: %v\n", output)
case <-timeout:
fmt.Printf("\nDing ding, time is up!\n")
break loop
case id := <-done: -> **this will get called**
//the mutex call is actually not very usefull as this only get called once per loop and is prefectly thread safe in this code
mu.Lock()
output = append(output, id)
fmt.Printf(".")
mu.Unlock()
case id := <-todo: -> **this will get called**
go concurrent(id, done, todo)
}
}
}
https://play.golang.org/p/SmlSIUIF5jn -> I have made some modifications to make your code work as expected..
try referring this to get a better understanding of golang channels and goroutine
https://tour.golang.org/concurrency/1
time.After(100*time.Millisecond) creates a brand new channel, with a brand new timer, which starts at the moment that function is called.
So, in your loop :
for {
select {
// this statement resets the 100ms timer each time you execute the loop :
case <-time.After(100*time.Millisecond):
...
Your branch never gets hit because, with 5 goroutines sending signals within less than 100ms on one of the other cases, this time.After(100ms) never reaches completion.
You need to choose a way to keep the same timer between iterations.
Here is one way to adapt your time.After(...) call :
// store the timer in a variable *outside* the loop :
statusTimer := time.After(100*time.Millisecond)
for {
select {
case <-statusTimer:
fmt.Printf("\nPolling status: %v\n", output)
// reset the timer :
statusTimer = time.After(100*time.Millisecond)
case <-timeout:
...
Another way is, as #blackgreen suggests, to use a time.Ticker :
statusTicker := time.NewTicker(100*time.Millisecond)
for {
select {
case <-statusTicker.C:
fmt.Printf("\nPolling status: %v\n", output)
case <-timeout:
...
side notes
a. if the output slice is not shared with other goroutines, you don't need a mutex around its access :
for {
select {
case <-statusTicker.C:
fmt.Printf("\nPolling status: %v\n", output)
...
case i <-done:
// no race condition here : all happens within the same goroutine,
// the 'select' statement makes sure that 'case's are executed
// one at a time
output = append(output, id)
fmt.Printf(".")
b. For your timeout channel :
Another generic way to "signal" that some event occurred with a channel is to close the channel instead of sending a value on it :
// if you don't actually care about the value you send over this channel :
// you can make it unbuffered, and use the empty 'struct{}' type
timeout := make(chan struct{})
go func(){
// wait for some condition ...
<-time.After(1*time.Second)
close(timeout)
}()
select {
case <-statusTimer:
...
case <-timeout: // this branch will also be taken once timeout is closed
fmt.Printf("\nDing ding, time is up!\n")
break loop
case ...
The bug you will avoid is the following : suppose you want to use that timeout channel in two goroutines
if you send a value over the timeout channel, only one goroutine will get signaled - it will "eat up" the value from the channel, and the other goroutine will only have a blocking channel,
if you close the channel, both goroutines will correctly "receive" the signal
In absence of a default case, when multiple cases are ready, it executes one of them at random. It's not deterministic.
To make sure the case runs, you should run it in a separate goroutine. (In that case, you must synchronize accesses to the output variable).
Moreover you say "I would like to see every 100ms", but time.After sends on the channel only once.
To execute the case periodically, use <-time.NewTicker(100 * time.Millis).C instead.
var mu sync.Mutex
var output []int
go func() {
ticker := time.NewTicker(100 * time.Millisecond)
defer ticker.Stop()
for {
select {
case <-ticker.C:
// TODO: must synchronize access
fmt.Printf("\nPolling status: %v\n", output)
case <-timeout:
return
}
}
}()
loop:
for {
select {
// other cases
}
}
Trying to get 2 Goroutines to play nicely within my project. Even though this is all sorta working, I am 100% sure it is being done very poorly... maybe even wrong...ly (is that even a word?).
Anyway, the basic concept of this project is to run some repeated work over a pre-selected amount of time while allowing the ability to abort the run before the time is up.
Here's the mess I have so far (I've only included the important parts for everyone's sanity... and to hide my horrible coding):
Two Goroutines:
BenchTimer() is a simple run-time countdown timer that does some repeating work for a set amount of time.
AbortTest() is a keyboard listener used to catch an 'ESC' (or whatever else I want) keypress from the keyboard to act as a "User Abort".
Each Goroutine, upon a successful run (i.e. BenchTimer() completes the countdown OR AbortTest() catches an abort keypress), sends a message down a common channel testAction. I use the one channel since this is an OR kinda thing (i.e. You can't get a completed countdown and an abort at the same time.). If BenchTimer() completes, then it sends "Complete" down the channel. If AbortTest() "completes" it sends "Abort" down the channel. [So far this all seems to be working...]
The next problem I ran into with this setup is how to kill the Goroutine that wasn't the "winner"... (i.e. If BenchTimer() completes normally, then I need to somehow kill AbortTest()... and vice-versa.) After a bunch of searching, I found that it isn't possible to kill a Goroutine externally, but it can be done internally... so I came up with using a second channel for each Goroutine to act as a sort of "kill signal" line: killAbortTest and killBenchTimer.
To tie this all together, I evaluate the result of the testAction channel. Because this channel will tell me which Goroutine "won", I can use this knowledge to send the correct (i.e. opposite) "kill signal" to have the "loser" Goroutine self-terminate.
Note: ... just means other code exists, but was removed due to not being needed for this post.
func main() {
...
testAction := make(chan string) // Action Result (Timer "Complete" or User "Abort")
killAbortTest := make(chan bool) // Kill AbortTest() Goroutine when BenchTimer() completes.
killBenchTimer := make(chan bool) // Kill BenchTimer() Goroutine when AbortTest() completes.
go BenchTimer(testAction, killBenchTimer) // Run BenchTimer() as Goroutine
go AbortTest(testAction, killAbortTest) // Run AbortTest() as Goroutine
// Program should wait here until it receives something on testAction channel.
actionVal := <-testAction
// Evaluate the testAction to kill the "loser" Goroutine
switch actionVal {
case "Abort":
killBenchTimer <- true // Abort received, signal BenchTimer() Goroutine to Quit
fmt.Println()
fmt.Println("Test Aborted")
case "Complete":
killAbortTest <- true // Countdown finished, signal AbortTest() Goroutine to Quit
fmt.Println()
fmt.Println("Test Completed")
}
...
}
// AbortTest - Listen for User Abort
func AbortTest(c chan<- string, k <-chan bool) {
if err := keyboard.Open(); err != nil {
panic(err)
}
defer func() {
_ = keyboard.Close()
}()
for {
select {
case <-k:
return
default:
_, key, err := keyboard.GetKey() // Poll for keypress
if err != nil {
panic(err)
}
if key == keyboard.KeyEsc { // ESC key was pressed
c <- "Abort"
return
}
}
}
}
// BenchTimer - Countdown Timer for BenchTest
func BenchTimer(c chan<- string, k <-chan bool) {
seconds := 0
switch testTime {
case "2-minute (fast)":
seconds = 120
case "5-minute (short)":
seconds = 300
case "10-minute (long)":
seconds = 600
case "20-minute (slow)":
seconds = 1200
}
ticker := time.Tick(time.Second)
for i := seconds; i >= 0; i-- {
select {
case <-k: // Kill Signal Received
return
default:
<-ticker
...
}
}
c <- "Complete"
}
There it is. My mess. There are many like it, but this one is my own. Like I said, it sorta works now, but I'm looking to make it better.
Am I just overthinking this whole process and making it way more complex than it needs to be?
Any help would be great.
This is the simplest example I can think of. I left out the keyboard part, but its basically the idea behind your code :
As mentioned by mh-cbon, this is safe :
package main
import (
"context"
"fmt"
"math/rand"
"sync"
"time"
)
var wg sync.WaitGroup
func main() {
rand.Seed(time.Now().UnixNano())
ctx, cancel := context.WithCancel(context.Background())
wg.Add(2)
go DoSomeTask(ctx, cancel)
go CancelTask(ctx, cancel)
wg.Wait()
}
func DoSomeTask(ctx context.Context, cancel func()) {
defer wg.Done()
defer cancel() // force cancellation
for i := 1; i < 10; i++ {
select {
case <-ctx.Done():
fmt.Println("context cancelled")
return
case <-time.After(time.Second):
}
fmt.Println("Done something!", i)
}
}
func CancelTask(ctx context.Context, cancel func()) {
defer wg.Done()
defer cancel() // force cancellation
duration := time.Second * time.Duration((rand.Intn(20-1) + 1))
fmt.Println("Will cancel in ", duration, " seconds!")
select {
case <-ctx.Done():
fmt.Println("context cancelled")
case <-time.After(duration):
}
}
I would like to pool for a token in on a timely base. The Token itself got also information about when it expires.
This should run forever until the user enters ctrl+c.
I tried the same with
span := timeLeft(*expDate)
timer := time.NewTimer(span).C
ticker := time.NewTicker(time.Second * 5).C
which also does not work (the application hangs after count down). So I decided to try it with <- time.After(...)
This is my code that does not work. You will see the count down but it never breaks on expiration.
This is is a small extract with the polling logic for simplicity sake in a main.go:
func refreshToken() (time.Time, error) {
//This should simulate a http request and returns the new target date for the next refresh
time.Sleep(2 * time.Second)
return time.Now().Add(10 * time.Second), nil
}
func timeLeft(d time.Time) time.Duration {
exactLeft := d.Sub(time.Now())
floorSeconds := math.Floor(exactLeft.Seconds())
return time.Duration(floorSeconds) * time.Second
}
func poller(expDate *time.Time) {
exp := timeLeft(*expDate)
done := make(chan bool)
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
for {
select {
// print time left on the screen
case <-time.After(3 * time.Second):
go func() {
fmt.Printf("\rNext Token refresh will be in: %v", timeLeft(*expDate))
}()
// mark as done when date is due
case <-time.After(exp):
fmt.Println("Refresh token now!")
done <- true
// exit app
case <-c:
os.Exit(0)
break
// exit function when done
case <-done:
break
}
}
}
func main() {
var expiration time.Time
expiration = time.Now().Add(10 * time.Second)
// loop and refresh token as long as the app does not exit
for {
poller(&expiration)
ex, err := refreshToken()
expiration = ex
if err != nil {
panic(err)
}
fmt.Println("next round poller")
}
}
I am also not sure if I need the done channel at all?
What is required to listen to two timers and call itself until someone hits ctrl+c?
Found a solution. While #ain was right wit the buffered done channel, it is not really required in the code now. It worked without it.
The trick did have the timeroutside of the for loop and the ticker within it. Reason is the time.Afteris a functhat return a new channel on every iteration. This seams perfectly fine for the ticker, but not for the timer.
With the following changes it worked =) ...
func poller(expDate *time.Time) {
exp := timeLeft(*expDate)
timer := time.After(exp)
fmt.Printf("Next Token refresh will be in: %v\n", exp)
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
for {
select {
// print time left on the screen
case <-time.After(3 * time.Second):
go func() {
fmt.Printf("\r ")
fmt.Printf("\r%v", timeLeft(*expDate))
}()
// mark as done when date is due
case <-timer:
fmt.Println("Refresh token now!")
return
// exit app
case <-c:
os.Exit(0)
break
}
}
}
This question already has answers here:
Timeout for WaitGroup.Wait()
(10 answers)
Closed 7 months ago.
I have come across a situation that i want to trace some goroutine to sync on a specific point, for example when all the urls are fetched. Then, we can put them all and show them in specific order.
I think this is the barrier comes in. It is in go with sync.WaitGroup. However, in real situation that we can not make sure that all the fetch operation will succeed in a short time. So, i want to introduce a timeout when wait for the fetch operations.
I am a newbie to Golang, so can someone give me some advice?
What i am looking for is like this:
wg := &sync.WaigGroup{}
select {
case <-wg.Wait():
// All done!
case <-time.After(500 * time.Millisecond):
// Hit timeout.
}
I know Wait do not support Channel.
If all you want is your neat select, you can easily convert blocking function to a channel by spawning a routine which calls a method and closes/sends on channel once done.
done := make(chan struct{})
go func() {
wg.Wait()
close(done)
}()
select {
case <-done:
// All done!
case <-time.After(500 * time.Millisecond):
// Hit timeout.
}
Send your results to a buffered channel enough to take all results, without blocking, and read them in for-select loop in the main thread:
func work(msg string, d time.Duration, ret chan<- string) {
time.Sleep(d) // Work emulation.
select {
case ret <- msg:
default:
}
}
// ...
const N = 2
ch := make(chan string, N)
go work("printed", 100*time.Millisecond, ch)
go work("not printed", 1000*time.Millisecond, ch)
timeout := time.After(500 * time.Millisecond)
loop:
for received := 0; received < N; received++ {
select {
case msg := <-ch:
fmt.Println(msg)
case <-timeout:
fmt.Println("timeout!")
break loop
}
}
Playground: http://play.golang.org/p/PxeEEJo2dz.
See also: Go Concurrency Patterns: Timing out, moving on.
Another way to do it would be to monitor it internally, your question is limited but I'm going to assume you're starting your goroutines through a loop even if you're not you can refactor this to work for you but you could do one of these 2 examples, the first one will timeout each request to timeout individually and the second one will timeout the entire batch of requests and move on if too much time has passed
var wg sync.WaitGroup
wg.Add(1)
go func() {
success := make(chan struct{}, 1)
go func() {
// send your request and wait for a response
// pretend response was received
time.Sleep(5 * time.Second)
success <- struct{}{}
// goroutine will close gracefully after return
fmt.Println("Returned Gracefully")
}()
select {
case <-success:
break
case <-time.After(1 * time.Second):
break
}
wg.Done()
// everything should be garbage collected and no longer take up space
}()
wg.Wait()
// do whatever with what you got
fmt.Println("Done")
time.Sleep(10 * time.Second)
fmt.Println("Checking to make sure nothing throws errors after limbo goroutine is done")
Or if you just want a general easy way to timeout ALL requests you could do something like
var wg sync.WaitGroup
waiter := make(chan int)
wg.Add(1)
go func() {
success := make(chan struct{}, 1)
go func() {
// send your request and wait for a response
// pretend response was received
time.Sleep(5 * time.Second)
success <- struct{}{}
// goroutine will close gracefully after return
fmt.Println("Returned Gracefully")
}()
select {
case <-success:
break
case <-time.After(1 * time.Second):
// control the timeouts for each request individually to make sure that wg.Done gets called and will let the goroutine holding the .Wait close
break
}
wg.Done()
// everything should be garbage collected and no longer take up space
}()
completed := false
go func(completed *bool) {
// Unblock with either wait
wg.Wait()
if !*completed {
waiter <- 1
*completed = true
}
fmt.Println("Returned Two")
}(&completed)
go func(completed *bool) {
// wait however long
time.Sleep(time.Second * 5)
if !*completed {
waiter <- 1
*completed = true
}
fmt.Println("Returned One")
}(&completed)
// block until it either times out or .Wait stops blocking
<-waiter
// do whatever with what you got
fmt.Println("Done")
time.Sleep(10 * time.Second)
fmt.Println("Checking to make sure nothing throws errors after limbo goroutine is done")
This way your WaitGroup will stay in sync and you won't have any goroutines left in limbo
http://play.golang.org/p/g0J_qJ1BUT try it here you can change the variables around to see it work differently
Edit: I'm on mobile If anybody could fix the formatting that would be great thanks.
If you would like to avoid mixing concurrency logic with business logic, I wrote this library https://github.com/shomali11/parallelizer to help you with that. It encapsulates the concurrency logic so you do not have to worry about it.
So in your example:
package main
import (
"github.com/shomali11/parallelizer"
"fmt"
)
func main() {
urls := []string{ ... }
results = make([]*HttpResponse, len(urls)
options := &Options{ Timeout: time.Second }
group := parallelizer.NewGroup(options)
for index, url := range urls {
group.Add(func(index int, url string, results *[]*HttpResponse) {
return func () {
...
results[index] = &HttpResponse{url, response, err}
}
}(index, url, &results))
}
err := group.Run()
fmt.Println("Done")
fmt.Println(fmt.Sprintf("Results: %v", results))
fmt.Printf("Error: %v", err) // nil if it completed, err if timed out
}