I have two tasks that are running in go routines. I am using errgroup. I am not sure how to use the errgroup.WithContext correctly.
In the following code, task1 is returning the error and I would like to terminate task2 (long running) when that happens. Please note that in this example time.sleep is added just to simulate my problem. In reality task1 and task2 are doing real work and does not have any sleep call.
package main
import (
"context"
"fmt"
"golang.org/x/sync/errgroup"
"time"
)
func task1(ctx context.Context) error {
time.Sleep(5 * time.Second)
fmt.Println("first finished, pretend error happened")
return ctx.Err()
}
func task2(ctx context.Context) error {
select {
case <-ctx.Done():
fmt.Println("task 1 is finished with error")
return ctx.Err()
default:
fmt.Println("second started")
time.Sleep(50 * time.Second)
fmt.Println("second finished")
}
return nil
}
func test() (err error) {
ctx := context.Background()
g, gctx := errgroup.WithContext(ctx)
g.Go(func() error {
return task1(gctx)
})
g.Go(func() error {
return task2(gctx)
})
err = g.Wait()
if err != nil {
fmt.Println("wait done")
}
return err
}
func main() {
fmt.Println("main")
err := test()
if err != nil {
fmt.Println("main err")
fmt.Println(err.Error())
}
}
It's up to your tasks to handle context cancellation properly and not time.Sleep inside a select.
As stated in errgroup documentation:
WithContext returns a new Group and an associated Context derived from ctx.
The derived Context is canceled the first time a function passed to Go returns a non-nil error or the first time Wait returns, whichever occurs first.
You are using error group right, but your context handling needs a refactor.
Here is a refacor of your task 2:
func task2(ctx context.Context) error {
errCh := make(chan bool)
go func() {
time.Sleep(50 * time.Second)
errCh <- true
}()
select {
case <-ctx.Done():
return fmt.Errorf("context done: %w", ctx.Err())
case <-errCh:
return errors.New("task 2 failed")
}
}
With such select, you wait for the first channel to emit. In this case, it is the context expiration, unless you modify time sleep to be lower. Example playground.
Related
I am learning golang and am confused about errgroup package when used with context.
Here is my simple code:
package main
import (
"context"
"errors"
"fmt"
"time"
"golang.org/x/sync/errgroup"
)
func main() {
fmt.Println("..................")
ctx := context.Background()
group, ctx := errgroup.WithContext(ctx)
ctx, cancel := context.WithCancel(ctx)
group.Go(func() error {
//return errors.New("Error 1")
time.Sleep(8 * time.Second)
fmt.Println("Sleep 1 ended..................")
cancel()
return errors.New("Error 1")
})
group.Go(func() error {
//return errors.New("Error 1")
time.Sleep(2 * time.Second)
fmt.Println("Sleep 2 ended..................")
cancel()
return errors.New("Error 2")
})
err := group.Wait()
if err != nil {
fmt.Println("Error: ", err)
}
fmt.Println("..................")
}
The output, as expected is:
..................
Sleep 2 ended..................
Sleep 1 ended..................
Error: Error 2
..................
group.Wait() "blocks until all function calls from the Go method have returned, then returns the first non-nil error (if any) from them."
Questions:
What if I want to use errgroup but want to wait until the
context shared by all the Go methods is cancelled or all
function calls from the Go method have returned?
What if I want to use errgroup but want to wait until one of the Go method has returned error, which method will cancel the context and not wait for all to finish?
Somehow I feel that errgroup package is too restrictive in its use. What am I missing?
It seems like this cannot be achieved just by using errGroup itself.
Maybe waitGroup can be used here.
Maybe call cancel only if error happened. Or use error channel and wait till first error.
This implementation answers both of your questions:
package main
import (
"context"
"errors"
"fmt"
"time"
"golang.org/x/sync/errgroup"
)
func main() {
ctx := context.Background()
g, gtx := errgroup.WithContext(ctx)
g.Go(func() error {
select {
case <-time.After(8 * time.Second):
fmt.Println("Sleep 1 ended..................")
return errors.New("Error 1")
case <-gtx.Done():
return gtx.Err()
}
})
g.Go(func() error {
select {
case <-time.After(2 * time.Second):
fmt.Println("Sleep 2 ended..................")
return errors.New("Error 2")
case <-gtx.Done():
return gtx.Err()
}
})
err := g.Wait()
if err != nil {
fmt.Println("Error: ", err)
}
fmt.Println("..................")
}
It will print the following:
..................
Sleep 2 ended..................
Error: Error 2
..................
Returning an error cancels the context implicitly. You need to wait for ctx.Done() to abandon the execution of the go routines explicitly.
Using time.Sleep might be considered as an anti-pattern, as it doesn't acknowledge context cancelation. In many cases you want to use:
select {
case <-time.After(x):
case <-ctx.Done():
return ctx.Err()
}
In go I have two callbacks that eventually do not fire.
registerCb(func() {...})
registerCb(func() {...})
/* Wait for both func to execute with timeout */
I want to wait for both of them but having a timeout if one is not executed.
sync.WaitGroup does not work, since it is blocking and not channel based. Also you call WaitGroup.Done() without the risk of panic outside the callbacks.
My current solution is using just two booleans and a busy wait loop. But that's not satisfying.
Is there any idiomatic way that do not use polling or busy waiting?
Update:
Here is some code that demonstrates a busy wait solution but should return as soon as both callbacks are fired or after the timeout, without using polling
package main
import (
"fmt"
"log"
"sync"
"time"
)
var cbOne func()
var cbTwo func()
func registerCbOne(cb func()) {
cbOne = cb
}
func registerCbTwo(cb func()) {
cbTwo = cb
}
func executeCallbacks() {
<-time.After(1 * time.Second)
cbOne()
// Might never happen
//<-time.After(1 * time.Second)
//cbTwo()
}
func main() {
// Some process in background will execute our callbacks
go func() {
executeCallbacks()
}()
err := WaitAllOrTimeout(3 * time.Second)
if err != nil {
fmt.Println("Error: ", err.Error())
}
fmt.Println("Hello, playground")
}
func WaitAllOrTimeout(to time.Duration) error {
cbOneDoneCh := make(chan bool, 1)
cbTwoDoneCh := make(chan bool, 1)
cbOneDone := false
cbTwoDone := false
registerCbOne(func() {
fmt.Println("cb One");
cbOneDoneCh <- true
})
registerCbTwo(func() {
fmt.Println("cb Two");
cbTwoDoneCh <- true
})
// Wait for cbOne and cbTwo to be executed or a timeout
// Busywait solution
for {
select {
case <-time.After(to):
if cbOneDone && cbTwoDone {
fmt.Println("Both CB executed (we could poll more often)")
return nil
}
fmt.Println("Timeout!")
return fmt.Errorf("Timeout")
case <-cbOneDoneCh:
cbOneDone = true
case <-cbTwoDoneCh:
cbTwoDone = true
}
}
}
This is a followup to my comment, added after you added your example solution. To be clearer than I can in comments, your example code is actually not that bad. Here is your original example:
// Busywait solution
for {
select {
case <-time.After(to):
if cbOneDone && cbTwoDone {
fmt.Println("Both CB executed (we could poll more often)")
return nil
}
fmt.Println("Timeout!")
return fmt.Errorf("Timeout")
case <-cbOneDoneCh:
cbOneDone = true
case <-cbTwoDoneCh:
cbTwoDone = true
}
}
This isn't a "busy wait" but it does have several bugs (including the fact that you need an only-once send semantic for the done channels, or maybe easier and at least as good, to just close them once when done, perhaps using sync.Once). What we want to do is:
Start a timer with to as the timeout.
Enter a select loop, using the timer's channel and the two "done" channels.
We want to exit the select loop when the first of the following events occurs:
the timer fires, or
both "done" channels have been signaled.
If we're going to close the two done channels we'll want to have the Ch variables cleared (set to nil) as well so that the selects don't spin—that would turn this into a true busy-wait—but for the moment let's just assume instead that we send exactly once on them on callback, and otherwise just leak the channels, so that we can use your code as written as those selects will only ever return once. Here's the updated code:
t := timer.NewTimer(to)
for !cbOneDone || !cbTwoDone {
select {
case <-t.C:
fmt.Println("Timeout!")
return fmt.Errorf("timeout")
}
case <-cbOneDoneCh:
cbOneDone = true
case <-cbTwoDoneCh:
cbTwoDone = true
}
}
// insert t.Stop() and receive here to drain t.C if desired
fmt.Println("Both CB executed")
return nil
Note that we will go through the loop at most two times:
If we receive from both Done channels, once each, the loop stops without a timeout. There's no spinning/busy-waiting: we never received anything from t.C. We return nil (no error).
If we receive from one Done channel, the loop resumes but blocks waiting for the timer or the other Done channel.
If we ever receive from t.C, it means we didn't get both callbacks yet. We may have had one, but there's been a timeout and we choose to give up, which was our goal. We return an error, without going back through the loop.
A real version needs a bit more work to clean up properly and avoid leaking "done" channels (and the timer channel and its goroutine; see comment), but this is the general idea. You're already turning the callbacks into channel operations, and you already have a timer with its channel.
func wait(ctx context.Context, wg *sync.WaitGroup) error {
done := make(chan struct{}, 1)
go func() {
wg.Wait()
done <- struct{}{}
}()
select {
case <-done:
// Counter is 0, so all callbacks completed.
return nil
case <-ctx.Done():
// Context cancelled.
return ctx.Err()
}
}
Alternatively, you can pass a time.Duration and block on <-time.After(d) rather than on <-ctx.Done(), but I would argue that using context is more idiomatic.
below code present two variations,
the first is the regular pattern, nothing fancy, it does the job and does it well. You launch your callbacks into a routine, you make them push to a sink, listen that sink for a result or timeout. Take care to the sink channel initial capacity, to prevent leaking a routine it must match the number of callbacks.
the second factories out the synchronization mechanisms into small functions to assemble, two wait methods are provided, waitAll and waitOne. Nice to write, but definitely less efficient, more allocations, more back and forth with more channels, more complex to reason about, more subtle.
package main
import (
"fmt"
"log"
"sync"
"time"
)
func main() {
ExampleOne()
ExampleTwo()
ExampleThree()
fmt.Println("Hello, playground")
}
func ExampleOne() {
log.Println("start reg")
errs := make(chan error, 2)
go func() {
fn := callbackWithOpts("reg: so slow", 2*time.Second, nil)
errs <- fn()
}()
go func() {
fn := callbackWithOpts("reg: too fast", time.Millisecond, fmt.Errorf("broke!"))
errs <- fn()
}()
select {
case err := <-errs: // capture only one result,
// the fastest to finish.
if err != nil {
log.Println(err)
}
case <-time.After(time.Second): // or wait that many amount of time,
// in case they are all so slow.
}
log.Println("done reg")
}
func ExampleTwo() {
log.Println("start wait")
errs := waitAll(
withTimeout(time.Second,
callbackWithOpts("waitAll: so slow", 2*time.Second, nil),
),
withTimeout(time.Second,
callbackWithOpts("waitAll: too fast", time.Millisecond, nil),
),
)
for err := range trim(errs) {
if err != nil {
log.Println(err)
}
}
log.Println("done wait")
}
func ExampleThree() {
log.Println("start waitOne")
errs := waitOne(
withTimeout(time.Second,
callbackWithOpts("waitOne: so slow", 2*time.Second, nil),
),
withTimeout(time.Second,
callbackWithOpts("waitOne: too fast", time.Millisecond, nil),
),
)
for err := range trim(errs) {
if err != nil {
log.Println(err)
}
}
log.Println("done waitOne")
}
// a configurable callback for playing
func callbackWithOpts(msg string, tout time.Duration, err error) func() error {
return func() error {
<-time.After(tout)
fmt.Println(msg)
return err
}
}
// withTimeout return a function that returns first error or times out and return nil
func withTimeout(tout time.Duration, h func() error) func() error {
return func() error {
d := make(chan error, 1)
go func() {
d <- h()
}()
select {
case err := <-d:
return err
case <-time.After(tout):
}
return nil
}
}
// wait launches all func() and return their errors into the returned error channel; (merge)
// It is the caller responsability to drain the output error channel.
func waitAll(h ...func() error) chan error {
d := make(chan error, len(h))
var wg sync.WaitGroup
for i := 0; i < len(h); i++ {
wg.Add(1)
go func(h func() error) {
defer wg.Done()
d <- h()
}(h[i])
}
go func() {
wg.Wait()
close(d)
}()
return d
}
// wait launches all func() and return the first error into the returned error channel
// It is the caller responsability to drain the output error channel.
func waitOne(h ...func() error) chan error {
d := make(chan error, len(h))
one := make(chan error, 1)
var wg sync.WaitGroup
for i := 0; i < len(h); i++ {
wg.Add(1)
go func(h func() error) {
defer wg.Done()
d <- h()
}(h[i])
}
go func() {
for err := range d {
one <- err
close(one)
break
}
}()
go func() {
wg.Wait()
close(d)
}()
return one
}
func trim(err chan error) chan error {
out := make(chan error)
go func() {
for e := range err {
out <- e
}
close(out)
}()
return out
}
I have a goroutine inside a loop and the way I am handling the error is that I add it to a channel and after all the goroutines are finished, I check if there was an error and I return accordingly.
The issue with this is that I want to return an error as soon as I get it so that I don't spend time waiting for all the goroutines to finish as it would be inefficient.
I tried adding the select statement but it doesn't work and I can't add the select statement inside the goroutines since I want to exit the for loop and the try function too.
How can I do this?
Here is the code:
package main
import (
"sync"
"runtime"
"fmt"
"errors"
)
func try() (bool, error) {
wg := new(sync.WaitGroup)
s := []int{0,1,2,3,4,5}
ec := make(chan error)
for i, val := range s {
/*
select {
case err, ok := <-ec:
if ok {
println("error 1", err.Error())
return false, err
}
default:
}
*/
wg.Add(1)
i := i
val := val
go func() {
err := func(i int, val int, wg *sync.WaitGroup) error {
defer wg.Done()
if i == 3 {
return errors.New("one error")
} else {
return nil
}
}(i, val, wg)
if err != nil {
ec <- err
return
}
}()
}
wg.Wait()
select {
case err, ok := <-ec:
if ok {
println("error 2", err.Error())
return false, err
}
default:
}
return true, nil
}
func main() {
runtime.GOMAXPROCS(runtime.NumCPU())
b, e := try()
if e != nil {
fmt.Println(e.Error(), b)
}
}
This is the go playground link
With wg.Wait() before your select statement, you are effectively waiting for all goroutines to return.
The issue with this is that I want to return an error as soon as I get it
I assume that with this you mean stopping running goroutines as soon as any one of them returns an error.
In this case, you could use context.Context to manage cancellation, but even better is an errgroup.Group, which nicely combines context functionality and synchronization:
Package errgroup provides synchronization, error propagation, and Context cancelation for groups of goroutines working on subtasks of a common task.
In particular Group.Go:
The first call to return a non-nil error cancels the group; its error will be returned by Wait.
import (
"sync"
"runtime"
"fmt"
"errors"
"golang.org/x/sync/errgroup"
)
func try() (bool, error) {
errg := new(errgroup.Group)
s := []int{0,1,2,3,4,5}
for i, val := range s {
i := i
val := val
errg.Go(func() error {
return func(i int, val int) error {
if i == 3 {
return errors.New("one error")
} else {
return nil
}
}(i, val)
})
}
if err := errg.Wait(); err != nil {
// handle error
}
return true, nil
}
https://play.golang.org/p/lSIIFJqXf0W
I have found tomb to be useful for this. Below is a stripped-down non-working example that shows the gist, without handling things like variable encapsulation in the loop. It should give you the idea, but I'm happy to clarify on any points.
package main
import (
"fmt"
"gopkg.in/tomb.v2"
"sync"
)
func main() {
ts := tomb.Tomb{}
s := []int{0,1,2,3,4,5}
for i, v := range s {
ts.Go(func() error {
// do some work here or return an error, make sure to watch the dying chan, if it closes,
//then one of the other go-routines failed.
select {
case <- ts.Dying():
return nil
case err := <- waitingForWork():
if err != nil {
return err
}
return nil
}
})
}
// If an error appears here, one of the go-routines must have failed
err := ts.Wait()
if err != nil {
fmt.Println(err)
}
}
I want to use context.WithTimeout() to handle a use case that I make an external request, and if the response of the request is too long, it will return an error.
I have implemented the pseudo code like the playground link attached below:
2 solution:
main -> not expected
main_1 -> expected
package main
import (
"context"
"fmt"
"time"
)
// I just dummy sleep in this func to produce use case this func
// need 10s to process and handle logic.
// And this assume will be out of timeOut expect (5s)
func makeHTTPRequest(ctx context.Context) (string, error) {
time.Sleep(time.Duration(10) * time.Second)
return "abc", nil
}
// In main Func, I will set timeout is 5 second.
func main() {
var strCh = make(chan string, 1)
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(5)*time.Second)
defer cancel()
fmt.Print("Begin make request\n")
abc, err := makeHTTPRequest(ctx)
if err != nil {
fmt.Print("Return error\n")
return
}
select {
case <-ctx.Done():
fmt.Printf("Return ctx error: %s\n", ctx.Err())
return
case strCh <- abc:
fmt.Print("Return response\n")
return
}
}
func main_1() {
var strCh = make(chan string, 1)
var errCh = make(chan error, 1)
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(5)*time.Second)
defer cancel()
go func() {
fmt.Print("Begin make request\n")
abc, err := makeHTTPRequest(ctx)
if err != nil {
fmt.Print("Return error\n")
errCh <- err
return
}
strCh <- abc
}()
select {
case err := <-errCh:
fmt.Printf("Return error: %s\n", err.Error())
return
case <-ctx.Done():
fmt.Printf("Return ctx error: %s\n", ctx.Err())
return
case str := <-strCh:
fmt.Printf("Return response: %s\n", str)
return
}
}
However, if with the main() function then it doesn't work as expected.
But if with the second main_1() implementation using goroutine then maybe the new context.WithTimeout() works as expected.
Can you help me to answer this problem?
https://play.golang.org/p/kZdlm_Tvljy
It's better to handle context in your makeHTTPRequest() function, so you can use it as a synchronous function in main().
https://play.golang.org/p/Bhl4qprIBgH
func makeHTTPRequest(ctx context.Context) (string, error) {
ch := make(chan string)
go func() {
time.Sleep(10 * time.Second)
select {
case ch <- "abc":
default:
// When context deadline exceeded, there is no receiver
// This case will prevent goroutine blocking forever
return
}
}()
select {
case <-ctx.Done():
return "", ctx.Err()
case result := <-ch:
return result, nil
}
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
fmt.Printf("[%v] Begin make request \n", time.Now())
abc, err := makeHTTPRequest(ctx)
if err != nil {
fmt.Printf("[%v] Return error: %v \n", time.Now(), err)
return
}
fmt.Printf("[%v] %s", time.Now(), abc)
}
If I got you right. There are two questiones.
You want to know why main() function not work?
What's the best practice?
Q1
main() blocked at makeHTTPRequest, and during that time, context had timeout. So, not work as expected.
Q2
This example can answer you. In main_1() , your code is already best practice.
I have the following go-duktape code:
package main
import (
"fmt"
"gopkg.in/olebedev/go-duktape.v3"
"time"
)
func main() {
code := "function test(){log('testing');log('testing2');done();done();};"
resp := make(chan string)
ctx := duktape.New()
go doExec(code, ctx, resp)
select {
case r := <-resp:
fmt.Printf("%s\n", r)
case <-time.After(5 * time.Second):
fmt.Printf("Execution timeout\n")
}
kill(ctx)
close(resp)
}
func kill(ctx *duktape.Context) {
ctx.DestroyHeap()
ctx.Destroy()
}
func doExec(code string, ctx *duktape.Context, resp chan string) {
ctx.PushGlobalGoFunction("done", func(c *duktape.Context) int {
resp <- "We're done!!"
return 0
})
ctx.PushGlobalGoFunction("log", func(c *duktape.Context) int {
fmt.Println(c.SafeToString(-1))
return 0
})
err := ctx.PevalString(code + ";try{test()}catch(e){log('Error in execution');}")
if err != nil {
fmt.Printf("Error is %s\n", err.Error())
resp <- "Error in exec"
}
}
My goal is to terminate the program after the FIRST done() function call and not allow any more funtions to be executed after that. But if I run the following code, it panics because done() is called twice and the second call tries to write on a channel that's closed. How do I make sure that it terminates after the first done() call?
Thanks!