I am learning golang and am confused about errgroup package when used with context.
Here is my simple code:
package main
import (
"context"
"errors"
"fmt"
"time"
"golang.org/x/sync/errgroup"
)
func main() {
fmt.Println("..................")
ctx := context.Background()
group, ctx := errgroup.WithContext(ctx)
ctx, cancel := context.WithCancel(ctx)
group.Go(func() error {
//return errors.New("Error 1")
time.Sleep(8 * time.Second)
fmt.Println("Sleep 1 ended..................")
cancel()
return errors.New("Error 1")
})
group.Go(func() error {
//return errors.New("Error 1")
time.Sleep(2 * time.Second)
fmt.Println("Sleep 2 ended..................")
cancel()
return errors.New("Error 2")
})
err := group.Wait()
if err != nil {
fmt.Println("Error: ", err)
}
fmt.Println("..................")
}
The output, as expected is:
..................
Sleep 2 ended..................
Sleep 1 ended..................
Error: Error 2
..................
group.Wait() "blocks until all function calls from the Go method have returned, then returns the first non-nil error (if any) from them."
Questions:
What if I want to use errgroup but want to wait until the
context shared by all the Go methods is cancelled or all
function calls from the Go method have returned?
What if I want to use errgroup but want to wait until one of the Go method has returned error, which method will cancel the context and not wait for all to finish?
Somehow I feel that errgroup package is too restrictive in its use. What am I missing?
It seems like this cannot be achieved just by using errGroup itself.
Maybe waitGroup can be used here.
Maybe call cancel only if error happened. Or use error channel and wait till first error.
This implementation answers both of your questions:
package main
import (
"context"
"errors"
"fmt"
"time"
"golang.org/x/sync/errgroup"
)
func main() {
ctx := context.Background()
g, gtx := errgroup.WithContext(ctx)
g.Go(func() error {
select {
case <-time.After(8 * time.Second):
fmt.Println("Sleep 1 ended..................")
return errors.New("Error 1")
case <-gtx.Done():
return gtx.Err()
}
})
g.Go(func() error {
select {
case <-time.After(2 * time.Second):
fmt.Println("Sleep 2 ended..................")
return errors.New("Error 2")
case <-gtx.Done():
return gtx.Err()
}
})
err := g.Wait()
if err != nil {
fmt.Println("Error: ", err)
}
fmt.Println("..................")
}
It will print the following:
..................
Sleep 2 ended..................
Error: Error 2
..................
Returning an error cancels the context implicitly. You need to wait for ctx.Done() to abandon the execution of the go routines explicitly.
Using time.Sleep might be considered as an anti-pattern, as it doesn't acknowledge context cancelation. In many cases you want to use:
select {
case <-time.After(x):
case <-ctx.Done():
return ctx.Err()
}
Related
I have two tasks that are running in go routines. I am using errgroup. I am not sure how to use the errgroup.WithContext correctly.
In the following code, task1 is returning the error and I would like to terminate task2 (long running) when that happens. Please note that in this example time.sleep is added just to simulate my problem. In reality task1 and task2 are doing real work and does not have any sleep call.
package main
import (
"context"
"fmt"
"golang.org/x/sync/errgroup"
"time"
)
func task1(ctx context.Context) error {
time.Sleep(5 * time.Second)
fmt.Println("first finished, pretend error happened")
return ctx.Err()
}
func task2(ctx context.Context) error {
select {
case <-ctx.Done():
fmt.Println("task 1 is finished with error")
return ctx.Err()
default:
fmt.Println("second started")
time.Sleep(50 * time.Second)
fmt.Println("second finished")
}
return nil
}
func test() (err error) {
ctx := context.Background()
g, gctx := errgroup.WithContext(ctx)
g.Go(func() error {
return task1(gctx)
})
g.Go(func() error {
return task2(gctx)
})
err = g.Wait()
if err != nil {
fmt.Println("wait done")
}
return err
}
func main() {
fmt.Println("main")
err := test()
if err != nil {
fmt.Println("main err")
fmt.Println(err.Error())
}
}
It's up to your tasks to handle context cancellation properly and not time.Sleep inside a select.
As stated in errgroup documentation:
WithContext returns a new Group and an associated Context derived from ctx.
The derived Context is canceled the first time a function passed to Go returns a non-nil error or the first time Wait returns, whichever occurs first.
You are using error group right, but your context handling needs a refactor.
Here is a refacor of your task 2:
func task2(ctx context.Context) error {
errCh := make(chan bool)
go func() {
time.Sleep(50 * time.Second)
errCh <- true
}()
select {
case <-ctx.Done():
return fmt.Errorf("context done: %w", ctx.Err())
case <-errCh:
return errors.New("task 2 failed")
}
}
With such select, you wait for the first channel to emit. In this case, it is the context expiration, unless you modify time sleep to be lower. Example playground.
The best way I figured out how to "multi-thread" requests with a different proxy each time was to nest a go func and for loop inside of another for loop, but I can't figure out how to stop all loops like a break normally would, I tried a regular break also tried break out and added out: above the loops but that didn't stop it.
package main
import (
"log"
"encoding/json"
"github.com/parnurzeal/gorequest"
)
func main(){
rep := 100
for i := 0; i < rep; i++ {
log.Println("starting loop")
go func() {
for{
request := gorequest.New()
resp, body, errs := request.Get("https://discord.com/api/v9/invites/family").End()
if errs != nil {
return
}
if resp.StatusCode == 200{
var result map[string]interface{}
json.Unmarshal([]byte(body), &result)
serverName := result["guild"].(map[string]interface{})["name"]
log.Println(sererName +" response 200, closing all loops")
//break all loops and goroutine here
}
}
}
}
log.Println("response 200,closed all loops")
Answering this would be complicated by your use of parnurzeal/gorequest because that package does not provide any obvious way to cancel requests (see this issue). Because your focus appears to be on the process rather than the specific function I've just used the standard library (http) instead (if you do need to use gorequest then perhaps ask a question specifically about that).
Anyway the below solution demonstrates a few things:
Uses a Waitgroup so it knows when all go routines are done (not essential here but often you want to know you have shutdown cleanly)
Passes the result out via a channel (updating shared variables from a goroutine leads to data races).
Uses a context for cancellation. The cancel function is called when we have a result and this will stop in progress requests.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"net/http"
"sync"
)
func main() {
// Get the context and a function to cancel it
ctx, cancel := context.WithCancel(context.Background())
defer cancel() // Not really required here but its good practice to ensure context is cancelled eventually.
results := make(chan string)
const goRoutineCount = 100
var wg sync.WaitGroup
wg.Add(goRoutineCount) // we will be waiting on 100 goRoutines
for i := 0; i < goRoutineCount; i++ {
go func() {
defer wg.Done() // Decrement WaitGroup when goRoutine exits
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://discord.com/api/v9/invites/family", nil)
if err != nil {
panic(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
if errors.Is(err, context.Canceled) {
return // The error is due to the context being cancelled so just shutdown
}
panic(err)
}
defer resp.Body.Close() // Ensure body is closed
if resp.StatusCode == 200 {
var result map[string]interface{}
if err = json.NewDecoder(resp.Body).Decode(&result); err != nil {
panic(err)
}
serverName := result["guild"].(map[string]interface{})["name"]
results <- serverName.(string) // Should error check this...
cancel() // We have a result so all goroutines can stop now!
}
}()
}
// We need to process results until everything has shutdown; simple approach is to just close the channel when done
go func() {
wg.Wait()
close(results)
}()
var firstResult string
requestsProcessed := 0
for x := range results {
fmt.Println("got result")
if requestsProcessed == 0 {
firstResult = x
}
requestsProcessed++ // Possible that we will get more than one result (remember that requests are running in parallel)
}
// At this point all goroutines have shutdown
if requestsProcessed == 0 {
log.Println("No results received")
} else {
log.Printf("xx%s response 200, closing all loops (requests processed: %d)", firstResult, requestsProcessed)
}
}
I am trying to read a constant stream of data, if the call to receive stream takes longer than 30 seconds I need to timeout and exit the program. I am not sure how to exit the go routine once a timeout has been received.
func ReceiveStreamMessages(strm Stream, msg chan<- []byte) error {
d := make(chan []byte, 1)
e := make(chan error)
tm := time.After(30 * time.Second)
go func() {
for {
//blocking call
data, err := strm.Recv()
if err != nil {
e <- err
return
}
select {
case d <- data.Result:
case <-tm:
//exit out go routine
return
}
}
}()
for {
select {
case message := <-d:
msg <- message
case err := <-e:
return err
case <-tm:
return nil
}
}
}
My code above is wrong as: in order for the select to run in the go routines for loop, the blocking function will have to return and data will be populated and therefore won't hit the timeout select case (or will do randomly as both will be ready). Is exiting the parent function enough to exit the go routine?
Use context package WithTimeout. Something like this:
package main
import (
"context"
"fmt"
"sync"
"time"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
// prepare
...
// wait group just for test
var wg sync.WaitGroup
wg.Add(1)
go func() {
for {
select {
case d <- data.Result:
// do something
case <-ctx.Done():
fmt.Println("Done")
wg.Done()
return
}
}
}()
wg.Wait()
cancel()
fmt.Println("Hello, playground")
}
You can see a working example here https://play.golang.org/p/agi1fimtEkJ
When I want ctx timeout, what should I do to completely terminate the method that is executing longRunningCalculation()?
package main
import (
"context"
"log"
"time"
)
func longRunningCalculation(timeCost int) chan string {
result := make(chan string)
go func() {
time.Sleep(time.Second * (time.Duration(timeCost)))
log.Println("Still doing other things...") //Even if it times out, this goroutine is still doing other tasks.
result <- "Done"
log.Println(timeCost)
}()
return result
}
func jobWithTimeout() {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
select {
case <-ctx.Done():
log.Println(ctx.Err())
return
case result := <-longRunningCalculation(3):
log.Println(result)
}
}
func main() {
jobWithTimeout()
time.Sleep(time.Second * 5)
}
What did you expect to see?
2019/09/25 11:00:16 context deadline exceeded
What did you see instead?
2019/09/25 11:00:16 context deadline exceeded
2019/09/25 11:00:17 Still doing other things...
To stop the goroutine started by longRunningCalculation when the caller's context times out, you need to pass ctx into longRunningCalculation and explicitly handle the context timing out, the same way you do in jobWithTimeout
Doing things that way also means instead of calling time.Sleep, that time.Tick will be a better choice, so both timers are running at the same time. Like so:
package main
import (
"context"
"log"
"time"
)
func longRunningCalculation(ctx context.Context, timeCost int) chan string {
result := make(chan string)
go func() {
calcDone := time.Tick(time.Second * time.Duration(timeCost))
log.Printf("entering select (longRunningCalculation)")
select {
case <-ctx.Done():
result <- "Caller timed out"
return
case <-calcDone:
log.Println("Still doing other things...") //Even if it times out, this goroutine is still doing other tasks.
result <- "Done"
}
log.Println(timeCost)
}()
return result
}
func jobWithTimeout() {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
result := longRunningCalculation(ctx, 3)
log.Printf("entering select (jobWithTimeout)")
select {
case <-ctx.Done():
log.Println(ctx.Err())
return
case res := <-result:
log.Println(res)
}
}
func main() {
jobWithTimeout()
}
In Go, I want to time.Sleep for some time (e.g. waiting between retries), but want to return quickly if the context gets canceled (not just from a deadline, but also manually).
What is the right or best way to do that? Thanks!
You can use select to acheive this:
package main
import (
"fmt"
"time"
"context"
)
func main() {
fmt.Println("Hello, playground")
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go func(){
t := time.Now()
select{
case <-ctx.Done(): //context cancelled
case <-time.After(2 * time.Second): //timeout
}
fmt.Printf("here after: %v\n", time.Since(t))
}()
cancel() //cancel manually, comment out to see timeout kick in
time.Sleep(3 * time.Second)
fmt.Println("done")
}
Here is the Go-playground link
Here is a sleepContext function that you can use instead of time.Sleep:
func sleepContext(ctx context.Context, delay time.Duration) {
select {
case <-ctx.Done():
case <-time.After(delay):
}
}
And some sample usage (full runnable code on the Go Playground):
func main() {
ctx := context.Background()
fmt.Println(time.Now())
sleepContext(ctx, 1*time.Second)
fmt.Println(time.Now())
ctxTimeout, cancel := context.WithTimeout(ctx, 500*time.Millisecond)
sleepContext(ctxTimeout, 1*time.Second)
cancel()
fmt.Println(time.Now())
}
You can use select as others have mentioned; however, the other answers have a bug since timer.After() will leak memory if not cleaned up.
func SleepWithContext(ctx context.Context, d time.Duration) {
timer := time.NewTimer(d)
select {
case <-ctx.Done():
if !timer.Stop() {
<-timer.C
}
case <-timer.C:
}
}
The time.After() function has this problem:
The underlying Timer is not recovered by the garbage collector until
the timer fires. If efficiency is a concern, use NewTimer instead and
call Timer.Stop if the timer is no longer needed.
It is better to use a Timer object and call Stop():
// Delay returns nil after the specified duration or error if interrupted.
func Delay(ctx context.Context, d time.Duration) error {
t := time.NewTimer(d)
select {
case <-ctx.Done():
t.Stop()
return fmt.Errorf("Interrupted")
case <-t.C:
}
return nil
}
I managed to do something similar by combining a CancelContext with a TimeoutContext...
Here is the sample code:
cancelCtx, cancel := context.WithCancel(context.Background())
defer cancel()
// The program "sleeps" for 5 seconds.
timeoutCtx, _ := context.WithTimeout(cancelCtx, 5*time.Second)
select {
case <-timeoutCtx.Done():
if cancelCtx.Err() != nil {
log.Printf("Context cancelled")
}
}
In this repo you can find the complete usage of the above code. Sorry for my short answer, I didn't turned on the computer yet, and is not that easy to answer from the phone...