What is the best practice when using with context.WithTimeout() in Go? - go

I want to use context.WithTimeout() to handle a use case that I make an external request, and if the response of the request is too long, it will return an error.
I have implemented the pseudo code like the playground link attached below:
2 solution:
main -> not expected
main_1 -> expected
package main
import (
"context"
"fmt"
"time"
)
// I just dummy sleep in this func to produce use case this func
// need 10s to process and handle logic.
// And this assume will be out of timeOut expect (5s)
func makeHTTPRequest(ctx context.Context) (string, error) {
time.Sleep(time.Duration(10) * time.Second)
return "abc", nil
}
// In main Func, I will set timeout is 5 second.
func main() {
var strCh = make(chan string, 1)
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(5)*time.Second)
defer cancel()
fmt.Print("Begin make request\n")
abc, err := makeHTTPRequest(ctx)
if err != nil {
fmt.Print("Return error\n")
return
}
select {
case <-ctx.Done():
fmt.Printf("Return ctx error: %s\n", ctx.Err())
return
case strCh <- abc:
fmt.Print("Return response\n")
return
}
}
func main_1() {
var strCh = make(chan string, 1)
var errCh = make(chan error, 1)
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(5)*time.Second)
defer cancel()
go func() {
fmt.Print("Begin make request\n")
abc, err := makeHTTPRequest(ctx)
if err != nil {
fmt.Print("Return error\n")
errCh <- err
return
}
strCh <- abc
}()
select {
case err := <-errCh:
fmt.Printf("Return error: %s\n", err.Error())
return
case <-ctx.Done():
fmt.Printf("Return ctx error: %s\n", ctx.Err())
return
case str := <-strCh:
fmt.Printf("Return response: %s\n", str)
return
}
}
However, if with the main() function then it doesn't work as expected.
But if with the second main_1() implementation using goroutine then maybe the new context.WithTimeout() works as expected.
Can you help me to answer this problem?
https://play.golang.org/p/kZdlm_Tvljy

It's better to handle context in your makeHTTPRequest() function, so you can use it as a synchronous function in main().
https://play.golang.org/p/Bhl4qprIBgH
func makeHTTPRequest(ctx context.Context) (string, error) {
ch := make(chan string)
go func() {
time.Sleep(10 * time.Second)
select {
case ch <- "abc":
default:
// When context deadline exceeded, there is no receiver
// This case will prevent goroutine blocking forever
return
}
}()
select {
case <-ctx.Done():
return "", ctx.Err()
case result := <-ch:
return result, nil
}
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
fmt.Printf("[%v] Begin make request \n", time.Now())
abc, err := makeHTTPRequest(ctx)
if err != nil {
fmt.Printf("[%v] Return error: %v \n", time.Now(), err)
return
}
fmt.Printf("[%v] %s", time.Now(), abc)
}

If I got you right. There are two questiones.
You want to know why main() function not work?
What's the best practice?
Q1
main() blocked at makeHTTPRequest, and during that time, context had timeout. So, not work as expected.
Q2
This example can answer you. In main_1() , your code is already best practice.

Related

How to exit a time-out task in goroutine

I want run (v, ctx) to exit when it runs out of time,Here is the code I wrote to help me see if there is a problem or Whether there is a better job?
package main
import (
"context"
"log"
"math/rand"
"time"
)
func main() {
do := make(chan int)
rand.Seed(time.Now().Unix())
s := []int{100, 200, 300, 400, 500, 600}
ctx, cancel := context.WithTimeout(context.Background(), time.Second*3)
defer cancel()
work := func(ctx context.Context) {
for v := range do {
run(v, ctx)
}
}
for range [3]struct{}{} {
go work(ctx)
}
for _, v := range s {
do <- v
}
time.Sleep(time.Second * 20)
}
func run(v int, ctx context.Context) {
select {
case <-ctx.Done():
log.Print("timeout")
return
default:
//do something
log.Print(v)
time.Sleep(time.Duration(rand.Intn(5)+1) * time.Second)
}
}
I'm a beginner,I have run my code, but I'm not sure if it will be a problem? Can you give me some advice?
There are a couple techniques you can use.
First, I would recommend avoiding range over the channel since it doesn't support contexts. Use a loop around select instead:
work := func(ctx context.Context) {
for {
select {
case <-ctx.Done():
return
case v := <-do:
run(ctx, v) // ctx should conventionally be first parameter.
}
}
}
Within run and descendent functions you can detect cancellation via ctx.Done() as above, or call ctx.Err() and return:
if err := ctx.Err(); err != nil {
return // err -- you should capture and return errors from your worker function.
}
golang.org/x/sync/errgroup.Group can be used to manage starting goroutines, waiting for them to complete, and capturing the first (if any) error worker functions:
g, ctx := errgroup.WithContext(ctx)
for i := 0; i < 3; i++ {
g.Go(func() error {
return work(ctx)
})
}
if err := g.Wait(); err != nil {
fmt.Println("Failed:", err)
}

terminating blocking goroutines with errgroup

I have two tasks that are running in go routines. I am using errgroup. I am not sure how to use the errgroup.WithContext correctly.
In the following code, task1 is returning the error and I would like to terminate task2 (long running) when that happens. Please note that in this example time.sleep is added just to simulate my problem. In reality task1 and task2 are doing real work and does not have any sleep call.
package main
import (
"context"
"fmt"
"golang.org/x/sync/errgroup"
"time"
)
func task1(ctx context.Context) error {
time.Sleep(5 * time.Second)
fmt.Println("first finished, pretend error happened")
return ctx.Err()
}
func task2(ctx context.Context) error {
select {
case <-ctx.Done():
fmt.Println("task 1 is finished with error")
return ctx.Err()
default:
fmt.Println("second started")
time.Sleep(50 * time.Second)
fmt.Println("second finished")
}
return nil
}
func test() (err error) {
ctx := context.Background()
g, gctx := errgroup.WithContext(ctx)
g.Go(func() error {
return task1(gctx)
})
g.Go(func() error {
return task2(gctx)
})
err = g.Wait()
if err != nil {
fmt.Println("wait done")
}
return err
}
func main() {
fmt.Println("main")
err := test()
if err != nil {
fmt.Println("main err")
fmt.Println(err.Error())
}
}
It's up to your tasks to handle context cancellation properly and not time.Sleep inside a select.
As stated in errgroup documentation:
WithContext returns a new Group and an associated Context derived from ctx.
The derived Context is canceled the first time a function passed to Go returns a non-nil error or the first time Wait returns, whichever occurs first.
You are using error group right, but your context handling needs a refactor.
Here is a refacor of your task 2:
func task2(ctx context.Context) error {
errCh := make(chan bool)
go func() {
time.Sleep(50 * time.Second)
errCh <- true
}()
select {
case <-ctx.Done():
return fmt.Errorf("context done: %w", ctx.Err())
case <-errCh:
return errors.New("task 2 failed")
}
}
With such select, you wait for the first channel to emit. In this case, it is the context expiration, unless you modify time sleep to be lower. Example playground.

How to return the error from the gouroutine inside a loop early?

I have a goroutine inside a loop and the way I am handling the error is that I add it to a channel and after all the goroutines are finished, I check if there was an error and I return accordingly.
The issue with this is that I want to return an error as soon as I get it so that I don't spend time waiting for all the goroutines to finish as it would be inefficient.
I tried adding the select statement but it doesn't work and I can't add the select statement inside the goroutines since I want to exit the for loop and the try function too.
How can I do this?
Here is the code:
package main
import (
"sync"
"runtime"
"fmt"
"errors"
)
func try() (bool, error) {
wg := new(sync.WaitGroup)
s := []int{0,1,2,3,4,5}
ec := make(chan error)
for i, val := range s {
/*
select {
case err, ok := <-ec:
if ok {
println("error 1", err.Error())
return false, err
}
default:
}
*/
wg.Add(1)
i := i
val := val
go func() {
err := func(i int, val int, wg *sync.WaitGroup) error {
defer wg.Done()
if i == 3 {
return errors.New("one error")
} else {
return nil
}
}(i, val, wg)
if err != nil {
ec <- err
return
}
}()
}
wg.Wait()
select {
case err, ok := <-ec:
if ok {
println("error 2", err.Error())
return false, err
}
default:
}
return true, nil
}
func main() {
runtime.GOMAXPROCS(runtime.NumCPU())
b, e := try()
if e != nil {
fmt.Println(e.Error(), b)
}
}
This is the go playground link
With wg.Wait() before your select statement, you are effectively waiting for all goroutines to return.
The issue with this is that I want to return an error as soon as I get it
I assume that with this you mean stopping running goroutines as soon as any one of them returns an error.
In this case, you could use context.Context to manage cancellation, but even better is an errgroup.Group, which nicely combines context functionality and synchronization:
Package errgroup provides synchronization, error propagation, and Context cancelation for groups of goroutines working on subtasks of a common task.
In particular Group.Go:
The first call to return a non-nil error cancels the group; its error will be returned by Wait.
import (
"sync"
"runtime"
"fmt"
"errors"
"golang.org/x/sync/errgroup"
)
func try() (bool, error) {
errg := new(errgroup.Group)
s := []int{0,1,2,3,4,5}
for i, val := range s {
i := i
val := val
errg.Go(func() error {
return func(i int, val int) error {
if i == 3 {
return errors.New("one error")
} else {
return nil
}
}(i, val)
})
}
if err := errg.Wait(); err != nil {
// handle error
}
return true, nil
}
https://play.golang.org/p/lSIIFJqXf0W
I have found tomb to be useful for this. Below is a stripped-down non-working example that shows the gist, without handling things like variable encapsulation in the loop. It should give you the idea, but I'm happy to clarify on any points.
package main
import (
"fmt"
"gopkg.in/tomb.v2"
"sync"
)
func main() {
ts := tomb.Tomb{}
s := []int{0,1,2,3,4,5}
for i, v := range s {
ts.Go(func() error {
// do some work here or return an error, make sure to watch the dying chan, if it closes,
//then one of the other go-routines failed.
select {
case <- ts.Dying():
return nil
case err := <- waitingForWork():
if err != nil {
return err
}
return nil
}
})
}
// If an error appears here, one of the go-routines must have failed
err := ts.Wait()
if err != nil {
fmt.Println(err)
}
}

Best approach to getting results out of goroutines

I have two functions that I cannot change (see first() and second() below). They are returning some data and errors (the output data is different, but in the examples below I use (string, error) for simplicity)
I would like to run them in separate goroutines - my approach:
package main
import (
"fmt"
"os"
)
func first(name string) (string, error) {
if name == "" {
return "", fmt.Errorf("empty name is not allowed")
}
fmt.Println("processing first")
return fmt.Sprintf("First hello %s", name), nil
}
func second(name string) (string, error) {
if name == "" {
return "", fmt.Errorf("empty name is not allowed")
}
fmt.Println("processing second")
return fmt.Sprintf("Second hello %s", name), nil
}
func main() {
firstCh := make(chan string)
secondCh := make(chan string)
go func() {
defer close(firstCh)
res, err := first("one")
if err != nil {
fmt.Printf("Failed to run first: %v\n", err)
}
firstCh <- res
}()
go func() {
defer close(secondCh)
res, err := second("two")
if err != nil {
fmt.Printf("Failed to run second: %v\n", err)
}
secondCh <- res
}()
resultsOne := <-firstCh
resultsTwo := <-secondCh
// It's important for my app to do error checking and stop if errors exist.
if resultsOne == "" || resultsTwo == "" {
fmt.Println("There was an ERROR")
os.Exit(1)
}
fmt.Println("ONE:", resultsOne)
fmt.Println("TWO:", resultsTwo)
}
I believe one caveat is that resultsOne := <- firstCh blocks until first goroutine finishes, but I don't care too much about this.
Can you please confirm that my approach is good? What other approaches would be better in my situation?
The example looks mostly good. A couple improvements are:
declaring your channels as buffered
firstCh := make(chan string, 1)
secondCh := make(chan string, 1)
With unbuffered channels, send operations block (until someone receives). If your goroutine #2 is much faster than the first, it will have to wait until the first finishes as well, since you receive in sequence:
resultsOne := <-firstCh // waiting on this one first
resultsTwo := <-secondCh // sender blocked because the main thread hasn't reached this point
use "golang.org/x/sync/errgroup".Group. The program will feel "less native" but it dispenses you from managing channels by hand — which trades, in a non-contrived setting, for sync'ing writes on the results:
func main() {
var (
resultsOne string
resultsTwo string
)
g := errgroup.Group{}
g.Go(func() error {
res, err := first("one")
if err != nil {
return err
}
resultsOne = res
return nil
})
g.Go(func() error {
res, err := second("two")
if err != nil {
return err
}
resultsTwo = res
return nil
})
err := g.Wait()
// ... handle err

Go routine leak fix

I am working on a small service at the moment. From my testing, the code I've written has the possibility of leaking go routines under certain circumstances pertaining to the context. Is there a good and/or idiomatic way to remedy this? I'm providing some sample code below.
func Handle(ctx context.Context, r *Req) (*Response, error) {
ctx, cancel := context.WithTimeout(ctx, time.Second * 5)
defer cancel()
resChan := make(chan Response)
errChan := make(chan error)
go process(r, resChan, errChan)
select {
case ctx.Done():
return nil, ctx.Err()
case res := <-resChan:
return &res, nil
case err := <-errChan:
return nil, err
}
}
func process(r *Req, resChan chan<- Response, errChan chan<- error) {
defer close(errChan)
defer close(resChan)
err := doSomeWork()
if err != nil {
errChan <- err
return
}
err = doSomeMoreWork()
if err != nil {
errChan <- err
return
}
res := Response{}
resChan <- res
}
Hypothetically, if the client cancelled the context or the timeout occurred before the process func had a chance to send on one of the unbuffered channels (resChan, errChan), there would be no channel readers left from Handle and sending on the channels would block indefinitely with no readers. Since process would not return in this case, the channels would also not be closed.
I came up with the process2 as a solution, but I can't help thinking I'm doing something wrong, or there's a better way to handle this.
func process2(ctx context.Context, r *Req, resChan chan<- Response, errChan chan<- error) {
defer close(errChan)
defer close(resChan)
err := doSomeWork()
select {
case <-ctx.Done():
return
default:
if err != nil {
errChan <- err
return
}
}
err = doSomeMoreWork()
select {
case <-ctx.Done():
return
default:
if err != nil {
errChan <- err
return
}
}
res := Response{}
select{
case <-ctx.Done():
return
default:
resChan <- res
}
}
This approach makes sure that each time a channel send is attempted, first the context is checked for having been completed or cancelled. If it was, then it does not attempt the send and returns. I'm pretty sure this fixes any go routine leaking happening in the first process func.
Is there a better way? Maybe I have this all wrong.

Resources