Context timeout not working as expected in golang - go

Helo All,
New to golang and was debugging timeout issues in a production environment. Before making a call to the server we add a timeout of 50ms to context and fire a server call. If the response is not received within 50 ms we expect the application to move on and not wait for the response.
But while debugging, I capture the duration between we fire a server call and the response received (or error out), to my surprise the value at the time is much higher than 50 ms.
Client syntax -
ctx, cancel := context.WithTimeout(ctx, e.opts.Timeout)
defer cancel()
fireServerCall(ctx)
..
..
def fireServerCall(ctx context:Context){
startTime:=time.Now()
//call to the server
res, err:=callToServer(ctx)
if err!=nil{
//capture failure latency
return ....
}
//capture success latency
return ....
}
Has anyone ever faced any similar issue? Is this expected behaviour? How did you handle such cases?
Am I doing something incorrectly? Suggestions are welcome :)
Edit:
I am passing context in my original code but forgot to mention it here, just added it. That mean, I am passing the same context on which my client is waiting for server to respond within 50 ms.

You should pass created context to fireServerCall and callToServer functions
callToServer should consider passed context and monitor ctx.Done() channel to stop its execution accordingly
Answering to comment by #Bishnu:
Don't think this is needed. Did a test and even without passing ctx to callToServer() it works. The behaviour is not as expected under high load. Can you kindly share some document/test what you have pointed here?
Context timeout just can't work without context passing and checking its Done() channel. Context is not some kind of magic — simplifying it is just a struct with done channel which is closed by calling cancel function or when timeout occurs. Monitoring this channel — is responsibility of the innermost function that accepts it.
Example:
package main
import (
"context"
"fmt"
"time"
)
func callToServer(ctx context.Context) {
now := time.Now()
select {
case <-ctx.Done(): // context cancelled via cancel or deadline
case <-time.After(1 * time.Second): // emulate external call
}
fmt.Printf("callToServer: %v\n", time.Since(now))
}
func callToServerContextAgnostic(ctx context.Context) {
now := time.Now()
select {
case <-time.After(2 * time.Second): // emulate external call
}
fmt.Printf("callToServerContextAgnostic: %v\n", time.Since(now))
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
callToServer(ctx)
ctx2, cancel2 := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel2()
callToServerContextAgnostic(ctx2)
}
Results:
callToServer: 100ms
callToServerContextAgnostic: 2s
You can launch it on Go Playground: https://go.dev/play/p/tIxjHxUzYfh
Note that many of the clients (from standard or third party libraries) monitors Done channel by themselves.
For example standard HTTP client:
c := &http.Client{} // client for all requests
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(time.Millisecond*100))
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://google.com", nil)
if err != nil {
log.Fatal(err)
}
resp, err := c.Do(req) // will monitor `Done` channel for you
Some docs and articles:
https://pkg.go.dev/context
https://www.digitalocean.com/community/tutorials/how-to-use-contexts-in-go

Related

Context.WithTimeout() and os.exit in gorilla/mux

I'm using the Golang gorilla/mux package, and one of the examples is as follows:
func main() {
var wait time.Duration
flag.DurationVar(&wait, "graceful-timeout", time.Second * 15, "the duration for which the server gracefully wait for existing connections to finish - e.g. 15s or 1m")
flag.Parse()
r := mux.NewRouter()
// Add your routes as needed
srv := &http.Server{
Addr: "0.0.0.0:8080",
// Good practice to set timeouts to avoid Slowloris attacks.
WriteTimeout: time.Second * 15,
ReadTimeout: time.Second * 15,
IdleTimeout: time.Second * 60,
Handler: r, // Pass our instance of gorilla/mux in.
}
// Run our server in a goroutine so that it doesn't block.
go func() {
if err := srv.ListenAndServe(); err != nil {
log.Println(err)
}
}()
c := make(chan os.Signal, 1)
// We'll accept graceful shutdowns when quit via SIGINT (Ctrl+C)
// SIGKILL, SIGQUIT or SIGTERM (Ctrl+/) will not be caught.
signal.Notify(c, os.Interrupt)
// Block until we receive our signal.
<-c
// Create a deadline to wait for.
ctx, cancel := context.WithTimeout(context.Background(), wait)
defer cancel()
// Doesn't block if no connections, but will otherwise wait
// until the timeout deadline.
srv.Shutdown(ctx)
// Optionally, you could run srv.Shutdown in a goroutine and block on
// <-ctx.Done() if your application should wait for other services
// to finalize based on context cancellation.
log.Println("shutting down")
os.Exit(0)
}
This seems simple enough, but my understanding is that defers do not run when os.Exit() is called (as per https://gobyexample.com/exit). I notice that there's a CancelFunc() returned by context.WithTimeout(), which is then deferred. My guess is that this is supposed to cancel the context created above if main() finishes before the deadline, but I don't see how that can happen with a call to os.Exit() at the end. Could anyone help me see what I'm missing?
You right, the deferred cancel function is never called. The author want probably point out that in real application, you should never forget to cancel your context.

How to include goroutine into a context?

I'm working on a Go project that require calling an initiation function (initFunction) in a separated goroutine (to ensure this function does not interfere with the rest of the project). initFunction must not take more than 30 seconds, so I thought I would use context.WithTimeout. Lastly, initFunction must be able to notify errors to the caller, so I thought of making an error channel and calling initFunction from an anonymous function, to recieve and report the error.
func RunInitGoRoutine(initFunction func(config string)error) error {
initErr := make(chan error)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Seconds)
go func() {
<-ctx.Done() // Line 7
err := initFunction(config)
initErr <-err
}()
select {
case res := <-initErr:
return res.err
case <-ctx.Done():
err := errors.New("Deadline")
return err
}
}
I'm quite new to Go, so I'm asking for feedbacks about the above code.
I have some doubt about Line 7. I used this to ensure the anonymous function is "included" under ctx and is therefore killed and freed and everything once timeout expires, but I'm not sure I have done the right thing.
Second thing is, I know I should be calling cancel( ) somewhere, but I can't put my finger around where.
Lastly, really any feedback is welcome, being it about efficency, style, correctness or anything.
In Go the pratice is to communicate via channels. So best thing is probably to share a channel on your context so others can consume from the channel.
As you are stating you are new to Go, I wrote a whole bunch of articles on Go (Beginner level) https://marcofranssen.nl/categories/golang.
Read from old to new to get familiar with the language.
Regarding the channel specifics you should have a look at this article.
https://marcofranssen.nl/concurrency-in-go
A pratical example of a webserver listening for ctrl+c and then gracefully shutting down the server using channels is described in this blog post.
https://marcofranssen.nl/improved-graceful-shutdown-webserver
In essence we run the server in a background routine
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
srv.l.Fatal("Could not listen on", zap.String("addr", srv.Addr), zap.Error(err))
}
}()
and then we have some code that is blocking the main routine by listening on a channel for the shutdown signal.
quit := make(chan os.Signal, 1)
signal.Notify(quit, os.Interrupt)
sig := <-quit
srv.l.Info("Server is shutting down", zap.String("reason", sig.String()))
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
srv.SetKeepAlivesEnabled(false)
if err := srv.Shutdown(ctx); err != nil {
srv.l.Fatal("Could not gracefully shutdown the server", zap.Error(err))
}
srv.l.Info("Server stopped")
This is very similar to your usecase. So running your init in a background routine and then consume the channel waiting for the result of this init routine.
package main
import (
"fmt"
"time"
)
type InitResult struct {
Message string
}
func main() {
initResult := make(chan InitResult, 0)
go func(c chan<- InitResult) {
time.Sleep(5 * time.Second)
// here we are publishing the result on the channel
c <- InitResult{Message: "Initialization succeeded"}
}(initResult)
fmt.Println("Started initializing")
// here we have a blocking operation consuming the channel
res := <-initResult
fmt.Printf("Init result: %s", res.Message)
}
https://play.golang.org/p/_YGIrdNVZx6
You could also add an error field on the struct so you could do you usual way of error checking.

Reusing parent context with context.WithTimeout in go?

Reusing parent context with context.WithTimeout with a new timeout
Hi there, I'm new to go. I was wondering if it's possible to reuse a parent context to create multiple context.withTimeout(). The rationale would be where I have to call multiple network requests in sequence and would like to set a timeout for each request at the same time using the parent's context.
Rationale
When the parent's context is cancelled, all the requests made would be cancelled too.
Problem
In the code below, it shows an example whereby LongProcess is the network request. However, the context is closed before the second LongProcess call can be made with a context deadline exceeded.
The documentation withDeadline states The returned context's Done channel is closed when the deadline expires, when the returned cancel function is called, or when the parent context's Done channel isclosed, whichever happens first.
So if that's the case, is there a way where I can reset the timer for withTimeout? Or do I have to create a new context context.Background() for every request? That would mean the parent context will not be passed. :(
// LongProcess refers to a long network request
func LongProcess(ctx context.Context, duration time.Duration, msg string) error {
c1 := make(chan string, 1)
go func() {
// Simulate processing
time.Sleep(duration)
c1 <- msg
}()
select {
case m := <-c1:
fmt.Println(m)
return nil
case <-ctx.Done():
return ctx.Err()
}
}
func main() {
ctx := context.Background()
t := 2 * time.Second
ctx, cancel := context.WithTimeout(ctx, t)
defer cancel()
// Simulate a 2 second process time
err := LongProcess(ctx, 2*time.Second, "first process")
fmt.Println(err)
// Reusing the context.
s, cancel := context.WithTimeout(ctx, t)
defer cancel()
// Simulate a 1 second process time
err = LongProcess(s, 1*time.Second, "second process")
fmt.Println(err) // context deadline exceeded
}
It looks like the first call to context.WithTimeout shadow the parent context ctx. The later process re-use this already canceled context hence the error. You have to re-use the parent one. Here is the example updated:
func main() {
// Avoid to shadow child contexts
parent := context.Background()
t := 2 * time.Second
// Use the parent context.
ctx1, cancel := context.WithTimeout(parent, t)
defer cancel()
err := LongProcess(ctx1, 2*time.Second, "first process")
fmt.Println(err)
// Use the parent context not the canceled one.
ctx2, cancel := context.WithTimeout(parent, t)
defer cancel()
err = LongProcess(ctx2, 1*time.Second, "second process")
fmt.Println(err)
}

Context confusion regarding cancellation

package main
import (
"context"
"fmt"
"sync"
"time"
)
func myfunc(ctx context.Context) {
for {
select {
case <-ctx.Done():
fmt.Printf("Ctx is kicking in with error:%+v\n", ctx.Err())
return
default:
time.Sleep(15 * time.Second)
fmt.Printf("I was not canceled\n")
return
}
}
}
func main() {
ctx, cancel := context.WithTimeout(
context.Background(),
time.Duration(3*time.Second))
defer cancel()
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
myfunc(ctx)
}()
wg.Wait()
fmt.Printf("In main, ctx err is %+v\n", ctx.Err())
}
I have the above snippet that does print the output like this
I was not canceled
In main, ctx err is context deadline exceeded
Process finished with exit code 0
I understand that context times-out after 3 seconds and hence it does give me the expected error when I call ctx.Err() in the end. I also get the fact that in my myfunc once select matches on the case for default, it won't match on the done. What I do not understand is that how do I make my go func myfunc get aborted in 3 seconds using the context logic. Basically, it won't terminate in 3 seconds so I am trying to understand how can golang's ctx help me with this?
If you want to use the timeout and cancellation feature from the context, then in your case the ctx.Done() need to be handled synchronously.
Explanation from https://golang.org/pkg/context/#Context
Done returns a channel that's closed when work is done on behalf of this context should be canceled. Done may return nil if this context can never be canceled. Successive calls to Done return the same value.
So basically the <-ctx.Done() will be called on two conditions:
when context timeout exceeds
when context canceled by force
And when that happens, the ctx.Err() will never be nil.
We can perform some checking on the error object to see whether the context is canceled by force or exceeding the timeout.
Context package provides two error objects, context.DeadlineExceeded and context.Timeout, this two will help us to identify why <-ctx.Done() is called.
Example #1 scenario: context cancelled by force (via cancel())
In the test, we'll try to make the context to be canceled before the timeout exceeds, so the <-ctx.Done() will be executed.
ctx, cancel := context.WithTimeout(
context.Background(),
time.Duration(3*time.Second))
go func(ctx context.Context) {
// simulate a process that takes 2 second to complete
time.Sleep(2 * time.Second)
// cancel context by force, assuming the whole process is complete
cancel()
}(ctx)
select {
case <-ctx.Done():
switch ctx.Err() {
case context.DeadlineExceeded:
fmt.Println("context timeout exceeded")
case context.Canceled:
fmt.Println("context cancelled by force. whole process is complete")
}
}
Output:
$ go run test.go
context cancelled by force
Example #2 scenario: context timeout exceeded
In this scenario, we make the process takes longer than context timeout, so ideally the <-ctx.Done() will also be executed.
ctx, cancel := context.WithTimeout(
context.Background(),
time.Duration(3*time.Second))
go func(ctx context.Context) {
// simulate a process that takes 4 second to complete
time.Sleep(4 * time.Second)
// cancel context by force, assuming the whole process is complete
cancel()
}(ctx)
select {
case <-ctx.Done():
switch ctx.Err() {
case context.DeadlineExceeded:
fmt.Println("context timeout exceeded")
case context.Canceled:
fmt.Println("context cancelled by force. whole process is complete")
}
}
Output:
$ go run test.go
context timeout exceeded
Example #3 scenario: context canceled by force due to error occurred
There might be a situation where we need to stop the goroutine in the middle of the process because error occurred. And sometimes, we might need to retrieve that error object on the main routine.
To achieve that, we need an additional channel to transport the error object from goroutine into main routine.
In the below example, I've prepared a channel called chErr. Whenever error happens in the middle of (goroutine) process, then we will send that error object through the channel and then stop process immediately from.
ctx, cancel := context.WithTimeout(
context.Background(),
time.Duration(3*time.Second))
chErr := make(chan error)
go func(ctx context.Context) {
// ... some process ...
if err != nil {
// cancel context by force, an error occurred
chErr <- err
return
}
// ... some other process ...
// cancel context by force, assuming the whole process is complete
cancel()
}(ctx)
select {
case <-ctx.Done():
switch ctx.Err() {
case context.DeadlineExceeded:
fmt.Println("context timeout exceeded")
case context.Canceled:
fmt.Println("context cancelled by force. whole process is complete")
}
case err := <-chErr:
fmt.Println("process fail causing by some error:", err.Error())
}
Additional info #1: calling cancel() right after context initialized
As per context documentation regarding the cancel() function:
Canceling this context releases resources associated with it, so code should call cancel as soon as the operations running in this Context complete.
It's good to always call cancel() function right after the context declaration. doesn't matter whether it's also called within the goroutine. This is due to ensure context is always cancelled when the whole process within the block are fully complete.
ctx, cancel := context.WithTimeout(
context.Background(),
time.Duration(3*time.Second))
defer cancel()
// ...
Additional info #2: defer cancel() call within goroutine
You can use defer on the cancel() statement within the goroutine (if you want).
// ...
go func(ctx context.Context) {
defer cancel()
// ...
}(ctx)
// ...
In your for ... select, you have 2 cases: case <-ctx.Done(): and default:. When your code reaches the select, it enters the default case because the context is not yet cancelled, where it sleeps for 15 seconds and then returns, breaking your loop. (in other words, it isn't blocking/waiting for your context to cancel)
If you want your code to do what you are describing, you need your select to have cases for the context being cancelled and your imposed timeout.
select {
case <-ctx.Done(): // context was cancelled
fmt.Printf("Ctx is kicking in with error:%+v\n", ctx.Err())
return
case <-time.After(15 * time.Second): // 15 seconds have elapsed
fmt.Printf("I was not canceled\n")
return
}
Now, your code will block when it hits select, rather than entering the default case and breaking your loop.

Context based termination of long running loop in a go routine

I'm implementing a feature where I need to read files from a directory, parse and export them to a REST service at a regular interval. As part of the same I would like to gracefully handle the program termination (SIGKILL, SIGQUIT etc).
Towards the same I would like to know how to implement Context based cancellation of process.
For executing the flow in regular interval I'm using gocron.
cmd/scheduler.go
func scheduleTask(){
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
s := gocron.NewScheduler()
s.Every(10).Minutes().Do(processTask, ctx)
s.RunAll() // run immediate
<-s.Start() // schedule
for {
select {
case <-(ctx).Done():
fmt.Print("context done")
s.Remove(processTask)
s.Clear()
cancel()
default:
}
}
}
func processTask(ctx *context.Context){
task.Export(ctx)
}
task/export.go
func Export(ctx *context.Context){
pendingFiles, err := filepath.Glob("/tmp/pending/" + "*_task.json")
//error handling
//as there can be 100s of files, I would like to break the loop when context.Done() to return asap & clean up the resources here as well
for _, fileName := range pendingFiles {
exportItem(fileName string)
}
}
func exportItem(fileName string){
data, err := ReadFile(fileName) //not shown here for brevity
//err handling
err = postHTTPData(string(data)) //not shown for brevity
//err handling
}
For the process management, I think the other component is the actual handling of signals, and managing the context from those signals.
I'm not sure of the specifics of go-cron (they have an example showing some of these concepts on their github) but in general I think that the steps involved are:
Registration of os signals handler
Waiting to receive a signal
Canceling top level context in response to a signal
Example:
sigCh := make(chan os.Signal, 1)
defer close(sigCh)
signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGQUIT, syscall.SIGINT)
<-sigCh
cancel()
I'm not sure how this will look in the context of go-cron, but the context that the signal handling code cancels should be a parent of the context that the task and job is given.
Worked this out myself just now. I've always felt the blog post on contexts was A LOT of material to try and understand so a simpler demonstration would be nice.
There are many scenarios you may encounter. Each one is different and will require adaptation. Here's one example:
Say you have a channel that could run for an indeterminate amount of time.
indeterminateChannel := make(chan string)
for s := range indeterminateChannel{
fmt.Println(s)
}
Your producer might look something like:
for {
indeterminateChannel <- "Terry"
}
We don't control the producer, so we need someway to cut out of your print loop if the producer exceeds your time limit.
indeterminateChannel := make(chan string)
// Close the channel when our code exits so OUR for loop no longer occupies
// resources and the goroutine exits.
// The producer will have a problem, but we don't care about their problems.
// In this instance.
defer close(indeterminateChannel)
// we wait for this context to time out below the goroutine.
ctx, cancel := context.WithTimeout(context.TODO(), time.Minute*1)
defer cancel()
go func() {
for s := range indeterminateChannel{
fmt.Println(s)
}
}()
<- ctx.Done // wait for the context to terminate based on a timeout.
You can also check ctx.Err to see if the context exited due to a timeout or because it was canceled.
You might also want to learn about how to properly check if the context failed due to a deadline: How to check if an error is "deadline exceeded" error?
Or if the context was canceled: How to check if a request was cancelled

Resources