Optional timeouts in golang - go

I have a function which runs a command with a timeout. It looks like this:
func run_command(cmdName string, cmdArgs []string, timeout int) (int, string) {
// the command we're going to run
cmd := exec.Command(cmdName, cmdArgs...)
// assign vars for output and stderr
var output bytes.Buffer
var stderr bytes.Buffer
// get the stdout and stderr and assign to pointers
cmd.Stderr = &stderr
cmd.Stdout = &output
// Start the command
if err := cmd.Start(); err != nil {
log.Fatalf("Command not found: %s", cmdName)
}
timer := time.AfterFunc(time.Second*time.Duration(timeout), func() {
err := cmd.Process.Kill()
if err != nil {
panic(err)
}
})
// Here's the good stuff
if err := cmd.Wait(); err != nil {
if exiterr, ok := err.(*exec.ExitError); ok {
// Command ! exit 0, capture it
if status, ok := exiterr.Sys().(syscall.WaitStatus); ok {
// Check it's nagios compliant
if status.ExitStatus() == 1 || status.ExitStatus() == 2 || status.ExitStatus() == 3 {
return status.ExitStatus(), stderr.String()
} else {
// If not, force an exit code 2
return 2, stderr.String()
}
}
} else {
log.Fatalf("cmd.Wait: %v", err)
}
timer.Stop()
}
// We didn't get captured, continue!
return 0, output.String()
}
Now I want to be able to make the timeout optional. In order to fudge this a bit, I tried simply allowing timeout to be set to 0 and then having an if statement around the timer. It ended up looking like this.
if timeout > 0 {
timer := time.AfterFunc(time.Second*time.Duration(timeout), func() {
err := cmd.Process.Kill()
if err != nil {
panic(err)
}
})
}
Of course, this failed because timer is no longer defined timer.Stop() isn't defined now.
So I wrapped the timer.Stop() with the if statement as well.
if timeout > 0 {
timer.Stop()
}
This also didn't work.
What is the correct way to do something like this? Golangs strict typing is new to me, so I'm struggling to get my head around it

Using the context package makes it easy to handle timeouts.
golang.org/x/net/context has become a standard library since Go 1.7.
The following is an example:
package main
import (
"context"
"os"
"os/exec"
"strconv"
"time"
)
func main() {
timeout, err := strconv.Atoi(os.Args[1])
if err != nil {
panic(err)
}
ctx := context.Background()
if timeout > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(context.Background(), time.Duration(timeout)*time.Second)
defer cancel()
}
cmd := exec.CommandContext(ctx, "sleep", "5")
if err := cmd.Run(); err != nil {
panic(err)
}
}
When timeout is set to 3 seconds, and run sleep 5:
$ go run main.go 3
panic: signal: killed
goroutine 1 [running]:
panic(0xc7040, 0xc42008c020)
/usr/local/Cellar/go/1.7.4_1/libexec/src/runtime/panic.go:500 +0x1a1
main.main()
/Users/m-morita/work/tmp/20170106/main.go:27 +0x11c
exit status 2
When it is set to 10 seconds or 0(= never timeout), it ends normally:
$ go run main.go 10
$ go run main.go 0

While you could replace the timer func with a noop if there's no duration, the usual solution is to simply defer the timer.Stop call when you create the timer:
timer := time.AfterFunc(time.Second*time.Duration(timeout), func() {
err := cmd.Process.Kill()
if err != nil {
panic(err)
}
})
defer timer.Stop()
Otherwise, you can declare timer at the function scope and check if it was assigned before calling timer.Stop()
if timer != nil {
timer.Stop()
}
You should also note that an exec.Cmd already makes use of a Context for timeouts, which is exposed via exec.CommandContext.

Simply define the timer variable before the first if timeout > 0 block and assign the timer to it using = instead of :=.
var timer *time.Timer
if timeout > 0 {
timer = time.AfterFunc(time.Second*time.Duration(timeout), func() {
err := cmd.Process.Kill()
if err != nil {
panic(err)
}
})
}
The check for timeout > 0 before timer.Stop() will still be necessary, or, to diminish dependencies, changed to timer != nil.
if timer != nil {
timer.Stop()
}

Related

go benchmark using rpc and exec.command but got stucked at cmd.Run()

I'm trying to do a benchmark with go using rpc and exec.command, here are parts of my code.
I have a master to send rpc to worker to do some job.
func main() {
var wg sync.WaitGroup
var clients []*rpc.Client
client, err := rpc.DialHTTP("tcp", "addr"+":1234")
if err != nil {
log.Fatal("dialing:", err)
}
reply := &Reply{}
args := &Args{}
clients = append(clients, client)
fmt.Println(clients)
err = clients[0].Call("Worker.Init", args, reply)
if err != nil {
log.Fatal("init error:", err)
}
// call for server to init channel
// err = client.Call("Worker.Init", args, reply)
args.A = 1
wg.Add(200)
fmt.Println(time.Now().UnixNano())
for i := 0; i < 200; i++ {
go func() {
defer wg.Done()
err = client.Call("Worker.DoJob", args, reply)
if err != nil {
log.Fatal("dojob error:", err)
}
fmt.Println("Done")
}()
}
wg.Wait()
fmt.Println(time.Now().UnixNano())
}
and worker's code
func (w *Worker) DoJob(args *Args, reply *Reply) error {
// find a channel to do it
w.c <- 1
runtime.LockOSThread()
fmt.Println("exec")
// cmd := exec.Command("docker", "run", "--rm", "ubuntu:16.04", "/bin/bash", "-c", "date +%s%N")
cmd := exec.Command("echo", "hello")
err := cmd.Run()
fmt.Println("exec done")
if err != nil {
reply.Err = err
fmt.Println(err)
}
fmt.Println("done")
<-w.c
return nil
}
I use a chan of size 12 to simulate that the machine has only 12 threads, and after I find it would stuck at cmd.Run(), I changed the command from running a docker to just simply echo hello, but it got still stucked between fmt.Println("exec") and fmt.Println("exec done").
I don'k know why is this happening? Am I sending out too many rpcs so a lot of rpcs will be dropped?

Calling an executable with a timeout

I am trying to use the context package to run a binary with a 10 second timeout, as such:
func RunQL(file db.File, flag string) string {
// 10-second timeout for the binary to run
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "qltool", "run", "-f", file.Path, "--rootfs", file.Root, "--args", flag)
out, err := cmd.Output()
// check to see if our timeout was executed
if ctx.Err() == context.DeadlineExceeded {
return ""
}
// no timeout (either completed successfully or errored)
if err != nil {
return ""
}
return string(out)
}
But for some reason, it still hangs if the process lasts longer than 10 seconds. Not sure what would be causing this, I also noticed that the documentation for the CommandContext() function appears to be wrong/misleading? It shows the following code:
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
if err := exec.CommandContext(ctx, "sleep", "5").Run(); err != nil {
// This will fail after 100 milliseconds. The 5 second sleep
// will be interrupted.
}
}
But CommandContext() returns type *Cmd not error .

Keep retrying a function in Golang

I am trying to make a functionality which would work in the following manner:
As soon as the service function is called, it uses the Fetch function to get records from a service (which come in the form of byte array), JSON unmarshal the byte array, populate the struct and then send the struct to a DB function to save to database.
Now, since this needs to be a continuous job, I have added two if conditions such that, if the records received are of length 0, then we use the retry function to retry pulling the records, else we just write to the database.
I have been trying to debug the retry function for a while now, but it is just not working, and basically stops after the first retry (even though I specify the attempts as 100). What can I do to make sure, it keeps retrying pulling the records ?
The code is as Follows:
// RETRY FUNCTION
func retry(attempts int, sleep time.Duration, f func() error) (err error) {
for i := 0; ; i++ {
err = f()
if err == nil {
return
}
if i >= (attempts - 1) {
break
}
time.Sleep(sleep)
sleep *= 2
log.Println("retrying after error:", err)
}
return fmt.Errorf("after %d attempts, last error: %s", attempts, err) }
//Save Data function
type Records struct {
Messages [][]byte
}
func (s *Service) SaveData(records Records, lastSentPlace uint) error {
//lastSentPlace is sent as 0 to begin with.
for i := lastSentPlace; i <= records.Place-1; i++ {
var msg Records
msg.Unmarshal(records.Messages[i])
order := MyStruct{
Fruit: msg.Fruit,
Burger: msg.Burger,
Fries: msg.Fries,
}
err := s.db.UpdateOrder(context.TODO(), nil , order)
if err != nil {
logging.Error("Error occured...")
}
}return nil}
//Service function (This runs as a batch, which is why we need retrying)
func (s *Service) MyServiceFunction(ctx context.Context, place uint, length uint) (err error) {
var lastSentPlace = place
records, err := s.Poll(context.Background(), place, length)
if err != nil {
logging.Info(err)
}
// if no records found then retry.
if len(records.Messages) == 0 {
err = retry(100, 2*time.Minute, func() (err error) {
records, err := s.Poll(context.Background(), place, length)
// if data received, write to DB
if len(records.Messages) != 0 {
err = s.SaveData(records, lastSentPlace)
}
return
})
// if data is not received, or if err is not null, retry
if err != nil || len(records.Messages) == 0 {
log.Println(err)
return
}
// if data received on first try, then no need to retry, write to db
} else if len(records.Messages) >0 {
err = s.SaveData(records, lastSentPlace)
if err != nil {
return err
}
}
return nil }
I think, the issue is with the way I am trying to implement the retry function, I have been trying to debug this for a while, but being new to the language, I am really stuck. What I wanted to do was, implement a backoff if no records are found. Any help is greatly appreciated.
Thanks !!!
I make a simpler retry.
Use simpler logic for loop to ensure correctness.
We sleep before executing a retry, so use i > 0 as the condition for the sleeping.
Here's the code:
func retry(attempts int, sleep time.Duration, f func() error) (err error) {
for i := 0; i < attempts; i++ {
if i > 0 {
log.Println("retrying after error:", err)
time.Sleep(sleep)
sleep *= 2
}
err = f()
if err == nil {
return nil
}
}
return fmt.Errorf("after %d attempts, last error: %s", attempts, err)
}
I know this is an old question, but came across it when searching for retries and used it as the base of a solution.
This version can accept a func with 2 return values and uses generics in golang 1.18 to make that possible. I tried it in 1.17, but couldn't figure out a way to make the method generic.
This could be extended to any number of return values of any type. I have used any here, but that could be limited to a list of types.
func retry[T any](attempts int, sleep int, f func() (T, error)) (result T, err error) {
for i := 0; i < attempts; i++ {
if i > 0 {
log.Println("retrying after error:", err)
time.Sleep(time.Duration(sleep) * time.Second)
sleep *= 2
}
result, err = f()
if err == nil {
return result, nil
}
}
return result, fmt.Errorf("after %d attempts, last error: %s", attempts, err)
}
Usage example:
var config Configuration
something, err := retry(config.RetryAttempts, config.RetrySleep, func() (Something, error) { return GetSomething(config.Parameter) })
func GetSomething(parameter string) (something Something, err error) {
// Do something flakey here that might need a retry...
return something, error
}
Hope that helps someone with the same use case as me.
The function you are calling is using a context. So it is important that you handle that context.
If you don't know what a context is and how to use it, I would recomend that post: https://blog.golang.org/context
Your retry function should also handle the context. Just to get you on the track I give you a simple implementation.
func retryMyServiceFunction(ctx context.Context, place uint, length uint, sleep time.Duration) {
for {
select {
case ctx.Done():
return
default:
err := MyServiceFunction(ctx, place, length)
if err != nil {
log.Println("handle error here!", err)
time.Sleep(sleep)
} else {
return
}
}
}
}
I don't like the sleep part. So you should analyse the returned error. Also you have to think about timeouts. When you let your service sleep to long there could be a timeout.
There is a library for the retry mechanism.
https://github.com/avast/retry-go
url := "http://example.com"
var body []byte
err := retry.Do(
func() error {
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
body, err = ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
return nil
},
)
fmt.Println(body)
In the GoPlayground in the comments of the accepted answer, there are some things I would consider adding. Using continue and break in the for loop would make the loop even simpler by not using the if i > 0 { statement. Furthermore I would use early return in all the functions to directly return on an error. And last I would consistently use errors to check if a function failed or not, checking the validity of a value should be inside the executed function itself.
This would be my little attempt:
package main
import (
"errors"
"fmt"
"log"
"time"
)
func main() {
var complicatedFunctionPassing bool = false
var attempts int = 5
// if complicatedFunctionPassing is true retry just makes one try
// if complicatedFunctionPassing is false retry makes ... attempts
err := retry(attempts, time.Second, func() (err error) {
if !complicatedFunctionPassing {
return errors.New("somthing went wrong in the important function")
}
log.Println("Complicated function passed")
return nil
})
if err != nil {
log.Printf("failed after %d attempts with error: %s", attempts, err.Error())
}
}
func retry(attempts int, sleep time.Duration, f func() error) (err error) {
for i := 0; i < attempts; i++ {
fmt.Println("This is attempt number", i+1)
// calling the important function
err = f()
if err != nil {
log.Printf("error occured after attempt number %d: %s", i+1, err.Error())
log.Println("sleeping for: ", sleep.String())
time.Sleep(sleep)
sleep *= 2
continue
}
break
}
return err
}
You can try it out here:
https://go.dev/play/p/Ag8ObCb980U

how to repeat shutting down and establish go routine?

every one,I am new to golang.I wanna get the data from log file generated by my application.cuz roll-back mechanism, I met some problem.For instance,my target log file is chats.log,it will be renamed to chats.log.2018xxx and a new chats.log will be created.so my go routine that read log file will fail to work.
so I need detect the change and shutdown the previous go routine and then establish the new go routine.
I looked for modules that can help me,and I found
func ExampleNewWatcher(fn string, createnoti chan string, wg sync.WaitGroup) {
wg.Add(1)
defer wg.Done()
watcher, err := fsnotify.NewWatcher()
if err != nil {
log.Fatal(err)
}
defer watcher.Close()
done := make(chan bool)
go func() {
for {
select {
case event := <-watcher.Events:
if event.Op == fsnotify.Create && event.Name==fn{
createnoti <- "has been created"
}
case err := <-watcher.Errors:
log.Println("error:", err)
}
}
}()
err = watcher.Add("./")
if err != nil {
log.Fatal(err)
}
<-done
}
I use fsnotify to detech the change,and make sure the event of file is my log file,and then send some message to a channel.
this is my worker go routine:
func tailer(fn string,isfollow bool, outchan chan string, done <-chan interface{},wg sync.WaitGroup) error {
wg.Add(1)
defer wg.Done()
_, err := os.Stat(fn)
if err != nil{
panic(err)
}
t, err := tail.TailFile(fn, tail.Config{Follow:isfollow})
if err != nil{
panic(err)
}
defer t.Stop()
for line := range t.Lines{
select{
case outchan <- line.Text:
case <- done:
return nil
}
}
return nil
}
I using tail module to read the log file,and I add a done channel to it to shutdown the cycle(I don't know whether I put it in the right way)
And I will send every log content to a channel to consuming it.
So here is the question:how should I put it together?
ps: Actually,I can use some tool to do this job.like apache-flume,but all of those tools need dependency.
Thank you a lot!
Here is a complete example that reloads and rereads the file as it changes or gets deleted and recreated:
package main
import (
"github.com/fsnotify/fsnotify"
"io/ioutil"
"log"
)
const filename = "myfile.txt"
func ReadFile(filename string) string {
data, err := ioutil.ReadFile(filename)
if err != nil {
log.Println(err)
}
return string(data)
}
func main() {
watcher, err := fsnotify.NewWatcher()
if err != nil {
log.Fatal(err)
}
defer watcher.Close()
err = watcher.Add("./")
if err != nil {
log.Fatal(err)
}
for {
select {
case event := <-watcher.Events:
if event.Op == fsnotify.Create && event.Name == filename {
log.Println(ReadFile(filename))
}
case err := <-watcher.Errors:
log.Println("error:", err)
}
}
}
Note this doesn't require goroutines, channels or a WaitGroup. Better to keep things simple and reserve those for when they're actually needed.

continuously reading from exec.Cmd output

Guys I am trying pick new lines as they come from command output, but always I end up doing it synchronous way (I have to wait until script is finished). I tired to use fsnotify but it is working only with regular files, do you have any idea how it can be done ?
package main
import (
"fmt"
"os/exec"
"bytes"
"os"
)
func main() {
cmd := exec.Command("scripts/long_script")
output := new(bytes.Buffer)
cmd.Stdout = output
cmd.Stderr = output
if err := cmd.Start(); err != nil{ // after Start program is continued and script is executing in background
fmt.Printf("Failed to start " + err.Error())
os.Exit(1)
}
fmt.Printf(" Before WAIT %s \n", output.String()) // script is writing but nothing can be read from output
cmd.Wait()
fmt.Printf(" After Wait %s \n", output.String()) // if we wait to finish execution, we can read all output
}
You should use os.StdoutPipe()
func main() {
for i := 10; i < 20; i++ {
go printName(`My name is Bob, I am ` + strconv.Itoa(i) + ` years old`)
// Adding delay so as to see incremental output
time.Sleep(60 * time.Millisecond)
}
// Adding delay so as to let program complete
// Please use channels or wait groups
time.Sleep(100 * time.Millisecond)
}
func printName(jString string) {
cmd := exec.Command("echo", "-n", jString)
cmdReader, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(cmdReader)
go func() {
for scanner.Scan() {
fmt.Println(scanner.Text())
}
}()
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
if err := cmd.Wait(); err != nil {
log.Fatal(err)
}
}
sources that helped me:
nathanleclaire.com
blog.kowalczyk.info
eventually I managed to do it with []bytes
stdout, err := cmd.StdoutPipe()
buff := make([]byte,10)
var n int
for err == nil {
n,err = stdout.Read(buff)
if n > 0{
fmt.Printf("taken %d chars %s",n,string(buff[:n]))
}
}
cmd.Wait()
if cmd.ProcessState.Success() {. // ProcessState is set after Wait
fmt.Println("Script success")
} else {
fmt.Println("Script failed")
}

Resources