I have a goroutine for a long running job. When the job is done, it pushes the results to a channel. In the meantime, while the job is running, I want to keep updating an API with the status RUNNING.
So far, I have the following code :
func getProgressTimeout() <-chan time.Time {
return time.After(5 * time.Minute)
}
func runCommand(arg *Request) {
chanResult := make(chan Results)
go func(args *Request, c chan Results) {
resp, err := execCommand(args)
c <- Results{
resp: resp,
err: err,
}
}(arg, chanResult)
var err error
progressLoop:
for {
select {
case <-getProgressTimeout():
updateProgress() // this method will send status= RUNNING to a REST API
case out := <-chanResult:
err = jobCompleted(request, out)
break progressLoop
}
}
return err
}
I am new to golang. And I have reached the above code after lot of trial and error, and googling. It's working now. Still it doesn't feel intuitive to me when I look at it (this may very well be because, I am still trying to learn the Go way of doing things). So my question is, can I refactor this into better shape? Is there some existing pattern which is applicable in this kind of scenario? Or if there is some totally different approach to keep sending periodic updates while job is running?
Also, any suggestions to improve upon my golang concurrency are also appreciated. :)
Thanks in advance!
Consider using time.NewTicker, which sends a periodic value to a channel. Here's an example from the documentation:
package main
import (
"fmt"
"time"
)
func main() {
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
done := make(chan bool)
go func() {
time.Sleep(10 * time.Second)
done <- true
}()
for {
select {
case <-done:
fmt.Println("Done!")
return
case t := <-ticker.C:
fmt.Println("Current time: ", t)
}
}
}
Note that the embedded goroutine invoking func emulates a long task by sleeping for 10 seconds, while the caller uses select to wait for the result, while also receiving periodic events from the ticker - this is where you can do the API progress update.
Related
I am writing code to control external LEDs over UDP, and need to keep a frame rate (e.g. 60Hz) that is as constant as possible. Too much jitter will look bad. I've written a simple test using time.NewTicker and the results are not ideal. I'm wondering if there is a different way to execute code on a more accurate interval. This test was run on macOS, but needs to run on Windows and Linux too. For what it's worth, it needs to sync to audio, so maybe there is an audio ticker API on each OS it could potentially sync with?
package main
import (
"encoding/csv"
"fmt"
"log"
"os"
"strconv"
"time"
)
var f *os.File
var testTimeSeconds = 30
func appendToCsv(t time.Time) {
w := csv.NewWriter(f)
defer w.Flush()
records := []string{strconv.FormatInt(t.UnixMicro(), 10)}
w.Write(records)
}
func init() {
var err error
f, err = os.Create("newTicker.csv")
if err != nil {
log.Fatal(err)
}
w := csv.NewWriter(f)
defer w.Flush()
records := []string{"timestamps"}
w.Write(records)
}
func main() {
usPerFrame := 16666
ticker := time.NewTicker(time.Duration(usPerFrame) * time.Microsecond)
defer ticker.Stop()
done := make(chan bool)
go func() {
time.Sleep(time.Duration(testTimeSeconds) * time.Second)
done <- true
}()
for {
select {
case <-done:
fmt.Println("Done!")
return
case t := <-ticker.C:
appendToCsv(t)
}
}
}
UPDATE:
I ran another test comparing the first method with the method in #jochen's answer, still not very accurate.
One idea would be to just use time.Sleep() instead of a ticker. This takes the channel send/receive out of the loop and may lead to more accurate timing. To do this, you could run a function like the following in a separate goroutine:
func ticker(step time.Duration, done <-chan struct{}) {
next := time.Now().Add(step)
for {
time.Sleep(time.Until(next))
appendToCsv(time.Now())
select { // check whether `done` was closed
case <-done:
return
default:
// pass
}
next = next.Add(step)
}
}
I'm working on a Go project that require calling an initiation function (initFunction) in a separated goroutine (to ensure this function does not interfere with the rest of the project). initFunction must not take more than 30 seconds, so I thought I would use context.WithTimeout. Lastly, initFunction must be able to notify errors to the caller, so I thought of making an error channel and calling initFunction from an anonymous function, to recieve and report the error.
func RunInitGoRoutine(initFunction func(config string)error) error {
initErr := make(chan error)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Seconds)
go func() {
<-ctx.Done() // Line 7
err := initFunction(config)
initErr <-err
}()
select {
case res := <-initErr:
return res.err
case <-ctx.Done():
err := errors.New("Deadline")
return err
}
}
I'm quite new to Go, so I'm asking for feedbacks about the above code.
I have some doubt about Line 7. I used this to ensure the anonymous function is "included" under ctx and is therefore killed and freed and everything once timeout expires, but I'm not sure I have done the right thing.
Second thing is, I know I should be calling cancel( ) somewhere, but I can't put my finger around where.
Lastly, really any feedback is welcome, being it about efficency, style, correctness or anything.
In Go the pratice is to communicate via channels. So best thing is probably to share a channel on your context so others can consume from the channel.
As you are stating you are new to Go, I wrote a whole bunch of articles on Go (Beginner level) https://marcofranssen.nl/categories/golang.
Read from old to new to get familiar with the language.
Regarding the channel specifics you should have a look at this article.
https://marcofranssen.nl/concurrency-in-go
A pratical example of a webserver listening for ctrl+c and then gracefully shutting down the server using channels is described in this blog post.
https://marcofranssen.nl/improved-graceful-shutdown-webserver
In essence we run the server in a background routine
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
srv.l.Fatal("Could not listen on", zap.String("addr", srv.Addr), zap.Error(err))
}
}()
and then we have some code that is blocking the main routine by listening on a channel for the shutdown signal.
quit := make(chan os.Signal, 1)
signal.Notify(quit, os.Interrupt)
sig := <-quit
srv.l.Info("Server is shutting down", zap.String("reason", sig.String()))
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
srv.SetKeepAlivesEnabled(false)
if err := srv.Shutdown(ctx); err != nil {
srv.l.Fatal("Could not gracefully shutdown the server", zap.Error(err))
}
srv.l.Info("Server stopped")
This is very similar to your usecase. So running your init in a background routine and then consume the channel waiting for the result of this init routine.
package main
import (
"fmt"
"time"
)
type InitResult struct {
Message string
}
func main() {
initResult := make(chan InitResult, 0)
go func(c chan<- InitResult) {
time.Sleep(5 * time.Second)
// here we are publishing the result on the channel
c <- InitResult{Message: "Initialization succeeded"}
}(initResult)
fmt.Println("Started initializing")
// here we have a blocking operation consuming the channel
res := <-initResult
fmt.Printf("Init result: %s", res.Message)
}
https://play.golang.org/p/_YGIrdNVZx6
You could also add an error field on the struct so you could do you usual way of error checking.
I need to read data from the Go channel for a certain period of time (say 5 seconds). The select statement with timeout doesn't work for me, as I need to read as many values as available and stop exactly after 5 seconds. So far, I've come up with a solution using an extra time channel https://play.golang.org/p/yev9CcvzRIL
package main
import "time"
import "fmt"
func main() {
// I have no control over dataChan
dataChan := make(chan string)
// this is a stub to demonstrate some data coming from dataChan
go func() {
for {
dataChan <- "some data"
time.Sleep(time.Second)
}
}()
// the following is the code I'm asking about
timeChan := time.NewTimer(time.Second * 5).C
for {
select {
case d := <-dataChan:
fmt.Println("Got:", d)
case <-timeChan:
fmt.Println("Time's up!")
return
}
}
}
I'm wondering is there any better or more idiomatic way for solving this problem?
That's pretty much it. But if you don't need to stop or reset the timer, simply use time.After() instead of time.NewTimer(). time.After() is "equivalent to NewTimer(d).C".
afterCh := time.After(5 * time.Second)
for {
select {
case d := <-dataChan:
fmt.Println("Got:", d)
case <-afterCh:
fmt.Println("Time's up!")
return
}
}
Alternatively (to your liking), you may declare the after channel in the for statement like this:
for afterCh := time.After(5 * time.Second); ; {
select {
case d := <-dataChan:
fmt.Println("Got:", d)
case <-afterCh:
fmt.Println("Time's up!")
return
}
}
Also I know this is just an example, but always think how a goroutine you start will properly end, as the goroutine producing data in your case will never terminate.
Also don't forget that if multiple cases may be executed without blocking, one is chosen randomly. So if dataChan is ready to receive from "non-stop", there is no guarantee that the loop will terminate immediately after the timeout. In practice this is usually not a problem (starting with that even the timeout is not a guarantee, see details at Golang Timers with 0 length), but you should not forget about it in "mission-critial" applications. For details, see related questions:
force priority of go select statement
golang: How the select worked when multiple channel involved?
It looks like context with deadline would be good fit, something like
func main() {
dataChan := make(chan string)
ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(5*time.Second))
defer cancel()
go func(ctx context.Context) {
for {
select {
case dataChan <- "some data":
time.Sleep(time.Second)
case <-ctx.Done():
fmt.Println(ctx.Err())
close(dataChan)
return
}
}
}(ctx)
for d := range dataChan {
fmt.Println("Got:", d)
}
}
This question already has answers here:
Timeout for WaitGroup.Wait()
(10 answers)
Closed 7 months ago.
I have come across a situation that i want to trace some goroutine to sync on a specific point, for example when all the urls are fetched. Then, we can put them all and show them in specific order.
I think this is the barrier comes in. It is in go with sync.WaitGroup. However, in real situation that we can not make sure that all the fetch operation will succeed in a short time. So, i want to introduce a timeout when wait for the fetch operations.
I am a newbie to Golang, so can someone give me some advice?
What i am looking for is like this:
wg := &sync.WaigGroup{}
select {
case <-wg.Wait():
// All done!
case <-time.After(500 * time.Millisecond):
// Hit timeout.
}
I know Wait do not support Channel.
If all you want is your neat select, you can easily convert blocking function to a channel by spawning a routine which calls a method and closes/sends on channel once done.
done := make(chan struct{})
go func() {
wg.Wait()
close(done)
}()
select {
case <-done:
// All done!
case <-time.After(500 * time.Millisecond):
// Hit timeout.
}
Send your results to a buffered channel enough to take all results, without blocking, and read them in for-select loop in the main thread:
func work(msg string, d time.Duration, ret chan<- string) {
time.Sleep(d) // Work emulation.
select {
case ret <- msg:
default:
}
}
// ...
const N = 2
ch := make(chan string, N)
go work("printed", 100*time.Millisecond, ch)
go work("not printed", 1000*time.Millisecond, ch)
timeout := time.After(500 * time.Millisecond)
loop:
for received := 0; received < N; received++ {
select {
case msg := <-ch:
fmt.Println(msg)
case <-timeout:
fmt.Println("timeout!")
break loop
}
}
Playground: http://play.golang.org/p/PxeEEJo2dz.
See also: Go Concurrency Patterns: Timing out, moving on.
Another way to do it would be to monitor it internally, your question is limited but I'm going to assume you're starting your goroutines through a loop even if you're not you can refactor this to work for you but you could do one of these 2 examples, the first one will timeout each request to timeout individually and the second one will timeout the entire batch of requests and move on if too much time has passed
var wg sync.WaitGroup
wg.Add(1)
go func() {
success := make(chan struct{}, 1)
go func() {
// send your request and wait for a response
// pretend response was received
time.Sleep(5 * time.Second)
success <- struct{}{}
// goroutine will close gracefully after return
fmt.Println("Returned Gracefully")
}()
select {
case <-success:
break
case <-time.After(1 * time.Second):
break
}
wg.Done()
// everything should be garbage collected and no longer take up space
}()
wg.Wait()
// do whatever with what you got
fmt.Println("Done")
time.Sleep(10 * time.Second)
fmt.Println("Checking to make sure nothing throws errors after limbo goroutine is done")
Or if you just want a general easy way to timeout ALL requests you could do something like
var wg sync.WaitGroup
waiter := make(chan int)
wg.Add(1)
go func() {
success := make(chan struct{}, 1)
go func() {
// send your request and wait for a response
// pretend response was received
time.Sleep(5 * time.Second)
success <- struct{}{}
// goroutine will close gracefully after return
fmt.Println("Returned Gracefully")
}()
select {
case <-success:
break
case <-time.After(1 * time.Second):
// control the timeouts for each request individually to make sure that wg.Done gets called and will let the goroutine holding the .Wait close
break
}
wg.Done()
// everything should be garbage collected and no longer take up space
}()
completed := false
go func(completed *bool) {
// Unblock with either wait
wg.Wait()
if !*completed {
waiter <- 1
*completed = true
}
fmt.Println("Returned Two")
}(&completed)
go func(completed *bool) {
// wait however long
time.Sleep(time.Second * 5)
if !*completed {
waiter <- 1
*completed = true
}
fmt.Println("Returned One")
}(&completed)
// block until it either times out or .Wait stops blocking
<-waiter
// do whatever with what you got
fmt.Println("Done")
time.Sleep(10 * time.Second)
fmt.Println("Checking to make sure nothing throws errors after limbo goroutine is done")
This way your WaitGroup will stay in sync and you won't have any goroutines left in limbo
http://play.golang.org/p/g0J_qJ1BUT try it here you can change the variables around to see it work differently
Edit: I'm on mobile If anybody could fix the formatting that would be great thanks.
If you would like to avoid mixing concurrency logic with business logic, I wrote this library https://github.com/shomali11/parallelizer to help you with that. It encapsulates the concurrency logic so you do not have to worry about it.
So in your example:
package main
import (
"github.com/shomali11/parallelizer"
"fmt"
)
func main() {
urls := []string{ ... }
results = make([]*HttpResponse, len(urls)
options := &Options{ Timeout: time.Second }
group := parallelizer.NewGroup(options)
for index, url := range urls {
group.Add(func(index int, url string, results *[]*HttpResponse) {
return func () {
...
results[index] = &HttpResponse{url, response, err}
}
}(index, url, &results))
}
err := group.Run()
fmt.Println("Done")
fmt.Println(fmt.Sprintf("Results: %v", results))
fmt.Printf("Error: %v", err) // nil if it completed, err if timed out
}
I want to ask several servers for data (e.g. multiple read replicas).
In this task most important is speed, so first result should be served
and all other can be ignored.
I have problem with idiomatic way of bypassing this data. Everything
with this problem is ok when it quits (all slower goroutines are not
finishing their work, because main process exists). But when we uncomment
last line (with Sleep) We can see that other goroutines are doing their work too.
Now I'm pushing data through channel is there any way to not push them?
What is good and safe way of dealing with this kind of problems?
package main
import (
"fmt"
"log"
"math/rand"
"time"
)
type Result int
type Conn struct {
Id int
}
func (c *Conn) DoQuery(params string) Result {
log.Println("Querying start", params, c.Id)
time.Sleep(time.Duration(rand.Int31n(1000)) * time.Millisecond)
log.Println("Querying end", params, c.Id)
return Result(1000 + c.Id*c.Id)
}
func Query(conns []Conn, query string) Result {
ch := make(chan Result)
for _, conn := range conns {
go func(c Conn) {
ch <- c.DoQuery(query)
}(conn)
}
return <-ch
}
func main() {
conns := []Conn{Conn{1}, Conn{2}, Conn{3}, Conn{4}, Conn{5}}
result := Query(conns, "query!")
fmt.Println(result)
// time.Sleep(time.Minute)
}
My recommendation would be to make ch a buffered channel with one space per query: ch := make(chan Result, len(conns)). This way each query can run to completion, and will not block on the channel write.
Query can read once and return the first result. When all other goroutines complete, the channel will eventually be garbage collected and everything will go away. With your unbuffered channel, you create a lot of goroutines that can never terminate.
EDIT:
If you want to cancel in-flight requests, it can become significantly harder. Some operations and apis provide cancellation, and others don't. With an http request you can use Cancel field on the request struct. Simply provide a channel that you can close to cancel:
func (c *Conn) DoQuery(params string, cancel chan struct{}) Result {
//error handling omitted. It is important to handle errors properly.
req, _ := http.NewRequest(...)
req.Cancel = cancel
resp, _ := http.DefaultClient.Do(req)
//On Cancellation, the request will return an error of some kind.
return readData(resp)
}
func Query(conns []Conn, query string) Result {
ch := make(chan Result)
cancel := make(chan struct{})
for _, conn := range conns {
go func(c Conn) {
ch <- c.DoQuery(query,cancel)
}(conn)
}
first := <-ch
close(cancel)
return first
}
This may help if there is a large request to read that you won't care about, but it may or may not actually cancel the request on the remote server. If your query is not http, but a database call or something else, you will need to look into if there is a similar cancellation mechanism you can use.