I'm working on a console music player in Go. Whenever the user selects and plays an album, I launch a goroutine to loop over a playlist.
playlist := make([]*Media, 0)
for _, path := range album.Paths {
media, err := NewMediaFromPath(path)
// return err
playlist = append(playlist, media)
}
for idx := range playlist {
player.SetMedia(playlist[idx])
err = player.Play()
// check err
status, err := player.MediaState()
// check err
for status != MediaEnded && status != MediaStopped {
// update UI and check player status
// loop until the song finishes
status, err = player.MediaState()
}
}
I need a way of canceling this goroutine when the user selects a new album. I'm using context.Context to do so (but I'm not convinced it's the best solution).
I create the ctx
ctx, cancel := context.WithCancel(context.Background())
So in the UI event handler, the play() func will cancel() the goroutine.
This works once I check inside the update UI for loop:
select {
case <-ctx.Done():
return err
default:
time.Sleep(50 * time.Millisecond)
}
Then the channel ctx.Done() is closed and the next albums played will always return instead of loop.
Is there a way to recancel a context.Context?
If not, is there a better way to cancel this goroutine (and the following goroutines)?
Alternatively I've tried to use waitgroups,
go func() {
wg.Wait()
wg.Add(1)
err = playAlbum(
done,
player,
*albums[s],
list,
status,
)
wg.Done()
// handle err
}()
But then I get a sync: WaitGroup is reused before previous Wait has returned panic
What about using a channel to cancel the goroutine?
select {
case <-chClose:
return
default:
}
Your cancel() call could simply close the channel:
close(chClose)
but then you cannot close it again! So you need to make sure your new album has a new chClose. Depending on your code structure this might be the cleaner solution.
Alternatively you can just send a value on chClose to initiate a stop of the go routine:
chClose <- 1
You can do that as often as you want.
Note that if there is no goroutine listening, this will block (or if you have a buffer, you will end up closing routines that have not even startet yet. --> You need a clean architecture!!)
Related
This one is a tricky issue that bugs me quite a bit.
Essentially, I wrote an integration microservice that provides data streams from Binance crypto exchange using the Go client. A client sends a start messages, starts data stream for a symbol, and at some point, sends a close message to stop the stream. My implementation looks basically like this:
func (c BinanceClient) StartDataStream(clientType bn.ClientType, symbol, interval string) error {
switch clientType {
case bn.SPOT_LIVE:
wsKlineHandler := c.handlers.klineHandler.SpotKlineHandler
wsErrHandler := c.handlers.klineHandler.ErrHandler
_, stopC, err := binance.WsKlineServe(symbol, interval, wsKlineHandler, wsErrHandler)
if err != nil {
fmt.Println(err)
return err
} else {
c.state.clientSymChanMap[clientType][symbol] = stopC
return nil
}
...
}
The clientSymChanMap stores the stopChannel in a nested hashmap so that I can retrieve the stop channel later to stop the data feed. The stop function has been implemented accordingly:
func (c BinanceClient) StopDataStream(clientType bn.ClientType, symbol string) {
//mtd := "StopDataStream: "
stopC := c.state.clientSymChanMap[clientType][symbol]
if isClosed(stopC) {
DbgPrint(" Channel is already closed. Do nothing for: " + symbol)
} else {
close(stopC)
}
// Delete channel from the map otherwise the next StopAll throws a NPE due to closing a dead channel
delete(c.state.clientSymChanMap[clientType], symbol)
return
}
To prevent panics from already closed channels, I use a check function that returns true in case the channel is already close.
func isClosed(ch <-chan struct{}) bool {
select {
case <-ch:
return true
default:
}
return false
}
Looks nice, but has a catch. When I run the code with starting data for just one symbol, it starts and closes the datafeed exactly as expected.
However, when starting multiple data feeds, then the above code somehow never closes the websocket and just keeps streaming data forever. Without the isClosed check, I get panics of trying to close a closed channel, but with the check in place, well, nothing gets closed.
When looking at the implementation of the above binance.WsKlineServe function, it's quite obvious that it just wraps a new websocket with each invocation and then returns the done & stop channel.
The documentation gives the following usage example:
wsKlineHandler := func(event *binance.WsKlineEvent) {
fmt.Println(event)
}
errHandler := func(err error) {
fmt.Println(err)
}
doneC, stopC, err := binance.WsKlineServe("LTCBTC", "1m", wsKlineHandler, errHandler)
if err != nil {
fmt.Println(err)
return
}
<-doneC
Because the doneC channel actually blocks, I removed it and thought that storing the stopC channel and then use it later to stop the datafeed would work. However, it only does so for one single instance. When multiple streams are open, this doesn't work anymore.
Any idea what that's the case and how to fix it?
Firstly, this is dangerous:
if isClosed(stopC) {
DbgPrint(" Channel is already closed. Do nothing for: " + symbol)
} else {
close(stopC) // <- can't be sure channel is still open
}
there is no guarantee that after your polling check of the channel state, that the channel will still be in that same state in the next line of code. So this code could in theory could panic if it's called concurrently.
If you want an asynchronous action to occur on the channel close - it's best to do this explicitly from its own goroutine. So you could try this:
go func() {
stopC := c.state.clientSymChanMap[clientType][symbol]
<-stopC
// stopC definitely closed now
delete(c.state.clientSymChanMap[clientType], symbol)
}()
P.S. you do need some sort of mutex on your map, since the delete is asynchronous - you need to ensure any adds to the map don't datarace with this.
P.P.S Channels are reclaimed by the GC when they go out of scope. If you are no longer reading from it - they do not need to be explicitly closed to be reclaimed by the GC.
Using channels for stopping a goroutine or closing something is very tricky. There are lots of things you can do wrong or forget to do.
context.WithCancel abstracts that complexity away, making the code more readable and maintainable.
Some code snippets:
ctx, cancel := context.WitchCancel(context.TODO())
TheThingToCancel(ctx, ...)
// Whenever you want to stop TheThingToCancel. Can be called multiple times.
cancel()
Then in a for loop you'd often have a select like this:
for {
select {
case <-ctx.Done():
return
default:
}
// do stuff
}
Here some code that is closer to your specific case of an open connection:
func TheThingToCancel(ctx context.Context) (context.CancelFunc, error) {
ctx, cancel := context.WithCancel(ctx)
conn, err := net.Dial("tcp", ":12345")
if err != nil {
cancel()
return nil, err
}
go func() {
<-ctx.Done()
_ = conn.Close()
}()
go func() {
defer func() {
_ = conn.Close()
// make sure context is always cancelled to avoid goroutine leak
cancel()
}()
var bts = make([]byte, 1024)
for {
n, err := conn.Read(bts)
if err != nil {
return
}
fmt.Println(bts[:n])
}
}()
return cancel, nil
}
It returns the cancel function to be able to close it from the outside.
Cancelling a context can be done many times over without a panic like would occur if a channel is closed multiple times. That is one advantage. Also you can derive contexts from other contexts and thereby close a lot of contexts that all stop different routines by closing a parent context. Carefully designed, this is very powerful for shutting down different routines belonging together that also need to be able to be shut down individually.
I'm new to Go and concurrency in Go. I'm trying to use a Go context to cancel a set of Go routines once I find a member with a given ID.
A Group stores a list of Clients, and each Client has a list of Members. I want to search in parallel all the Clients and all their Members to find a Member with a given ID. Once this Member is found, I want to cancel all the other Go routines and return the discovered Member.
I've tried the following implementation, using a context.WithCancel and a WaitGroup.
This doesn't work however, and hangs indefinitely, never getting past the line waitGroup.Wait(), but I'm not sure why exactly.
func (group *Group) MemberWithID(ID string) (*models.Member, error) {
found := make(chan *models.Member)
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
defer cancel()
var waitGroup sync.WaitGroup
for _, client := range group.Clients {
waitGroup.Add(1)
go func(clientToQuery Client) {
defer waitGroup.Done()
select {
case <-ctx.Done():
return
default:
}
member, _ := client.ClientMemberWithID(ID)
if member != nil {
found <- member
cancel()
return
}
} (client)
}
waitGroup.Wait()
if len(found) > 0 {
return <-found, nil
}
return nil, fmt.Errorf("no member found with given id")
}
found is an unbuffered channel, so sending on it blocks until there is someone ready to receive from it.
Your main() function would be the one to receive from it, but only after waitGroup.Wait() returns. But that will block until all launched goroutines call waitGroup.Done(). But that won't happen until they return, which won't happen until they can send on found. It's a deadlock.
If you change found to be buffered, that will allow sending values on it even if main() is not ready to receive from it (as many values as big the buffer is).
But you should receive from found before waitGroup.Wait() returns.
Another solution is to use a buffer of 1 for found, and use non-blocking send on found. That way the first (fastest) goroutine will be able to send the result, and the rest (given we're using non-blocking send) will simply skip sending.
Also note that it should be the main() that calls cancel(), not each launched goroutines individually.
For this kind of use case I think a sync.Once is probably a better fit than a channel. When you find the first non-nil member, you want to do two different things:
Record the member you found.
Cancel the remaining goroutines.
A buffered channel can easily do (1), but makes (2) a bit more complicated. But a sync.Once is perfect for doing two different things the first time something interesting happens!
I would also suggest aggregating non-trivial errors, so that you can report something more useful than no member found if, say, your database connection fails or some other nontrivial error occurs. You can use a sync.Once for that, too!
Putting it all together, I would want to see something like this (https://play.golang.org/p/QZXUUnbxOv5):
func (group *Group) MemberWithID(ctx context.Context, id string) (*Member, error) {
ctx, cancel := context.WithCancel(ctx)
defer cancel()
var (
wg sync.WaitGroup
member *Member
foundOnce sync.Once
firstNontrivialErr error
errOnce sync.Once
)
for _, client := range group.Clients {
wg.Add(1)
client := client // https://golang.org/doc/faq#closures_and_goroutines
go func() {
defer wg.Done()
m, err := client.ClientMemberWithID(ctx, id)
if m != nil {
foundOnce.Do(func() {
member = m
cancel()
})
} else if nf := (*MemberNotFoundError)(nil); !errors.As(err, &nf) {
errOnce.Do(func() {
firstNontrivialErr = err
})
}
}()
}
wg.Wait()
if member == nil {
if firstNontrivialErr != nil {
return nil, firstNontrivialErr
}
return nil, &MemberNotFoundError{ID: id}
}
return member, nil
}
I'm working on a Go project that require calling an initiation function (initFunction) in a separated goroutine (to ensure this function does not interfere with the rest of the project). initFunction must not take more than 30 seconds, so I thought I would use context.WithTimeout. Lastly, initFunction must be able to notify errors to the caller, so I thought of making an error channel and calling initFunction from an anonymous function, to recieve and report the error.
func RunInitGoRoutine(initFunction func(config string)error) error {
initErr := make(chan error)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Seconds)
go func() {
<-ctx.Done() // Line 7
err := initFunction(config)
initErr <-err
}()
select {
case res := <-initErr:
return res.err
case <-ctx.Done():
err := errors.New("Deadline")
return err
}
}
I'm quite new to Go, so I'm asking for feedbacks about the above code.
I have some doubt about Line 7. I used this to ensure the anonymous function is "included" under ctx and is therefore killed and freed and everything once timeout expires, but I'm not sure I have done the right thing.
Second thing is, I know I should be calling cancel( ) somewhere, but I can't put my finger around where.
Lastly, really any feedback is welcome, being it about efficency, style, correctness or anything.
In Go the pratice is to communicate via channels. So best thing is probably to share a channel on your context so others can consume from the channel.
As you are stating you are new to Go, I wrote a whole bunch of articles on Go (Beginner level) https://marcofranssen.nl/categories/golang.
Read from old to new to get familiar with the language.
Regarding the channel specifics you should have a look at this article.
https://marcofranssen.nl/concurrency-in-go
A pratical example of a webserver listening for ctrl+c and then gracefully shutting down the server using channels is described in this blog post.
https://marcofranssen.nl/improved-graceful-shutdown-webserver
In essence we run the server in a background routine
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
srv.l.Fatal("Could not listen on", zap.String("addr", srv.Addr), zap.Error(err))
}
}()
and then we have some code that is blocking the main routine by listening on a channel for the shutdown signal.
quit := make(chan os.Signal, 1)
signal.Notify(quit, os.Interrupt)
sig := <-quit
srv.l.Info("Server is shutting down", zap.String("reason", sig.String()))
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
srv.SetKeepAlivesEnabled(false)
if err := srv.Shutdown(ctx); err != nil {
srv.l.Fatal("Could not gracefully shutdown the server", zap.Error(err))
}
srv.l.Info("Server stopped")
This is very similar to your usecase. So running your init in a background routine and then consume the channel waiting for the result of this init routine.
package main
import (
"fmt"
"time"
)
type InitResult struct {
Message string
}
func main() {
initResult := make(chan InitResult, 0)
go func(c chan<- InitResult) {
time.Sleep(5 * time.Second)
// here we are publishing the result on the channel
c <- InitResult{Message: "Initialization succeeded"}
}(initResult)
fmt.Println("Started initializing")
// here we have a blocking operation consuming the channel
res := <-initResult
fmt.Printf("Init result: %s", res.Message)
}
https://play.golang.org/p/_YGIrdNVZx6
You could also add an error field on the struct so you could do you usual way of error checking.
I am in doubt whether all of my spawned goroutines are dying after doing their assigned work.
I have to make two HTTP calls(always), but based on a flag, read the response from either one of them.
what I have done so far is ->
var result error
resultChannel := make(chan error)
var wg sync.WaitGroup
wg.Add(1) // only adding 1, as I don't need to wait for other to complete.
go func() {
_, err := // HTTP call ONE
if flagIsTrue {
defer wg.Done()
resultChannel <- err
}
}()
go func() {
_, err := // HTTP call TWO
if !flagIsTrue {
defer wg.Done()
resultChannel <- err
}
}()
go func() {
wg.Wait()
close(resultChannel)
}()
for err := range resultChannel {
result = err
}
Hence, I will wait for the corresponding call, and listen to its response only. This is working well, but since the app is deployed on the server, where I guess the main goroutine won't die(henceforth killing other goroutines), my main concern is whether the other ignorable thread will die or not after it will get the response from HTTP call(afaik, we need to tell go that a goroutine needs to die).
My concerns:
The assumption(true acc to me) that the main thread does not terminate after serving one of these calls.
Will the ignorable(response is, but necessary to trigger the API call) thread die or not?
Should I use a select case to handle this, if yes then how(other suggestions are welcome)?
If the flagIsTrue is set before creating the goroutines, then only one of the goroutines will be able to write to the channel. The other one will not attempt to write to the channel, and thus will terminate.
You could simply move the check for the flag outside, and create one goroutine based on the flag.
Reusing parent context with context.WithTimeout with a new timeout
Hi there, I'm new to go. I was wondering if it's possible to reuse a parent context to create multiple context.withTimeout(). The rationale would be where I have to call multiple network requests in sequence and would like to set a timeout for each request at the same time using the parent's context.
Rationale
When the parent's context is cancelled, all the requests made would be cancelled too.
Problem
In the code below, it shows an example whereby LongProcess is the network request. However, the context is closed before the second LongProcess call can be made with a context deadline exceeded.
The documentation withDeadline states The returned context's Done channel is closed when the deadline expires, when the returned cancel function is called, or when the parent context's Done channel isclosed, whichever happens first.
So if that's the case, is there a way where I can reset the timer for withTimeout? Or do I have to create a new context context.Background() for every request? That would mean the parent context will not be passed. :(
// LongProcess refers to a long network request
func LongProcess(ctx context.Context, duration time.Duration, msg string) error {
c1 := make(chan string, 1)
go func() {
// Simulate processing
time.Sleep(duration)
c1 <- msg
}()
select {
case m := <-c1:
fmt.Println(m)
return nil
case <-ctx.Done():
return ctx.Err()
}
}
func main() {
ctx := context.Background()
t := 2 * time.Second
ctx, cancel := context.WithTimeout(ctx, t)
defer cancel()
// Simulate a 2 second process time
err := LongProcess(ctx, 2*time.Second, "first process")
fmt.Println(err)
// Reusing the context.
s, cancel := context.WithTimeout(ctx, t)
defer cancel()
// Simulate a 1 second process time
err = LongProcess(s, 1*time.Second, "second process")
fmt.Println(err) // context deadline exceeded
}
It looks like the first call to context.WithTimeout shadow the parent context ctx. The later process re-use this already canceled context hence the error. You have to re-use the parent one. Here is the example updated:
func main() {
// Avoid to shadow child contexts
parent := context.Background()
t := 2 * time.Second
// Use the parent context.
ctx1, cancel := context.WithTimeout(parent, t)
defer cancel()
err := LongProcess(ctx1, 2*time.Second, "first process")
fmt.Println(err)
// Use the parent context not the canceled one.
ctx2, cancel := context.WithTimeout(parent, t)
defer cancel()
err = LongProcess(ctx2, 1*time.Second, "second process")
fmt.Println(err)
}