I work the first time with goroutines and channels in go and do not come any further.
I have a websocket connection where every time a user connect a new goroutine is spawned. Now I want to stop this goroutine if the user disconnect from the websocket connection.
To manage the stop signale I have create a map of channels. Each entry can be identified by the users websocket connection. I pass the websocket connection, the map of channels for the stop signal and two other parameters to the goroutine. But the goroutine doesn't receive any values from the quit channel and I don't know why.
Here is the relevant code for the main package:
package main
import (
"net/http"
"time"
"github.com/gorilla/websocket"
)
func wsHandler(w http.ResponseWriter, r *http.Request) {
...
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
return
}
defer ws.Close()
data.Quit[ws] = make(chan bool)
data.DB.ListenToTable(data.GetTableName(source), channel, data.Quit, ws)
for {
if _, _, err := ws.NextReader(); err != nil {
data.Quit[ws] <- true
ws.Close()
break
}
}
}
And the code of the data package where the goroutine is created:
package data
var Quit = make(map[*websocket.Conn](chan bool))
func (db *rethinkDB) ListenToTable(name string, ch chan Data, quit map[*websocket.Conn](chan bool), ws *websocket.Conn) {
go func(name string, ws *websocket.Conn) {
for {
select {
case <-quit[ws]:
fmt.Println("QUIT GOROUTINE")
break
default:
res, err := r.Table(name).Changes().Run(db.session)
if err != nil {
log.Fatalln(err)
}
var response DataFeed
for res.Next(&response) {
response.NewVal.WS = ws
ch <- response.NewVal
}
if res.Err() != nil {
log.Println(res.Err())
}
}
}
}(name, ws)
}
I have also tried buffered channels or pass the channel instead of the map of channels to the goroutine but without success. The fmt.Println("QUIT GOROUTINE") command is never called and the goroutine isn't sopped.
I hope someone can help me and sorry if this question was already ask but I haven't found a solution that solves my problem.
First to make things easier:
As far as I can see you do not need a global register for the quit channels. Just create a ch := make(chan bool) in main, pass it to ListenToTable (instead of the whole map of channels) and use it in the select. In main close(ch) it if you want to exit. But as you said, that doesn't solve your problem.
Theoretically you are on the right track with closing the go routine. I took your sample code and made the following runnable code from it:
package main
import (
"fmt"
"time"
)
func main() {
chClose := make(chan bool)
channel := make(chan string)
ListenToTable("somestring", channel, chClose)
time.Sleep(3 * time.Second)
chClose <- true
time.Sleep(1 * time.Second)
}
func ListenToTable(name string, ch chan string, chClose chan bool) {
go func(name string) {
for {
select {
case <-chClose:
fmt.Println("QUIT GOROUTINE")
return // VERY IMPORTANT: not break!
default:
}
}
}(name)
}
The problem must be with something else in you code, probably blocked by something in the default section and not even executing the select. Try printing fmt.Println("something") before the select {. If that is not printed regularly then you have your answer.
One more thing: As commented in the code above you cannot break out of a for { select { ... } } with a single break. You need to use a return (to exit the function) or another tactic (like a break with a label as Adrian suggested in the comments). The break will only exit the select, but not the for loop.
Related
I face an error while working with channel and context in golang.
So my idea is,
I want to run 3 functions with goroutines and pass a channel into that function. If one of that functions is already sending a value into that channel, I want to stop other goroutines from sending it to the channel and close the channel.
My current problem is after a channel value is received, the other functions tried to send the value to that channel again
here is my code
package main
import (
"context"
"fmt"
"log"
"time"
)
func main() {
var err string
ctx, cancel := context.WithCancel(context.Background())
errChan := make(chan string)
go one(ctx, errChan, cancel)
go two(ctx, errChan, cancel)
go three(ctx, errChan, cancel)
err = <-errChan
fmt.Println("recevied:", err)
close(errChan)
time.Sleep(10 * time.Second)
}
func one(ctx context.Context, c chan<- string, done context.CancelFunc) {
go func() {
for {
select {
case <-ctx.Done():
log.Println("a goutines already sending to channel, one is lose.")
return
case c <- "from one":
done()
return
}
}
}()
}
func two(ctx context.Context, c chan<- string, done context.CancelFunc) {
go func() {
for {
select {
case <-ctx.Done():
log.Println("a goutines already sending to channel, two is lose.")
return
case c <- "from two":
done()
return
}
}
}()
}
func three(ctx context.Context, c chan<- string, done context.CancelFunc) {
go func() {
for {
select {
case <-ctx.Done():
log.Println("a goutines already sending to channel, three is lose.")
return
case c <- "from three":
done()
return
}
}
}()
}
What should I change from my code so I could only send one (exactly one) time to the channel and stop the whole operation?
I have two goroutines: the main worker and a helper that it spins off for some help. helper can encounter errors, so I use a channel to communicate errors over from the helper to the worker.
func helper(c chan <- error) (){
//do some work
c <- err // send errors/nil on c
}
Here is how helper() is called:
func worker() error {
//do some work
c := make(chan error, 1)
go helper(c)
err := <- c
return err
}
Questions:
Is the statement err := <- c blocking worker? I don't think so, since the channel is buffered.
If it is blocking, how do I make it non-blocking? My requirement is to have worker and its caller continue with rest of the work, without waiting for the value to appear on the channel.
Thanks.
You can easily verify
func helper(c chan<- error) {
time.Sleep(5 * time.Second)
c <- errors.New("") // send errors/nil on c
}
func worker() error {
fmt.Println("do one")
c := make(chan error, 1)
go helper(c)
err := <-c
fmt.Println("do two")
return err
}
func main() {
worker()
}
Q: Is the statement err := <- c blocking worker? I don't think so, since the channel is buffered.
A: err := <- c will block worker.
Q: If it is blocking, how do I make it non-blocking? My requirement is to have worker and its caller continue with rest of the work, without waiting for the value to appear on the channel.
A: If you don't want blocking, just remove err := <-c. If you need err at the end, just move err := <-c to the end.
You can not read channel without blocking, if you go through without blocking, can can no more exec this code, unless your code is in a loop.
Loop:
for {
select {
case <-c:
break Loop
default:
//default will go through without blocking
}
// do something
}
And have you ever seen errgroup or waitgroup?
It use atomic, cancel context and sync.Once to implement this.
https://github.com/golang/sync/blob/master/errgroup/errgroup.go
https://github.com/golang/go/blob/master/src/sync/waitgroup.go
Or you can just use it, go you func and then wait for error in any place you want.
In your code, the rest of the work is independent of whether the helper encountered an error. You can simply receive from the channel after the rest of the work is completed.
func worker() error {
//do some work
c := make(chan error, 1)
go helper(c)
//do rest of the work
return <-c
}
I think you need this code..
run this code
package main
import (
"log"
"sync"
)
func helper(c chan<- error) {
for {
var err error = nil
// do job
if err != nil {
c <- err // send errors/nil on c
break
}
}
}
func worker(c chan error) error {
log.Println("first log")
go func() {
helper(c)
}()
count := 1
Loop:
for {
select {
case err := <- c :
return err
default:
log.Println(count, " log")
count++
isFinished := false
// do your job
if isFinished {
break Loop // remove this when you test
}
}
}
return nil
}
func main() {
wg := sync.WaitGroup{}
wg.Add(1)
go func() {
c := make(chan error, 1)
worker(c)
wg.Done()
}()
wg.Wait()
}
In go I have two callbacks that eventually do not fire.
registerCb(func() {...})
registerCb(func() {...})
/* Wait for both func to execute with timeout */
I want to wait for both of them but having a timeout if one is not executed.
sync.WaitGroup does not work, since it is blocking and not channel based. Also you call WaitGroup.Done() without the risk of panic outside the callbacks.
My current solution is using just two booleans and a busy wait loop. But that's not satisfying.
Is there any idiomatic way that do not use polling or busy waiting?
Update:
Here is some code that demonstrates a busy wait solution but should return as soon as both callbacks are fired or after the timeout, without using polling
package main
import (
"fmt"
"log"
"sync"
"time"
)
var cbOne func()
var cbTwo func()
func registerCbOne(cb func()) {
cbOne = cb
}
func registerCbTwo(cb func()) {
cbTwo = cb
}
func executeCallbacks() {
<-time.After(1 * time.Second)
cbOne()
// Might never happen
//<-time.After(1 * time.Second)
//cbTwo()
}
func main() {
// Some process in background will execute our callbacks
go func() {
executeCallbacks()
}()
err := WaitAllOrTimeout(3 * time.Second)
if err != nil {
fmt.Println("Error: ", err.Error())
}
fmt.Println("Hello, playground")
}
func WaitAllOrTimeout(to time.Duration) error {
cbOneDoneCh := make(chan bool, 1)
cbTwoDoneCh := make(chan bool, 1)
cbOneDone := false
cbTwoDone := false
registerCbOne(func() {
fmt.Println("cb One");
cbOneDoneCh <- true
})
registerCbTwo(func() {
fmt.Println("cb Two");
cbTwoDoneCh <- true
})
// Wait for cbOne and cbTwo to be executed or a timeout
// Busywait solution
for {
select {
case <-time.After(to):
if cbOneDone && cbTwoDone {
fmt.Println("Both CB executed (we could poll more often)")
return nil
}
fmt.Println("Timeout!")
return fmt.Errorf("Timeout")
case <-cbOneDoneCh:
cbOneDone = true
case <-cbTwoDoneCh:
cbTwoDone = true
}
}
}
This is a followup to my comment, added after you added your example solution. To be clearer than I can in comments, your example code is actually not that bad. Here is your original example:
// Busywait solution
for {
select {
case <-time.After(to):
if cbOneDone && cbTwoDone {
fmt.Println("Both CB executed (we could poll more often)")
return nil
}
fmt.Println("Timeout!")
return fmt.Errorf("Timeout")
case <-cbOneDoneCh:
cbOneDone = true
case <-cbTwoDoneCh:
cbTwoDone = true
}
}
This isn't a "busy wait" but it does have several bugs (including the fact that you need an only-once send semantic for the done channels, or maybe easier and at least as good, to just close them once when done, perhaps using sync.Once). What we want to do is:
Start a timer with to as the timeout.
Enter a select loop, using the timer's channel and the two "done" channels.
We want to exit the select loop when the first of the following events occurs:
the timer fires, or
both "done" channels have been signaled.
If we're going to close the two done channels we'll want to have the Ch variables cleared (set to nil) as well so that the selects don't spin—that would turn this into a true busy-wait—but for the moment let's just assume instead that we send exactly once on them on callback, and otherwise just leak the channels, so that we can use your code as written as those selects will only ever return once. Here's the updated code:
t := timer.NewTimer(to)
for !cbOneDone || !cbTwoDone {
select {
case <-t.C:
fmt.Println("Timeout!")
return fmt.Errorf("timeout")
}
case <-cbOneDoneCh:
cbOneDone = true
case <-cbTwoDoneCh:
cbTwoDone = true
}
}
// insert t.Stop() and receive here to drain t.C if desired
fmt.Println("Both CB executed")
return nil
Note that we will go through the loop at most two times:
If we receive from both Done channels, once each, the loop stops without a timeout. There's no spinning/busy-waiting: we never received anything from t.C. We return nil (no error).
If we receive from one Done channel, the loop resumes but blocks waiting for the timer or the other Done channel.
If we ever receive from t.C, it means we didn't get both callbacks yet. We may have had one, but there's been a timeout and we choose to give up, which was our goal. We return an error, without going back through the loop.
A real version needs a bit more work to clean up properly and avoid leaking "done" channels (and the timer channel and its goroutine; see comment), but this is the general idea. You're already turning the callbacks into channel operations, and you already have a timer with its channel.
func wait(ctx context.Context, wg *sync.WaitGroup) error {
done := make(chan struct{}, 1)
go func() {
wg.Wait()
done <- struct{}{}
}()
select {
case <-done:
// Counter is 0, so all callbacks completed.
return nil
case <-ctx.Done():
// Context cancelled.
return ctx.Err()
}
}
Alternatively, you can pass a time.Duration and block on <-time.After(d) rather than on <-ctx.Done(), but I would argue that using context is more idiomatic.
below code present two variations,
the first is the regular pattern, nothing fancy, it does the job and does it well. You launch your callbacks into a routine, you make them push to a sink, listen that sink for a result or timeout. Take care to the sink channel initial capacity, to prevent leaking a routine it must match the number of callbacks.
the second factories out the synchronization mechanisms into small functions to assemble, two wait methods are provided, waitAll and waitOne. Nice to write, but definitely less efficient, more allocations, more back and forth with more channels, more complex to reason about, more subtle.
package main
import (
"fmt"
"log"
"sync"
"time"
)
func main() {
ExampleOne()
ExampleTwo()
ExampleThree()
fmt.Println("Hello, playground")
}
func ExampleOne() {
log.Println("start reg")
errs := make(chan error, 2)
go func() {
fn := callbackWithOpts("reg: so slow", 2*time.Second, nil)
errs <- fn()
}()
go func() {
fn := callbackWithOpts("reg: too fast", time.Millisecond, fmt.Errorf("broke!"))
errs <- fn()
}()
select {
case err := <-errs: // capture only one result,
// the fastest to finish.
if err != nil {
log.Println(err)
}
case <-time.After(time.Second): // or wait that many amount of time,
// in case they are all so slow.
}
log.Println("done reg")
}
func ExampleTwo() {
log.Println("start wait")
errs := waitAll(
withTimeout(time.Second,
callbackWithOpts("waitAll: so slow", 2*time.Second, nil),
),
withTimeout(time.Second,
callbackWithOpts("waitAll: too fast", time.Millisecond, nil),
),
)
for err := range trim(errs) {
if err != nil {
log.Println(err)
}
}
log.Println("done wait")
}
func ExampleThree() {
log.Println("start waitOne")
errs := waitOne(
withTimeout(time.Second,
callbackWithOpts("waitOne: so slow", 2*time.Second, nil),
),
withTimeout(time.Second,
callbackWithOpts("waitOne: too fast", time.Millisecond, nil),
),
)
for err := range trim(errs) {
if err != nil {
log.Println(err)
}
}
log.Println("done waitOne")
}
// a configurable callback for playing
func callbackWithOpts(msg string, tout time.Duration, err error) func() error {
return func() error {
<-time.After(tout)
fmt.Println(msg)
return err
}
}
// withTimeout return a function that returns first error or times out and return nil
func withTimeout(tout time.Duration, h func() error) func() error {
return func() error {
d := make(chan error, 1)
go func() {
d <- h()
}()
select {
case err := <-d:
return err
case <-time.After(tout):
}
return nil
}
}
// wait launches all func() and return their errors into the returned error channel; (merge)
// It is the caller responsability to drain the output error channel.
func waitAll(h ...func() error) chan error {
d := make(chan error, len(h))
var wg sync.WaitGroup
for i := 0; i < len(h); i++ {
wg.Add(1)
go func(h func() error) {
defer wg.Done()
d <- h()
}(h[i])
}
go func() {
wg.Wait()
close(d)
}()
return d
}
// wait launches all func() and return the first error into the returned error channel
// It is the caller responsability to drain the output error channel.
func waitOne(h ...func() error) chan error {
d := make(chan error, len(h))
one := make(chan error, 1)
var wg sync.WaitGroup
for i := 0; i < len(h); i++ {
wg.Add(1)
go func(h func() error) {
defer wg.Done()
d <- h()
}(h[i])
}
go func() {
for err := range d {
one <- err
close(one)
break
}
}()
go func() {
wg.Wait()
close(d)
}()
return one
}
func trim(err chan error) chan error {
out := make(chan error)
go func() {
for e := range err {
out <- e
}
close(out)
}()
return out
}
I am trying to read a constant stream of data, if the call to receive stream takes longer than 30 seconds I need to timeout and exit the program. I am not sure how to exit the go routine once a timeout has been received.
func ReceiveStreamMessages(strm Stream, msg chan<- []byte) error {
d := make(chan []byte, 1)
e := make(chan error)
tm := time.After(30 * time.Second)
go func() {
for {
//blocking call
data, err := strm.Recv()
if err != nil {
e <- err
return
}
select {
case d <- data.Result:
case <-tm:
//exit out go routine
return
}
}
}()
for {
select {
case message := <-d:
msg <- message
case err := <-e:
return err
case <-tm:
return nil
}
}
}
My code above is wrong as: in order for the select to run in the go routines for loop, the blocking function will have to return and data will be populated and therefore won't hit the timeout select case (or will do randomly as both will be ready). Is exiting the parent function enough to exit the go routine?
Use context package WithTimeout. Something like this:
package main
import (
"context"
"fmt"
"sync"
"time"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
// prepare
...
// wait group just for test
var wg sync.WaitGroup
wg.Add(1)
go func() {
for {
select {
case d <- data.Result:
// do something
case <-ctx.Done():
fmt.Println("Done")
wg.Done()
return
}
}
}()
wg.Wait()
cancel()
fmt.Println("Hello, playground")
}
You can see a working example here https://play.golang.org/p/agi1fimtEkJ
I got a problem using sync.WaitGroup and select together. If you take a look at following http request pool you will notice that if an error occurs it will never be reported as wg.Done() will block and there is no read from the channel anymore.
package pool
import (
"fmt"
"log"
"net/http"
"sync"
)
var (
MaxPoolQueue = 100
MaxPoolWorker = 10
)
type Pool struct {
wg *sync.WaitGroup
queue chan *http.Request
errors chan error
}
func NewPool() *Pool {
return &Pool{
wg: &sync.WaitGroup{},
queue: make(chan *http.Request, MaxPoolQueue),
errors: make(chan error),
}
}
func (p *Pool) Add(r *http.Request) {
p.wg.Add(1)
p.queue <- r
}
func (p *Pool) Run() error {
for i := 0; i < MaxPoolWorker; i++ {
go p.doWork()
}
select {
case err := <-p.errors:
return err
default:
p.wg.Wait()
}
return nil
}
func (p *Pool) doWork() {
for r := range p.queue {
fmt.Printf("Request to %s\n", r.Host)
p.wg.Done()
_, err := http.DefaultClient.Do(r)
if err != nil {
log.Fatal(err)
p.errors <- err
} else {
fmt.Printf("no error\n")
}
}
}
Source can be found here
How can I still use WaitGroup but also get errors from go routines?
Just got the answer my self as I wrote the question and as I think it is an interesting case I would like to share it with you.
The trick to use sync.WaitGroup and chan together is that we wrap:
select {
case err := <-p.errors:
return err
default:
p.wg.Done()
}
Together in a for loop:
for {
select {
case err := <-p.errors:
return err
default:
p.wg.Done()
}
}
In this case select will always check for errors and wait if nothing happens :)
It looks a bit like the fail-fast mechanism enabled by the Tomb library (Tomb V2 GoDoc):
The tomb package handles clean goroutine tracking and termination.
If any of the tracked goroutines returns a non-nil error, or the Kill or Killf method is called by any goroutine in the system (tracked or not), the tomb Err is set, Alive is set to false, and the Dying channel is closed to flag that all tracked goroutines are supposed to willingly terminate as soon as possible.
Once all tracked goroutines terminate, the Dead channel is closed, and Wait unblocks and returns the first non-nil error presented to the tomb via a result or an explicit Kill or Killf method call, or nil if there were no errors.
You can see an example in this playground:
(extract)
// start runs all the given functions concurrently
// until either they all complete or one returns an
// error, in which case it returns that error.
//
// The functions are passed a channel which will be closed
// when the function should stop.
func start(funcs []func(stop <-chan struct{}) error) error {
var tomb tomb.Tomb
var wg sync.WaitGroup
allDone := make(chan struct{})
// Start all the functions.
for _, f := range funcs {
f := f
wg.Add(1)
go func() {
defer wg.Done()
if err := f(tomb.Dying()); err != nil {
tomb.Kill(err)
}
}()
}
// Start a goroutine to wait for them all to finish.
go func() {
wg.Wait()
close(allDone)
}()
// Wait for them all to finish, or one to fail
select {
case <-allDone:
case <-tomb.Dying():
}
tomb.Done()
return tomb.Err()
}
A simpler implementation would be like below. (Check in play.golang: https://play.golang.org/p/TYxxsDRt5Wu)
package main
import "fmt"
import "sync"
import "time"
type Error struct {
message string
}
func (e Error) Error() string {
return e.message
}
func main() {
var wg sync.WaitGroup
waitGroupLength := 8
errChannel := make(chan error, 1)
// Setup waitgroup to match the number of go routines we'll launch off
wg.Add(waitGroupLength)
finished := make(chan bool, 1) // this along with wg.Wait() are why the error handling works and doesn't deadlock
for i := 0; i < waitGroupLength; i++ {
go func(i int) {
fmt.Printf("Go routine %d executed\n", i+1)
time.Sleep(time.Duration(waitGroupLength - i))
time.Sleep(0) // only here so the time import is needed
if i%4 == 1 {
errChannel <- Error{fmt.Sprintf("Errored on routine %d", i+1)}
}
// Mark the wait group as Done so it does not hang
wg.Done()
}(i)
}
go func() {
wg.Wait()
close(finished)
}()
L:
for {
select {
case <-finished:
break L // this will break from loop
case err := <-errChannel:
if err != nil {
fmt.Println("error ", err)
// handle your error
}
}
}
fmt.Println("Executed all go routines")
}