I'm having trouble wrangling go routines and getting them to communicate back to a channel on the main go routine. To simplify, my code looks something like this:
func main() {
channel := make(chan string)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go performTest(channel, &wg, i)
}
wg.Wait()
close(channel)
for line := range channel {
fmt.Print(line)
}
}
func performTest(channel chan string, wg *sync.WaitGroup, i int) {
defer wg.Done()
// perform some work here
result := fmt.sprintf("Pretend result %d", i)
channel <- result
}
This seems to enter into some kind of a deadlock, but I don't understand why. It gets stuck on wg.Wait(), even though I would expect it to continue once all the goroutines have called Done on the wait group. What am I missing here? I'd like to wait for the goroutines, and then iterate over all results in the channel.
You can wait for the group and close the channel in a separate go routine. If the channel is closed, your range over the channel will end after the last sent value has been received.
If you just wait, nothing will receive from the channel. Since the channel is unbuffered, the performTest goroutines won't be able to send. For an unbuffered channel, the send operation will block until it has been received. Therefore, the deferred wg.Done call would never happen, and your program is deadlocked. Since Done is only called after the forever-blocking send has been performed.
func main() {
channel := make(chan string)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go performTest(channel, &wg, i)
}
// this is the trick
go func() {
wg.Wait()
close(channel)
}()
for line := range channel {
fmt.Print(line)
}
}
func performTest(channel chan string, wg *sync.WaitGroup, i int) {
defer wg.Done()
// perform some work here
result := fmt.Sprintf("Pretend result %d\n", i)
channel <- result
}
https://play.golang.com/p/5pACJzwL4Hi
Related
Im not 100% clear on why my code doesnt work
package main
import (
"fmt"
"sync"
)
//var wg sync.WaitGroup
func main() {
c := make(chan int)
send(c)
receive(c)
}
func send(c chan<- int) {
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func() {
for j := 0; j < 10; j++ {
c <- j
}
wg.Done()
}()
}
wg.Wait()
close(c)
}
func receive(c <-chan int) {
for v := range c {
fmt.Println(v)
}
}
If I put Go before send it works fine. But i thought having the wait in the send would mean the program would wait there until the go routines created in that function completed and would only go to the receive function once they were all done and so the channel was populated?
The channel is not buffered. A write to an unbuffered channel will only complete if there is a read from that channel in another goroutine. So, all the goroutines created in the send are stuck waiting to write to the channel, never reaching wg.Done, and thus, wg.Wait waits indefinitely. When you put the send in a separate goroutine, receive can run, thus reading from the channel, releasing the waiting goroutines.
I'm trying to build a generic pipeline library using worker pools. I created an interface for a source, pipe, and sink. You see, the pipe's job is to receive data from an input channel, process it, and output the result onto a channel. Here is its intended behavior:
Receive data from an input channel.
Delegate the data to an available worker.
The worker sends the result to the output channel.
Close the output channel once all workers are finished.
func (p *pipe) Process(in chan interface{}) (out chan interface{}) {
var wg sync.WaitGroup
out = make(chan interface{}, 100)
go func() {
for i := 1; i <= 100; i++ {
go p.work(in, out, &wg)
}
wg.Wait()
close(out)
}()
return
}
func (p *pipe) work(jobs <-chan interface{}, out chan<- interface{}, wg *sync.WaitGroup) {
for j := range jobs {
func(j Job) {
defer wg.Done()
wg.Add(1)
res := doSomethingWith(j)
out <- res
}(j)
}
}
However, running it may either exit without processing all of the inputs or panic with a send on closed channel message. Building the source with the -race flag gives out a data race warning between close(out) and out <- res.
Here's what I think might happen. Once a number of workers have finished their jobs, there's a split second where wg's counter reach zero. Hence, wg.Wait() is done and the program proceeds to close(out). Meanwhile, the job channel isn't finished producing data, meaning some workers are still running in another goroutine. Since the out channel is already closed, it results in a panic.
Should the wait group be placed somewhere else? Or is there a better way to wait for all workers to finish?
It's not clear why you want one worker per job, but if you do, you can restructure your outer loop setup (see untested code below). This kind of obviates the need for worker pools in the first place.
Always, though, do a wg.Add before spinning off any worker. Right here, you are spinning off exactly 100 workers:
var wg sync.WaitGroup
out = make(chan interface{}, 100)
go func() {
for i := 1; i <= 100; i++ {
go p.work(in, out, &wg)
}
wg.Wait()
close(out)
}()
You could therefore do this:
var wg sync.WaitGroup
out = make(chan interface{}, 100)
go func() {
wg.Add(100) // ADDED - count the 100 workers
for i := 1; i <= 100; i++ {
go p.work(in, out, &wg)
}
wg.Wait()
close(out)
}()
Note that you can now move wg itself down into the goroutine that spins off the workers. This can make things cleaner, if you give up on the notion of having each worker spin off jobs as new goroutines. But if each worker is going to spin off another goroutine, that worker itself must also use wg.Add, like this:
for j := range jobs {
wg.Add(1) // ADDED - count the spun-off goroutines
func(j Job) {
res := doSomethingWith(j)
out <- res
wg.Done() // MOVED (for illustration only, can defer as before)
}(j)
}
wg.Done() // ADDED - our work in `p.work` is now done
That is, each anonymous function is another user of the channel, so increment the users-of-channel count (wg.Add(1)) before spinning off a new goroutine. When you have finished reading the input channel jobs, call wg.Done() (perhaps via an earlier defer, but I showed it at the end here).
The key to thinking about this is that wg counts the number of active goroutines that could, at this point, write to the channel. It only goes to zero when no goroutines intend to write any more. That makes it safe to close the channel.
Consider using the rather simpler (but untested):
func (p *pipe) Process(in chan interface{}) (out chan interface{}) {
out = make(chan interface{})
var wg sync.WaitGroup
go func() {
defer close(out)
for j := range in {
wg.Add(1)
go func(j Job) {
res := doSomethingWith(j)
out <- res
wg.Done()
}(j)
}
wg.Wait()
}()
return out
}
You now have one goroutine that is reading the in channel as fast as it can, spinning off jobs as it goes. You'll get one goroutine per incoming job, except when they finish their work early. There is no pool, just one worker per job (same as your code except that we knock out the pools that aren't doing anything useful).
Or, since there are only some number of CPUs available, spin off some number of goroutines as you did before at the start, but have each one run one job to completion, and deliver its result, then go back to reading the next job:
func (p *pipe) Process(in chan interface{}) (out chan interface{}) {
out = make(chan interface{})
go func() {
defer close(out)
var wg sync.WaitGroup
ncpu := runtime.NumCPU() // or something fancier if you like
wg.Add(ncpu)
for i := 0; i < ncpu; i++ {
go func() {
defer wg.Done()
for j := range in {
out <- doSomethingWith(j)
}
}()
}
wg.Wait()
}
return out
}
By using runtime.NumCPU() we get only as many workers reading jobs as there are CPUs to run jobs. Those are the pools and they only do one job at a time.
There's generally no need to buffer the output channel, if the output-channel readers are well-structured (i.e., don't cause the pipeline to constipate). If they're not, the depth of buffering here limits how many jobs you can "work ahead" of whoever is consuming the results. Set it based on how useful it is to do this "working ahead"—not necessarily the number of CPUs, or the number of expected jobs, or whatever.
It's possible that the jobs are being completed just as fast as they're being sent. In this case the WaitGroup will be floating near zero even while there's many more items to process.
One fix for this is to add one before sending jobs, and decrement that one after sending them all, effectively consider the sender to be one of the 'jobs'. In this case, it's better if we do the wg.Add in the sender:
func (p *pipe) Process(in chan interface{}) (out chan interface{}) {
var wg sync.WaitGroup
out = make(chan interface{}, 100)
go func() {
for i := 1; i <= 100; i++ {
wg.Add(1)
go p.work(in, out, &wg)
}
wg.Wait()
close(out)
}()
return
}
func (p *pipe) work(jobs <-chan interface{}, out chan<- interface{}, wg *sync.WaitGroup) {
for j := range jobs {
func(j Job) {
res := doSomethingWith(j)
out <- res
wg.Done()
}(j)
}
}
One thing I notice in the code is that a goroutine is started for each job. At the same time each job processes the jobs channel in a loop until empty/closed. It doesn't seem necessary to do both.
I need several goroutines to write in the same channel. Then all the data is read in one place until all the goroutines complete the process. But I'm not sure how best to close this channel.
this is my example implementation:
func main() {
ch := make(chan data)
wg := &sync.WaitGroup{}
for instance := range dataSet {
wg.Add(1)
go doStuff(ch, instance)
}
go func() {
wg.Wait()
close(ch)
}()
for v := range ch { //range until it closes
//proceed v
}
}
func doStuff(ch chan data, instance data) {
//do some stuff with instance...
ch <- instance
}
but I'm not sure that it is idiomatic.
As you are using WaitGroup and increase the counter at the time of starting new goroutine, you have to notify WaitGroup when a goroutine is finished by calling the Done() method. Also you have to pass the same WaitGroup to the goroutine. You can do it by passing the address of WaitGroup. Otherwise each goroutine will use it's own WaitGroup which will be on different scope.
func main() {
ch := make(chan data)
wg := &sync.WaitGroup{}
for _, instance := range dataSet {
wg.Add(1)
go doStuff(ch, instance, wg)
}
go func() {
wg.Wait()
close(ch)
}()
for v := range ch { //range until it closes
//proceed v
}
}
func doStuff(ch chan data, instance data, wg *sync.WaitGroup) {
//do some stuff with instance...
ch <- instance
// call done method to decrease the counter of WaitGroup
wg.Done()
}
go version go1.11.4 darwin/amd64
A new channel and goroutine were created, and the content of the old channel was transferred to the new channel through goroutine. It should not block, but after testing, it was found to be blocked.
thanks.
type waiter struct {
ch1 chan struct{}
ch2 <-chan time.Time
limit int
count int
}
func (w *waiter) recv1Block() chan struct{} {
ch := make(chan struct{})
go func() {
for m := range w.ch1 {
ch <- m
}
}()
return ch
}
func (w *waiter) runBlock(wg *sync.WaitGroup) {
defer wg.Done()
i := 0
for i < w.limit {
select {
case <-w.recv1Block(): **// why block here?**
i++
case <-w.recv2():
}
}
w.count = i
}
why recv1Block will be block.
Every time you call recv1Block(), it creates a new channel and launches a background goroutine that tries to read all of the data from it. Since you're calling it in a loop, you will have many things all trying to consume the data from the channel; since the channel never closes, all of the goroutines will run forever.
As an exercise, you might try changing your code to pass around a chan int instead of a chan struct{}, and write a series of sequential numbers, and print them out as they're ultimately received. A sequence like this is valid:
run on goroutine #1 calls recv1Block().
recv1Block() on GR#1 spawns GR#2, and returns channel#2.
run on GR#1 blocks receiving on channel#2.
recv1Block() on GR#2 reads 0 from w.c1.
recv1Block() on GR#2 writes 0 to channel#2 (where run on GR#1 is ready to read).
recv1Block() on GR#2 reads 1 from w.c1.
recv1Block() on GR#2 wants to write 0 to channel#2 but blocks.
run on GR#1 wakes up, and receives the 0.
run on GR#1 calls recv1Block().
recv1Block() on GR#1 spawns GR#3, and returns channel #3.
recv1Block() on GR#3 reads 2 from w.c1.
...
Notice that the value 1 in this sequence will never be written anywhere, and in fact there is nothing left that could read it.
The easy solution here is to not call the channel-creating function in a loop:
func (w *waiter) runBlock(wg *sync.WaitGroup) {
defer wg.Done()
ch1 := w.recv1Block()
ch2 := w.recv2()
for {
select {
case _, ok := <-ch1:
if !ok {
return
}
w.count++
case <-ch2:
}
}
It's also standard practice to close channels when you're done with them. This will terminate a for ... range ch loop, and it will appear as readable to a select statement. In your top-level generator function:
for i := 0; i < w.limit; i++ {
w.ch1 <- struct{}{}
}
close(w.ch1)
And in your "copy the channel" function:
func (w *waiter) recv1Block() chan struct{} {
ch := make(chan struct{})
go func() {
for m := range w.ch1 {
ch <- m
}
close(ch)
}()
return ch
}
This also means that you don't need to run the main loop by "dead reckoning", expecting it to produce exactly 100 items then stop; you can stop whenever its channel exits. The consumer loop I show above does this.
I have 3 channels that get data from together . My code run for first time perfect but when after that I send another data , there is error : send on closed channel.In func FillNotCheckedDeliveryCh that line I closed channel .If I do not close it app stopped and not continue.
func main() {
receiveBulkIdsCh := make(chan int64, 100)
NotCheckedDeliveryCh := make(chan CommonType.BasicRequestParameters, 100)
ResultCh := make(chan CommonType.MessageStateResult, 100)
var wg sync.WaitGroup
wg.Add(4)
/* Every BulkIds that receive from queue , save in receiveBulkIdsChan channel*/
go func() {
defer wg.Done()
PopQueue(receiveBulkIdsCh)
}()
go func() {
defer wg.Done()
for BulkId := range receiveBulkIdsCh {
FillNotCheckedDeliveryCh(data, NotCheckedDeliveryCh)
}
}()
go func() {
defer wg.Done()
for item := range NotCheckedDeliveryCh {
for msgStatus := range DoFuncGetMessageState(item) {
ResultCh <- msgStatus
}
close(ResultCh)
}
}()
go func() {
defer wg.Done()
for Result := range ResultCh {
ReadResultCh(Result)
}
}()
wg.Wait()
}
func FillNotCheckedDeliveryCh(data IntegrateRowsFields, ch chan<- CommonType.BasicRequestParameters) {
ch <- PackerForNotFinishedCh(data)
defer close(ch)
}
can help what's wrong ?
When you work with channels in Go always the sender should close the channel. Because that signals that no more data will be send over the channel.
In your code the receiver is closing it. Just leave the channel here opened and remove the close(ch) there.
func FillNotCheckedDeliveryCh(data IntegrateRowsFields, ch chan<- CommonType.BasicRequestParameters) {
ch <- PackerForNotFinishedCh(data)
defer close(ch)
}
If something is blocking at that part you could use select:
func FillNotCheckedDeliveryCh(data IntegrateRowsFields, ch chan<- CommonType.BasicRequestParameters) {
select {
case ch <- PackerForNotFinishedCh(data):
default:
}
}
Closing the channels on the receiver side, and then trying to send on the same channel gives you the error. Better is to close the channel outside the go routine only after all values are sent on the channels.
Close the channel after waiting for all go routines to be finished.
wg.Wait()
close(ch)
Always close the channel when there are no more values to be sent on the channel.
Receivers can test whether a channel has been closed by assigning a
second parameter to the receive expression: after
v, ok := <-ch
In your case too you can check on the receiver side by using ok if there are more values in the channel or if it is closed.
Channels aren't like files; you don't usually need to close them.
Closing is only necessary when the receiver must be told there are no
more values coming, such as to terminate a range loop.
Go playground example for generating the error when not closing a channel and range over it.
Playground example for closing the channels when all values are send.