How do I wait for all go routines to finish and then read all the data from a channel?
Why is this example stuck waiting for the go routines to finish?
Go Playground
package main
import (
"fmt"
"sync"
"time"
)
func doStuff(i int, wg *sync.WaitGroup, messages chan<- string) {
defer wg.Done()
time.Sleep(time.Duration(i) * time.Second)
messages <- fmt.Sprintf("Doing stuff...%d", i)
}
func doMoreStuff(i int, wg *sync.WaitGroup, messages chan<- string) {
defer wg.Done()
time.Sleep(time.Duration(i) * time.Second)
messages <- fmt.Sprintf("Doing more stuff...%d", i)
}
func main() {
var wg sync.WaitGroup
var messages = make(chan string)
for start := time.Now();; {
elapsedTime := time.Since(start)
fmt.Println(elapsedTime)
if elapsedTime > time.Duration(3) * time.Second {
fmt.Println("BREAK")
break
}
for i := 0; i < 10; i++ {
wg.Add(1)
go doStuff(i, &wg, messages)
wg.Add(1)
go doMoreStuff(i, &wg, messages)
}
time.Sleep(time.Duration(1) * time.Second)
}
fmt.Println("WAITING")
wg.Wait()
fmt.Println("DONE")
for message := range messages {
fmt.Println(message)
}
}
If you want to wait for all goroutines to finish that send messages on a channel, and you want to start reading the messages after that, then you have no choice but to use a buffered channel that can "host" all the messages that can be sent on it by the goroutines.
This isn't something practical. And even if you'd go down this path, you would be able to receive and print the messages, but the for range loop doing so would never terminate because that only terminates when it receives all messages that were sent on the channel before it was closed, but you never close the channel. You may check this "half-working" solution on the Go Playground.
Instead launch a goroutine that waits for others to finish, and then close the channel:
go func() {
fmt.Println("WAITING")
wg.Wait()
close(messages)
}()
So now you may use for range to receive messages in the main goroutine:
for message := range messages {
fmt.Println(message)
}
fmt.Println("DONE")
Try this one on the Go Playground.
This solution is still not perfect: it first has to launch all goroutines, and only then it tries to receive values, and all the launched goroutines will be blocked on their send operation (until main is ready to receive those).
A better solution may be to launch a goroutine that receives the values, preferably before the goroutines that send messages on the channel (else they would be blocked on their sends anyway):
go func() {
for message := range messages {
fmt.Println(message)
}
}()
And wait for the goroutines and close the channel at the end of main():
fmt.Println("WAITING")
wg.Wait()
close(messages)
The problem with this is that when main() ends, so does your app, it does not wait for other non-main goroutines to finish. And this means it will not wait for the "consumer" goroutine to receive the messages.
To wait for that "consumer", you may use an additional sync.WaitGroup:
var wg2 sync.WaitGroup
wg2.Add(1)
go func() {
defer wg2.Done()
for message := range messages {
fmt.Println(message)
}
}()
// And the end of `main()`:
fmt.Println("WAITING")
wg.Wait()
close(messages)
fmt.Println("DONE")
wg2.Wait()
Try this one on the Go Playground.
The example is stuck because you are using an unbuffered channel. Both go-routines are blocked waiting to write to the channel since nothing is ready to read from it.
You can use a buffered channel but then you would just be using it as a data storage. Channels are more useful for communication. The question is why do you want to wait until all the writers are finished? In the general case of an unknown (unlimited) number of writers you would no know how big to make your channel.
Related
I'm having trouble wrangling go routines and getting them to communicate back to a channel on the main go routine. To simplify, my code looks something like this:
func main() {
channel := make(chan string)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go performTest(channel, &wg, i)
}
wg.Wait()
close(channel)
for line := range channel {
fmt.Print(line)
}
}
func performTest(channel chan string, wg *sync.WaitGroup, i int) {
defer wg.Done()
// perform some work here
result := fmt.sprintf("Pretend result %d", i)
channel <- result
}
This seems to enter into some kind of a deadlock, but I don't understand why. It gets stuck on wg.Wait(), even though I would expect it to continue once all the goroutines have called Done on the wait group. What am I missing here? I'd like to wait for the goroutines, and then iterate over all results in the channel.
You can wait for the group and close the channel in a separate go routine. If the channel is closed, your range over the channel will end after the last sent value has been received.
If you just wait, nothing will receive from the channel. Since the channel is unbuffered, the performTest goroutines won't be able to send. For an unbuffered channel, the send operation will block until it has been received. Therefore, the deferred wg.Done call would never happen, and your program is deadlocked. Since Done is only called after the forever-blocking send has been performed.
func main() {
channel := make(chan string)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go performTest(channel, &wg, i)
}
// this is the trick
go func() {
wg.Wait()
close(channel)
}()
for line := range channel {
fmt.Print(line)
}
}
func performTest(channel chan string, wg *sync.WaitGroup, i int) {
defer wg.Done()
// perform some work here
result := fmt.Sprintf("Pretend result %d\n", i)
channel <- result
}
https://play.golang.com/p/5pACJzwL4Hi
package main
import (
"fmt"
"sync"
)
func main(){
ch1 := make(chan int,100)
ct := 0
var wg sync.WaitGroup
wg.Add(1)
go func(){
//defer close(ch1)
for i:= 0; i < 10;i ++{
ch1 <- i
}
}()
go func(){
defer wg.Done()
for x := range ch1{
fmt.Println(x)
}
}()
wg.Wait()
fmt.Println("numbers:",ct)
}
Why this code will return
fatal error: all goroutines are asleep - deadlock!
I've found if I closed the channel there will be no deadlock, but I don't know why it's like that.
Do I have to close the channel after I inputted all items into the channel?
for range over a channel only terminates if / when the channel is closed. If you don't close the channel and you don't send more values on it, the for range statement will block forever, and so will the main goroutine at wg.Wait().
The "sender" party should close the channel once it sent all values, signalling the "receiver" party that no more values will come on the channel.
So yes, you should close the channel:
go func() {
defer close(ch1)
for i := 0; i < 10; i++ {
ch1 <- i
}
}()
The for loop that reads from the channel will continue reading unless you close the channel. That is the reason for the deadlock, because the goroutine that reads from the channel is the only active goroutine, and there is no other goroutines that can write to it. When you close, the for loop terminates.
range unclosed ch1 will causing block;
even close ch1, you can still receive data from ch1;
It can be said, close(ch1) will send a special message to ch1, which can be used to notify the ch1 receiver that no more data will be received. So even if there is data in the ch1, you can close() without causing the receiver to not receive the remaining data.
The channel does not need to release resources through close. As long as there is no goroutine holding the channel, the related resources will be automatically released.
uncomment defer close(ch1) will solve this problems.
here
I am trying to wrap my mind around go. I want to make a simple program that basically
starts a bunch of go routines
process messages
sends the processed result to a channel
have the main thread collect these results
shut down.
Seems simple. I started with no logic at all. I just send a number and try to get that number back.
issue: I'm deadlocking and I'm not sure why. I think I might be misusing wait groups with channels, because they work individually, but I'm not sure how to get the main thread to block on an arbitrary number of initiated go routines.
package main
import (
"fmt"
"sync"
"time"
)
func main() {
queue := make(chan int)
start := time.Now()
var wg sync.WaitGroup
for i := 0; i < 10; i += 1 {
wg.Add(1)
go count(i, queue, &wg)
}
wg.Wait()
for value := range queue {
println(value)
}
close(queue)
fmt.Println(time.Now().Sub(start))
// fmt.Println(summation)
}
func count(number int, queue chan int, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("Starting ", number)
queue <- number
fmt.Println("ending")
}
Your goroutines block on queue <- number because queue is an unbuffered channel and nobody is reading from it, as main blocks on wg.Wait.
Declare queue as a buffered channel instead. For example: queue := make(chan int, 10)
From the Go Tour (concurrency) and subsequent page:
By default, sends and receives block until the other side is ready. This allows goroutines to synchronize without explicit locks or condition variables.
Sends to a buffered channel block only when the buffer is full. Receives block when the buffer is empty.
Alternatively, move wg.Wait after the for v := range queue loop.
This should help.
package main
import (
"fmt"
"sync"
"time"
)
type event struct {
data chan string
numWorker int
}
func (e event) Send() {
var wg sync.WaitGroup
// Spaw numWorker goroutines that sends message to
// the same channel.
for i := 0; i < e.numWorker; i++ {
wg.Add(1)
go func(id int) {
// Do some fake work
time.Sleep(1 * time.Second)
e.data <- fmt.Sprintf("message from go #%d", id)
wg.Done()
}(i)
}
// Wait for goroutines to finish their work.
wg.Wait()
// Close the channel to signal Recv to stop ranging
// over the channel.
close(e.data)
}
func (e event) Recv() {
// Range over the data channel to receive message(s).
for msg := range e.data {
fmt.Println(msg)
}
}
func main() {
e := event{
numWorker: 10, // Number of worker goroutine(s)
data: make(chan string, 5 /* Buffer Size */),
}
// Spawn a goroutine for Send
go e.Send()
// Recv receives data from Send
e.Recv()
}
To avoid deadlocking you can manage the channel and wait groups in separate goroutine. Try change that:
wg.Wait()
for value := range queue {
println(value)
}
close(queue)
with this:
go func() {
wg.Wait()
close(queue)
}()
for value := range queue {
println(value)
}
I am trying to implement a simple logic where a Producer sends data to a channel ch with an forever for loop and a Consumer reads from the channel ch.
The Producer stops producing and exit the forever loop when it receives a signal on the channel quit.
The code is this (see also this playground)
func main() {
ch := make(chan int)
quit := make(chan bool)
var wg sync.WaitGroup
wg.Add(1)
go produce(ch, quit, &wg)
go consume(ch)
time.Sleep(1 * time.Millisecond)
fmt.Println("CLOSE")
close(quit)
wg.Wait()
}
func produce(ch chan int, quit chan bool, wg *sync.WaitGroup) {
for i := 0; ; i++ {
select {
case <-quit:
close(ch)
fmt.Println("exit")
wg.Done()
return //we exit
default:
ch <- i
fmt.Println("Producer sends", i)
}
}
}
func consume(ch chan int) {
for {
runtime.Gosched() // give the opportunity to the main goroutine to close the "quit" channel
select {
case i, more := <-ch:
if !more {
fmt.Println("exit consumer")
return
}
fmt.Println("Consumer receives", i)
}
}
}
If I run this piece of code on my machine (a Mac with 4 cores) everything works fine. If I try the same code on the Go Playgroud it always times out. I guess that this because the Go Playground is a single core and so the infinite loop does not give the chance to other goroutines to run, but then I do not understand why the instruction runtime.Gosched() does not have any effect.
Just to complete the picture I have seen that, if I set GOMAXPROCS=1 on my Mac, the program still works fine and exits as expected. If I set GOMAXPROCS=1 on my Mac and remove the runtime.Gosched() instruction, the behavior gets brittle: sometimes the program terminates as expected, some other times it seems to never exit the infinite loop.
You created a pathological situation that shouldn't happen in a real program, so the scheduler is not optimized to handle this. Combined with the fake time implementation in the playground, and you get far too many cycles of the producer and consumer before hitting a timeout.
The producer goroutine is creating values as fast as possible, while the consumer is always ready to receive them. With GOMAPXPROCS=1, the scheduler spends all its time bouncing between the two before it is forced to preempt the available work to check on the main goroutine, which takes longer than the playground will allow.
If we add something for the producer-consumer pair to do, we can limit the amount of time they have to monopolize the scheduler. For example, adding a time.Sleep(time.Microsecond) to the consumer will cause the playground to print 1000 values. This also goes to show how "accurate" the simulated time is in the playground, since that would not be possible with normal hardware which takes a non-zero amount time to process each message.
While an interesting case, this has little bearing on real programs.
A few notes, you can range over a channel to receive all values, you should always defer wg.Done at the start of the goroutine when possible, you can send values in the select case which allows you to actually cancel the for-select loop when the send isn't ready, and if you want the "exit consumer" message you need to send the WaitGroup to the consumer as well.
https://play.golang.org/p/WyPmpY9pFl7
func main() {
ch := make(chan int)
quit := make(chan bool)
var wg sync.WaitGroup
wg.Add(2)
go produce(ch, quit, &wg)
go consume(ch, &wg)
time.Sleep(50 * time.Microsecond)
fmt.Println("CLOSE")
close(quit)
wg.Wait()
}
func produce(ch chan int, quit chan bool, wg *sync.WaitGroup) {
defer wg.Done()
for i := 0; ; i++ {
select {
case <-quit:
close(ch)
fmt.Println("exit")
return
case ch <- i:
fmt.Println("Producer sends", i)
}
}
}
func consume(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
for i := range ch {
fmt.Println("Consumer receives", i)
time.Sleep(time.Microsecond)
}
fmt.Println("exit consumer")
return
}
I have 3 channels that get data from together . My code run for first time perfect but when after that I send another data , there is error : send on closed channel.In func FillNotCheckedDeliveryCh that line I closed channel .If I do not close it app stopped and not continue.
func main() {
receiveBulkIdsCh := make(chan int64, 100)
NotCheckedDeliveryCh := make(chan CommonType.BasicRequestParameters, 100)
ResultCh := make(chan CommonType.MessageStateResult, 100)
var wg sync.WaitGroup
wg.Add(4)
/* Every BulkIds that receive from queue , save in receiveBulkIdsChan channel*/
go func() {
defer wg.Done()
PopQueue(receiveBulkIdsCh)
}()
go func() {
defer wg.Done()
for BulkId := range receiveBulkIdsCh {
FillNotCheckedDeliveryCh(data, NotCheckedDeliveryCh)
}
}()
go func() {
defer wg.Done()
for item := range NotCheckedDeliveryCh {
for msgStatus := range DoFuncGetMessageState(item) {
ResultCh <- msgStatus
}
close(ResultCh)
}
}()
go func() {
defer wg.Done()
for Result := range ResultCh {
ReadResultCh(Result)
}
}()
wg.Wait()
}
func FillNotCheckedDeliveryCh(data IntegrateRowsFields, ch chan<- CommonType.BasicRequestParameters) {
ch <- PackerForNotFinishedCh(data)
defer close(ch)
}
can help what's wrong ?
When you work with channels in Go always the sender should close the channel. Because that signals that no more data will be send over the channel.
In your code the receiver is closing it. Just leave the channel here opened and remove the close(ch) there.
func FillNotCheckedDeliveryCh(data IntegrateRowsFields, ch chan<- CommonType.BasicRequestParameters) {
ch <- PackerForNotFinishedCh(data)
defer close(ch)
}
If something is blocking at that part you could use select:
func FillNotCheckedDeliveryCh(data IntegrateRowsFields, ch chan<- CommonType.BasicRequestParameters) {
select {
case ch <- PackerForNotFinishedCh(data):
default:
}
}
Closing the channels on the receiver side, and then trying to send on the same channel gives you the error. Better is to close the channel outside the go routine only after all values are sent on the channels.
Close the channel after waiting for all go routines to be finished.
wg.Wait()
close(ch)
Always close the channel when there are no more values to be sent on the channel.
Receivers can test whether a channel has been closed by assigning a
second parameter to the receive expression: after
v, ok := <-ch
In your case too you can check on the receiver side by using ok if there are more values in the channel or if it is closed.
Channels aren't like files; you don't usually need to close them.
Closing is only necessary when the receiver must be told there are no
more values coming, such as to terminate a range loop.
Go playground example for generating the error when not closing a channel and range over it.
Playground example for closing the channels when all values are send.