Golang : send on closed channel error in multi goroutines - go

I have 3 channels that get data from together . My code run for first time perfect but when after that I send another data , there is error : send on closed channel.In func FillNotCheckedDeliveryCh that line I closed channel .If I do not close it app stopped and not continue.
func main() {
receiveBulkIdsCh := make(chan int64, 100)
NotCheckedDeliveryCh := make(chan CommonType.BasicRequestParameters, 100)
ResultCh := make(chan CommonType.MessageStateResult, 100)
var wg sync.WaitGroup
wg.Add(4)
/* Every BulkIds that receive from queue , save in receiveBulkIdsChan channel*/
go func() {
defer wg.Done()
PopQueue(receiveBulkIdsCh)
}()
go func() {
defer wg.Done()
for BulkId := range receiveBulkIdsCh {
FillNotCheckedDeliveryCh(data, NotCheckedDeliveryCh)
}
}()
go func() {
defer wg.Done()
for item := range NotCheckedDeliveryCh {
for msgStatus := range DoFuncGetMessageState(item) {
ResultCh <- msgStatus
}
close(ResultCh)
}
}()
go func() {
defer wg.Done()
for Result := range ResultCh {
ReadResultCh(Result)
}
}()
wg.Wait()
}
func FillNotCheckedDeliveryCh(data IntegrateRowsFields, ch chan<- CommonType.BasicRequestParameters) {
ch <- PackerForNotFinishedCh(data)
defer close(ch)
}
can help what's wrong ?

When you work with channels in Go always the sender should close the channel. Because that signals that no more data will be send over the channel.
In your code the receiver is closing it. Just leave the channel here opened and remove the close(ch) there.
func FillNotCheckedDeliveryCh(data IntegrateRowsFields, ch chan<- CommonType.BasicRequestParameters) {
ch <- PackerForNotFinishedCh(data)
defer close(ch)
}
If something is blocking at that part you could use select:
func FillNotCheckedDeliveryCh(data IntegrateRowsFields, ch chan<- CommonType.BasicRequestParameters) {
select {
case ch <- PackerForNotFinishedCh(data):
default:
}
}

Closing the channels on the receiver side, and then trying to send on the same channel gives you the error. Better is to close the channel outside the go routine only after all values are sent on the channels.
Close the channel after waiting for all go routines to be finished.
wg.Wait()
close(ch)
Always close the channel when there are no more values to be sent on the channel.
Receivers can test whether a channel has been closed by assigning a
second parameter to the receive expression: after
v, ok := <-ch
In your case too you can check on the receiver side by using ok if there are more values in the channel or if it is closed.
Channels aren't like files; you don't usually need to close them.
Closing is only necessary when the receiver must be told there are no
more values coming, such as to terminate a range loop.
Go playground example for generating the error when not closing a channel and range over it.
Playground example for closing the channels when all values are send.

Related

Channels and Wait Groups Entering Deadlock

I'm having trouble wrangling go routines and getting them to communicate back to a channel on the main go routine. To simplify, my code looks something like this:
func main() {
channel := make(chan string)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go performTest(channel, &wg, i)
}
wg.Wait()
close(channel)
for line := range channel {
fmt.Print(line)
}
}
func performTest(channel chan string, wg *sync.WaitGroup, i int) {
defer wg.Done()
// perform some work here
result := fmt.sprintf("Pretend result %d", i)
channel <- result
}
This seems to enter into some kind of a deadlock, but I don't understand why. It gets stuck on wg.Wait(), even though I would expect it to continue once all the goroutines have called Done on the wait group. What am I missing here? I'd like to wait for the goroutines, and then iterate over all results in the channel.
You can wait for the group and close the channel in a separate go routine. If the channel is closed, your range over the channel will end after the last sent value has been received.
If you just wait, nothing will receive from the channel. Since the channel is unbuffered, the performTest goroutines won't be able to send. For an unbuffered channel, the send operation will block until it has been received. Therefore, the deferred wg.Done call would never happen, and your program is deadlocked. Since Done is only called after the forever-blocking send has been performed.
func main() {
channel := make(chan string)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go performTest(channel, &wg, i)
}
// this is the trick
go func() {
wg.Wait()
close(channel)
}()
for line := range channel {
fmt.Print(line)
}
}
func performTest(channel chan string, wg *sync.WaitGroup, i int) {
defer wg.Done()
// perform some work here
result := fmt.Sprintf("Pretend result %d\n", i)
channel <- result
}
https://play.golang.com/p/5pACJzwL4Hi

Exiting forever loop using channels - issues with Go Playground

I am trying to implement a simple logic where a Producer sends data to a channel ch with an forever for loop and a Consumer reads from the channel ch.
The Producer stops producing and exit the forever loop when it receives a signal on the channel quit.
The code is this (see also this playground)
func main() {
ch := make(chan int)
quit := make(chan bool)
var wg sync.WaitGroup
wg.Add(1)
go produce(ch, quit, &wg)
go consume(ch)
time.Sleep(1 * time.Millisecond)
fmt.Println("CLOSE")
close(quit)
wg.Wait()
}
func produce(ch chan int, quit chan bool, wg *sync.WaitGroup) {
for i := 0; ; i++ {
select {
case <-quit:
close(ch)
fmt.Println("exit")
wg.Done()
return //we exit
default:
ch <- i
fmt.Println("Producer sends", i)
}
}
}
func consume(ch chan int) {
for {
runtime.Gosched() // give the opportunity to the main goroutine to close the "quit" channel
select {
case i, more := <-ch:
if !more {
fmt.Println("exit consumer")
return
}
fmt.Println("Consumer receives", i)
}
}
}
If I run this piece of code on my machine (a Mac with 4 cores) everything works fine. If I try the same code on the Go Playgroud it always times out. I guess that this because the Go Playground is a single core and so the infinite loop does not give the chance to other goroutines to run, but then I do not understand why the instruction runtime.Gosched() does not have any effect.
Just to complete the picture I have seen that, if I set GOMAXPROCS=1 on my Mac, the program still works fine and exits as expected. If I set GOMAXPROCS=1 on my Mac and remove the runtime.Gosched() instruction, the behavior gets brittle: sometimes the program terminates as expected, some other times it seems to never exit the infinite loop.
You created a pathological situation that shouldn't happen in a real program, so the scheduler is not optimized to handle this. Combined with the fake time implementation in the playground, and you get far too many cycles of the producer and consumer before hitting a timeout.
The producer goroutine is creating values as fast as possible, while the consumer is always ready to receive them. With GOMAPXPROCS=1, the scheduler spends all its time bouncing between the two before it is forced to preempt the available work to check on the main goroutine, which takes longer than the playground will allow.
If we add something for the producer-consumer pair to do, we can limit the amount of time they have to monopolize the scheduler. For example, adding a time.Sleep(time.Microsecond) to the consumer will cause the playground to print 1000 values. This also goes to show how "accurate" the simulated time is in the playground, since that would not be possible with normal hardware which takes a non-zero amount time to process each message.
While an interesting case, this has little bearing on real programs.
A few notes, you can range over a channel to receive all values, you should always defer wg.Done at the start of the goroutine when possible, you can send values in the select case which allows you to actually cancel the for-select loop when the send isn't ready, and if you want the "exit consumer" message you need to send the WaitGroup to the consumer as well.
https://play.golang.org/p/WyPmpY9pFl7
func main() {
ch := make(chan int)
quit := make(chan bool)
var wg sync.WaitGroup
wg.Add(2)
go produce(ch, quit, &wg)
go consume(ch, &wg)
time.Sleep(50 * time.Microsecond)
fmt.Println("CLOSE")
close(quit)
wg.Wait()
}
func produce(ch chan int, quit chan bool, wg *sync.WaitGroup) {
defer wg.Done()
for i := 0; ; i++ {
select {
case <-quit:
close(ch)
fmt.Println("exit")
return
case ch <- i:
fmt.Println("Producer sends", i)
}
}
}
func consume(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
for i := range ch {
fmt.Println("Consumer receives", i)
time.Sleep(time.Microsecond)
}
fmt.Println("exit consumer")
return
}

Wait for go routines to finish then read from channel

How do I wait for all go routines to finish and then read all the data from a channel?
Why is this example stuck waiting for the go routines to finish?
Go Playground
package main
import (
"fmt"
"sync"
"time"
)
func doStuff(i int, wg *sync.WaitGroup, messages chan<- string) {
defer wg.Done()
time.Sleep(time.Duration(i) * time.Second)
messages <- fmt.Sprintf("Doing stuff...%d", i)
}
func doMoreStuff(i int, wg *sync.WaitGroup, messages chan<- string) {
defer wg.Done()
time.Sleep(time.Duration(i) * time.Second)
messages <- fmt.Sprintf("Doing more stuff...%d", i)
}
func main() {
var wg sync.WaitGroup
var messages = make(chan string)
for start := time.Now();; {
elapsedTime := time.Since(start)
fmt.Println(elapsedTime)
if elapsedTime > time.Duration(3) * time.Second {
fmt.Println("BREAK")
break
}
for i := 0; i < 10; i++ {
wg.Add(1)
go doStuff(i, &wg, messages)
wg.Add(1)
go doMoreStuff(i, &wg, messages)
}
time.Sleep(time.Duration(1) * time.Second)
}
fmt.Println("WAITING")
wg.Wait()
fmt.Println("DONE")
for message := range messages {
fmt.Println(message)
}
}
If you want to wait for all goroutines to finish that send messages on a channel, and you want to start reading the messages after that, then you have no choice but to use a buffered channel that can "host" all the messages that can be sent on it by the goroutines.
This isn't something practical. And even if you'd go down this path, you would be able to receive and print the messages, but the for range loop doing so would never terminate because that only terminates when it receives all messages that were sent on the channel before it was closed, but you never close the channel. You may check this "half-working" solution on the Go Playground.
Instead launch a goroutine that waits for others to finish, and then close the channel:
go func() {
fmt.Println("WAITING")
wg.Wait()
close(messages)
}()
So now you may use for range to receive messages in the main goroutine:
for message := range messages {
fmt.Println(message)
}
fmt.Println("DONE")
Try this one on the Go Playground.
This solution is still not perfect: it first has to launch all goroutines, and only then it tries to receive values, and all the launched goroutines will be blocked on their send operation (until main is ready to receive those).
A better solution may be to launch a goroutine that receives the values, preferably before the goroutines that send messages on the channel (else they would be blocked on their sends anyway):
go func() {
for message := range messages {
fmt.Println(message)
}
}()
And wait for the goroutines and close the channel at the end of main():
fmt.Println("WAITING")
wg.Wait()
close(messages)
The problem with this is that when main() ends, so does your app, it does not wait for other non-main goroutines to finish. And this means it will not wait for the "consumer" goroutine to receive the messages.
To wait for that "consumer", you may use an additional sync.WaitGroup:
var wg2 sync.WaitGroup
wg2.Add(1)
go func() {
defer wg2.Done()
for message := range messages {
fmt.Println(message)
}
}()
// And the end of `main()`:
fmt.Println("WAITING")
wg.Wait()
close(messages)
fmt.Println("DONE")
wg2.Wait()
Try this one on the Go Playground.
The example is stuck because you are using an unbuffered channel. Both go-routines are blocked waiting to write to the channel since nothing is ready to read from it.
You can use a buffered channel but then you would just be using it as a data storage. Channels are more useful for communication. The question is why do you want to wait until all the writers are finished? In the general case of an unknown (unlimited) number of writers you would no know how big to make your channel.

Panic while trying to avoid goroutine leak

I'm spawning some goroutines and want to hand them a channel to send back errors. In the parent goroutine, I select the first error and return that, or the wg.Done() condition, which is synchronized with closing a done channel.
The closing of errc is deferred to avoid a goroutine leak; but it causes a race condition.
package main
import (
"log"
"sync"
"time"
)
func f(ch chan<- bool, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(1 * time.Second)
log.Println("f sending a value")
ch <- true
log.Println("f sent a value")
}
func g(ch chan<- bool, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(2 * time.Second)
log.Println("g sending a value")
ch <- true
log.Println("g sent a value")
}
func main() {
var wg sync.WaitGroup
ch := make(chan bool)
bufc := make(chan bool, 2)
defer func() {
log.Println("Closing bufc")
close(bufc)
log.Println("Closed bufc")
time.Sleep(5 * time.Second)
}()
wg.Add(2)
go f(bufc, &wg)
go g(bufc, &wg)
go func() {
wg.Wait()
close(ch)
}()
select {
case done, ok := <-bufc:
log.Printf("bufc closed: %v %v", done, ok)
case <-ch:
log.Println("ch was closed")
}
}
Result:
❗ ~/c/scrap
(i) go run test.go
2018/05/01 20:28:03 f sending a value
2018/05/01 20:28:03 f sent a value
2018/05/01 20:28:03 bufc closed: true true
2018/05/01 20:28:03 Closing bufc
2018/05/01 20:28:03 Closed bufc
2018/05/01 20:28:04 g sending a value
panic: send on closed channel
goroutine 19 [running]:
main.g(0xc42009c000, 0xc42008a010)
/Users/yangmillstheory/code/scrap/test.go:23 +0xb2
created by main.main
/Users/yangmillstheory/code/scrap/test.go:42 +0x11e
exit status 2
Is there any way to do proper cleanup of the errc channel without causing a panic? Do I even need to close errc? Given that it's buffered, senders on that channel wouldn't block, so I would guess the answer is no?
Your error is clear enough--the channel bufc (which I assume you refer to as errc) is closed before g can send the value to it because the select statement receives only once from bufc and it's closed by defer. Instead of deferring the closing of bufc you'd have to make some synchronization, possibly using a sync.WaitGroup to make sure all the values are sent before closing it, for example by just moving close(bufc) to after wg.Wait():
go func() {
wg.Wait()
close(ch)
close(bufc)
}()
In your case since bufc is buffered you don't have to close it because it's not blocking on the receiving end, but once you have more than two goroutines sending you'll still need to close it to signal properly.
I ended up with the following implementation, which drains the buffered channel bufc and works correctly in all cases.
var (
err error
wg sync.WaitGroup
)
ch := make(chan bool)
bufc := make(chan error, 2)
wg.Add(2)
go f(bufc, &wg)
go g(bufc, &wg)
go func() {
wg.Wait()
close(ch)
close(bufc)
}()
<-ch
log.Println("Goroutines are done")
for err = range bufc {
log.Printf("Got an error: %v", err)
}
log.Println("Returning.")
return err

What's the best practice to synchronise channels and wait groups?

What's the best practice to synchronise wait groups and channels? I want to handle messages and block on a loop, and it appears that delegating the closing of the channel to another go routine seems to be a weird solution?
func Crawl(url string, depth int, fetcher Fetcher) {
ch := make(chan string)
var waitGroup sync.WaitGroup
waitGroup.Add(1)
go crawlTask(&waitGroup, ch, url, depth, fetcher)
go func() {
waitGroup.Wait()
close(ch)
}()
for message := range ch {
// I want to handle the messages here
fmt.Println(message)
}
}
func crawlTask(waitGroup *sync.WaitGroup, ch chan string, url string, depth int, fetcher Fetcher) {
defer waitGroup.Done()
if depth <= 0 {
return
}
body, urls, err := fetcher.Fetch(url)
if err != nil {
ch <- err.Error()
return
}
ch <- fmt.Sprintf("found: %s %q\n", url, body)
for _, u := range urls {
waitGroup.Add(1)
go crawlTask(waitGroup, ch, u, depth-1, fetcher)
}
}
func main() {
Crawl("http://golang.org/", 4, fetcher)
}
// truncated from https://tour.golang.org/concurrency/10 webCrawler
As an alternative to using a waitgroup and extra goroutine, you can use a separate channel for ending goroutines.
This is (also) idiomatic in Go. It involves blocking using a select control group.
So you'd have to make a new channel, typically with an empty struct as it's value (eg closeChan := make(chan struct{}) which, when closed (close(closeChan)) would end the goroutine itself.
Instead of ranging over a chan, you can use a select to block until either fed data or closed.
The code in Crawl could look something like this:
for { // instead of ranging over a to-be closed chan
select {
case message := <-ch:
// handle message
case <-closeChan:
break // exit goroutine, can use return instead
}
}
And then in crawlTask, you could close the closeChan (passed in as another parameter, like ch when you return (I figure that's when you want the other goroutine to end, and stop handling messages?)
if depth <= 0 {
close(closeChan)
return
}
Using a separate 'closer' go-routine prevents a deadlock.
If the wait/close operation were in the main go-routine before the for-range loop, it would never end, because all of the 'worker' go-routines would block in the absence of a receiver on the channel. And if it were placed in the main go-routine after the for-range loop, it would be unreachable, because the loop would block with no-one to close the channel.
This explanation was borrowed from 'The Go Programming Language' book (8.5 Looping in parallel).

Resources