"Consume or put back" Go channels [duplicate] - go

This question already has answers here:
Go channels and deadlock
(3 answers)
Closed 7 years ago.
I am trying to have two separate consumer go routines, that would filter out even and odd numbers from the input channel. This is just a toy example in order to see if it is possible to have consumers do something with the message read from the input channel if it matches certain condition, otherwise put back onto the input channel.
My current code is as follows:
package main
func filterOdd(ch chan int, out chan int) {
val := <- ch
if val % 2 == 0 {
ch <- val
} else {
out <- val
}
}
func filterEven(ch chan int, out chan int) {
val := <- ch
if val % 2 != 0 {
ch <- val
} else {
out <- val
}
}
func main() {
even := make(chan int)
odd := make(chan int)
input := make(chan int)
go filterOdd(input, odd)
go filterEven(input, even)
for i:=1; i <= 10; i++ {
input <- i
}
println("Even...")
for i := range even {
println(i)
}
println("Odd...")
for i := range odd {
println(i)
}
}
However, this produces the following output:
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.main()
/tmp/sandbox594577124/main.go:27 +0x140
goroutine 4 [chan send]:
main.filterOdd(0x10336100, 0x103360c0)
/tmp/sandbox594577124/main.go:8 +0xc0
created by main.main
/tmp/sandbox594577124/main.go:24 +0xc0
Link to the Go Playground: https://play.golang.org/p/9RIvFsGKI-

You have a deadlock because your even and odd goroutines are blocked on sending to out because nothing is reading from it. Why is nothing reading out? Because the main goroutine is blocked on sending to input because nothing is reading from it. Why is nothing reading from input? Because the two goroutines that would read from it are blocked.
Also, both filterEven and filterOdd will only run once unless you wrap their contents in for { } (but then they'll never stop until you break). On the other hand, range even will block (and range odd never happens) when nothing is left to write to even , because range over a channel only stops when the channel is closed or break is called.
In general, these aren't hard problems to solve as long as you know when you can close a channel. With what you're describing, that gets more difficult. None of the goroutines know when it's OK to close input, because all three write to it and two also read from it. You can use a sync.WaitGroup to make sure that everything you've put into the input channel has been processed before closing it. Once it's closed, the two other goroutines can use that as a signal to close their own channels and break or return to finish running.
However, writes to the in and out channels will still block until there is a corresponding read, because they are unbuffered. However, if you buffer them by specifying a size as the second argument to make, writes won't block until the channel is full. Since you know neither even or odd will have more written to them than what main send to input, you can use that as a safe buffer capacity.
Here's an example of using a WaitGroup with buffered channels for your code: https://play.golang.org/p/VXqfwUwRcx
If you don't want buffered channels, you can also use another pair of goroutines to capture the values and send them back to main as slices once finished. This way writes on the even and odd channels don't block: https://play.golang.org/p/i5vLDcsK1v
Otherwise, if don't need to print the contents of each channel all at once, you can use those two extra goroutines to read from the channels and print right away: https://play.golang.org/p/OCaUTcJkKB

Related

Why is the order of channels receiving causing/resolving a deadlock in Golang?

I've boiled my issue down to this simple example below. I am invoking a goroutine that takes two channels and sends one message to each. Then I am attempting to receive those messages further along. However, the order of channels receiving matters. If I use the same order I sent the messages, the program runs. If I switch, it does not.
I would have expected the goroutine to run independently from retrieving the messages, allowing me to receive from whichever channel I wanted to first.
I can solve this by sending messages to a single channel per goroutine (2 goroutines).
Could someone explain why there is an order dependence here and why 2 separate goroutines resolves that dependence?
package main
import "fmt"
func main() {
chanA := make(chan string)
chanB := make(chan string)
go func() {
chanA <- "el"
chanB <- "el"
}()
// if B is received before A, fatal error
// if A is received before B, completes
<-chanB
<-chanA
fmt.Println("complete")
}
You will need to buffer your channels. A buffered channel can store so many elements before it will block.
chanA := make(chan string, 1)
chanA <- "el" // This will not block
fmt.Println("Hello World")
When you do chanA <- "el" on the buffered channel above, the element gets placed into the buffer and the thread does not block. If you add a second element, it will then block as there is no room in the buffer:
chanA := make(chan string, 1)
chanA <- "el"
chanA <- "el" // <- This will block, as the buffer is full
In your example, you have a buffer of 0. So the first write to the channel is blocked, and requires another thread to read the value to unblock.
https://go.dev/play/p/6GbsVW4d0Mg
chanA := make(chan string)
go func() {
time.Sleep(time.Second)
fmt.Println("Pop:", <-chanA) // Unblock the writer
}()
chanA <- "el"
Extra knowledge
If you do not want a thread to block, you can wrap a channel insert in a select. This will ensure if the channel is full, your application does not deadlock. One cheap way of fixing this is a larger buffer...
https://go.dev/play/p/kKR-lrCO4FX
select {
case chanA <- "el":
default:
return fmt.Errorf("value not written: %s", value)
}
this is how goroutine works:
a goroutine will be blocked read/write on a channel unless if find another goroutine which write/read from the same channel.
Pay attention to read/write and write/read in the above blocked quote.
In your case, your anon goroutine(which you kicked off with go) waits to write on channelA until it finds a goroutine which reads from channelA.
The main goroutine waits to read from channelB unless it finds a goroutine that reads from it.
You can think it this way, any line written after read/write to channel won't be considered unless go finds another routine which write/read from the same channel.
So, if you change either read or write order you will not have deadlock or as you said another goroutine will do the job too.
Hope it's clear.
A write to or read from an unbuffered channel will block until there is a goroutine to write to or read from that channel. When the goroutine writes to a, it will block until the main goroutine can read from a, but main goroutine is also blocked waiting to read from b, hence deadlock.

Why does my `for-select` statement keeps receiving from my channel even after I close it?

I have the following code:
package main
import (
"fmt"
"time"
)
func main() {
ch := make(chan int)
ch2 := make(chan int)
go func(c chan int, c2 chan int) {
for {
select {
case v := <-c:
fmt.Println(v)
case v := <-c2:
fmt.Println(v)
default:
}
}
}(ch, ch2)
ch <- 1
close(ch)
close(ch2)
time.Sleep(10 * time.Second)
}
When I run this, it prints 1 to the stdout, and then keeps printing 0. Why is this?
I know I can check whether the channel is closed in my goroutine, but I just want to know the reason for this.
Also, suppose I want to exit from a goroutine after all(multiple) channels are closed, is that possible? I was assuming that once all channels are closed, I could potentially exit from the goroutine in the default case after all channels are closed
That's the expected behavior per Spec: Receive operator:
A receive operation on a closed channel can always proceed immediately, yielding the element type's zero value after any previously sent values have been received.
See How does a non initialized channel behave?
If you want to terminate a loop once all values have been received from a channel (that were sent on it before it was closed), use the for ... range construct, for example:
c := make(chan int) // Initialize some channel
for v := range c {
fmt.Println("Received:", v)
}
If you have multiple channels and you want to receive from all, you may use multiple goroutines, each having a for range for a designated channel.
Another solution is:
A function can read from multiple inputs and proceed until all are closed by multiplexing the input channels onto a single channel that's closed when all the inputs are closed. This is called fan-in.
Read more about it in The Go Blog
Go Concurrency Patterns: Pipelines and cancellation: Fan-out- fain-in.
The Go Programming Language Specification
Close
After calling close, and after any previously sent values have been
received, receive operations will return the zero value for the
channel's type without blocking.
Receive operator
A receive operation on a closed channel can always proceed
immediately, yielding the element type's zero value after any
previously sent values have been received.

golang channels deadlock before receiving values

I'm not understanding why this doesn't work https://play.golang.org/p/_ALPii0pXV6 but this https://play.golang.org/p/vCOjAr-o54e works.
As I understand the goroutine asynchronously sends to value true to a and 12 to b. While in the main function, a is blocked, until it receives a value. Why is it that when I rearrange it to have b is blocked before a, it results in a deadlock?
Go channels are unbuffered by default. That means that it cannot send on a channel until the receiver is reading the channel. This is actually the Go preferred mode. It's more efficient than buffered channels in most cases.
What that means for your first code is that the goroutine cannot proceed to write to channel b until it completes the write to channel a. It cannot do that until the main goroutine reads a.
Go by Example explains that, by default, channel sending and receiving waits until both the sending routine and the receiving routine are ready. This blocking is made obvious by the following example:
func main() {
ch := make(chan int)
ch <- 1
fmt.Println(<-ch)
}
This code results in a deadlock because the only goroutine (the main one) is stuck at ch <- 1, waiting for another goroutine to receive. Little does it know that we are expecting it to be the receiver at the next line.
This explains why your first example does not work, because the other goroutine doesn't send on b until its send operation on a has completed. But the main routine won't receive on a until it's received on b! So both are stuck waiting forever.
To read more about this kind of operation (called a synchronous operation), check out this explanation.
If you rewrite the code the way it is going to be executed sequentially then it becomes clearer what's going on.
Original code:
func main() {
a := make(chan bool)
b := make(chan int64)
go func(a chan bool, b chan int64) {
fmt.Println("Here1")
a <- true
b <- 12
} (a,b)
fmt.Println("Here2")
fmt.Println(fmt.Sprintf("%d", <-b))
fmt.Println(fmt.Sprintf("%v", <-a))
}
Close representation of sequential execution of the same code:
a := make(chan bool)
b := make(chan int64)
fmt.Println("Here2") // Prints
// Pass control to new goroutine
fmt.Println("Here1")
a <- true // Write to channel a and block goroutine here and pass control to main
fmt.Println(fmt.Sprintf("%d", <-b)) // Tries to read from b but nothing has been written to it so blocks. At this point all your goroutines are blocked hence the deadlock.
fmt.Println(fmt.Sprintf("%v", <-a)) // doesn't even reach here.
b <- 12
}

Spread sequential tests into 4 go routines and terminate all if one fails

Suppose I have a simple loop which does sequential tests like this.
for f := 1; f <= 1000; f++ {
if doTest(f) {
break
}
}
I loop through range of numbers and do a test for each number. If test fails for one number, I break and exit the main thread. Simple enough.
Now, how do correctly feed the test numbers in say four or several go routines. Basically, I want to test the numbers from 1 to 1000 in batches of 4 (or whatever number of go routines is).
Do I create 4 routines reading from one channel and feed the numbers sequentially into this channel? Or do I make 4 routines with an individual channel?
And another question. How do I stop all 4 routines if one of them fails the test? I've been reading some texts on channels but I cannot put the pieces together.
You can create a producer/consumer system: https://play.golang.org/p/rks0gB3aDb
func main() {
ch := make(chan int)
clients := 4
// make it buffered, so all clients can fail without hanging
notifyCh := make(chan struct{}, clients)
go produce(100, ch, notifyCh)
var wg sync.WaitGroup
wg.Add(clients)
for i := 0; i < clients; i++ {
go func() {
consumer(ch, notifyCh)
wg.Done()
}()
}
wg.Wait()
}
func consumer(in chan int, notifyCh chan struct{}) {
fmt.Printf("Start consumer\n")
for i := range in {
<-time.After(100 * time.Millisecond)
if i == 42 {
fmt.Printf("%d fails\n", i)
notifyCh <- struct{}{}
return
} else {
fmt.Printf("%d\n", i)
}
}
fmt.Printf("Consumer stopped working\n")
}
func produce(N int, out chan int, notifyCh chan struct{}) {
for i := 0; i < N; i++ {
select {
case out <- i:
case <-notifyCh:
close(out)
return
}
}
close(out)
}
The producer pushes numbers from 0 to 99 to the channel, the consumer consumes until the channel is closed. In main we create 4 clients and add them to a waitgroup to reliably check if every goroutine returned.
Every consumer can signal on the notifyCh, the producer stops working and no further numbers are generated, therefor all consumers return after their current number.
There's also an option to create 4 go routines, wait for all of them to return, start the next 4 go routines. But this adds quite an overhead on waiting.
Since you mentioned prime numbers, here's a really cool prime seive: https://golang.org/doc/play/sieve.go
Whether you will create one channel common or a channel per routines depend on what you want.
If you want only put some numbers (or more general - requests) inside and you don't care which goroutine serve that, than of course is better to share a channel. In case when you want for example first 250 request to be served by goroutine1, than of course you cannot share a channel.
For channel is a good practice use it as input or output. And the simples thing how sender can sent, that he is finished is close the channel. Good article about that is https://blog.golang.org/pipelines
What is not mentiond in the question - is you need also another channel (or channels) or or any other communication primitive to get results. And here is the channel most interesting than to feeding.
What information should be sent - it should be sent, a bool after every doTest, or just know when everthing was done (it this case neither bool is not necessary just close a channel)?
If you prefer program at first fail. Than I would prefer use buffered shared channel to feed the numbers. Don't forget to close it, when all numbers will be feed.
And another unbuffered chan to let main thread know, that tests are done. It can be channel, there you only put the number, where test failed, or if you also want a positive result - channel of struct containing number and result, or any other informantion returned from doTest.
Very good article about channel is also http://dave.cheney.net/2014/03/19/channel-axioms
Each of your four goroutines can report a failure (by sending error and closing channel). But gotcha is what goroutines should do, when all numbers passed and feeding channel is closed. And about that is also nice article http://nathanleclaire.com/blog/2014/02/15/how-to-wait-for-all-goroutines-to-finish-executing-before-continuing/

How does make(chan bool) behave differently from make(chan bool, 1)?

My question arises from trying to read a channel, if I can, or write it, if I can, using a select statement.
I know that channels specified like make(chan bool, 1) are buffered, and part of my question is what is the difference between that, and make(chan bool) -- which this page says is the same thing as make(chan bool, 0) --- what is the point of a channel that can fit 0 values in it?
See playground A:
chanFoo := make(chan bool)
for i := 0; i < 5; i++ {
select {
case <-chanFoo:
fmt.Println("Read")
case chanFoo <- true:
fmt.Println("Write")
default:
fmt.Println("Neither")
}
}
A output:
Neither
Neither
Neither
Neither
Neither
(Removing the default case results in a deadlock!!)
Now see playground B:
chanFoo := make(chan bool, 1) // the only difference is the buffer size of 1
for i := 0; i < 5; i++ {
select {
case <-chanFoo:
fmt.Println("Read")
case chanFoo <- true:
fmt.Println("Write")
default:
fmt.Println("Neither")
}
}
B output:
Write
Read
Write
Read
Write
In my case, B output is what I want. What good are unbuffered channels? All the examples I see on golang.org appear to use them to send one signal/value at a time (which is all I need) -- but as in playground A, the channel never gets read or written. What am I missing here in my understanding of channels?
what is the point of a channel that can fit 0 values in it
First I want to point out that the second parameter here means buffer size, so that is simply a channel without buffers (un-buffered channel).
Actually that's the reason why your problem is generated. Un-buffered channels are only writable when there's someone blocking to read from it, which means you shall have some coroutines to work with -- instead of this single one.
Also see The Go Memory Model:
A receive from an unbuffered channel happens before the send on that channel completes.
An unbuffered channel means that read and writes from and to the channel are blocking.
In a select statement:
the read would work if some other goroutine was currently blocked in writing to the channel
the write would work if some other goroutine was currently blocked in reading to the channel
otherwise the default case is executed, which happens in your case A.
Unbuffered channels (created without capacity) will block the sender until somebody can read from them, so to make it work as you expect, you should use two goroutines to avoid the deadlock in the same thread.
For example, with this code: http://play.golang.org/p/KWJ1gbdSqf
It also includes a mechanism for the main func to detect when both goroutines have finished.
package main
import "fmt"
func send(out, finish chan bool) {
for i := 0; i < 5; i++ {
out <- true
fmt.Println("Write")
}
finish <- true
close(out)
}
func recv(in, finish chan bool) {
for _ = range in {
fmt.Println("Read")
}
finish <- true
}
func main() {
chanFoo := make(chan bool)
chanfinish := make(chan bool)
go send(chanFoo, chanfinish)
go recv(chanFoo, chanfinish)
<-chanfinish
<-chanfinish
}
It won't show the alternate Read and Write, because as soon as send writes to the channel, it is blocked, before it can print the "Write", so the execution moves to recv that receives the channel and prints "Read". It will try to read the channel again but it will be blocked and the execution moves to send. Now send can write the first "Write", send to the channel (without blocking because now there is a receiver ready) and write the second "Write". In any case, this is not deterministic and the scheduler may move the execution at any point to any other running goroutine (at least in the latest 1.2 release).

Resources