Channel buffer taking one more value than expected in Go - go

I thought that channels in Go only hold 1 value by default unless the buffer size if specified. I read that here. But when I run this:
func main (){
for i := range numGen(6) {
log.Println("taking from channel", i)
}
}
func numGen(num int) chan int {
c := make(chan string)
go func() {
for i := 0; i < num; i++ {
log.Println("passing to channel", i)
c <- i
}
close(c)
}
return c
}
my output is:
2017/06/13 18:09:08 passing to channel 0
2017/06/13 18:09:08 passing to channel 1
2017/06/13 18:09:08 taking from channel 0
2017/06/13 18:09:08 taking from channel 1
2017/06/13 18:09:08 passing to channel 2
2017/06/13 18:09:08 passing to channel 3
2017/06/13 18:09:08 taking from channel 2
2017/06/13 18:09:08 taking from channel 3
2017/06/13 18:09:08 passing to channel 4
2017/06/13 18:09:08 passing to channel 5
2017/06/13 18:09:08 taking from channel 4
2017/06/13 18:09:08 taking from channel 5
which shows that the channel is holding 2 values at a time. Specifying a buffer size like this
c := make(chan int, 0)
does nothing. Any way I could make it only hold 1, value, not 2?

which shows that the channel is holding 2 values at a time.
Thats not the case. This is how the code executes:
The main goroutine blocks on a read on the channel
The second goroutine writes to the channel and continues executing.
The second goroutine blocks at the second write attempt because noone is reading
The main goroutine continues executing, prints the read number
The main goroutine reads another number since someone is writing to it
The main goroutine prints the read number and blocks on the next read
The second goroutine continues executing at step 2.
There is no buffer, just concurrency.

package main
import (
"log"
)
func main() {
seq := make(chan bool)
for i := range numGen(6, seq) {
<-seq
log.Println("taking from channel", i)
}
}
func numGen(num int, seq chan bool) chan int {
c := make(chan int)
go func() {
for i := 0; i < num; i++ {
c <- i
log.Println("passing to channel", i)
seq <- true // 要保证顺序,这里发送一个信号量。
}
close(c)
}()
return c
}

Related

Gracefully closing channel and not sending on closed channel

I am new to Golang concurrency and have been working to understand this piece of code mentioned below.
I witness few things which I am unable to explain why it happens:
when using i smaller than equal to 100000 for i <= 100000 { in main function, it sometimes prints different values for nResults and countWrites (in last two statements)
fmt.Printf("number of result writes %d\n", nResults) fmt.Printf("Number of job writes %d\n", jobWrites)
when using i more than 1000000 it gives panic: send on closed channel
How can I make sure that the values send to jobs is not on closed channel and later after all values are received in results we can close the channel without deadlock?
package main
import (
"fmt"
"sync"
)
func worker(wg *sync.WaitGroup, id int, jobs <-chan int, results chan<- int, countWrites *int64) {
defer wg.Done()
for j := range jobs {
*countWrites += 1
go func(j int) {
if j%2 == 0 {
results <- j * 2
} else {
results <- j
}
}(j)
}
}
func main() {
wg := &sync.WaitGroup{}
jobs := make(chan int)
results := make(chan int)
var i int = 1
var jobWrites int64 = 0
for i <= 10000000 {
go func(j int) {
if j%2 == 0 {
i += 99
j += 99
}
jobWrites += 1
jobs <- j
}(i)
i += 1
}
var nResults int64 = 0
for w := 1; w < 1000; w++ {
wg.Add(1)
go worker(wg, w, jobs, results, &nResults)
}
close(jobs)
wg.Wait()
var sum int32 = 0
var count int64 = 0
for r := range results {
count += 1
sum += int32(r)
if count == nResults {
close(results)
}
}
fmt.Println(sum)
fmt.Printf("number of result writes %d\n", nResults)
fmt.Printf("Number of job writes %d\n", jobWrites)
}
Quite a few problems in your code.
Sending on closed channel
One general principle of using Go channels is
don't close a channel from the receiver side and don't close a channel if the channel has multiple concurrent senders
(https://go101.org/article/channel-closing.html)
The solution for you is simple: don't have multiple concurrent senders, and then you can close the channel from the sender side.
Instead of starting millions of separate goroutine for each job you add to the channel, run one goroutine that executes the whole loop to add all jobs to the channel. And close the channel after the loop. The workers will consume the channel as fast as they can.
Data races by modifying shared variables in multiple goroutines
You're modifying two shared variables without taking special steps:
nResults, which you pass to the countWrites *int64 in the worker.
i in the loop that writes to the jobs channel: you're adding 99 to it from multiple goroutines, making it unpredictable how many values you actually write to the jobs channel
To solve 1, there are many options, including using sync.Mutex. However since you're just adding to it, the easiest solution is to use atomic.AddInt64(countWrites, 1) instead of *countWrites += 1
To solve 2, don't use one goroutine per write to the channel, but one goroutine for the entire loop (see above)

buffered channel not blocking for write when reached its capacity

https://play.golang.org/p/5A-dnZVy2aA
func closeChannel(stream chan int) {
fmt.Println("closing the channel")
close(stream)
}
func main() {
chanOwner := func() <-chan int {
resultStream := make(chan int, 5)
go func() {
defer closeChannel(resultStream)
for i := 0; i <= 5; i++ {
resultStream <- i
}
}()
return resultStream
}
resultStream := chanOwner()
for result := range resultStream { //this blocks until the channel is closed
fmt.Printf("Received: %d\n", result)
}
fmt.Println("done receiving")
}
Output of program
closing the channel
Received: 0
Received: 1
Received: 2
Received: 3
Received: 4
Received: 5
done receiving
why is the closing the channel statement printed before any Received in the above program. since the channel is buffered with a capacity of 5, and we are inserting 6 elements into it, I was expecting it to block at the resultStream <- i before a value was read and cleared up space in the buffer.
The generator goroutine fills the channel to its capacity and blocks. The receiver for loops receives the first item from the channel, which enables the generator goroutine again. The generator goroutine runs to completion before the receiver for-loop can print the Received message, printing the closing the channel message. Then the receiver loop receives all remaining messages.

Incomprehension of buffered channels described in the "Concurrency in Go" book

I read the book "Concurrency in Go" written by Katherine Cox-Buday and I don't understand comments for examples of buffered channels.
The author says:
if a goroutine making writes to a channel has knowledge of how many writes it will make,
it can be useful to create a buffered channel whose capacity is the number of writes to be made
It sounds clear, but examples are confusing.
First example - source: https://github.com/kat-co/concurrency-in-go-src/blob/master/gos-concurrency-building-blocks/channels/fig-using-buffered-chans.go#L13
var stdoutBuff bytes.Buffer // <1>
defer stdoutBuff.WriteTo(os.Stdout) // <2>
intStream := make(chan int, 4) // <3>
go func() {
defer close(intStream)
defer fmt.Fprintln(&stdoutBuff, "Producer Done.")
for i := 0; i < 5; i++ {
fmt.Fprintf(&stdoutBuff, "Sending: %d\n", i)
intStream <- i
}
}()
for integer := range intStream {
fmt.Fprintf(&stdoutBuff, "Received %v.\n", integer)
}
The line with comment <3> has the following explanation:
Here we create a buffered channel with a capacity of one.
There is 4, not 1. Is it a mistake?
Second example - channel ownership, source: https://github.com/kat-co/concurrency-in-go-src/blob/master/gos-concurrency-building-blocks/channels/fig-chan-ownership.go
chanOwner := func() <-chan int {
resultStream := make(chan int, 5) // <1>
go func() { // <2>
defer close(resultStream) // <3>
for i := 0; i <= 5; i++ {
resultStream <- i
}
}()
return resultStream // <4>
}
The line marked as <1>, has the following comment:
Since we know we will produce six results, we create a buffered channel of five
so that the goroutine can complete as quickly as possible.
I completely don't understand this comment. Goroutine will be blocked, because the channel has the capacity of 5 messages and there will be produced 6 messages, so it will wait until a receiver takes the first message.
Since we know we will produce six results, we create a buffered channel of five so that the goroutine can complete as quickly as possible.
You are correct that the goroutine will block until a value is received.
It doesn't make sense to create a channel with capacity one less than the number of values to be sent. The goroutine can be eliminated if the channel capacity is equal to or greater than the number of values:
chanOwner := func() <-chan int {
resultStream := make(chan int, 6)
for i := 0; i < cap(resultStream); i++ {
resultStream <- i
}
close(resultStream)
return resultStream
}()
or this by eliminating the anonymous function:
chanOwner := make(chan int, 6)
for i := 0; i < cap(chanOwner); i++ {
resultStream <- i
}
close(chanOwner)
Yes, it sounds like this book needs a better editor!
the channel capacity is indeed indicated as the 2nd argument to make:
intStream := make(chan int, 4) // buffered-channel of capacity 4 (not 1!)
If no reads are done on the channel - then yes the writing goroutine will write 5 times to the buffered channel (of capacity 5) without issue (i.e. without blocking). The 6th write will indeed block - and the goroutine will not return until the buffer congestion reduces.
If some other goroutine does read from the channel - even just once - then the buffer frees up and the writer goroutine will be able to complete the final write.
There is 4, not 1. Is it a mistake?
It seems a typo. As clearly stated in the documentation, the second argument to make is the channel capacity:
Channel: The channel's buffer is initialized with the specified
buffer capacity. If zero, or the size is omitted, the channel is unbuffered.
Therefore make(chan int, 4) is a chan with capacity 4.
Goroutine will be blocked, because the channel has the capacity of 5 messages and there will be produced 6 messages, so it will wait until a receiver takes the first message.
Correct, the declared chan has capacity 5, and if there's no receiver, the sixth and last send operation will indeed block, as the channel buffer is full.
Assuming good faith on all sides, the book probably missed a round of proofreading.

Deadlock with Multiple goroutines with multiple channels

I am working on a sample program to print sum of odd and sum of even number between 1 to 100 using goroutine with multiple channels.
you can find my code
here
output
sum of even number = 2550
sum of odd number = 2500
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan receive]:
main.print(0x434100, 0x11db7c)
/tmp/sandbox052575152/main.go:18 +0xc0
main.main()
/tmp/sandbox052575152/main.go:14 +0x120
The code works but with deadlock.
I am not sure what is wrong in my code
We can iterate through values sent over a channel. To break such iteration channel needs to be closed explicitly. Otherwise range would block forever in the same way as for nil channel. In your code you did't close the sum(for print function sumValues channel) channel. That's why following function will be blocked for forever.
func print(sumValues <-chan string ){
for val := range sumValues {
fmt.Println(val)
}
}
So you have to close the sum channel in the doSum function after all the go routine in the doSum function are complete (otherwise sum channel might be closed before go routines are complete). You can use sync.WaitGroup to do that. See the updated doSum function below:
func doSum(sum chan<- string, oddChan <-chan int, evenChan <-chan int) {
var waitGroup sync.WaitGroup
waitGroup.Add(2) // Must wait for 2 calls to 'done' before moving on
go func(sum chan<- string) {
s1 := 0
for val := range oddChan {
s1 += val
}
sum <- fmt.Sprint("sum of odd number = ", s1)
waitGroup.Done()
}(sum)
go func(sum chan<- string) {
s1 := 0
for val := range evenChan {
s1 += val
}
sum <- fmt.Sprint("sum of even number = ", s1)
waitGroup.Done()
}(sum)
// Waiting for all goroutines to exit
waitGroup.Wait()
// all goroutines are complete now close the sum channel
close(sum)
}

Goroutines channels and "stopping short"

I'm reading/working through Go Concurrency Patterns: Pipelines and cancellation, but i'm having trouble understanding the Stopping short section. We have the following functions:
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n
}
close(out)
}()
return out
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func merge(cs ...<-chan int) <-chan int {
var wg sync.WaitGroup
out := make(chan int, 1) // enough space for the unread inputs
// Start an output goroutine for each input channel in cs. output
// copies values from c to out until c is closed, then calls wg.Done.
output := func(c <-chan int) {
for n := range c {
out <- n
}
wg.Done()
}
wg.Add(len(cs))
for _, c := range cs {
go output(c)
}
// Start a goroutine to close out once all the output goroutines are
// done. This must start after the wg.Add call.
go func() {
wg.Wait()
close(out)
}()
return out
}
func main() {
in := gen(2, 3)
// Distribute the sq work across two goroutines that both read from in.
c1 := sq(in)
c2 := sq(in)
// Consume the first value from output.
out := merge(c1, c2)
fmt.Println(<-out) // 4 or 9
return
// Apparently if we had not set the merge out buffer size to 1
// then we would have a hanging go routine.
}
Now, if you notice line 2 in merge, it says we make the out chan with buffer size 1, because this is enough space for the unread inputs. However, I'm almost positive that we should allocate a chan with buffer size 2. In accordance with this code sample:
c := make(chan int, 2) // buffer size 2
c <- 1 // succeeds immediately
c <- 2 // succeeds immediately
c <- 3 // blocks until another goroutine does <-c and receives 1
Because this section implies that a chan of buffer size 3 would not block. Can anyone please clarify/assist my understanding?
The program sends two values to the channel out and reads one value from the channel out. One of the values is not received.
If the channel is unbuffered (capacity 0), then one of the sending goroutines will block until the program exits. This is a leak.
If the channel is created with a capacity of 1, then both goroutines can send to the channel and exit. The first value sent to the channel is received by main. The second value remains in the channel.
If the main function does not receive a value from the channel out, then a channel of capacity 2 is required to prevent the goroutines from blocking indefinitely.

Resources