Incomprehension of buffered channels described in the "Concurrency in Go" book - go

I read the book "Concurrency in Go" written by Katherine Cox-Buday and I don't understand comments for examples of buffered channels.
The author says:
if a goroutine making writes to a channel has knowledge of how many writes it will make,
it can be useful to create a buffered channel whose capacity is the number of writes to be made
It sounds clear, but examples are confusing.
First example - source: https://github.com/kat-co/concurrency-in-go-src/blob/master/gos-concurrency-building-blocks/channels/fig-using-buffered-chans.go#L13
var stdoutBuff bytes.Buffer // <1>
defer stdoutBuff.WriteTo(os.Stdout) // <2>
intStream := make(chan int, 4) // <3>
go func() {
defer close(intStream)
defer fmt.Fprintln(&stdoutBuff, "Producer Done.")
for i := 0; i < 5; i++ {
fmt.Fprintf(&stdoutBuff, "Sending: %d\n", i)
intStream <- i
}
}()
for integer := range intStream {
fmt.Fprintf(&stdoutBuff, "Received %v.\n", integer)
}
The line with comment <3> has the following explanation:
Here we create a buffered channel with a capacity of one.
There is 4, not 1. Is it a mistake?
Second example - channel ownership, source: https://github.com/kat-co/concurrency-in-go-src/blob/master/gos-concurrency-building-blocks/channels/fig-chan-ownership.go
chanOwner := func() <-chan int {
resultStream := make(chan int, 5) // <1>
go func() { // <2>
defer close(resultStream) // <3>
for i := 0; i <= 5; i++ {
resultStream <- i
}
}()
return resultStream // <4>
}
The line marked as <1>, has the following comment:
Since we know we will produce six results, we create a buffered channel of five
so that the goroutine can complete as quickly as possible.
I completely don't understand this comment. Goroutine will be blocked, because the channel has the capacity of 5 messages and there will be produced 6 messages, so it will wait until a receiver takes the first message.

Since we know we will produce six results, we create a buffered channel of five so that the goroutine can complete as quickly as possible.
You are correct that the goroutine will block until a value is received.
It doesn't make sense to create a channel with capacity one less than the number of values to be sent. The goroutine can be eliminated if the channel capacity is equal to or greater than the number of values:
chanOwner := func() <-chan int {
resultStream := make(chan int, 6)
for i := 0; i < cap(resultStream); i++ {
resultStream <- i
}
close(resultStream)
return resultStream
}()
or this by eliminating the anonymous function:
chanOwner := make(chan int, 6)
for i := 0; i < cap(chanOwner); i++ {
resultStream <- i
}
close(chanOwner)

Yes, it sounds like this book needs a better editor!
the channel capacity is indeed indicated as the 2nd argument to make:
intStream := make(chan int, 4) // buffered-channel of capacity 4 (not 1!)
If no reads are done on the channel - then yes the writing goroutine will write 5 times to the buffered channel (of capacity 5) without issue (i.e. without blocking). The 6th write will indeed block - and the goroutine will not return until the buffer congestion reduces.
If some other goroutine does read from the channel - even just once - then the buffer frees up and the writer goroutine will be able to complete the final write.

There is 4, not 1. Is it a mistake?
It seems a typo. As clearly stated in the documentation, the second argument to make is the channel capacity:
Channel: The channel's buffer is initialized with the specified
buffer capacity. If zero, or the size is omitted, the channel is unbuffered.
Therefore make(chan int, 4) is a chan with capacity 4.
Goroutine will be blocked, because the channel has the capacity of 5 messages and there will be produced 6 messages, so it will wait until a receiver takes the first message.
Correct, the declared chan has capacity 5, and if there's no receiver, the sixth and last send operation will indeed block, as the channel buffer is full.
Assuming good faith on all sides, the book probably missed a round of proofreading.

Related

How to signal if a value has been read from a channel in Go

I am reading values that are put into a channel ch via an infinite for. I would like some way to signal if a value has been read and operated upon (via the sq result) and add it to some sort of counter variable upon success. That way I have a way to check if my channel has been exhausted so that I can properly exit my infinite for loop.
Currently it is incrementing regardless if a value was read, thus causing it to exit early when the counter == num. I only want it to count when the value has been squared.
EDIT: Another approach I have tested is to receive the ok val out of the channel upon reading and setting val and then check if !ok { break }. However I receive a deadlock panic since the for did has not properly break. Example here: https://go.dev/play/p/RYNtTix2nm2
package main
import "fmt"
func main() {
num := 5
// Buffered channel with 5 values.
ch := make(chan int, num)
defer close(ch)
for i := 0; i < num; i++ {
go func(val int) {
fmt.Printf("Added value: %d to the channel\n", val)
ch <- val
}(i)
}
// Read from our channel infinitely and increment each time a value has been read and operated upon
counter := 0
for {
// Check our counter and if its == num then break the infinite loop
if counter == num {
break
}
val := <-ch
counter++
go func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * i
fmt.Println(sq)
}(val)
}
}
let me try to help you in figuring out the issue.
Reading issue
The latest version of the code you put in the question is working except when you're about to read values from the ch channel. I mean with the following code snippet:
go func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * I
fmt.Println(sq)
}(val)
In fact, it's not needed to spawn a new goroutine for each read. You can consume the messages as soon as they arrived in the ch channel. This is possible due to writing done inside goroutines. Thanks to them, the code can go ahead and reach the reading phase without being blocked.
Buffered vs unbuffered
In this scenario, you used a buffered channel with 5 slots for data. However, if you're relying on the buffered channel you should signal when you finish sending data to it. This is done with a close(ch) invocation after all of the Go routines finish their job. If you use an unbuffered channel it's fine to invoke defer close(ch) next to the channel initialization. In fact, this is done for cleanup and resource optimization tasks. Back to your example, you can change the implementation to use unbuffered channels.
Final Code
Just to recap, the two small changes that you've to do are:
Use an unbuffered channel instead of a buffered one.
Do Not use a Go routine when reading the messages from the channel.
Please be sure to understand exactly what's going on. Another tip can be to issue the statement: fmt.Println("NumGoroutine:", runtime.NumGoroutine()) to print the exact number of Go routines running in that specific moment.
The final code:
package main
import (
"fmt"
"runtime"
)
func main() {
num := 5
// Buffered channel with 5 values.
ch := make(chan int)
defer close(ch)
for i := 0; i < num; i++ {
go func(val int) {
fmt.Printf("Added value: %d to the channel\n", val)
ch <- val
}(i)
}
fmt.Println("NumGoroutine:", runtime.NumGoroutine())
// Read from our channel infinitely and increment each time a value has been read and operated upon
counter := 0
for {
// Check our counter and if its == num then break the infinite loop
if counter == num {
break
}
val := <-ch
counter++
func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * i
fmt.Println(sq)
}(val)
}
}
Let me know if this helps you, thanks!
package main
import "fmt"
func main() {
c := make(chan int)
done := make(chan bool)
go func() {
for i := 0; i < 10; i++ {
c <- i
}
close(c)
}()
go func() {
for i := range c {
fmt.Println(i)
done <- true
}
close(done)
}()
for i := 0; i < 10; i++ {
<-done
}
}
In this example, the done channel is used to signal that a value has been read from the c channel. After each value is read from c, a signal is sent on the done channel. The main function blocks on the done channel, waiting for a signal before continuing. This ensures that all values from c have been processed before the program terminates.

How to recover the input of multiple go routines called in a loop

I have a loop throwing multiple go routines, they call a function that makes a http get petition and calculate and object.
I want to recover the result of all those routines.
I tried using channels, but hey are empty, even if I force wait for all the routines to be done.
This is the code that starts the routines:
func main() {
pairs := getPairs() //Returns an array of strings
c := make(chan result)
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
fmt.Println(len(c))
time.Sleep(5000 * time.Millisecond)
fmt.Println(len(c))
}
And at the end of getScore() I do this, c being the name of the channel in the function and res the result of the function
c <- res
The length of the channel is 0 in both prints.
What's the best way to get the result of the functions?
A channel is a synchronization prototype against a shared memory (in simple point of view). A buffered channel has a length but not a regular channel. Buffered channel is useful in little bit cases but not as a general approaches.
The simplest way to Just add a loop by range of pair or len of pairs:
// start processing
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
// getting a result
for i:=0; i<len(pairs); i ++ {
result := <-c
// process a result value
}
Or another way is collecting result in another grouting:
// result and sync variable
var (
wait sync.WaitGroup
result int32
)
// start processing
for _, product := range pairs {
wait.Add(1)
go getScore(product.Symbol, 1, c)
go func() {
defer wait.Done()
// simple accumulate or maybe more complicated actions
atomic.AddInt32(&result, <-c)
}()
}
// wait finishing
wait.Wait()
c := make(chan result)
Creates an unbuffered channel. Therefore send statements, such as
c <- res
cannot proceed until another goroutine is attempting a receive operation.
In other words, execute the number of receive operations in your main goroutine matching the number of sends that will be attempted from other goroutines. Like this:
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
for x := 0; x < len(pairs); x++ {
fmt.Println(<-c)
}
See the Go Tour section on channels, and the Effective Go section on channels for more information.

Write to channel blocked forever

I am stuck in a strange situation where the write operation to the channel never happens.
package main
import (
"fmt"
"time"
)
func main() {
c := make(chan int)
s := make(chan bool)
k := make(chan bool)
fmt.Println("start")
go func() {
fmt.Println("routine 1")
s <- true
}()
go func() {
fmt.Println("routine 2")
for {
select {
case <-s :
for i := 0; i < 3; i++ {
fmt.Println("before")
c <- i
fmt.Println("after")
}
case <- k :
fmt.Println("k ready")
break
}
}
}()
go func() {
fmt.Println("routine 3")
for {
select {
case x := <- c :
fmt.Println("x=", x)
k <- true
fmt.Println("k done")
}
}
}()
time.Sleep(1000 * time.Millisecond)
}
And here is the output:
start
routine 1
routine 2
before
routine 3
x= 0
after
before
I wonder why the write to the channel k blocks, but the log statement fmt.Println("k ready") is never printed.
Here is what I think :
go routine 1 writes true to channel s
go routine 2 writes 0 to
channel c and waits because buffer size is 0, it will not be able to
write '1' to it unless someone reads channel c
go routine 3 comes into the picture, reads channel c (now go routine 2
can write to c once go routine 2 resumes) prints the value of x. NOW IT SHOULD BE ABLE TO WRITE TO CHANNEL K but that is not happening
According to me it should be able to write to channel k then case 2 of goroutine should execute and print "k ready"
Can anyone explain me why write to the channel blocked?
As a fix I know I can increase the buffer size of channel c and everything will get printed but I am not interested in fixing this, instead I want to understand this scenario.
A nice blog to understand the above case.
You have a deadlock.
goroutine 1 writes to s then quits
goroutine 2 reads from s, and writes to c
goroutine 3 reads from c, and writes to k, and this blocks because nothing is reading from k, because goroutine 2 is blocked in the write to k above.
goroutine 2 writes to c again which blocks as goroutine 3 is still trying to write to k and thus is not reading from c
Contrary to what you say, you don't have a buffer size of 1. You have a buffer size of zero (i.e. an unbuffered channel), so a write will block until something reads. This is probably the source of your misunderstanding. Per the language specification:
A new, initialized channel value can be made using the built-in function make, which takes the channel type and an optional capacity as arguments:
make(chan int, 100)
The capacity, in number of elements, sets the size of the buffer in the channel. If the capacity is zero or absent, the channel is unbuffered and communication succeeds only when both a sender and receiver are ready. Otherwise, the channel is buffered and communication succeeds without blocking if the buffer is not full (sends) or not empty (receives). A nil channel is never ready for communication.

Goroutines channels and "stopping short"

I'm reading/working through Go Concurrency Patterns: Pipelines and cancellation, but i'm having trouble understanding the Stopping short section. We have the following functions:
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n
}
close(out)
}()
return out
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func merge(cs ...<-chan int) <-chan int {
var wg sync.WaitGroup
out := make(chan int, 1) // enough space for the unread inputs
// Start an output goroutine for each input channel in cs. output
// copies values from c to out until c is closed, then calls wg.Done.
output := func(c <-chan int) {
for n := range c {
out <- n
}
wg.Done()
}
wg.Add(len(cs))
for _, c := range cs {
go output(c)
}
// Start a goroutine to close out once all the output goroutines are
// done. This must start after the wg.Add call.
go func() {
wg.Wait()
close(out)
}()
return out
}
func main() {
in := gen(2, 3)
// Distribute the sq work across two goroutines that both read from in.
c1 := sq(in)
c2 := sq(in)
// Consume the first value from output.
out := merge(c1, c2)
fmt.Println(<-out) // 4 or 9
return
// Apparently if we had not set the merge out buffer size to 1
// then we would have a hanging go routine.
}
Now, if you notice line 2 in merge, it says we make the out chan with buffer size 1, because this is enough space for the unread inputs. However, I'm almost positive that we should allocate a chan with buffer size 2. In accordance with this code sample:
c := make(chan int, 2) // buffer size 2
c <- 1 // succeeds immediately
c <- 2 // succeeds immediately
c <- 3 // blocks until another goroutine does <-c and receives 1
Because this section implies that a chan of buffer size 3 would not block. Can anyone please clarify/assist my understanding?
The program sends two values to the channel out and reads one value from the channel out. One of the values is not received.
If the channel is unbuffered (capacity 0), then one of the sending goroutines will block until the program exits. This is a leak.
If the channel is created with a capacity of 1, then both goroutines can send to the channel and exit. The first value sent to the channel is received by main. The second value remains in the channel.
If the main function does not receive a value from the channel out, then a channel of capacity 2 is required to prevent the goroutines from blocking indefinitely.

Number of elements in a channel

Using a buffered channel, how do measure how many elements are in the channel? For example, I'm creating and sending on a channel like this:
send_ch := make(chan []byte, 100)
// code
send_ch <- msg
I want to measure how many msgs are in the channel send_ch.
I'm aware that due to concurrency the measurement won't be exact, as pre-emption could occur between measurement and action (eg discussed in this video Google I/O 2012 - Go Concurrency Patterns). I'll be using this for flow control between producers and consumers ie once I've passed through a high watermark, changing some behaviour until I pass back through a low watermark.
http://golang.org/pkg/builtin/#len
func len(v Type) int
The len built-in function returns the length of v, according to its type:
Array: the number of elements in v.
Pointer to array: the number of elements in *v (even if v is nil).
Slice, or map: the number of elements in v; if v is nil, len(v) is zero.
String: the number of bytes in v.
Channel: the number of elements queued (unread) in the channel buffer; if v is nil, len(v) is zero.
package main
import "fmt"
func main() {
c := make(chan int, 100)
for i := 0; i < 34; i++ {
c <- 0
}
fmt.Println(len(c))
}
will output:
34
To get the length of a buffered channel that is sending in a go routine:
done := make(chan bool)
ch1 := make(chan string, 3)
go func() {
for i := 0; i < 3; i++ {
ch1 <- "hello"
}
done <- true
}()
<-done
fmt.Println(len(ch1))

Resources