I'm reading/working through Go Concurrency Patterns: Pipelines and cancellation, but i'm having trouble understanding the Stopping short section. We have the following functions:
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n
}
close(out)
}()
return out
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func merge(cs ...<-chan int) <-chan int {
var wg sync.WaitGroup
out := make(chan int, 1) // enough space for the unread inputs
// Start an output goroutine for each input channel in cs. output
// copies values from c to out until c is closed, then calls wg.Done.
output := func(c <-chan int) {
for n := range c {
out <- n
}
wg.Done()
}
wg.Add(len(cs))
for _, c := range cs {
go output(c)
}
// Start a goroutine to close out once all the output goroutines are
// done. This must start after the wg.Add call.
go func() {
wg.Wait()
close(out)
}()
return out
}
func main() {
in := gen(2, 3)
// Distribute the sq work across two goroutines that both read from in.
c1 := sq(in)
c2 := sq(in)
// Consume the first value from output.
out := merge(c1, c2)
fmt.Println(<-out) // 4 or 9
return
// Apparently if we had not set the merge out buffer size to 1
// then we would have a hanging go routine.
}
Now, if you notice line 2 in merge, it says we make the out chan with buffer size 1, because this is enough space for the unread inputs. However, I'm almost positive that we should allocate a chan with buffer size 2. In accordance with this code sample:
c := make(chan int, 2) // buffer size 2
c <- 1 // succeeds immediately
c <- 2 // succeeds immediately
c <- 3 // blocks until another goroutine does <-c and receives 1
Because this section implies that a chan of buffer size 3 would not block. Can anyone please clarify/assist my understanding?
The program sends two values to the channel out and reads one value from the channel out. One of the values is not received.
If the channel is unbuffered (capacity 0), then one of the sending goroutines will block until the program exits. This is a leak.
If the channel is created with a capacity of 1, then both goroutines can send to the channel and exit. The first value sent to the channel is received by main. The second value remains in the channel.
If the main function does not receive a value from the channel out, then a channel of capacity 2 is required to prevent the goroutines from blocking indefinitely.
Related
This question already has answers here:
Close multiple goroutine if an error occurs in one in go
(3 answers)
Handle goroutine termination and error handling via error group?
(1 answer)
Closed 3 months ago.
I'm trying to collect errors from goroutines in loop, but dont't understand how it must correctly work
https://go.dev/play/p/WrxE0vH6JSG
func init() {
rand.Seed(1500929006430687579)
}
func goroutine(n int, wg *sync.WaitGroup, ch chan error) {
defer wg.Done()
defer fmt.Println("defer done")
fmt.Println("num ", n)
if n == 1 {
ch <- fmt.Errorf("error")
} else {
ch <- nil
}
}
func main() {
var wg sync.WaitGroup
var err error
errs := make(chan error)
platforms := 2
types := 3
for j := 0; j < platforms; j++ {
wg.Add(1)
for k := 0; k < types; k++ {
wg.Add(1)
n := rand.Intn(2)
go goroutine(n, &wg, errs)
}
for k := 0; k < types; k++ {
wg.Add(1)
n := rand.Intn(2)
go goroutine(n, &wg, errs)
}
}
wg.Wait()
err = <-errs
fmt.Println(err)
}
how should I collect array of errors correctly and done all wait groups?
In Golang channels is similar to pipes in bash (|). But in contrast to bash pipes which are used to transport output of one command to input of another command, Go channels are used to transport some data between goroutines. You can read more about channels here.
Channels have capacity. When you don't specify capacity for the channel go assumes that it has 0 capacity. Channels with zero capacity often called unbuffered channels while channels with non-zero capacity called buffered. When channel is full (number of elements in channel is equal to channel's capacity) than all write operations on the channel (->errs) block execution flow until read operation (<-errs) will be presented.
In your particular example you have unbuffered channel (the channel with 0 capacity). Thus any write operation (->errs) on your channel will block the execution until some read operation would be provided, therefore all goroutines that you launched will be blocked despite the only one goroutine that would be able to proceed write operation when the flow of the main function moved forward to read operation (err = <-errs).
To solve your issue you could create one extra goroutine that would read from channel concurrently with goroutines that would write to channel. It will look like that:
func init() {
rand.Seed(1500929006430687579)
}
func goroutine(n int, wg *sync.WaitGroup, ch chan error) {
defer fmt.Println("defer done")
defer wg.Done()
fmt.Println("num ", n)
if n == 1 {
ch <- fmt.Errorf("error")
}
}
func main() {
var wg sync.WaitGroup
errs := make(chan error)
platforms := 2
types := 3
go func() {
for e := range errs {
fmt.Println(e)
}
}()
for j := 0; j < platforms; j++ {
for k := 0; k < types; k++ {
wg.Add(1)
n := rand.Intn(2)
go goroutine(n, &wg, errs)
}
for k := 0; k < types; k++ {
wg.Add(1)
n := rand.Intn(2)
go goroutine(n, &wg, errs)
}
}
wg.Wait()
}
In addition you have several bugs and inaccuracies that I refactored in your code:
You shouldn't write nil in channel with errors. If you want errs chan to comprise only errors so write there only if your function executed with non-nil error.
You had one extra wd.Add(1) as the beginning of j loop so there was disbalance between Add functions and Done function. 3.
Furthemore, you add defer fmt.Println("defer done") after defer wg.Done() but defers constructions are executed in reversed order than they were specified so it would be more correct to put defer fmt.Println("defer done") before defer wg.Done() so that "defer done" would really signalize that all previous defers had been executed.
(New to concurrent programming)
Had a doubt about why is the flow of execution of goroutines a bit weird here?
Beginner to goroutines and channels stuff in golang.
func main() {
// Set up the pipeline.
c := gen(2, 3)
out := sq(c)
// Consume the output.
fmt.Println(<-out) // 4
fmt.Println(<-out) // 9
}
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n
}
close(out)
}()
return out
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
Hope it helps. It goroutine pipeline diagram. So there are three goroutines and two channels
Once calls to gen and sq are finished there are 3 goroutines running concurrently. They pass data between with channels and therefore the execution produces the same results.
gen-inner
sq-inner
main
They always pass at least 2 pieces of information through and therefore run their code in the order below
gen-inner out <- n -2-> sq-inner out <- n * n -4-> main println(<-out)
gen-inner out <- n -3-> sq-inner out <- n * n -9-> main println(<-out)
There is also a third pass that may happen but it may be skipped as main goroutine ends.
gen-inner close() -close-> sq-inner close(out) -close->
I am experimenting with channel concept in Go. I wrote the below program playground to implement counter using channels. But I am not getting any output, although I am doing some printing in the goroutine body.
func main() {
wg := sync.WaitGroup{}
ch := make(chan int)
count := func(ch chan int) {
var last int
last = <-ch
last = last + 1
fmt.Println(last)
ch <- last
wg.Done()
}
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
}
I expect at least some output but I am getting none at all.
When the main() function (the main goroutine) ends, your program ends as well (it doesn't wait for other non-main goroutines to finish). You must add a wg.Wait() call to the end. See No output from goroutine in Go.
Once you do this, you'll hit a deadlock. This is because all goroutines start with attempting to receive a value from the channel, and only then would they send something.
So you should first send something on the channel to let at least one of the goroutines to proceed.
Once you do that, you'll see numbers printed 10 times, and again deadlock. This is because when the last goroutine tries to send its incremented number, there will be no one to receive that. An easy way to fix that is to give a buffer to the channel.
Final, working example:
wg := sync.WaitGroup{}
ch := make(chan int, 2)
count := func(ch chan int) {
var last int
last = <-ch
last = last + 1
fmt.Println(last)
ch <- last
wg.Done()
}
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
go func() {
ch <- 0
}()
wg.Wait()
Outputs (try it on the Go Playground):
1
2
3
4
5
6
7
8
9
10
Also note that since we made the channel buffered, it's not necessary to use another goroutine to send an initial value, we can do that in the main goroutine:
ch <- 0
wg.Wait()
This will output the same. Try it on the Go Playground.
func main() {
wg := sync.WaitGroup{}
ch := make(chan int)
count := func(ch chan int) {
var last int
last, ok := <-ch // 这里做一层保护
if !ok {
return
}
last = last + 1
fmt.Println(last)
go func(ch chan int, res int) {
ch <- res
}(ch, last)
wg.Done()
}
go func() {
ch <- 0
}()
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
wg.Wait()
fmt.Println("main finish")
close(ch)
}
I am working on a sample program to print sum of odd and sum of even number between 1 to 100 using goroutine with multiple channels.
you can find my code
here
output
sum of even number = 2550
sum of odd number = 2500
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan receive]:
main.print(0x434100, 0x11db7c)
/tmp/sandbox052575152/main.go:18 +0xc0
main.main()
/tmp/sandbox052575152/main.go:14 +0x120
The code works but with deadlock.
I am not sure what is wrong in my code
We can iterate through values sent over a channel. To break such iteration channel needs to be closed explicitly. Otherwise range would block forever in the same way as for nil channel. In your code you did't close the sum(for print function sumValues channel) channel. That's why following function will be blocked for forever.
func print(sumValues <-chan string ){
for val := range sumValues {
fmt.Println(val)
}
}
So you have to close the sum channel in the doSum function after all the go routine in the doSum function are complete (otherwise sum channel might be closed before go routines are complete). You can use sync.WaitGroup to do that. See the updated doSum function below:
func doSum(sum chan<- string, oddChan <-chan int, evenChan <-chan int) {
var waitGroup sync.WaitGroup
waitGroup.Add(2) // Must wait for 2 calls to 'done' before moving on
go func(sum chan<- string) {
s1 := 0
for val := range oddChan {
s1 += val
}
sum <- fmt.Sprint("sum of odd number = ", s1)
waitGroup.Done()
}(sum)
go func(sum chan<- string) {
s1 := 0
for val := range evenChan {
s1 += val
}
sum <- fmt.Sprint("sum of even number = ", s1)
waitGroup.Done()
}(sum)
// Waiting for all goroutines to exit
waitGroup.Wait()
// all goroutines are complete now close the sum channel
close(sum)
}
I am trying to understand, how golang channel works. I read a book about the go language and found the following example.
package main
import (
"fmt"
)
// Send the sequence 2, 3, 4, ... to returned channel
func generate() chan int {
ch := make(chan int)
go func() {
for i := 2; i <= 100 ; i++ {
ch <- i
}
}()
return ch
}
// Filter out input values divisible by 'prime', send rest to returned channel
func filter(in chan int, prime int) chan int {
out := make(chan int)
go func() {
for {
if i := <-in; i%prime != 0 {
out <- i
}
}
}()
return out
}
func sieve() chan int {
out := make(chan int)
go func() {
ch := generate()
for {
prime := <-ch
ch = filter(ch, prime)
out <- prime
}
}()
return out
}
func main() {
primes := sieve()
for {
fmt.Println(<-primes)
}
}
When I run this programm, I've got a deadlock, but when I change the generate function to
// Send the sequence 2, 3, 4, ... to returned channel
func generate() chan int {
ch := make(chan int)
go func() {
for i := 2; ; i++ {
ch <- i
}
}()
return ch
}
Then the programm will run the infinite loop, but not deadlock. Why do I get deadlock, when I remove the condition in for loop?
What do you mean with blocking principle?
You can see it illustrated in the blog post "The Nature Of Channels In Go "
for an unbuffered channel:
(Illustration from blog post "The Nature Of Channels In Go ", written by William Kennedy, Feb. 2014)
Unbuffered channels have no capacity and therefore require both goroutines to be ready to make any exchange.
When a goroutine attempts to write a resource to an unbuffered channel and there is no goroutine waiting to receive the resource, the channel will lock the goroutine and make it wait.
When a goroutine attempts to read from an unbuffered channel, and there is no goroutine waiting to send a resource, the channel will lock the goroutine and make it wait.
That is what happens in your case with your reader:
func main() {
primes := sieve()
for {
fmt.Println(<-primes)
}
}
since primes is never closed, main remains blocked.
It (main) is in step 3:
in step 3, the goroutine on the right places his hand into the channel or performs a read.
That goroutine is also locked in the channel until the exchange is complete.
The sender never calls close(primes).
Let's consider a simpler example:
func generate() chan int {
ch := make(chan int)
go func() {
for i := 2; /*i < 100*/; i++ {
ch <- i
}
}()
return ch
}
func main() {
for i := range generate() {
fmt.Println(i)
}
}
With the condition i < 100 uncommented, the goroutine spawned by generate stops after sending 98 numbers. However, it does not close the channel, so main has no way of knowing that no more numbers are going to be sent, and it just keeps blocking on the channel. Since main is now the only goroutine still in existence (the other one has returned), and it's blocking, you have a deadlock.