This question already has answers here:
Why does Go handle closures differently in goroutines?
(2 answers)
Cannot assign variable to anonymous func in for loop
(1 answer)
Captured Closure (for Loop Variable) in Go
(1 answer)
go vet - loop variable i captured by func literal
(1 answer)
Closed 8 months ago.
I created the following simple program to test the fan-in-fan-out pattern using channel. What it does is generate a few go routines to calculate the square of a number coming from an input channel and send the square into an output channel. All output channels will then be merged into a single channel to print the square in main.
func calculateSquare(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for num := range in {
fmt.Printf("Receving num %v\n", num)
out <- num * num
fmt.Printf("Sending square %v\n", num * num)
}
fmt.Println("Closing out")
close(out)
}()
return out
}
func fanOut(in <-chan int, workerCount int) []<-chan int {
outs := make([]<-chan int, 0, workerCount)
for i := 0 ; i < workerCount ; i++ {
outs = append(outs, calculateSquare(in))
}
return outs
}
func fanIn(outs []<-chan int) <-chan int {
var wg sync.WaitGroup
merge := make(chan int)
for _, out := range outs {
wg.Add(1)
go func() {
for result := range out {
merge <- result
}
wg.Done()
}()
}
go func() {
wg.Wait()
fmt.Println("Closing merge")
close(merge)
}()
return merge
}
func main() {
in := make(chan int)
go func() {
for i := 0 ; i < 4 ; i++ {
fmt.Printf("Sending num %v\n", i)
in <- i
}
close(in)
}()
outs := fanOut(in, 5)
merge := fanIn(outs)
for num := range merge {
fmt.Printf("Final square %v\n", num)
}
}
In the main function, I'm sending in 4 numbers 0 -> 3 into the input channel and I expect to see 4 square printed in the console. However, when I ran the program, even though the output fluctuates a bit but I never ever see 4 square numbers printed in the console.
Below is a sample output I'm seeing.
Sending num 0
Sending num 1
Sending num 2
Sending num 3
Closing out
Receving num 0
Receving num 1
Receving num 2
Sending square 4
Closing out
Receving num 3
Final square 4
Closing merge
I'd be very grateful if someone could explain to me why Receving num 1 was printed but Sending square 1 is never coming. In addition, if Sending square 1 is not printed, how did the output channel get closed. I'm only seeing 2 Closing out, yet, the wait group where I was merging the result ended its Wait().
I must have done something wrong somewhere.
To fix:
for _, out := range outs {
wg.Add(1)
out := out // <- add this
Why?
https://golang.org/doc/effective_go is an excellent resource and covers the exact closure bug (that #JimB mentioned) towards the end of the channels section:
It may seem odd to write
req := req
but it's legal and idiomatic in Go to do this. You get a
fresh version of the variable with the same name, deliberately
shadowing the loop variable locally but unique to each goroutine.
your issue is in the code below, for loop in fanIn function.
for _, out := range outs {
wg.Add(1)
go func() {
for result := range out {
merge <- result
}
wg.Done()
}()
}
Reason for this is you using out iterator variable in gofunc, when gofunc going to use it, loop is gone to it's end.
This is describe in go/wiki/CommonMistakes under the sub topic Using goroutines on loop iterator variables
For more example - read this
corrected loop should be as below,
for _, out := range outs {
wg.Add(1)
go func(c <- chan int) {
for result := range c {
merge <- result
}
wg.Done()
}(out)
}
Related
I am reading values that are put into a channel ch via an infinite for. I would like some way to signal if a value has been read and operated upon (via the sq result) and add it to some sort of counter variable upon success. That way I have a way to check if my channel has been exhausted so that I can properly exit my infinite for loop.
Currently it is incrementing regardless if a value was read, thus causing it to exit early when the counter == num. I only want it to count when the value has been squared.
EDIT: Another approach I have tested is to receive the ok val out of the channel upon reading and setting val and then check if !ok { break }. However I receive a deadlock panic since the for did has not properly break. Example here: https://go.dev/play/p/RYNtTix2nm2
package main
import "fmt"
func main() {
num := 5
// Buffered channel with 5 values.
ch := make(chan int, num)
defer close(ch)
for i := 0; i < num; i++ {
go func(val int) {
fmt.Printf("Added value: %d to the channel\n", val)
ch <- val
}(i)
}
// Read from our channel infinitely and increment each time a value has been read and operated upon
counter := 0
for {
// Check our counter and if its == num then break the infinite loop
if counter == num {
break
}
val := <-ch
counter++
go func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * i
fmt.Println(sq)
}(val)
}
}
let me try to help you in figuring out the issue.
Reading issue
The latest version of the code you put in the question is working except when you're about to read values from the ch channel. I mean with the following code snippet:
go func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * I
fmt.Println(sq)
}(val)
In fact, it's not needed to spawn a new goroutine for each read. You can consume the messages as soon as they arrived in the ch channel. This is possible due to writing done inside goroutines. Thanks to them, the code can go ahead and reach the reading phase without being blocked.
Buffered vs unbuffered
In this scenario, you used a buffered channel with 5 slots for data. However, if you're relying on the buffered channel you should signal when you finish sending data to it. This is done with a close(ch) invocation after all of the Go routines finish their job. If you use an unbuffered channel it's fine to invoke defer close(ch) next to the channel initialization. In fact, this is done for cleanup and resource optimization tasks. Back to your example, you can change the implementation to use unbuffered channels.
Final Code
Just to recap, the two small changes that you've to do are:
Use an unbuffered channel instead of a buffered one.
Do Not use a Go routine when reading the messages from the channel.
Please be sure to understand exactly what's going on. Another tip can be to issue the statement: fmt.Println("NumGoroutine:", runtime.NumGoroutine()) to print the exact number of Go routines running in that specific moment.
The final code:
package main
import (
"fmt"
"runtime"
)
func main() {
num := 5
// Buffered channel with 5 values.
ch := make(chan int)
defer close(ch)
for i := 0; i < num; i++ {
go func(val int) {
fmt.Printf("Added value: %d to the channel\n", val)
ch <- val
}(i)
}
fmt.Println("NumGoroutine:", runtime.NumGoroutine())
// Read from our channel infinitely and increment each time a value has been read and operated upon
counter := 0
for {
// Check our counter and if its == num then break the infinite loop
if counter == num {
break
}
val := <-ch
counter++
func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * i
fmt.Println(sq)
}(val)
}
}
Let me know if this helps you, thanks!
package main
import "fmt"
func main() {
c := make(chan int)
done := make(chan bool)
go func() {
for i := 0; i < 10; i++ {
c <- i
}
close(c)
}()
go func() {
for i := range c {
fmt.Println(i)
done <- true
}
close(done)
}()
for i := 0; i < 10; i++ {
<-done
}
}
In this example, the done channel is used to signal that a value has been read from the c channel. After each value is read from c, a signal is sent on the done channel. The main function blocks on the done channel, waiting for a signal before continuing. This ensures that all values from c have been processed before the program terminates.
This question already has answers here:
Close multiple goroutine if an error occurs in one in go
(3 answers)
Handle goroutine termination and error handling via error group?
(1 answer)
Closed 3 months ago.
I'm trying to collect errors from goroutines in loop, but dont't understand how it must correctly work
https://go.dev/play/p/WrxE0vH6JSG
func init() {
rand.Seed(1500929006430687579)
}
func goroutine(n int, wg *sync.WaitGroup, ch chan error) {
defer wg.Done()
defer fmt.Println("defer done")
fmt.Println("num ", n)
if n == 1 {
ch <- fmt.Errorf("error")
} else {
ch <- nil
}
}
func main() {
var wg sync.WaitGroup
var err error
errs := make(chan error)
platforms := 2
types := 3
for j := 0; j < platforms; j++ {
wg.Add(1)
for k := 0; k < types; k++ {
wg.Add(1)
n := rand.Intn(2)
go goroutine(n, &wg, errs)
}
for k := 0; k < types; k++ {
wg.Add(1)
n := rand.Intn(2)
go goroutine(n, &wg, errs)
}
}
wg.Wait()
err = <-errs
fmt.Println(err)
}
how should I collect array of errors correctly and done all wait groups?
In Golang channels is similar to pipes in bash (|). But in contrast to bash pipes which are used to transport output of one command to input of another command, Go channels are used to transport some data between goroutines. You can read more about channels here.
Channels have capacity. When you don't specify capacity for the channel go assumes that it has 0 capacity. Channels with zero capacity often called unbuffered channels while channels with non-zero capacity called buffered. When channel is full (number of elements in channel is equal to channel's capacity) than all write operations on the channel (->errs) block execution flow until read operation (<-errs) will be presented.
In your particular example you have unbuffered channel (the channel with 0 capacity). Thus any write operation (->errs) on your channel will block the execution until some read operation would be provided, therefore all goroutines that you launched will be blocked despite the only one goroutine that would be able to proceed write operation when the flow of the main function moved forward to read operation (err = <-errs).
To solve your issue you could create one extra goroutine that would read from channel concurrently with goroutines that would write to channel. It will look like that:
func init() {
rand.Seed(1500929006430687579)
}
func goroutine(n int, wg *sync.WaitGroup, ch chan error) {
defer fmt.Println("defer done")
defer wg.Done()
fmt.Println("num ", n)
if n == 1 {
ch <- fmt.Errorf("error")
}
}
func main() {
var wg sync.WaitGroup
errs := make(chan error)
platforms := 2
types := 3
go func() {
for e := range errs {
fmt.Println(e)
}
}()
for j := 0; j < platforms; j++ {
for k := 0; k < types; k++ {
wg.Add(1)
n := rand.Intn(2)
go goroutine(n, &wg, errs)
}
for k := 0; k < types; k++ {
wg.Add(1)
n := rand.Intn(2)
go goroutine(n, &wg, errs)
}
}
wg.Wait()
}
In addition you have several bugs and inaccuracies that I refactored in your code:
You shouldn't write nil in channel with errors. If you want errs chan to comprise only errors so write there only if your function executed with non-nil error.
You had one extra wd.Add(1) as the beginning of j loop so there was disbalance between Add functions and Done function. 3.
Furthemore, you add defer fmt.Println("defer done") after defer wg.Done() but defers constructions are executed in reversed order than they were specified so it would be more correct to put defer fmt.Println("defer done") before defer wg.Done() so that "defer done" would really signalize that all previous defers had been executed.
(New to concurrent programming)
Had a doubt about why is the flow of execution of goroutines a bit weird here?
Beginner to goroutines and channels stuff in golang.
func main() {
// Set up the pipeline.
c := gen(2, 3)
out := sq(c)
// Consume the output.
fmt.Println(<-out) // 4
fmt.Println(<-out) // 9
}
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n
}
close(out)
}()
return out
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
Hope it helps. It goroutine pipeline diagram. So there are three goroutines and two channels
Once calls to gen and sq are finished there are 3 goroutines running concurrently. They pass data between with channels and therefore the execution produces the same results.
gen-inner
sq-inner
main
They always pass at least 2 pieces of information through and therefore run their code in the order below
gen-inner out <- n -2-> sq-inner out <- n * n -4-> main println(<-out)
gen-inner out <- n -3-> sq-inner out <- n * n -9-> main println(<-out)
There is also a third pass that may happen but it may be skipped as main goroutine ends.
gen-inner close() -close-> sq-inner close(out) -close->
I am experimenting with channel concept in Go. I wrote the below program playground to implement counter using channels. But I am not getting any output, although I am doing some printing in the goroutine body.
func main() {
wg := sync.WaitGroup{}
ch := make(chan int)
count := func(ch chan int) {
var last int
last = <-ch
last = last + 1
fmt.Println(last)
ch <- last
wg.Done()
}
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
}
I expect at least some output but I am getting none at all.
When the main() function (the main goroutine) ends, your program ends as well (it doesn't wait for other non-main goroutines to finish). You must add a wg.Wait() call to the end. See No output from goroutine in Go.
Once you do this, you'll hit a deadlock. This is because all goroutines start with attempting to receive a value from the channel, and only then would they send something.
So you should first send something on the channel to let at least one of the goroutines to proceed.
Once you do that, you'll see numbers printed 10 times, and again deadlock. This is because when the last goroutine tries to send its incremented number, there will be no one to receive that. An easy way to fix that is to give a buffer to the channel.
Final, working example:
wg := sync.WaitGroup{}
ch := make(chan int, 2)
count := func(ch chan int) {
var last int
last = <-ch
last = last + 1
fmt.Println(last)
ch <- last
wg.Done()
}
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
go func() {
ch <- 0
}()
wg.Wait()
Outputs (try it on the Go Playground):
1
2
3
4
5
6
7
8
9
10
Also note that since we made the channel buffered, it's not necessary to use another goroutine to send an initial value, we can do that in the main goroutine:
ch <- 0
wg.Wait()
This will output the same. Try it on the Go Playground.
func main() {
wg := sync.WaitGroup{}
ch := make(chan int)
count := func(ch chan int) {
var last int
last, ok := <-ch // 这里做一层保护
if !ok {
return
}
last = last + 1
fmt.Println(last)
go func(ch chan int, res int) {
ch <- res
}(ch, last)
wg.Done()
}
go func() {
ch <- 0
}()
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
wg.Wait()
fmt.Println("main finish")
close(ch)
}
I'm reading/working through Go Concurrency Patterns: Pipelines and cancellation, but i'm having trouble understanding the Stopping short section. We have the following functions:
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n
}
close(out)
}()
return out
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func merge(cs ...<-chan int) <-chan int {
var wg sync.WaitGroup
out := make(chan int, 1) // enough space for the unread inputs
// Start an output goroutine for each input channel in cs. output
// copies values from c to out until c is closed, then calls wg.Done.
output := func(c <-chan int) {
for n := range c {
out <- n
}
wg.Done()
}
wg.Add(len(cs))
for _, c := range cs {
go output(c)
}
// Start a goroutine to close out once all the output goroutines are
// done. This must start after the wg.Add call.
go func() {
wg.Wait()
close(out)
}()
return out
}
func main() {
in := gen(2, 3)
// Distribute the sq work across two goroutines that both read from in.
c1 := sq(in)
c2 := sq(in)
// Consume the first value from output.
out := merge(c1, c2)
fmt.Println(<-out) // 4 or 9
return
// Apparently if we had not set the merge out buffer size to 1
// then we would have a hanging go routine.
}
Now, if you notice line 2 in merge, it says we make the out chan with buffer size 1, because this is enough space for the unread inputs. However, I'm almost positive that we should allocate a chan with buffer size 2. In accordance with this code sample:
c := make(chan int, 2) // buffer size 2
c <- 1 // succeeds immediately
c <- 2 // succeeds immediately
c <- 3 // blocks until another goroutine does <-c and receives 1
Because this section implies that a chan of buffer size 3 would not block. Can anyone please clarify/assist my understanding?
The program sends two values to the channel out and reads one value from the channel out. One of the values is not received.
If the channel is unbuffered (capacity 0), then one of the sending goroutines will block until the program exits. This is a leak.
If the channel is created with a capacity of 1, then both goroutines can send to the channel and exit. The first value sent to the channel is received by main. The second value remains in the channel.
If the main function does not receive a value from the channel out, then a channel of capacity 2 is required to prevent the goroutines from blocking indefinitely.