I define unbuffered channel and do these
1.Write some values into it in one go routine.
2.Read values from it in main go routine.
Following the example
package main
import "fmt"
func main() {
c := make(chan int)
go func() {
for i := 0;i < 3; i++ {
c <- i
fmt.Printf("write %v to channel\n", i)
}
}()
for i := 0; i < 3; i++ {
res := <-c
fmt.Printf("read %v from channel\n", res)
}
}
Output
write 0 to channel
read 0 from channel
read 1 from channel
write 1 to channel
write 2 to channel
read 2 from channel
I do not understand the ordering.
After "read 0 from channel", how does it prints "read 1 from channel" after that event it has not been pushed in channel ?
I read the statements like once value is pushed into it and go routine gets blocked until some other go routine reads value from it. But at what point (or which line in above example) it gets blocked ?
Can someone explain ?
The problem is with fmt.Printf, there is no guarantee your output will not be mixed.
Related
I am reading values that are put into a channel ch via an infinite for. I would like some way to signal if a value has been read and operated upon (via the sq result) and add it to some sort of counter variable upon success. That way I have a way to check if my channel has been exhausted so that I can properly exit my infinite for loop.
Currently it is incrementing regardless if a value was read, thus causing it to exit early when the counter == num. I only want it to count when the value has been squared.
EDIT: Another approach I have tested is to receive the ok val out of the channel upon reading and setting val and then check if !ok { break }. However I receive a deadlock panic since the for did has not properly break. Example here: https://go.dev/play/p/RYNtTix2nm2
package main
import "fmt"
func main() {
num := 5
// Buffered channel with 5 values.
ch := make(chan int, num)
defer close(ch)
for i := 0; i < num; i++ {
go func(val int) {
fmt.Printf("Added value: %d to the channel\n", val)
ch <- val
}(i)
}
// Read from our channel infinitely and increment each time a value has been read and operated upon
counter := 0
for {
// Check our counter and if its == num then break the infinite loop
if counter == num {
break
}
val := <-ch
counter++
go func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * i
fmt.Println(sq)
}(val)
}
}
let me try to help you in figuring out the issue.
Reading issue
The latest version of the code you put in the question is working except when you're about to read values from the ch channel. I mean with the following code snippet:
go func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * I
fmt.Println(sq)
}(val)
In fact, it's not needed to spawn a new goroutine for each read. You can consume the messages as soon as they arrived in the ch channel. This is possible due to writing done inside goroutines. Thanks to them, the code can go ahead and reach the reading phase without being blocked.
Buffered vs unbuffered
In this scenario, you used a buffered channel with 5 slots for data. However, if you're relying on the buffered channel you should signal when you finish sending data to it. This is done with a close(ch) invocation after all of the Go routines finish their job. If you use an unbuffered channel it's fine to invoke defer close(ch) next to the channel initialization. In fact, this is done for cleanup and resource optimization tasks. Back to your example, you can change the implementation to use unbuffered channels.
Final Code
Just to recap, the two small changes that you've to do are:
Use an unbuffered channel instead of a buffered one.
Do Not use a Go routine when reading the messages from the channel.
Please be sure to understand exactly what's going on. Another tip can be to issue the statement: fmt.Println("NumGoroutine:", runtime.NumGoroutine()) to print the exact number of Go routines running in that specific moment.
The final code:
package main
import (
"fmt"
"runtime"
)
func main() {
num := 5
// Buffered channel with 5 values.
ch := make(chan int)
defer close(ch)
for i := 0; i < num; i++ {
go func(val int) {
fmt.Printf("Added value: %d to the channel\n", val)
ch <- val
}(i)
}
fmt.Println("NumGoroutine:", runtime.NumGoroutine())
// Read from our channel infinitely and increment each time a value has been read and operated upon
counter := 0
for {
// Check our counter and if its == num then break the infinite loop
if counter == num {
break
}
val := <-ch
counter++
func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * i
fmt.Println(sq)
}(val)
}
}
Let me know if this helps you, thanks!
package main
import "fmt"
func main() {
c := make(chan int)
done := make(chan bool)
go func() {
for i := 0; i < 10; i++ {
c <- i
}
close(c)
}()
go func() {
for i := range c {
fmt.Println(i)
done <- true
}
close(done)
}()
for i := 0; i < 10; i++ {
<-done
}
}
In this example, the done channel is used to signal that a value has been read from the c channel. After each value is read from c, a signal is sent on the done channel. The main function blocks on the done channel, waiting for a signal before continuing. This ensures that all values from c have been processed before the program terminates.
This question already has answers here:
Why does Go handle closures differently in goroutines?
(2 answers)
Cannot assign variable to anonymous func in for loop
(1 answer)
Captured Closure (for Loop Variable) in Go
(1 answer)
go vet - loop variable i captured by func literal
(1 answer)
Closed 8 months ago.
I created the following simple program to test the fan-in-fan-out pattern using channel. What it does is generate a few go routines to calculate the square of a number coming from an input channel and send the square into an output channel. All output channels will then be merged into a single channel to print the square in main.
func calculateSquare(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for num := range in {
fmt.Printf("Receving num %v\n", num)
out <- num * num
fmt.Printf("Sending square %v\n", num * num)
}
fmt.Println("Closing out")
close(out)
}()
return out
}
func fanOut(in <-chan int, workerCount int) []<-chan int {
outs := make([]<-chan int, 0, workerCount)
for i := 0 ; i < workerCount ; i++ {
outs = append(outs, calculateSquare(in))
}
return outs
}
func fanIn(outs []<-chan int) <-chan int {
var wg sync.WaitGroup
merge := make(chan int)
for _, out := range outs {
wg.Add(1)
go func() {
for result := range out {
merge <- result
}
wg.Done()
}()
}
go func() {
wg.Wait()
fmt.Println("Closing merge")
close(merge)
}()
return merge
}
func main() {
in := make(chan int)
go func() {
for i := 0 ; i < 4 ; i++ {
fmt.Printf("Sending num %v\n", i)
in <- i
}
close(in)
}()
outs := fanOut(in, 5)
merge := fanIn(outs)
for num := range merge {
fmt.Printf("Final square %v\n", num)
}
}
In the main function, I'm sending in 4 numbers 0 -> 3 into the input channel and I expect to see 4 square printed in the console. However, when I ran the program, even though the output fluctuates a bit but I never ever see 4 square numbers printed in the console.
Below is a sample output I'm seeing.
Sending num 0
Sending num 1
Sending num 2
Sending num 3
Closing out
Receving num 0
Receving num 1
Receving num 2
Sending square 4
Closing out
Receving num 3
Final square 4
Closing merge
I'd be very grateful if someone could explain to me why Receving num 1 was printed but Sending square 1 is never coming. In addition, if Sending square 1 is not printed, how did the output channel get closed. I'm only seeing 2 Closing out, yet, the wait group where I was merging the result ended its Wait().
I must have done something wrong somewhere.
To fix:
for _, out := range outs {
wg.Add(1)
out := out // <- add this
Why?
https://golang.org/doc/effective_go is an excellent resource and covers the exact closure bug (that #JimB mentioned) towards the end of the channels section:
It may seem odd to write
req := req
but it's legal and idiomatic in Go to do this. You get a
fresh version of the variable with the same name, deliberately
shadowing the loop variable locally but unique to each goroutine.
your issue is in the code below, for loop in fanIn function.
for _, out := range outs {
wg.Add(1)
go func() {
for result := range out {
merge <- result
}
wg.Done()
}()
}
Reason for this is you using out iterator variable in gofunc, when gofunc going to use it, loop is gone to it's end.
This is describe in go/wiki/CommonMistakes under the sub topic Using goroutines on loop iterator variables
For more example - read this
corrected loop should be as below,
for _, out := range outs {
wg.Add(1)
go func(c <- chan int) {
for result := range c {
merge <- result
}
wg.Done()
}(out)
}
I have the following excerpt from an example given in a programming course, and I am not sure why a deadlock is occurring.
package main
import (
"fmt"
)
var (
Count int = 0
nFunctions int = 2
)
var sema = make(chan int)
func increment(ch chan int, nSteps int) {
for i := 0; i < nSteps; i++ {
<- sema
cnt := Count
Count = cnt + 1
ch <- 1
sema <- 1
}
return
}
func main() {
ch := make(chan int)
sema <- 1
go increment(ch, 1000)
go increment(ch, 1000)
for i := 0; i < nFunctions*1000; i++ {
<-ch
}
fmt.Printf("Count = %d\n", Count)
}
Strangely, the deadlock does not occur when I change the statement, in the main, from sema <- 1 to
go func () {
sema <- 1
}()
Any explanations for this much appreciated. The error message is :
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.main()
Channels block both the sender and the receiver. If you send something, you are blocked until it is received. You could reduce your code even further, all you need is a channel and then write to that channel. Note that you could also use a buffered channel, which allows writes up to the buffer length without blocking. If the buffer is full however, it does still block.
When writing data into same channel using multiple go routines with waitgroup after waiting wg.Wait() getting exception saying all go routines are asleep or deedlock.
package main
import (
"fmt"
"runtime"
"sync"
)
var wg sync.WaitGroup
func CreateMultipleRoutines() {
ch := make(chan int)
for i := 0; i < 10; i++ { // creates 10 go routines and adds to waitgroup
wg.Add(1)
go func() {
for j := 0; j < 10; j++ {
ch <- j
}
wg.Done() // indication of go routine is done to main routine
}()
}
fmt.Println(runtime.NumGoroutine())
wg.Wait() //wait for all go routines to complete
close(ch) // closing channel after completion of wait fo go routines
for v := range ch { // range can be used since channel is closed
fmt.Println(v)
}
fmt.Println("About to exit program ...")
}
When tried to implement this without waitgroup I am able to read data from channel by looping exact number of times data pushed to channel but i cant range since there will be panic when we close channel. here is the example code
package main
import (
"fmt"
"runtime"
)
func main() {
ch := make(chan int)
for i := 0; i < 10; i++ { // creates 10 go routines and adds to waitgroup
go func(i int) {
for j := 0; j < 10; j++ {
ch <- j * i
}
}(i)
}
fmt.Println(runtime.NumGoroutine())
for v := 0; v < 100; v++ {
fmt.Println(<-ch)
}
fmt.Println("About to exit program ...")
}
I want to understand why waitgroup in wait state is still waiting even though all go routines are signalled Done() which inturn makes number of go routines to zero
I think your original code has some problems.
You are closing the channel before reading from it.
You are not getting the advantage of using 10 goroutines because of your channel is 1 "sized". So one goroutine is producing one result per once.
My solution would be to spawn a new goroutine to monitor if the 10 goroutines finished its jobs. There you will use your WaitGroup.
Then the code would be like:
package main
import (
"fmt"
"runtime"
"sync"
)
var wg sync.WaitGroup
func main() {
ch := make(chan int, 10)
for i := 0; i < 10; i++ { // creates 10 go routines and adds to waitgroup
wg.Add(1)
go func() {
for j := 0; j < 10; j++ {
ch <- j
}
wg.Done() // indication of go routine is done to main routine
}()
}
go func(){
wg.Wait()
close(ch)
}()
fmt.Println(runtime.NumGoroutine())
for v := range ch { // range can be used since channel is closed
fmt.Println(v)
}
fmt.Println("About to exit program ...")
}
By default a chan holds no items, so all go routines are blocked on sending, until something reads from it. They never actually reach the wg.Done() statement.
A solution would be to close the channel in it's own go routine. Wrap your wg.Wait() and close(ch) lines like this:
go func() {
wg.Wait() //wait for all go routines to complete
close(ch) // closing channel after completion of wait fo go routines
}()
Then you can range over the channel, which will only close after all of the sending go routines have finished (and implicitly all values have been received).
Playing around with go, I threw together this code:
package main
import "fmt"
const N = 10
func main() {
ch := make(chan int, N)
done := make(chan bool)
for i := 0; i < N; i++ {
go (func(n int, ch chan int, done chan bool) {
for i := 0; i < N; i++ {
ch <- n*N + i
}
done <- true
})(i, ch, done)
}
numDone := 0
for numDone < N {
select {
case i := <-ch:
fmt.Println(i)
case <-done:
numDone++
}
}
for {
select {
case i := <-ch:
fmt.Println(i)
default:
return
}
}
}
Basically I have N channels doing some work and reporting it on the same channel -- I want to know when all the channels are done. So I have this other done channel that each worker goroutine sends a message on (message doesn't matter), and this causes main to count that thread as done. When the count gets to N, we're actually done.
Is this "good" go? Is there a more go-idiomatic way of doing this?
edit: To clarify a bit, I'm doubtful because the done channel seems to be doing a job that channel closing seems to be for, but of course I can't actually close the channel in any goroutine because all the routines share the same channel. So I'm using done to simulate a channel that does some kind of "buffered closing".
edit2: Original code wasn't really working since sometimes the done signal from a routine was read before the int it just put on ch. Needs a "cleanup" loop.
Here is an idiomatic use of sync.WaitGroup for you to study
(playground link)
package main
import (
"fmt"
"sync"
)
const N = 10
func main() {
ch := make(chan int, N)
var wg sync.WaitGroup
for i := 0; i < N; i++ {
wg.Add(1)
go func(n int) {
defer wg.Done()
for i := 0; i < N; i++ {
ch <- n*N + i
}
}(i)
}
go func() {
wg.Wait()
close(ch)
}()
for i := range ch {
fmt.Println(i)
}
}
Note the use of closures in the two go routine definitions and note the second go statement to wait for all the routines to finish, then close the channel, so range can be used.
looks like you want a sync.WaitGroup (http://golang.org/pkg/sync/#WaitGroup)
Just use a WaitGroup! They are the built-in primitive that essentially let you wait for stuff in different goroutines to finish up.
http://golang.org/pkg/sync/#WaitGroup
As for your doubts, The way to thing about is that being done by closing a channel (done permanently) and being done with work (temporarily) are different.
In the first approximation the code seems more or less okay to me.
Wrt the details, the 'ch' should be buffered. Also the 'done' channel goroutine "accounting" might be possibly replaced with sync.WaitGroup.
If you're iterating over values generated from goroutines, you can iterate directly over the
communication channel:
for value := range ch {
println(value)
}
The only thing necessary for this is, that the channel ch is closed later on, or else the
loop would wait for new values forever.
This would effectively replace your for numDone < N when used in combination with sync.WaitGroup.
I was dealing with the same issue in some code of mine and found this to be a more than adequate solution.
The answer provides Go's idiom for handling multiple goroutines all sending across a single channel.