I have made a simple code example to understand the usage of pipeline, here it is.
package main
import (
"fmt"
"sync"
"time"
)
func main() {
ch1 := make(chan int, 10) // Use buffered channel so as to avoid clogging
ch2 := make(chan string, 10)
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func1(i, ch1, &wg)
go func2(ch1, ch2)
}
wg.Wait()
close(ch1)
for val := range ch2 {
fmt.Println(val)
}
}
func func1(seconds int, ch chan<- int, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(time.Duration(seconds) * time.Second)
fmt.Println(seconds)
ch <- seconds
}
func func2(ch1 chan int, ch2 chan string) {
for range ch1 {
ch2 <- "hello"
}
close(ch2)
}
Now, the problem is I don't get consistent output ( I understand it's some concurrency issue, which I haven't fully understood ).
Output
> go run pipeline-loop.go
0
1
2
hello
hello
> go run pipeline-loop.go
0
1
2
hello
hello
hello
> go run pipeline-loop.go
0
1
2
hello
hello
> go run pipeline-loop.go
0
1
2
hello
hello
> go run pipeline-loop.go
0
1
2
hello
hello
panic: close of closed channel
goroutine 6 [running]:
main.func2(0xc00006c000, 0xc000056180)
/home/projects/go-tuts/pipeline-loop.go:36 +0x72
created by main.main
/home/projects/go-tuts/pipeline-loop.go:16 +0x10f
exit status 2
Another guy changed the code ( and it was working ) and put func2 outside the loop but I want func2 for each iteration of the func1.
Problem
So, I want to understand as to where should the WaitGroup and close(ch) be used ?
Thanks.
Temporarya
( A golang noobie )
Update
Based on a user's answer, I changed the code, now I get output as expected ( but not the solution to this question ) but still there's a deadlock.
https://play.golang.org/p/O_rp_FLvNh8
package main
import (
"fmt"
"sync"
"time"
)
func main() {
ch1 := make(chan int, 10) // Use buffered channel so as to avoid clogging
ch2 := make(chan string, 10)
var wg1 sync.WaitGroup
// var wg2 sync.WaitGroup
for i := 0; i < 3; i++ {
wg1.Add(1)
go func1(i, ch1)
go func2(ch1, ch2, &wg1)
}
for val := range ch2 {
fmt.Println(val)
}
wg1.Wait()
close(ch1)
close(ch2)
}
// func func1(seconds int, ch chan<- int, wg *sync.WaitGroup) {
func func1(seconds int, ch chan<- int) {
// defer wg.Done()
time.Sleep(time.Duration(seconds) * time.Second)
fmt.Println(seconds)
ch <- seconds
}
func func2(ch1 chan int, ch2 chan string, wg *sync.WaitGroup) {
defer wg.Done()
for range ch1 {
ch2 <- "hello"
}
}
There are multiple problems in your code.
In the loop, you are spawning multiple (3) goroutines that runs func2, and in func2, you send data to ch2 and call close(ch2). This is a problem. It can happen that when one goroutine is seding data to ch2, the other has closed that channel, which causes:
panic: close of closed channel
goroutine 6 [running]:
main.func2(0xc00006c000, 0xc000056180)
/home/projects/go-tuts/pipeline-loop.go:36 +0x72
created by main.main
/home/projects/go-tuts/pipeline-loop.go:16 +0x10f
exit status 2
In general, you don't need close a chanel multiple times - you only need to shut them once they are all finished. You need another WaitGroup for this; you need to pass both functions a WaitGroup.
Further reading: https://blog.golang.org/pipelines
UPDATE:
Personally I use a pattern for "works" that produce data into a same channel and the channel needs to be closed after all works are done:
for something {
wg.Add(1)
go func(i int) {
work(ch)
wg.Done()
}
}
go func() {
wg.Wait()
close()
}()
I think it is a good idea that keeps the API clean from WorkGroup as WorkGroup is about how you sync the work instead of how the work is done.
I have changed your code into this pattern: https://play.golang.org/p/vdCNsxWhgyQ
I suspect you only want one channel to read from ch1 and write to ch2. There is not much point in creating 3 go-routines to do the same thing (and you also end up closing the same channel mutliple time which causes a panic as leaf bebop pointed out)
func main() {
ch1 := make(chan int, 10) // Use buffered channel so as to avoid clogging
ch2 := make(chan string, 10)
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func1(i, ch1, &wg)
}
go func2(ch1, ch2)
wg.Wait()
close(ch1)
for val := range ch2 {
fmt.Println(val)
}
}
Related
I was trying to debug this code but am stuck here. I wanted to access ch1, ch2 but found printed nothing.
package main
import (
"fmt"
)
type degen struct {
i, j string
}
func (x degen) CVIO(ch1, ch2 chan string, quit chan int, m, n string) {
for {
select {
case ch1 <- m:
fmt.Println(x.i)
case ch2 <- n:
fmt.Println("ok")
case <-quit:
fmt.Println("quit")
return
}
}
}
func main() {
ch1 := make(chan string)
ch2 := make(chan string)
quit := make(chan int)
x := degen{"goosebump", "ok"}
go x.CVIO(ch1, ch2, quit, "goosebump", "ok")
}
Desired:
It should print the channel data as to be produced.
Its not really clear what you expect your code to do:
main() ends without waiting for the go routine to exit (its quite possible it the loop will not run at all).
in the select the sends will not proceed because there is no receiver (spec - "if the capacity is zero or absent, the channel is unbuffered and communication succeeds only when both a sender and receiver are ready.").
Nothing is sent to the quit channel.
I suspect that the following (playground) might do what you were expecting.
package main
import (
"fmt"
"sync"
)
type degen struct {
i, j string
}
func (x degen) CVIO(ch1, ch2 chan string, quit chan int, m, n string) {
for {
select {
case ch1 <- m:
fmt.Println(x.i)
case ch2 <- n:
fmt.Println("ok")
case <-quit:
fmt.Println("quit")
return
}
}
}
func main() {
ch1 := make(chan string)
ch2 := make(chan string)
quit := make(chan int)
x := degen{"goosebump", "ok"}
var wg sync.WaitGroup
wg.Add(1)
go func() {
x.CVIO(ch1, ch2, quit, "goosebump", "ok")
wg.Done()
}()
<-ch1 // Receive from CH1 (allowing "ch1 <- m" in go routine to proceed)
<-ch2 // Receive from CH2 (allowing "ch2 <- n" in go routine to proceed)
quit <- 1
wg.Wait() // Wait for CVIO to end (which it should do due to above send)
}
I want to send a value in a channel to go routines from main function. What happens is which go routine will receive the value from the channel first.
package main
import (
"fmt"
"math/rand"
//"runtime"
"strconv"
"time"
)
func main() {
var ch chan int
ch = make(chan int)
ch <- 1
receive(ch)
}
func receive(ch chan int){
for i := 0; i < 4; i++ {
// Create some threads
go func(i int) {
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
fmt.Println(<-ch)
}(i)
}
}
My current implementation is giving an error.
fatal error: all goroutines are asleep - deadlock!
How can I know that which go routine will receive the value from the channel first. And what happens to other go routine If those will run or throw an error since there is no channel to receive the value. As it is already received by one of them.
If a create a buffered channel my code works. So I don't get it what has happened behind the scene which is making it work when creating a buffered channel like below:
func main() {
var ch chan int
ch = make(chan int, 10)
ch <- 1
receive(ch)
}
If we look at below code. I can see that we can send values through channels directly there is no need of creating a go routine to send a value thorugh a channel to another go routines.
package main
import "fmt"
func main() {
// We'll iterate over 2 values in the `queue` channel.
queue := make(chan string, 2)
queue <- "one"
queue <- "two"
close(queue)
for elem := range queue {
fmt.Println(elem)
}
}
Then what is wrong with my code. Why is it creating a deadlock.
If all you need is to start several workers and send a task to any of them, then you'd better run workers before sending a value to a channel, because as #mkopriva said above, writing to a channel is a blocking operation. You always have to have a consumer, or the execution will freeze.
func main() {
var ch chan int
ch = make(chan int)
receive(ch)
ch <- 1
}
func receive(ch chan int) {
for i := 0; i < 4; i++ {
// Create some threads
go func(i int) {
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
fmt.Printf("Worker no %d is processing the value %d\n", i, <-ch)
}(i)
}
}
Short answer for the question "Which go routine will receive it?" - Whatever. :) Any of them, you can't say for sure.
However I have no idea what is time.Sleep(...) for there, kept it as is.
An unbuffered channel (without a length) blocks until the value has been received. This means the program that wrote to the channel will stop after writing to the channel until it has been read from. If that happens in the main thread, before your call to receive, it causes a deadlock.
There are two more issues: you need to use a WaitGroup to pause the completion until finished, and a channel behaves like a concurrent queue. In particular, it has push and pop operations, which are both performed using <-. For example:
//Push to channel, channel contains 1 unless other things were there
c <- 1
//Pop from channel, channel is empty
x := <-c
Here is a working example:
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
func main() {
var ch chan int
ch = make(chan int)
go func() {
ch <- 1
ch <- 1
ch <- 1
ch <- 1
}()
receive(ch)
}
func receive(ch chan int) {
wg := &sync.WaitGroup{}
for i := 0; i < 4; i++ {
// Create some threads
wg.Add(1)
go func(i int) {
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
fmt.Println(<-ch)
wg.Done()
}(i)
}
wg.Wait()
fmt.Println("done waiting")
}
Playground Link
As you can see the WaitGroup is quite simple as well. You declare it at a higher scope. It's essentially a fancy counter, with three primary methods. When you call wg.Add(1) the counter is increased, when you call wg.Done() the counter is decreased, and when you call wg.Wait(), the execution is halted until the counter reaches 0.
I've been trying to solve this simple problem I encountered in Golang concurrency. I've been searching all possible solutions, but found nothing specific to my problem(or I might be missed one). Here's my code:
package main
import (
"fmt"
"time"
)
func producer(ch chan int, d time.Duration, num int) {
for i:=0; i<num; i++ {
ch <- i
time.Sleep(d)
}
}
func main() {
ch := make(chan int)
go producer(ch, 100*time.Millisecond, 2)
go producer(ch, 200*time.Millisecond, 5)
for {
fmt.Println(<-ch)
}
close(ch)
}
It prints error:
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan receive]:
main.main()
D:/Code/go/src/testconcurrency/main.go:23 +0xca
exit status 2
What is the efficient way to avoid this error?, Thank you.
You have producers which are "short-lived", they only send values on the channel for a finite amount of time, and you have an endless for loop which receives values from the channel endlessly, without a termination condition, and the channel is only closed after this endless loop. Once the producers stop sending values, it's a deadlock.
Channels must be closed by the producer(s), signalling that no more values will be sent on it. Since you have multiple producers without synchronization (producers are not synchronized with each other), in general you can't tell which one will finish first, so you can't designate one to close the channel (and a channel can only be closed once, see Why Go's channel can close twice?; and Closing channel of unknown length).
You have to "coordinate" the producers, and when all have finished their jobs, the coordinator should close the channel.
And the consumer should use a for range on the channel, as the for range construct receives all values from the channel that were sent on it before it was closed, then it terminates automatically.
For the coordination it is recommended to use sync.WaitGroup. Whether you use a global one in this case or a local one and you pass it to producers is up to you. Using a local will make the solution more general and easier to extend. One thing to note is that you must pass a pointer to sync.WaitGroup. Whenever you spin up a new producer, increment the waitgroup using WaitGroup.Add(). When a producer is done, it can signal this using WaitGroup.Done(), preferably using defer (so it runs no matter what, mitigating the deadlock in case of abnormal circumstances). And the controller can wait for all producers to finish using WaitGroup.Wait().
Here's a complete solution:
func producer(ch chan int, d time.Duration, num int, wg *sync.WaitGroup) {
defer wg.Done()
for i := 0; i < num; i++ {
ch <- i
time.Sleep(d)
}
}
func main() {
wg := &sync.WaitGroup{}
ch := make(chan int)
wg.Add(1)
go producer(ch, 100*time.Millisecond, 2, wg)
wg.Add(1)
go producer(ch, 200*time.Millisecond, 5, wg)
go func() {
wg.Wait()
close(ch)
}()
for v := range ch {
fmt.Println(v)
}
}
Output (try it on the Go Playground):
0
0
1
1
2
3
4
See related question: Prevent the main() function from terminating before goroutines finish in Golang
This problem can be solved in an elegant way using two wait groups. By closing channel ch we signal to the consumers that there is no more data.
The solutions scales well with more consumers.
package main
import (
"fmt"
"sync"
"time"
)
func producer(ch chan<- int, d time.Duration, num int, wg *sync.WaitGroup) {
defer wg.Done()
for i := 0; i < num; i++ {
ch <- i
time.Sleep(d)
}
}
func consumer(ch <-chan int, wg *sync.WaitGroup) {
defer wg.Done()
for x := range ch {
fmt.Println(x)
}
}
func main() {
ch := make(chan int)
producers := &sync.WaitGroup{}
consumers := &sync.WaitGroup{}
producers.Add(2)
go producer(ch, 100*time.Millisecond, 2, producers)
go producer(ch, 200*time.Millisecond, 5, producers)
consumers.Add(1)
go consumer(ch, consumers)
producers.Wait()
close(ch)
consumers.Wait()
}
The problem is that <-ch is blocking, so if you don't add any new values to the channel it will block forever. One way is to replace it with a switch select which is also blocking but allows to listen on multiple channels. You would also have to add an exit channel. In your example, as soon as the exit channel received two values we can break. The break statement needs a label because we wanna exit from the switch and the for loop.
https://play.golang.org/p/wGdCulZDnrx
Another way is to have multiple input channels and close them as soon as they are finished sending. For this, each goroutine needs it's own channel, otherwise we will exit when the first goroutine is finished.
A third option is to create a merge function which merges multiple channels into one. This allows for moving the creation of the channels into the producers, so they are created, filled and closed in one location. The merge function is relatively complex but it's removed from the business logic code and can separately be understood and tested. the main code is then reduced to just:
ch1 := producer(100*time.Millisecond, 2)
ch2 := producer(200*time.Millisecond, 5)
for i := range merge(ch1, ch2) {
fmt.Println(i)
}
https://play.golang.org/p/2mv8ILhJPIB
merge func is from https://blog.golang.org/pipelines
You need to synchronize all the asynchronous process in your goroutines. Your main thread and the goroutine threads are not synchronous process. Your main thread will never knew when to stop invoking channel from goroutines. Since your main thread loop over the channel, it always invoke the value from channel, and when the goroutines finished and the channel stop sending value, your main thread cannot get anymore value from the channel, hence the condition become deadlock. To avoid this use sync.WaitGroup to synchronize the asynchronous process.
Here's the code:
package main
import (
"fmt"
"time"
"sync"
)
func producer(ch chan int, d time.Duration, num int, wg *sync.WaitGroup) {
for i:=0; i<num; i++ {
ch <- i;
time.Sleep(d);
}
defer wg.Done();
}
func main() {
wg := &sync.WaitGroup{}
ch := make(chan int);
wg.Add(2);
go producer(ch, 100*time.Millisecond, 2, wg);
go producer(ch, 200*time.Millisecond, 5, wg);
go func() {
wg.Wait()
close(ch)
}()
// print the outputs
for i:= range ch {
fmt.Println(i);
}
}
https://play.golang.org/p/euMTGTIs83g
Hope it helps.
Since my solution looks a little similar to already answered, I change it to my original answer before modification to suit OP question.
Here's the code:
package main
import (
"fmt"
"time"
"sync"
)
// producer produce values tobe sent to consumer
func producer(ch chan int, d time.Duration, num int, wg *sync.WaitGroup) {
defer wg.Done();
for i:=0; i<num; i++ {
ch <- i;
time.Sleep(d);
}
}
// consumer consume all values from producers
func consumer(ch chan int, out chan int, wg *sync.WaitGroup) {
defer wg.Done();
for i:= range ch {
out <- i
}
}
// synchronizer synchronize all goroutines to avoid deadlocks
func synchronizer(ch chan int, out chan int, wgp *sync.WaitGroup, wgc *sync.WaitGroup) {
wgp.Wait()
close(ch)
wgc.Wait()
close(out)
}
func main() {
wgp := &sync.WaitGroup{}
wgc := &sync.WaitGroup{}
ch := make(chan int);
out := make(chan int);
wgp.Add(2);
go producer(ch, 100*time.Millisecond, 2, wgp);
go producer(ch, 200*time.Millisecond, 5, wgp);
wgc.Add(1);
go consumer(ch, out, wgc)
go synchronizer(ch, out, wgp, wgc)
// print the outputs
for i:= range out {
fmt.Println(i);
}
}
Using consumer goroutine to fan-in all input from multiple goroutines and read all values from the consumer goroutine.
Hope it helps.
Simpler answer- one of the producers needs to close the channel, and the consumer can just range over the channel.
package main
import (
"fmt"
"time"
)
func producer(ch chan int, d time.Duration, num int, closer bool) {
for i:=0; i<num; i++ {
ch <- i
time.Sleep(d)
}
if closer {
close(ch)
}
}
func main() {
ch := make(chan int)
go producer(ch, 100*time.Millisecond, 2, false)
go producer(ch, 200*time.Millisecond, 5, true)
for i := range ch {
fmt.Println(i)
}
}
Of course, unless you have a situation where you know which producer will always finish last, you would not want to do this in real code. Better designs are in the WaitGroup-based patterns in the other answers. But this is the simplest way for this code to avoid deadlock.
In the following code I am trying to spawn MaxOutstanding number of handlers. Each handler loops over items in the queue queue and prints them out, I also write true to the done channel.
In my main function, I start the handlers and write 9 elements to the queue and wait for the 1st element to be written to the done queue.
package main
import "fmt"
type Request struct {
int32
}
var MaxOutstanding = 5
func handle(queue chan *Request, i int, done chan bool) {
for r := range queue {
fmt.Println(i, "---", r)
done <- true
}
}
func Serve(clientRequests chan *Request, quit, done chan bool) {
// Start handlers
for i := 0; i < MaxOutstanding; i++ {
go handle(clientRequests, i, done)
}
<-quit // Wait to be told to exit.
}
func main() {
clientRequests := make(chan *Request)
quit := make(chan bool)
done := make(chan bool)
go Serve(clientRequests, quit, done)
clientRequests <- &Request{4}
clientRequests <- &Request{1}
clientRequests <- &Request{2}
clientRequests <- &Request{3}
clientRequests <- &Request{5}
clientRequests <- &Request{6}
clientRequests <- &Request{7}
clientRequests <- &Request{8}
clientRequests <- &Request{9}
fmt.Println( "...........>", <- done )
close(clientRequests)
close(done)
}
On execution I get the following error. I don't see whats wrong with the implementation, I am even closing the channel.
4 --- &{4}
0 --- &{1}
1 --- &{2}
2 --- &{3}
3 --- &{5}
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.main()
/home/ubuntu/digs-svc/src/digs/go1.go:45 +0x251
goroutine 5 [chan receive]:
main.Serve(0xc82004c060, 0xc82004c0c0, 0xc82004c120)
/home/ubuntu/digs-svc/src/digs/go1.go:28 +0x92
created by main.main
/home/ubuntu/digs-svc/src/digs/go1.go:37 +0xb9
goroutine 6 [chan send]:
main.handle(0xc82004c060, 0x0, 0xc82004c120)
/home/ubuntu/digs-svc/src/digs/go1.go:16 +0x23b
created by main.Serve
/home/ubuntu/digs-svc/src/digs/go1.go:25 +0x5b
EDIT:
Apparently, the fmt.Println("....", <- done) wasn't enough to signify that there is a consumer to the done channel. I moved the code up in the execution order. A consumer needs to be "listening" on the channel when data is written to it. In my earlier code, the when the first data was written there were no consumer.
Working code.
https://play.golang.org/p/98l2M4XO9t
You're blocking the iteration over the channel in your handle function with the send on the done channel, because nothing is receiving on the other side.
Those extra channels aren't really doing anything, and you could just add a WaitGroup to synchronize the handler's exit, then you can remove the done channel which will allow the handler to continue.
func handle(queue chan *Request, i int, wg *sync.WaitGroup) {
defer wg.Done()
for r := range queue {
fmt.Println(i, "---", r)
}
}
func Serve(clientRequests chan *Request, wg *sync.WaitGroup) {
// Start handlers
for i := 0; i < MaxOutstanding; i++ {
wg.Add(1)
go handle(clientRequests, i, wg)
}
}
func main() {
clientRequests := make(chan *Request)
var wg sync.WaitGroup
go Serve(clientRequests, &wg)
for i := int32(0); i < 50; i++ {
clientRequests <- &Request{i}
}
close(clientRequests)
wg.Wait()
}
https://play.golang.org/p/oUFjZONjhk (note that in the playground, this example seems to currently favor a single goroutine being the receiver. Normally the blocked goroutines will receive randomly, and you can see that behavior if you compile and run normally)
Inside the for loop you are handling the channel operation only to the 5th element, however in the main function you are trying to send over the value to the channel, which is closed.
To overcome this situation you can send the request value inside a for loop:
for i := 0; i < MaxOutstanding; i++ {
clientRequests <- &Request{int32(i)}
}
Here is the working code:
package main
import (
"fmt"
)
type Request struct {
int32
}
var MaxOutstanding = 10
func handle(queue chan *Request, i int, done chan bool) {
for r := range queue {
fmt.Println(i, "---", r)
done <- true
}
}
func Serve(clientRequests chan *Request, quit, done chan bool) {
// Start handlers
for i := 0; i < MaxOutstanding; i++ {
go handle(clientRequests, i, done)
}
<-quit // Wait to be told to exit.
}
func main() {
clientRequests := make(chan *Request)
quit := make(chan bool)
done := make(chan bool)
go Serve(clientRequests, quit, done)
for i := 0; i < MaxOutstanding; i++ {
clientRequests <- &Request{int32(i)}
}
fmt.Println("...........>", <-done)
close(clientRequests)
close(done)
}
https://play.golang.org/p/L5Y2YoFNvz
FATAL Error All go routines are asleep. Deadlock.
This is what I tried. I am calling wg.Done(). What is missing?
package main
import (
"fmt"
"strconv"
"sync"
)
func sender(wg *sync.WaitGroup, cs chan int) {
defer wg.Done()
for i := 0; i < 2; i++ {
fmt.Println(i)
cs <- i
}
}
func reciever(wg *sync.WaitGroup, cs chan int) {
x, ok := <-cs
for ok {
fmt.Println("Retrieved" + strconv.Itoa(x))
x, ok = <-cs
if !ok {
wg.Done()
break
}
}
}
func main() {
wg := &sync.WaitGroup{}
cs := make(chan int, 1000)
wg.Add(1)
go sender(wg, cs)
for i := 1; i < 30; i++ {
wg.Add(1)
go reciever(wg, cs)
}
wg.Wait()
close(cs)
}
You should to close channel before wg.Wait.
All your receivers are waiting for data from channel. That's why you have deadlock.
You can close channel at defer statement of sender function.
Also you need to wg.Done() if the first attempt of receiving from channel was unsuccessful (because channel already closed)
http://play.golang.org/p/qdEIEfY-kl
There are couple of things:
You need to close the channel once sender is completed.
In receiver, range over channel
Don't need to add 1 to wait group and call Done in sender
Please refer to http://play.golang.org/p/vz39RY6WA7