In the following code I am trying to spawn MaxOutstanding number of handlers. Each handler loops over items in the queue queue and prints them out, I also write true to the done channel.
In my main function, I start the handlers and write 9 elements to the queue and wait for the 1st element to be written to the done queue.
package main
import "fmt"
type Request struct {
int32
}
var MaxOutstanding = 5
func handle(queue chan *Request, i int, done chan bool) {
for r := range queue {
fmt.Println(i, "---", r)
done <- true
}
}
func Serve(clientRequests chan *Request, quit, done chan bool) {
// Start handlers
for i := 0; i < MaxOutstanding; i++ {
go handle(clientRequests, i, done)
}
<-quit // Wait to be told to exit.
}
func main() {
clientRequests := make(chan *Request)
quit := make(chan bool)
done := make(chan bool)
go Serve(clientRequests, quit, done)
clientRequests <- &Request{4}
clientRequests <- &Request{1}
clientRequests <- &Request{2}
clientRequests <- &Request{3}
clientRequests <- &Request{5}
clientRequests <- &Request{6}
clientRequests <- &Request{7}
clientRequests <- &Request{8}
clientRequests <- &Request{9}
fmt.Println( "...........>", <- done )
close(clientRequests)
close(done)
}
On execution I get the following error. I don't see whats wrong with the implementation, I am even closing the channel.
4 --- &{4}
0 --- &{1}
1 --- &{2}
2 --- &{3}
3 --- &{5}
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.main()
/home/ubuntu/digs-svc/src/digs/go1.go:45 +0x251
goroutine 5 [chan receive]:
main.Serve(0xc82004c060, 0xc82004c0c0, 0xc82004c120)
/home/ubuntu/digs-svc/src/digs/go1.go:28 +0x92
created by main.main
/home/ubuntu/digs-svc/src/digs/go1.go:37 +0xb9
goroutine 6 [chan send]:
main.handle(0xc82004c060, 0x0, 0xc82004c120)
/home/ubuntu/digs-svc/src/digs/go1.go:16 +0x23b
created by main.Serve
/home/ubuntu/digs-svc/src/digs/go1.go:25 +0x5b
EDIT:
Apparently, the fmt.Println("....", <- done) wasn't enough to signify that there is a consumer to the done channel. I moved the code up in the execution order. A consumer needs to be "listening" on the channel when data is written to it. In my earlier code, the when the first data was written there were no consumer.
Working code.
https://play.golang.org/p/98l2M4XO9t
You're blocking the iteration over the channel in your handle function with the send on the done channel, because nothing is receiving on the other side.
Those extra channels aren't really doing anything, and you could just add a WaitGroup to synchronize the handler's exit, then you can remove the done channel which will allow the handler to continue.
func handle(queue chan *Request, i int, wg *sync.WaitGroup) {
defer wg.Done()
for r := range queue {
fmt.Println(i, "---", r)
}
}
func Serve(clientRequests chan *Request, wg *sync.WaitGroup) {
// Start handlers
for i := 0; i < MaxOutstanding; i++ {
wg.Add(1)
go handle(clientRequests, i, wg)
}
}
func main() {
clientRequests := make(chan *Request)
var wg sync.WaitGroup
go Serve(clientRequests, &wg)
for i := int32(0); i < 50; i++ {
clientRequests <- &Request{i}
}
close(clientRequests)
wg.Wait()
}
https://play.golang.org/p/oUFjZONjhk (note that in the playground, this example seems to currently favor a single goroutine being the receiver. Normally the blocked goroutines will receive randomly, and you can see that behavior if you compile and run normally)
Inside the for loop you are handling the channel operation only to the 5th element, however in the main function you are trying to send over the value to the channel, which is closed.
To overcome this situation you can send the request value inside a for loop:
for i := 0; i < MaxOutstanding; i++ {
clientRequests <- &Request{int32(i)}
}
Here is the working code:
package main
import (
"fmt"
)
type Request struct {
int32
}
var MaxOutstanding = 10
func handle(queue chan *Request, i int, done chan bool) {
for r := range queue {
fmt.Println(i, "---", r)
done <- true
}
}
func Serve(clientRequests chan *Request, quit, done chan bool) {
// Start handlers
for i := 0; i < MaxOutstanding; i++ {
go handle(clientRequests, i, done)
}
<-quit // Wait to be told to exit.
}
func main() {
clientRequests := make(chan *Request)
quit := make(chan bool)
done := make(chan bool)
go Serve(clientRequests, quit, done)
for i := 0; i < MaxOutstanding; i++ {
clientRequests <- &Request{int32(i)}
}
fmt.Println("...........>", <-done)
close(clientRequests)
close(done)
}
https://play.golang.org/p/L5Y2YoFNvz
Related
I'm having trouble wrangling go routines and getting them to communicate back to a channel on the main go routine. To simplify, my code looks something like this:
func main() {
channel := make(chan string)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go performTest(channel, &wg, i)
}
wg.Wait()
close(channel)
for line := range channel {
fmt.Print(line)
}
}
func performTest(channel chan string, wg *sync.WaitGroup, i int) {
defer wg.Done()
// perform some work here
result := fmt.sprintf("Pretend result %d", i)
channel <- result
}
This seems to enter into some kind of a deadlock, but I don't understand why. It gets stuck on wg.Wait(), even though I would expect it to continue once all the goroutines have called Done on the wait group. What am I missing here? I'd like to wait for the goroutines, and then iterate over all results in the channel.
You can wait for the group and close the channel in a separate go routine. If the channel is closed, your range over the channel will end after the last sent value has been received.
If you just wait, nothing will receive from the channel. Since the channel is unbuffered, the performTest goroutines won't be able to send. For an unbuffered channel, the send operation will block until it has been received. Therefore, the deferred wg.Done call would never happen, and your program is deadlocked. Since Done is only called after the forever-blocking send has been performed.
func main() {
channel := make(chan string)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go performTest(channel, &wg, i)
}
// this is the trick
go func() {
wg.Wait()
close(channel)
}()
for line := range channel {
fmt.Print(line)
}
}
func performTest(channel chan string, wg *sync.WaitGroup, i int) {
defer wg.Done()
// perform some work here
result := fmt.Sprintf("Pretend result %d\n", i)
channel <- result
}
https://play.golang.com/p/5pACJzwL4Hi
Im not 100% clear on why my code doesnt work
package main
import (
"fmt"
"sync"
)
//var wg sync.WaitGroup
func main() {
c := make(chan int)
send(c)
receive(c)
}
func send(c chan<- int) {
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func() {
for j := 0; j < 10; j++ {
c <- j
}
wg.Done()
}()
}
wg.Wait()
close(c)
}
func receive(c <-chan int) {
for v := range c {
fmt.Println(v)
}
}
If I put Go before send it works fine. But i thought having the wait in the send would mean the program would wait there until the go routines created in that function completed and would only go to the receive function once they were all done and so the channel was populated?
The channel is not buffered. A write to an unbuffered channel will only complete if there is a read from that channel in another goroutine. So, all the goroutines created in the send are stuck waiting to write to the channel, never reaching wg.Done, and thus, wg.Wait waits indefinitely. When you put the send in a separate goroutine, receive can run, thus reading from the channel, releasing the waiting goroutines.
I have made a simple code example to understand the usage of pipeline, here it is.
package main
import (
"fmt"
"sync"
"time"
)
func main() {
ch1 := make(chan int, 10) // Use buffered channel so as to avoid clogging
ch2 := make(chan string, 10)
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func1(i, ch1, &wg)
go func2(ch1, ch2)
}
wg.Wait()
close(ch1)
for val := range ch2 {
fmt.Println(val)
}
}
func func1(seconds int, ch chan<- int, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(time.Duration(seconds) * time.Second)
fmt.Println(seconds)
ch <- seconds
}
func func2(ch1 chan int, ch2 chan string) {
for range ch1 {
ch2 <- "hello"
}
close(ch2)
}
Now, the problem is I don't get consistent output ( I understand it's some concurrency issue, which I haven't fully understood ).
Output
> go run pipeline-loop.go
0
1
2
hello
hello
> go run pipeline-loop.go
0
1
2
hello
hello
hello
> go run pipeline-loop.go
0
1
2
hello
hello
> go run pipeline-loop.go
0
1
2
hello
hello
> go run pipeline-loop.go
0
1
2
hello
hello
panic: close of closed channel
goroutine 6 [running]:
main.func2(0xc00006c000, 0xc000056180)
/home/projects/go-tuts/pipeline-loop.go:36 +0x72
created by main.main
/home/projects/go-tuts/pipeline-loop.go:16 +0x10f
exit status 2
Another guy changed the code ( and it was working ) and put func2 outside the loop but I want func2 for each iteration of the func1.
Problem
So, I want to understand as to where should the WaitGroup and close(ch) be used ?
Thanks.
Temporarya
( A golang noobie )
Update
Based on a user's answer, I changed the code, now I get output as expected ( but not the solution to this question ) but still there's a deadlock.
https://play.golang.org/p/O_rp_FLvNh8
package main
import (
"fmt"
"sync"
"time"
)
func main() {
ch1 := make(chan int, 10) // Use buffered channel so as to avoid clogging
ch2 := make(chan string, 10)
var wg1 sync.WaitGroup
// var wg2 sync.WaitGroup
for i := 0; i < 3; i++ {
wg1.Add(1)
go func1(i, ch1)
go func2(ch1, ch2, &wg1)
}
for val := range ch2 {
fmt.Println(val)
}
wg1.Wait()
close(ch1)
close(ch2)
}
// func func1(seconds int, ch chan<- int, wg *sync.WaitGroup) {
func func1(seconds int, ch chan<- int) {
// defer wg.Done()
time.Sleep(time.Duration(seconds) * time.Second)
fmt.Println(seconds)
ch <- seconds
}
func func2(ch1 chan int, ch2 chan string, wg *sync.WaitGroup) {
defer wg.Done()
for range ch1 {
ch2 <- "hello"
}
}
There are multiple problems in your code.
In the loop, you are spawning multiple (3) goroutines that runs func2, and in func2, you send data to ch2 and call close(ch2). This is a problem. It can happen that when one goroutine is seding data to ch2, the other has closed that channel, which causes:
panic: close of closed channel
goroutine 6 [running]:
main.func2(0xc00006c000, 0xc000056180)
/home/projects/go-tuts/pipeline-loop.go:36 +0x72
created by main.main
/home/projects/go-tuts/pipeline-loop.go:16 +0x10f
exit status 2
In general, you don't need close a chanel multiple times - you only need to shut them once they are all finished. You need another WaitGroup for this; you need to pass both functions a WaitGroup.
Further reading: https://blog.golang.org/pipelines
UPDATE:
Personally I use a pattern for "works" that produce data into a same channel and the channel needs to be closed after all works are done:
for something {
wg.Add(1)
go func(i int) {
work(ch)
wg.Done()
}
}
go func() {
wg.Wait()
close()
}()
I think it is a good idea that keeps the API clean from WorkGroup as WorkGroup is about how you sync the work instead of how the work is done.
I have changed your code into this pattern: https://play.golang.org/p/vdCNsxWhgyQ
I suspect you only want one channel to read from ch1 and write to ch2. There is not much point in creating 3 go-routines to do the same thing (and you also end up closing the same channel mutliple time which causes a panic as leaf bebop pointed out)
func main() {
ch1 := make(chan int, 10) // Use buffered channel so as to avoid clogging
ch2 := make(chan string, 10)
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func1(i, ch1, &wg)
}
go func2(ch1, ch2)
wg.Wait()
close(ch1)
for val := range ch2 {
fmt.Println(val)
}
}
FATAL Error All go routines are asleep. Deadlock.
This is what I tried. I am calling wg.Done(). What is missing?
package main
import (
"fmt"
"strconv"
"sync"
)
func sender(wg *sync.WaitGroup, cs chan int) {
defer wg.Done()
for i := 0; i < 2; i++ {
fmt.Println(i)
cs <- i
}
}
func reciever(wg *sync.WaitGroup, cs chan int) {
x, ok := <-cs
for ok {
fmt.Println("Retrieved" + strconv.Itoa(x))
x, ok = <-cs
if !ok {
wg.Done()
break
}
}
}
func main() {
wg := &sync.WaitGroup{}
cs := make(chan int, 1000)
wg.Add(1)
go sender(wg, cs)
for i := 1; i < 30; i++ {
wg.Add(1)
go reciever(wg, cs)
}
wg.Wait()
close(cs)
}
You should to close channel before wg.Wait.
All your receivers are waiting for data from channel. That's why you have deadlock.
You can close channel at defer statement of sender function.
Also you need to wg.Done() if the first attempt of receiving from channel was unsuccessful (because channel already closed)
http://play.golang.org/p/qdEIEfY-kl
There are couple of things:
You need to close the channel once sender is completed.
In receiver, range over channel
Don't need to add 1 to wait group and call Done in sender
Please refer to http://play.golang.org/p/vz39RY6WA7
package main
import (
"fmt"
"runtime"
"sync"
"time"
)
func main() {
intInputChan := make(chan int, 50)
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go worker(intInputChan, wg)
}
for i := 1; i < 51; i++ {
fmt.Printf("Inputs. %d \n", i)
intInputChan <- i
}
close(intInputChan)
wg.Wait()
fmt.Println("Existing Main App... ")
panic("---------------")
}
func worker(input chan int, wg sync.WaitGroup) {
defer func() {
fmt.Println("Executing defer..")
wg.Done()
}()
for {
select {
case intVal, ok := <-input:
time.Sleep(100 * time.Millisecond)
if !ok {
input = nil
return
}
fmt.Printf("%d %v\n", intVal, ok)
default:
runtime.Gosched()
}
}
}
error thrown is.
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.(*WaitGroup).Wait(0xc082004600)
c:/go/src/sync/waitgroup.go:132 +0x170
main.main()
E:/Go/go_projects/go/src/Test.go:22 +0x21a
I just tried it (playground) passing a wg *sync.WaitGroup and it works.
Passing sync.WaitGroup means passing a copy of the sync.WaitGroup (passing by value): the goroutine mentions Done() to a different sync.WaitGroup.
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go worker(intInputChan, &wg)
}
Note the &wg: you are passing by value the pointer to the original sync.WaitGroup, for the goroutine to use.
As mentioned, don't pass types from the sync package around by value, right near the top of the sync package documentation: "Values containing the types defined in this package should not be copied." That also includes the types themselves (sync.Mutex, sync.WaitGroup, etc).
However, several notes:
You can use just a single call to wg.Add if you know how many you're going to add (but as documented make sure it's done before anything can call Wait).
You don't want to call runtime.Gosched like that; it makes the workers busy loop.
You can use range to read from the channel to simplify stopping when it's closed.
For small functions you can use a closure and not bother to pass the channel or wait group at all.
That turns it into this:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
const numWorkers = 3
c := make(chan int, 10)
var wg sync.WaitGroup
wg.Add(numWorkers)
for i := 0; i < numWorkers; i++ {
go func() {
defer func() {
fmt.Println("Executing defer…")
wg.Done()
}()
for v := range c {
fmt.Println("recv:", v)
time.Sleep(100 * time.Millisecond)
}
}()
}
for i := 1; i < 51; i++ {
fmt.Println("send:", i)
c <- i
}
fmt.Println("closing…")
close(c)
fmt.Println("waiting…")
wg.Wait()
fmt.Println("Exiting Main App... ")
}
playground