All goroutines are asleep when reading from buffered channel - go

func writeToChan(wg *sync.WaitGroup, ch chan int, stop int) {
defer wg.Done()
for i := 0; i < stop; i++ {
ch <- i
}
}
func readToChan(wg *sync.WaitGroup, ch chan int) {
defer wg.Done()
for n := range ch {
fmt.Println(n)
}
}
func main() {
ch := make(chan int, 3)
wg := new(sync.WaitGroup)
wg.Add(2)
go writeToChan(wg, ch, 5)
go readToChan(wg, ch)
wg.Wait()
}
0
1
2
3
4
fatal error: all goroutines are asleep - deadlock!
I assume that the readToChan always reads continuously, and the writeToChan write to the channel and waits while the channel is read.
I don't know why the output showed deadlock while I added two 'wait' to the WaitGroup.

You need to close channel at the sender side.
By using
for n := range ch {
fmt.Println(n)
}
The loop will only stop when ch is closed
correct example:
package main
import (
"fmt"
"sync"
)
func writeToChan(wg *sync.WaitGroup, ch chan int, stop int) {
defer wg.Done()
for i := 0; i < stop; i++ {
ch <- i
}
close(ch)
}
func readToChan(wg *sync.WaitGroup, ch chan int) {
defer wg.Done()
for n := range ch {
fmt.Println(n)
}
}
func main() {
ch := make(chan int, 3)
wg := new(sync.WaitGroup)
wg.Add(2)
go writeToChan(wg, ch, 5)
go readToChan(wg, ch)
wg.Wait()
}

If close is not called on buffered channel, reader doesn't know when to stop reading.
Check this example with for and select calls(to handle multi channels).
https://go.dev/play/p/Lx5g9o4RsqW
package main
import (
"fmt"
"sync"
"time")
func writeToChan(wg *sync.WaitGroup, ch chan int, stop int, quit chan<- bool) {
defer func() {
wg.Done()
close(ch)
fmt.Println("write wg done")
}()
for i := 0; i < stop; i++ {
ch <- i
fmt.Println("write:", i)
}
fmt.Println("write done")
fmt.Println("sleeping for 5 sec")
time.Sleep(5 * time.Second)
quit <- true
close(quit)
}
func readToChan(wg *sync.WaitGroup, ch chan int, quit chan bool) {
defer func() {
wg.Done()
fmt.Println("read wg done")
}()
//using rang over
//for n := range ch {
// fmt.Println(n)
//}
//using Select if you have multiple channels.
for {
//fmt.Println("waiting for multiple channels")
select {
case n := <-ch:
fmt.Println("read:", n)
// if ok == false {
// fmt.Println("read done")
// //return
// }
case val := <-quit:
fmt.Println("received quit :", val)
return
// default:
// fmt.Println("default")
}
}
}
func main() {
ch := make(chan int, 5)
ch2 := make(chan bool)
wg := new(sync.WaitGroup)
wg.Add(2)
go writeToChan(wg, ch, 3, ch2)
go readToChan(wg, ch, ch2)
wg.Wait()
}
Output:
write: 0
write: 1
write: 2
write done
sleeping for 5 sec
read: 0
read: 1
read: 2
write wg done
received quit : true
read wg done
Program exited.

Related

WaitGroup - fatal error: all goroutines are asleep - deadlock

Can someone explain why this code throws an "fatal error: all goroutines are asleep - deadlock!"?
I can't seem to find what is wrong. I've seen some questions about this specific error, but the reason was mostly looping through a channel without closing it.
Thank you!
package main
import (
"fmt"
"sync"
"time"
)
func main() {
ch := make(chan time.Duration)
var wg sync.WaitGroup
for _, v := range []time.Duration{5, 1} {
wg.Add(1)
go wait(v, ch, wg)
fmt.Println(<-ch)
}
wg.Wait()
}
func wait(seconds time.Duration, c chan time.Duration, wg sync.WaitGroup) {
defer wg.Done()
time.Sleep(seconds * time.Second)
c <- seconds
}
You have to pass the WaitGroup by reference, not by value. Otherwise the Done has no effect. The documentation of this type says:
A WaitGroup must not be copied after first use.
Fix your code to this and it will work:
func main() {
ch := make(chan time.Duration)
var wg sync.WaitGroup
for _, v := range []time.Duration{5, 1} {
wg.Add(1)
go wait(v, ch, &wg)
fmt.Println(<-ch)
}
wg.Wait()
}
func wait(seconds time.Duration, c chan time.Duration, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(seconds * time.Second)
c <- seconds
}
It is also common to express this pattern as follows:
func main() {
ch := make(chan time.Duration)
var wg sync.WaitGroup
for _, v := range []time.Duration{5, 1} {
wg.Add(1)
go func() {
defer wg.Done()
wait(v, ch)
}()
fmt.Println(<-ch)
}
wg.Wait()
}
func wait(seconds time.Duration, c chan time.Duration) {
time.Sleep(seconds * time.Second)
c <- seconds
}
What's nice about this case is that wait doesn't have to be aware of any wait groups (it could be a 3rd-party function, for example), and there's no confusion about passing a wait group by value or reference.

How to fix the deadlock in this code, and extend it so that it can wait on a channel which has no writers?

I'm trying to learn channels in Go, so this is a contrived example of having a channel where there are multiple writers, but only one reader. This is a very basic example, but I would like to take it further, by imagining its a http server, where there is a goroutine created for each new request, and each goroutine makes a write on a channel.
func Start2(){
var wg sync.WaitGroup
wg.Add(2)
c := make(chan int)
go func(){
defer wg.Done()
i := 0
for {
i+=2
c<-i
time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
if i > 10 {
break
}
}
}()
go func(){
defer wg.Done()
i := 1
for {
i+=2
c<-i
time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
if i > 10 {
break
}
}
}()
for a := range c {
log.Println(a)
}
wg.Wait()
}
Why does this deadlock?
How could I organise the logic of reading from the channel, so that the code doesn't deadlock when there are no writers on the channel? (For instance, in the end goal example, if there isn't currently a request being made to the http server)?
Unbuffered channel needs at least two goroutines to operate and by exiting one of then. the Go runtime is smart enough to detect the deadlock, so you have two options here:
Since you want to use the channel in the http server - there is no deadlock here:
package main
import (
"fmt"
"log"
"net/http"
"time"
)
func main() {
http.HandleFunc("/", home)
go func() {
err := http.ListenAndServe(":8080", nil)
if err != nil {
log.Fatal(err)
}
}()
count := 0
for a := range c {
count++
fmt.Println(count, a)
}
}
var c = make(chan time.Time)
func home(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hi")
c <- time.Now()
}
Or close the channel to force the main goroutine to exit too - just for your current example code - try it:
package main
import (
"log"
"math/rand"
"sync"
"time"
)
func main() {
wg := &sync.WaitGroup{}
wg.Add(2)
c := make(chan int)
go routine(0, c, wg)
go routine(1, c, wg)
go func() {
wg.Wait()
close(c)
}()
for a := range c {
log.Println(a)
}
}
func routine(i int, c chan int, wg *sync.WaitGroup) {
defer wg.Done()
for {
i += 2
c <- i
time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
if i > 10 {
break
}
}
}
You are not closing your channel, you need to close it when you are done.
You can create a goroutine for reading from your channel and close it after the wait group.
Example:
var wg sync.WaitGroup
wg.Add(2)
c := make(chan int)
go func() {
defer wg.Done()
i := 0
for {
i += 2
c <- i
time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
if i > 10 {
return
}
}
}()
go func() {
defer wg.Done()
i := 1
for {
i += 2
c <- i
time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
if i > 10 {
break
}
}
}()
//read channel in a goroutine
go func() {
for a := range c {
log.Println(a)
}
}()
wg.Wait()
//close channel when done
close(c)

Wait for concurrent workers to finish before exiting

The following code has an obvious problem: the program will exit before all the work is finished by the workers.
Goroutines of the workers are launched before the sender starts sending data, which must remain. Starting these goroutines from the sender function is not an option. It would be easy to do so, however, need to learn a more complex synchronization technique.
What would be the correct way to wait for the workers to finish?
Have tried closing the worker1CH and worker2CH channels, as well as adding dedicated sync.WaitGroups to each worker.
package main
import (
"log"
"math/rand"
"sync"
)
func main() {
worker1CH := make(chan int, 1)
worker2CH := make(chan int, 1)
// worker for even numbers
go func(in chan int) {
for i := range in {
log.Print(i)
}
}(worker1CH)
// worker for odd numbers
go func(in chan int) {
for i := range in {
log.Print(i)
}
}(worker2CH)
// sender which sends even numbers to worker1CH, and odd numbers to worker2CH
var wg sync.WaitGroup
wg.Add(1)
go func(wg *sync.WaitGroup, evenChan chan int, oddChan chan int) {
defer wg.Done()
data := rand.Perm(10)
for _, i := range data {
switch i%2 {
case 0:
evenChan <- i
default:
oddChan <- i
}
}
}(&wg, worker1CH, worker2CH)
wg.Wait()
}
Wait for the two receiving goroutines to complete using a wait group. Use one wait group to wait for both goroutines.
Close the channels after sending all values so that the loops in the receiving goroutines exit.
There's no need to wait for the sending goroutine. The grouting competes all of it's work before the other coroutines complete.
worker1CH := make(chan int, 1)
worker2CH := make(chan int, 1)
var wg sync.WaitGroup
wg.Add(2) // <-- wait for the two receiving goroutines.
// worker for even numbers
go func(wg *sync.WaitGroup, in chan int) {
defer wg.Done() // <--- add this line
for i := range in {
log.Print(i)
}
}(&wg, worker1CH)
// worker for odd numbers
go func(wg *sync.WaitGroup, in chan int) {
defer wg.Done() <-- add this line
for i := range in {
log.Print(i)
}
}(&wg, worker2CH)
// sender which sends even numbers to worker1CH, and odd numbers to worker2CH
go func(evenChan chan int, oddChan chan int) {
defer close(evenChan) // <-- close channel so that receiver exits loop
defer close(oddChan) // <-- ditto
data := rand.Perm(10)
for _, i := range data {
switch i % 2 {
case 0:
evenChan <- i
default:
oddChan <- i
}
}
}(worker1CH, worker2CH)
wg.Wait()
Run the example on the Go Playground.
Have Been able to create worker1Done and worker2Done channels, then waiting for the work to finish.
Also had to add close(evenChan) and close(oddChan) to the sender function to avoid the fatal error: all goroutines are asleep - deadlock! error
package main
import (
"log"
"math/rand"
"sync"
)
func main() {
worker1CH := make(chan int, 1)
worker2CH := make(chan int, 1)
worker1Done := make(chan bool)
worker2Done := make(chan bool)
// worker for even numbers
go func(in chan int, done chan bool) {
for i := range in {
log.Print(i)
}
done <- true
}(worker1CH, worker1Done)
// worker for odd numbers
go func(in chan int, done chan bool) {
for i := range in {
log.Print(i)
}
done <- true
}(worker2CH, worker2Done)
// sender which sends even numbers to worker1CH, and odd numbers to worker2CH
var wg sync.WaitGroup
wg.Add(1)
go func(wg *sync.WaitGroup, evenChan chan int, oddChan chan int) {
defer wg.Done()
data := rand.Perm(10)
for _, i := range data {
switch i%2 {
case 0:
evenChan <- i
default:
oddChan <- i
}
}
close(evenChan)
close(oddChan)
}(&wg, worker1CH, worker2CH)
wg.Wait()
<- worker1Done
<- worker2Done
}
Since your sender has fix size so it will be exiton its own and you can just close the channel for reader and wait
package main
import (
"log"
"math/rand"
"sync"
)
func reader(in chan int, wg *sync.WaitGroup) {
defer wg.Done()
for i := range in {
log.Print(i)
}
}
func main() {
var wg sync.WaitGroup
worker1CH := make(chan int, 1)
worker2CH := make(chan int, 1)
wg.Add(1)
// worker for even numbers
go reader(worker1CH, &wg)
wg.Add(1)
// worker for odd numbers
go reader(worker2CH, &wg)
// sender which sends even numbers to worker1CH, and odd numbers to worker2CH
sender(worker1CH, worker2CH)
close(worker2CH)
close(worker1CH)
wg.Wait()
}
func sender(evenChan chan int, oddChan chan int) {
data := rand.Perm(10)
for _, i := range data {
switch i % 2 {
case 0:
evenChan <- i
default:
oddChan <- i
}
}
}
playground link https://play.golang.org/p/JJ9ngCHUvbS

golang goroutines, chanel and sync.WaitGroup

I use simple code, but I get deadlock all the time.
Please explain to the beginner what I am doing wrong.
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
ok := make(chan int, 3)
for i := 0; i < 2; i++ {
wg.Add(1)
go func(i int, wg *sync.WaitGroup) {
for x := range ok {
fmt.Println(i, x)
if x >= 5 {
ok <- x - 1
}
}
wg.Done()
}(i, &wg)
}
ok <- 1
ok <- 3
ok <- 2
ok <- 5
ok <- 3
ok <- 9
wg.Wait()
}
fatal error: all goroutines are asleep - deadlock!
The background of the problem is that the range operator reads from the channel until the channel is closed. Hence the for-range loop keeps waiting for more input from the channel ok even though there are no more inputs to come at some point. At the same time wg.Wait() is waiting for the goroutines to finish. Hence a deadlock!
Either you'll have to close the ok channel at some point, where
there are no more incoming values to the channel ok.
Or you may use the worker pool just like,
package main
import (
"fmt"
"strconv"
"sync"
)
func worker(wg *sync.WaitGroup, ch chan string, i int) {
defer wg.Done()
ch <- "worker process value " + strconv.Itoa(i)
}
func workerMonitor(wg *sync.WaitGroup, ch chan string) {
wg.Wait()
close(ch)
}
func doWork(ch <-chan string, done chan<- bool) {
for i := range ch {
fmt.Println(i)
}
done <- true
}
func main() {
var wg sync.WaitGroup
ch := make(chan string)
var arr = [6]int{1, 3, 2, 5, 3, 9}
for i := 0; i < len(arr); i++ {
wg.Add(1)
if arr[i] >= 5 {
for j := arr[i]; j >= 5; j-- {
wg.Add(1)
go worker(&wg, ch, j-1)
}
}
go worker(&wg, ch, arr[i])
}
go workerMonitor(&wg, ch)
done := make(chan bool, 1)
go doWork(ch, done)
<-done
}
Play ground url

go routine for range over channels

I have been working in Golang for a long time. But still I am facing this problem though I know the solution to my problem. But never figured out why is it happening.
For example If I have a pipeline situation for inbound and outbound channels like below:
package main
import (
"fmt"
)
func main() {
for n := range sq(sq(gen(3, 4))) {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n
}
close(out)
}()
return out
}
It does not give me a deadlock situation. But if I remove the go routine inside the outbound code as below:
func sq(in <-chan int) <-chan int {
out := make(chan int)
for n := range in {
out <- n * n
}
close(out)
return out
}
I received a deadlock error. Why is it so that looping over channels using range without go routine gives a deadlock.
This situation caused of output channel of sq function is not buffered. So sq is waiting until next function will read from output, but if sq is not async, it will not happen (Playground link):
package main
import (
"fmt"
"sync"
)
var wg sync.WaitGroup
func main() {
numsCh := gen(3, 4)
sqCh := sq(numsCh) // if there is no sq in body - we are locked here until input channel will be closed
result := sq(sqCh) // but if output channel is not buffered, so `sq` is locked, until next function will read from output channel
for n := range result {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int, 100)
for n := range in {
out <- n * n
}
close(out)
return out
}
Your function creates a channel, writes to it, then returns it. The writing will block until somebody can read the corresponding value, but that's impossible because nobody outside this function has the channel yet.
func sq(in <-chan int) <-chan int {
// Nobody else has this channel yet...
out := make(chan int)
for n := range in {
// ...but this line will block until somebody reads the value...
out <- n * n
}
close(out)
// ...and nobody else can possibly read it until after this return.
return out
}
If you wrap the loop in a goroutine then both the loop and the sq function are allowed to continue; even if the loop blocks, the return out statement can still go and eventually you'll be able to connect up a reader to the channel.
(There's nothing intrinsically bad about looping over channels outside of goroutines; your main function does it harmlessly and correctly.)
The reason of the deadlock is because the main is waiting for the sq return and finish, but the sq is waiting for someone read the chan then it can continue.
I simplified your code by removing layer of sq call, and split one sentence into 2 :
func main() {
result := sq(gen(3, 4)) // <-- block here, because sq doesn't return
for n := range result {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int)
for n := range in {
out <- n * n // <-- block here, because no one is reading from the chan
}
close(out)
return out
}
In sq method, if you put code in goroutine, then the sq will returned, and main func will not block, and consume the result queue, and the goroutine will continue, then there is no block any more.
func main() {
result := sq(gen(3, 4)) // will not blcok here, because the sq just start a goroutine and return
for n := range result {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n // will not block here, because main will continue and read the out chan
}
close(out)
}()
return out
}
The code is a bit complicated,
Let's simplify
First eq below, not has deadLock
func main() {
send := make(chan int)
receive := make(chan int)
go func() {
send<-3
send<-4
close(send)
}()
go func() {
receive<- <-send
receive<- <-send
close(receive)
}()
for v := range receive{
fmt.Println(v)
}
}
Second eq below,remove "go" has deadLock
func main() {
send := make(chan int)
receive := make(chan int)
go func() {
send<-3
send<-4
close(send)
}()
receive<- <-send
receive<- <-send
close(receive)
for v := range receive{
fmt.Println(v)
}
}
Let's simplify second code again
func main() {
ch := make(chan int)
ch <- 3
ch <- 4
close(ch)
for v := range ch{
fmt.Println(v)
}
}
The reason of the deadlock is no buffer channel waiting in main goroutine.
Two Solutions
// add more cap then "channel<-" time
func main() {
ch := make(chan int,2)
ch <- 3
ch <- 4
close(ch)
for v := range ch{
fmt.Println(v)
}
}
//async "<-channel"
func main() {
ch := make(chan int)
go func() {
for v := range ch {
fmt.Println(v)
}
}()
ch <- 3
ch <- 4
close(ch)
}
My understanding is
when the main thread is blocked for waiting for the chan to be writen or read, Go will detect if any other Go routine is running. If there is no any other Go routine running, it will have "fatal error: all goroutines are asleep - deadlock!"
I tested if by using the below simple case
func main() {
c := make(chan int)
go func() {
time.Sleep(10 * time.Second)
}()
c <- 1
}
The deadlock error is reported after 10 seconds.

Resources