Wait for concurrent workers to finish before exiting - go

The following code has an obvious problem: the program will exit before all the work is finished by the workers.
Goroutines of the workers are launched before the sender starts sending data, which must remain. Starting these goroutines from the sender function is not an option. It would be easy to do so, however, need to learn a more complex synchronization technique.
What would be the correct way to wait for the workers to finish?
Have tried closing the worker1CH and worker2CH channels, as well as adding dedicated sync.WaitGroups to each worker.
package main
import (
"log"
"math/rand"
"sync"
)
func main() {
worker1CH := make(chan int, 1)
worker2CH := make(chan int, 1)
// worker for even numbers
go func(in chan int) {
for i := range in {
log.Print(i)
}
}(worker1CH)
// worker for odd numbers
go func(in chan int) {
for i := range in {
log.Print(i)
}
}(worker2CH)
// sender which sends even numbers to worker1CH, and odd numbers to worker2CH
var wg sync.WaitGroup
wg.Add(1)
go func(wg *sync.WaitGroup, evenChan chan int, oddChan chan int) {
defer wg.Done()
data := rand.Perm(10)
for _, i := range data {
switch i%2 {
case 0:
evenChan <- i
default:
oddChan <- i
}
}
}(&wg, worker1CH, worker2CH)
wg.Wait()
}

Wait for the two receiving goroutines to complete using a wait group. Use one wait group to wait for both goroutines.
Close the channels after sending all values so that the loops in the receiving goroutines exit.
There's no need to wait for the sending goroutine. The grouting competes all of it's work before the other coroutines complete.
worker1CH := make(chan int, 1)
worker2CH := make(chan int, 1)
var wg sync.WaitGroup
wg.Add(2) // <-- wait for the two receiving goroutines.
// worker for even numbers
go func(wg *sync.WaitGroup, in chan int) {
defer wg.Done() // <--- add this line
for i := range in {
log.Print(i)
}
}(&wg, worker1CH)
// worker for odd numbers
go func(wg *sync.WaitGroup, in chan int) {
defer wg.Done() <-- add this line
for i := range in {
log.Print(i)
}
}(&wg, worker2CH)
// sender which sends even numbers to worker1CH, and odd numbers to worker2CH
go func(evenChan chan int, oddChan chan int) {
defer close(evenChan) // <-- close channel so that receiver exits loop
defer close(oddChan) // <-- ditto
data := rand.Perm(10)
for _, i := range data {
switch i % 2 {
case 0:
evenChan <- i
default:
oddChan <- i
}
}
}(worker1CH, worker2CH)
wg.Wait()
Run the example on the Go Playground.

Have Been able to create worker1Done and worker2Done channels, then waiting for the work to finish.
Also had to add close(evenChan) and close(oddChan) to the sender function to avoid the fatal error: all goroutines are asleep - deadlock! error
package main
import (
"log"
"math/rand"
"sync"
)
func main() {
worker1CH := make(chan int, 1)
worker2CH := make(chan int, 1)
worker1Done := make(chan bool)
worker2Done := make(chan bool)
// worker for even numbers
go func(in chan int, done chan bool) {
for i := range in {
log.Print(i)
}
done <- true
}(worker1CH, worker1Done)
// worker for odd numbers
go func(in chan int, done chan bool) {
for i := range in {
log.Print(i)
}
done <- true
}(worker2CH, worker2Done)
// sender which sends even numbers to worker1CH, and odd numbers to worker2CH
var wg sync.WaitGroup
wg.Add(1)
go func(wg *sync.WaitGroup, evenChan chan int, oddChan chan int) {
defer wg.Done()
data := rand.Perm(10)
for _, i := range data {
switch i%2 {
case 0:
evenChan <- i
default:
oddChan <- i
}
}
close(evenChan)
close(oddChan)
}(&wg, worker1CH, worker2CH)
wg.Wait()
<- worker1Done
<- worker2Done
}

Since your sender has fix size so it will be exiton its own and you can just close the channel for reader and wait
package main
import (
"log"
"math/rand"
"sync"
)
func reader(in chan int, wg *sync.WaitGroup) {
defer wg.Done()
for i := range in {
log.Print(i)
}
}
func main() {
var wg sync.WaitGroup
worker1CH := make(chan int, 1)
worker2CH := make(chan int, 1)
wg.Add(1)
// worker for even numbers
go reader(worker1CH, &wg)
wg.Add(1)
// worker for odd numbers
go reader(worker2CH, &wg)
// sender which sends even numbers to worker1CH, and odd numbers to worker2CH
sender(worker1CH, worker2CH)
close(worker2CH)
close(worker1CH)
wg.Wait()
}
func sender(evenChan chan int, oddChan chan int) {
data := rand.Perm(10)
for _, i := range data {
switch i % 2 {
case 0:
evenChan <- i
default:
oddChan <- i
}
}
}
playground link https://play.golang.org/p/JJ9ngCHUvbS

Related

All goroutines are asleep when reading from buffered channel

func writeToChan(wg *sync.WaitGroup, ch chan int, stop int) {
defer wg.Done()
for i := 0; i < stop; i++ {
ch <- i
}
}
func readToChan(wg *sync.WaitGroup, ch chan int) {
defer wg.Done()
for n := range ch {
fmt.Println(n)
}
}
func main() {
ch := make(chan int, 3)
wg := new(sync.WaitGroup)
wg.Add(2)
go writeToChan(wg, ch, 5)
go readToChan(wg, ch)
wg.Wait()
}
0
1
2
3
4
fatal error: all goroutines are asleep - deadlock!
I assume that the readToChan always reads continuously, and the writeToChan write to the channel and waits while the channel is read.
I don't know why the output showed deadlock while I added two 'wait' to the WaitGroup.
You need to close channel at the sender side.
By using
for n := range ch {
fmt.Println(n)
}
The loop will only stop when ch is closed
correct example:
package main
import (
"fmt"
"sync"
)
func writeToChan(wg *sync.WaitGroup, ch chan int, stop int) {
defer wg.Done()
for i := 0; i < stop; i++ {
ch <- i
}
close(ch)
}
func readToChan(wg *sync.WaitGroup, ch chan int) {
defer wg.Done()
for n := range ch {
fmt.Println(n)
}
}
func main() {
ch := make(chan int, 3)
wg := new(sync.WaitGroup)
wg.Add(2)
go writeToChan(wg, ch, 5)
go readToChan(wg, ch)
wg.Wait()
}
If close is not called on buffered channel, reader doesn't know when to stop reading.
Check this example with for and select calls(to handle multi channels).
https://go.dev/play/p/Lx5g9o4RsqW
package main
import (
"fmt"
"sync"
"time")
func writeToChan(wg *sync.WaitGroup, ch chan int, stop int, quit chan<- bool) {
defer func() {
wg.Done()
close(ch)
fmt.Println("write wg done")
}()
for i := 0; i < stop; i++ {
ch <- i
fmt.Println("write:", i)
}
fmt.Println("write done")
fmt.Println("sleeping for 5 sec")
time.Sleep(5 * time.Second)
quit <- true
close(quit)
}
func readToChan(wg *sync.WaitGroup, ch chan int, quit chan bool) {
defer func() {
wg.Done()
fmt.Println("read wg done")
}()
//using rang over
//for n := range ch {
// fmt.Println(n)
//}
//using Select if you have multiple channels.
for {
//fmt.Println("waiting for multiple channels")
select {
case n := <-ch:
fmt.Println("read:", n)
// if ok == false {
// fmt.Println("read done")
// //return
// }
case val := <-quit:
fmt.Println("received quit :", val)
return
// default:
// fmt.Println("default")
}
}
}
func main() {
ch := make(chan int, 5)
ch2 := make(chan bool)
wg := new(sync.WaitGroup)
wg.Add(2)
go writeToChan(wg, ch, 3, ch2)
go readToChan(wg, ch, ch2)
wg.Wait()
}
Output:
write: 0
write: 1
write: 2
write done
sleeping for 5 sec
read: 0
read: 1
read: 2
write wg done
received quit : true
read wg done
Program exited.

Expected output as well as deadlock in a worker pool

I'm learning Go concurrency and wrote the obligatory worker pool example, where there are N pieces of work and M workers (N > M). I'm running into a deadlock (all goroutines are asleep), which I can't figure out; however, I'm also getting the expected output before the deadlock occurs, which has me even more confused. Can someone please point out the things I'm doing wrong?
My code is this:
package main
import (
"fmt"
"sync"
)
// A simple worker pool implementation using channels and WaitGroups.
// Our workers simply read from a channel of integers from an input
// channel and write their squares to an output channel.
func addJobs(jobsCh chan<- int, wg *sync.WaitGroup) {
// 100 numbers to crunch (jobs)
for i := 1; i < 101; i++ {
jobsCh <- i
}
wg.Done()
}
func worker(jobsCh <-chan int, resultsCh chan<- int, wg2 *sync.WaitGroup) {
for num := range jobsCh {
resultsCh <- num * num
}
wg2.Done()
}
func addWorkers(jobsCh <-chan int, resultsCh chan<- int, wg *sync.WaitGroup) {
var wg2 sync.WaitGroup
// 10 workers
for i := 0; i < 10; i++ {
wg2.Add(1)
go worker(jobsCh, resultsCh, &wg2)
}
wg.Done()
}
func readResults(resultsCh <-chan int, wg *sync.WaitGroup) {
for sq := range resultsCh {
fmt.Printf("%v ", sq)
}
wg.Done()
}
func main() {
var wg sync.WaitGroup
jobsCh := make(chan int)
resultsCh := make(chan int)
wg.Add(1)
go addJobs(jobsCh, &wg)
wg.Add(1)
go addWorkers(jobsCh, resultsCh, &wg)
wg.Add(1)
go readResults(resultsCh, &wg)
wg.Wait()
}
This prints the squares of the numbers (in random order), as expected, but also runs into a deadlock. Please see this playground link. :(
Close jobsCh to cause workers to exit:
func addJobs(jobsCh chan<- int, wg *sync.WaitGroup) {
// 100 numbers to crunch (jobs)
for i := 1; i < 101; i++ {
jobsCh <- i
}
close(jobsCh) // <-- add this line
wg.Done()
}
After workers are done, close resultsCh to cause results loop to exit:
func addWorkers(jobsCh <-chan int, resultsCh chan<- int, wg *sync.WaitGroup) {
var wg2 sync.WaitGroup
// 10 workers
for i := 0; i < 10; i++ {
wg2.Add(1)
go worker(jobsCh, resultsCh, &wg2)
}
wg2.Wait() // <-- add this line
close(resultsCh) // and this line
wg.Done()
}

How to execute multiple goroutines in parallel without a Deadlock

So I have been trying to run mutliple goroutines in parallel using a WaitGroup. Whatever I try I always end up with a "fatal error: all goroutines are asleep - deadlock!"
This is what my code looks like right now:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
ch := make(chan time.Duration)
var wg sync.WaitGroup
for _, v := range []time.Duration{5, 1} {
wg.Add(1)
go func() {
defer wg.Done()
wait(v, ch)
}()
}
wg.Wait()
}
func wait(seconds time.Duration, c chan time.Duration) {
time.Sleep(seconds * time.Second)
c <- seconds
}
However this results in a deadlock and I can't figure out why.
I have been trying to read the values after the WaitGroup with the following code:
close(ch)
for v := range ch {
fmt.Println(v)
}
However it seems it wouldn't even reach this part.
Thank you!
There are two problems withe the code:
You must close the channel only after all go-routines have finished
The value of v is being lost during the iteration
In go, the for loop will reuse v, so all go-routines will have the same value on the wait call.
package main
import (
"fmt"
"sync"
"time"
)
func main() {
ch := make(chan time.Duration)
var wg sync.WaitGroup
for _, v := range []time.Duration{5, 1} {
wg.Add(1)
v := v // <- this
go func() {
defer wg.Done()
wait(v, ch)
}()
}
go func() { // Close the channel only after go-routines finish
wg.Wait()
close(ch)
}()
for v := range ch { // Will loop until channel is closed
fmt.Println(v)
}
}
func wait(seconds time.Duration, c chan time.Duration) {
time.Sleep(seconds * time.Second)
c <- seconds
}
You need to close the worker channel after all the workers are done. You are closing it immediately from the main goroutine.
So do this:
go func() {
wg.Wait()
close(ch)
}()
Also your wait function already takes a time.Duration so really it should be in the real time at that point & not multiplied by time.Second. if you want to pass in units of seconds, consider changing the input type to int to avoid confusion.

How to set a max number of goroutines to complete the work?

var wg sync.WaitGroup
wg.Add(len(work))
sem := make(chan struct{}, 10)
wgDone := make(chan bool)
for i < len(work)-1 {
go func() {
defer wg.Done()
sem <- struct{}{}
defer func() {
<-sem
}()
worker(work[i])
}()
i = i + 1
}
go func() {
wg.Wait()
close(wgDone)
}()
I only want 10 new goroutines at a time performing the work. This is my current solution, it blocks goroutines from continuing so there is only 10 at a time. How can I change this so it doesn't create an abundance of goroutines that are blocked waiting to work and instead only creates 10 that complete all the work?
Based on the use case one of these methods is useful:
Using max number of new goroutines and a channel as a queue (The Go playground):
package main
import (
"fmt"
"sync"
)
func main() {
const max = 10
queue := make(chan int, max)
wg := &sync.WaitGroup{}
for i := 0; i < max; i++ {
wg.Add(1)
go worker(wg, queue)
}
for i := 0; i < 100; i++ {
queue <- i
}
close(queue)
wg.Wait()
fmt.Println("Done")
}
func worker(wg *sync.WaitGroup, queue chan int) {
defer wg.Done()
for job := range queue {
fmt.Print(job, " ") // a job
}
}
Using a buffered channel as a semaphore to limits the new goroutines number to the max number (The Go playground):
package main
import (
"fmt"
"sync"
)
func main() {
const max = 10
semaphore := make(chan struct{}, max)
wg := &sync.WaitGroup{}
for i := 0; i < 1000; i++ {
semaphore <- struct{}{} // acquire
wg.Add(1)
go limited(i, wg, semaphore)
}
wg.Wait()
fmt.Println("Done")
}
func limited(i int, wg *sync.WaitGroup, semaphore chan struct{}) {
defer wg.Done()
fmt.Println("i =", i) // a job
<-semaphore // release
}
Using a buffered channel as a semaphore to limits the number of jobs to the max number - here number of goroutines are more than max number (The Go playground):
package main
import (
"fmt"
"sync"
)
func main() {
const max = 10
semaphore := make(chan struct{}, max)
wg := &sync.WaitGroup{}
for i := 0; i < 1000; i++ {
wg.Add(1)
go limited(i, wg, semaphore)
}
wg.Wait()
fmt.Println("Done")
}
func limited(i int, wg *sync.WaitGroup, semaphore chan struct{}) {
defer wg.Done()
semaphore <- struct{}{} // acquire
fmt.Println("i =", i) // a job
<-semaphore // release
}
So if you want only say 10 workers you should spawn 10 workers listening to a job Queue this can be a channel you can push the inputs to this channel and workers will pick it
Now it will only block jobs when the queue is full so you can decide the queue size based on your use case
package main
import (
"fmt"
"sync"
)
var jobQ chan int
var wg sync.WaitGroup
func main() {
jobQ = make(chan int, 100)
go func(){
wg.Add(1)
defer wg.Done()
//Spawn 10 workers
for i:=0;i<10;i++ {
fmt.Println("Spawn :", i)
wg.Add(1)
go worker(jobQ)
}
}()
for i := 0; i< 1000;i++ {
jobQ<- i
}
close(jobQ)
wg.Wait()
}
func worker(jobs chan int) {
defer wg.Done()
for job:=range jobs {
fmt.Println(job)
}
}
Now you can customize this and find other worker pool implementations; worker pools are used a lot and you will find different implementaions
Playground : https://play.golang.org/p/lzIMRUCvqR9

golang goroutines, chanel and sync.WaitGroup

I use simple code, but I get deadlock all the time.
Please explain to the beginner what I am doing wrong.
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
ok := make(chan int, 3)
for i := 0; i < 2; i++ {
wg.Add(1)
go func(i int, wg *sync.WaitGroup) {
for x := range ok {
fmt.Println(i, x)
if x >= 5 {
ok <- x - 1
}
}
wg.Done()
}(i, &wg)
}
ok <- 1
ok <- 3
ok <- 2
ok <- 5
ok <- 3
ok <- 9
wg.Wait()
}
fatal error: all goroutines are asleep - deadlock!
The background of the problem is that the range operator reads from the channel until the channel is closed. Hence the for-range loop keeps waiting for more input from the channel ok even though there are no more inputs to come at some point. At the same time wg.Wait() is waiting for the goroutines to finish. Hence a deadlock!
Either you'll have to close the ok channel at some point, where
there are no more incoming values to the channel ok.
Or you may use the worker pool just like,
package main
import (
"fmt"
"strconv"
"sync"
)
func worker(wg *sync.WaitGroup, ch chan string, i int) {
defer wg.Done()
ch <- "worker process value " + strconv.Itoa(i)
}
func workerMonitor(wg *sync.WaitGroup, ch chan string) {
wg.Wait()
close(ch)
}
func doWork(ch <-chan string, done chan<- bool) {
for i := range ch {
fmt.Println(i)
}
done <- true
}
func main() {
var wg sync.WaitGroup
ch := make(chan string)
var arr = [6]int{1, 3, 2, 5, 3, 9}
for i := 0; i < len(arr); i++ {
wg.Add(1)
if arr[i] >= 5 {
for j := arr[i]; j >= 5; j-- {
wg.Add(1)
go worker(&wg, ch, j-1)
}
}
go worker(&wg, ch, arr[i])
}
go workerMonitor(&wg, ch)
done := make(chan bool, 1)
go doWork(ch, done)
<-done
}
Play ground url

Resources