Await multiple goroutine responses [duplicate] - go

I have a slice of integers, which are manipulated concurrently:
ints := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
I'm using a buffered channel as semaphore in order to have an upper bound of concurrently running go routines:
sem := make(chan struct{}, 2)
for _, i := range ints {
// acquire semaphore
sem <- struct{}{}
// start long running go routine
go func(id int, sem chan struct{}) {
// do something
// release semaphore
<- sem
}(i, sem)
}
The code above works pretty well until the last or last two integers are reached, because the program ends before those last go routines are finished.
Question: how do I wait for the buffered channel to drain?

You can't use a semaphore (channel in this case) in that manner. There's no guarantee it won't be empty any point while you are processing values and dispatching more goroutines. That's not a concern in this case specifically since you're dispatching work synchronously, but because there's no race-free way to check a channel's length, there's no primitive to wait for a channel's length to reach 0.
Use a sync.WaitGroup to wait for all goroutines to complete
sem := make(chan struct{}, 2)
var wg sync.WaitGroup
for _, i := range ints {
wg.Add(1)
// acquire semaphore
sem <- struct{}{}
// start long running go routine
go func(id int) {
defer wg.Done()
// do something
// release semaphore
<-sem
}(i)
}
wg.Wait()

Use "worker pool" to process you data. It is cheeper than run goroutine for each int, allocate memory for variables inside it and so on...
ints := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
ch := make(chan int)
var wg sync.WaitGroup
// run worker pool
for i := 2; i > 0; i-- {
wg.Add(1)
go func() {
defer wg.Done()
for id := range ch {
// do something
fmt.Println(id)
}
}()
}
// send ints to workers
for _, i := range ints {
ch <- i
}
close(ch)
wg.Wait()

Clearly there is no one waiting for your go-routines to complete. Thus the program ends before the last 2 go-routines are completed. You may use a workgroup to wait for all your go-routines complete before the program ends. This tells it better - https://nathanleclaire.com/blog/2014/02/15/how-to-wait-for-all-goroutines-to-finish-executing-before-continuing/

You can wait your "sub-goroutines" with the current goroutine in a for loop
semLimit := 2
sem := make(chan struct{}, semLimit)
for _, i := range ints {
// acquire semaphore
sem <- struct{}{}
// start long running go routine
go func(id int, sem chan struct{}) {
// do something
// release semaphore
<- sem
}(i, sem)
}
// wait semaphore
for i := 0; i < semLimit; i++ {
wg<-struct{}{}
}
Optionnaly it is also possible to program a minimalistic "semaphored waitgroup" with the economy of import sync
semLimit := 2
// mini semaphored waitgroup
wg := make(chan struct{}, semLimit)
// mini methods
wgAdd := func(){ wg<-struct{}{} }
wgDone := func(){ <-wg }
wgWait := func(){ for i := 0; i < semLimit; i++ { wgAdd() } }
for _, i := range ints {
// acquire semaphore
wgAdd()
// start long running go routine
go func(id int, sem chan struct{}) {
// do something
// release semaphore
wgDone()
}(i, sem)
}
// wait semaphore
wgWait()

Here is a working example. The for loop at the end forces the program to wait
until the jobs are done:
package main
import "time"
func w(n int, e chan error) {
// do something
println(n)
time.Sleep(time.Second)
// release semaphore
<-e
}
func main() {
a := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
e := make(chan error, 2)
for _, n := range a {
// acquire semaphore
e <- nil
// start long running go routine
go w(n, e)
}
for n := cap(e); n > 0; n-- {
e <- nil
}
}

Related

Channels concurrency issue

I am experimenting with channel concept in Go. I wrote the below program playground to implement counter using channels. But I am not getting any output, although I am doing some printing in the goroutine body.
func main() {
wg := sync.WaitGroup{}
ch := make(chan int)
count := func(ch chan int) {
var last int
last = <-ch
last = last + 1
fmt.Println(last)
ch <- last
wg.Done()
}
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
}
I expect at least some output but I am getting none at all.
When the main() function (the main goroutine) ends, your program ends as well (it doesn't wait for other non-main goroutines to finish). You must add a wg.Wait() call to the end. See No output from goroutine in Go.
Once you do this, you'll hit a deadlock. This is because all goroutines start with attempting to receive a value from the channel, and only then would they send something.
So you should first send something on the channel to let at least one of the goroutines to proceed.
Once you do that, you'll see numbers printed 10 times, and again deadlock. This is because when the last goroutine tries to send its incremented number, there will be no one to receive that. An easy way to fix that is to give a buffer to the channel.
Final, working example:
wg := sync.WaitGroup{}
ch := make(chan int, 2)
count := func(ch chan int) {
var last int
last = <-ch
last = last + 1
fmt.Println(last)
ch <- last
wg.Done()
}
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
go func() {
ch <- 0
}()
wg.Wait()
Outputs (try it on the Go Playground):
1
2
3
4
5
6
7
8
9
10
Also note that since we made the channel buffered, it's not necessary to use another goroutine to send an initial value, we can do that in the main goroutine:
ch <- 0
wg.Wait()
This will output the same. Try it on the Go Playground.
func main() {
wg := sync.WaitGroup{}
ch := make(chan int)
count := func(ch chan int) {
var last int
last, ok := <-ch // 这里做一层保护
if !ok {
return
}
last = last + 1
fmt.Println(last)
go func(ch chan int, res int) {
ch <- res
}(ch, last)
wg.Done()
}
go func() {
ch <- 0
}()
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
wg.Wait()
fmt.Println("main finish")
close(ch)
}

All goroutines are asleep

I write some code which aims to synchronize using channel.
var counter int64 // shared resource
var wg sync.WaitGroup
func main() {
ch := make(chan int64)
wg.Add(2)
go incCounter(ch)
go incCounter(ch)
ch <- counter
wg.Wait()
fmt.Println("Final Counter:", counter) // expected value is 4
}
func incCounter(ch chan int64) {
defer wg.Done()
for count := 0; count < 2; count++ {
value := <-ch
value++
counter = value
ch <- counter
}
}
When I ran this program, an error happened: all goroutines are asleep - deadlock!. However I can't fix the problem and I don't know what is wrong. Could anyone help?
Channels make(chan int) has implicit size zero ( ref: https://golang.org/ref/spec#Making_slices_maps_and_channels)
A channel of size zero is unbuffered. A channel of specified size make(chan int, n) is buffered. See http://golang.org/ref/spec#Send_statements for a discussion on buffered vs. unbuffered channels. The example at http://play.golang.org/p/VZAiN1V8-P illustrates the difference.
Here, channel <-ch or ch <- will be blocked until someone processes it (concurrently). If you try the flow of this program in pen and paper, you will figure it out why it is blocked. Below figure shows possible data flow through channel ch:
So if you make your ch := make(chan int64) to ch := make(chan int64,1), it will work.
var counter int64 // shared resource
var wg sync.WaitGroup
func main() {
ch := make(chan int64, 1)
wg.Add(2)
go incCounter(ch)
go incCounter(ch)
ch <- counter
wg.Wait()
fmt.Println("Final Counter:", counter) // expected value is 4
}
func incCounter(ch chan int64) {
defer wg.Done()
for count := 0; count < 2; count++ {
value := <-ch
value++
counter = value
ch <- counter
}
}
If we analyse how program works when you are using ch := make(chan int64), we can see that one go routine is blocked in this program(another one is exited). With the help of time.Sleep(n) and receiving the last data from the channel in the blocked go routine, we can overcome the deadlock. See the code below:
var counter int64 // shared resource
var wg sync.WaitGroup
func main() {
ch := make(chan int64)
wg.Add(2)
go incCounter(ch)
go incCounter(ch)
ch <- counter
// to ensure one go routine 'incCounter' is completed and one go routine is blocked for unbuffered channel
time.Sleep(3*time.Second)
<-ch // to unblock the last go routine
wg.Wait()
fmt.Println("Final Counter:", counter) // expected value is 4
}
func incCounter(ch chan int64) {
defer wg.Done()
for count := 0; count < 2; count++ {
value := <-ch
value++
counter = value
ch <- counter
}
}

goroutines order of execution

I'm trying to understand this piece of code, not sure why the 2nd go is executed before the 1st one. It'd be great if someone can really help me out with this!
func sum(a []int, c chan int) {
fmt.Println("summing: ", a)
total := 0
for _, v := range a {
total += v
}
//fmt.Println("send to c",total)
c <- total // send total to c
}
func main() {
//a := []int{7, 2, 8,134,23,23,1,23,1234,143, -9, 4, 0, 1234}
c := make(chan int)
go sum([]int{1,2,3}, c)
go sum([]int{4,5,6}, c)
x := <-c
fmt.Println(x)
x = <-c
fmt.Println(x)
}
OUTPUT:
summing: [4 5 6]
15
summing: [1 2 3]
6
You have nothing explicitly synchronizing the order of the two goroutines. If you run this enough times, you will see the calls to fmt.Println print in different sequences. When executing goroutines, as they are concurrent operations, you have no guarantees when they will execute and/or complete. You need to use various standard library packages, or channels themselves to synchronize the execution of concurrently running goroutines.
For example (by leveraging the blocking nature of channels, you could do something like):
func main() {
c := make(chan int)
go sum([]int{1, 2, 3}, c)
//use the channel to block until it receives a send
x := <-c
fmt.Println(x)
//then execute the next routine
go sum([]int{4, 5, 6}, c)
x = <-c
fmt.Println(x)
}
Another example (significantly less practical, but here to look at other common go synchronization features) you could introduce a wait group, and a range over a channel:
func sum(a []int, c chan int, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("summing: ", a)
total := 0
for _, v := range a {
total += v
}
//fmt.Println("send to c",total)
c <- total // send total to c
}
func main() {
c := make(chan int)
wg := new(sync.WaitGroup)
//concurrently call the concurrent calls to sum, allowing execution to continue to the range of the channel
go func() {
//increment the wait group, and pass it to the sum func to decrement it when it is complete
wg.Add(1)
go sum([]int{1, 2, 3}, c, wg)
//wait for the above call to sum to complete
wg.Wait()
//and repeat...
wg.Add(1)
go sum([]int{4, 5, 6}, c, wg)
wg.Wait()
//all calls are complete, close the channel to allow the program to exit cleanly
close(c)
}()
//range of the channel
for theSum := range c {
x := theSum
fmt.Println(x)
}
}

How to wait until buffered channel (semaphore) is empty?

I have a slice of integers, which are manipulated concurrently:
ints := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
I'm using a buffered channel as semaphore in order to have an upper bound of concurrently running go routines:
sem := make(chan struct{}, 2)
for _, i := range ints {
// acquire semaphore
sem <- struct{}{}
// start long running go routine
go func(id int, sem chan struct{}) {
// do something
// release semaphore
<- sem
}(i, sem)
}
The code above works pretty well until the last or last two integers are reached, because the program ends before those last go routines are finished.
Question: how do I wait for the buffered channel to drain?
You can't use a semaphore (channel in this case) in that manner. There's no guarantee it won't be empty any point while you are processing values and dispatching more goroutines. That's not a concern in this case specifically since you're dispatching work synchronously, but because there's no race-free way to check a channel's length, there's no primitive to wait for a channel's length to reach 0.
Use a sync.WaitGroup to wait for all goroutines to complete
sem := make(chan struct{}, 2)
var wg sync.WaitGroup
for _, i := range ints {
wg.Add(1)
// acquire semaphore
sem <- struct{}{}
// start long running go routine
go func(id int) {
defer wg.Done()
// do something
// release semaphore
<-sem
}(i)
}
wg.Wait()
Use "worker pool" to process you data. It is cheeper than run goroutine for each int, allocate memory for variables inside it and so on...
ints := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
ch := make(chan int)
var wg sync.WaitGroup
// run worker pool
for i := 2; i > 0; i-- {
wg.Add(1)
go func() {
defer wg.Done()
for id := range ch {
// do something
fmt.Println(id)
}
}()
}
// send ints to workers
for _, i := range ints {
ch <- i
}
close(ch)
wg.Wait()
Clearly there is no one waiting for your go-routines to complete. Thus the program ends before the last 2 go-routines are completed. You may use a workgroup to wait for all your go-routines complete before the program ends. This tells it better - https://nathanleclaire.com/blog/2014/02/15/how-to-wait-for-all-goroutines-to-finish-executing-before-continuing/
You can wait your "sub-goroutines" with the current goroutine in a for loop
semLimit := 2
sem := make(chan struct{}, semLimit)
for _, i := range ints {
// acquire semaphore
sem <- struct{}{}
// start long running go routine
go func(id int, sem chan struct{}) {
// do something
// release semaphore
<- sem
}(i, sem)
}
// wait semaphore
for i := 0; i < semLimit; i++ {
wg<-struct{}{}
}
Optionnaly it is also possible to program a minimalistic "semaphored waitgroup" with the economy of import sync
semLimit := 2
// mini semaphored waitgroup
wg := make(chan struct{}, semLimit)
// mini methods
wgAdd := func(){ wg<-struct{}{} }
wgDone := func(){ <-wg }
wgWait := func(){ for i := 0; i < semLimit; i++ { wgAdd() } }
for _, i := range ints {
// acquire semaphore
wgAdd()
// start long running go routine
go func(id int, sem chan struct{}) {
// do something
// release semaphore
wgDone()
}(i, sem)
}
// wait semaphore
wgWait()
Here is a working example. The for loop at the end forces the program to wait
until the jobs are done:
package main
import "time"
func w(n int, e chan error) {
// do something
println(n)
time.Sleep(time.Second)
// release semaphore
<-e
}
func main() {
a := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
e := make(chan error, 2)
for _, n := range a {
// acquire semaphore
e <- nil
// start long running go routine
go w(n, e)
}
for n := cap(e); n > 0; n-- {
e <- nil
}
}

Why happen here a deadlock

I am trying to understand, how golang channel works. I read a book about the go language and found the following example.
package main
import (
"fmt"
)
// Send the sequence 2, 3, 4, ... to returned channel
func generate() chan int {
ch := make(chan int)
go func() {
for i := 2; i <= 100 ; i++ {
ch <- i
}
}()
return ch
}
// Filter out input values divisible by 'prime', send rest to returned channel
func filter(in chan int, prime int) chan int {
out := make(chan int)
go func() {
for {
if i := <-in; i%prime != 0 {
out <- i
}
}
}()
return out
}
func sieve() chan int {
out := make(chan int)
go func() {
ch := generate()
for {
prime := <-ch
ch = filter(ch, prime)
out <- prime
}
}()
return out
}
func main() {
primes := sieve()
for {
fmt.Println(<-primes)
}
}
When I run this programm, I've got a deadlock, but when I change the generate function to
// Send the sequence 2, 3, 4, ... to returned channel
func generate() chan int {
ch := make(chan int)
go func() {
for i := 2; ; i++ {
ch <- i
}
}()
return ch
}
Then the programm will run the infinite loop, but not deadlock. Why do I get deadlock, when I remove the condition in for loop?
What do you mean with blocking principle?
You can see it illustrated in the blog post "The Nature Of Channels In Go "
for an unbuffered channel:
(Illustration from blog post "The Nature Of Channels In Go ", written by William Kennedy, Feb. 2014)
Unbuffered channels have no capacity and therefore require both goroutines to be ready to make any exchange.
When a goroutine attempts to write a resource to an unbuffered channel and there is no goroutine waiting to receive the resource, the channel will lock the goroutine and make it wait.
When a goroutine attempts to read from an unbuffered channel, and there is no goroutine waiting to send a resource, the channel will lock the goroutine and make it wait.
That is what happens in your case with your reader:
func main() {
primes := sieve()
for {
fmt.Println(<-primes)
}
}
since primes is never closed, main remains blocked.
It (main) is in step 3:
in step 3, the goroutine on the right places his hand into the channel or performs a read.
That goroutine is also locked in the channel until the exchange is complete.
The sender never calls close(primes).
Let's consider a simpler example:
func generate() chan int {
ch := make(chan int)
go func() {
for i := 2; /*i < 100*/; i++ {
ch <- i
}
}()
return ch
}
func main() {
for i := range generate() {
fmt.Println(i)
}
}
With the condition i < 100 uncommented, the goroutine spawned by generate stops after sending 98 numbers. However, it does not close the channel, so main has no way of knowing that no more numbers are going to be sent, and it just keeps blocking on the channel. Since main is now the only goroutine still in existence (the other one has returned), and it's blocking, you have a deadlock.

Resources