I'm trying to implement a simple parallelization example in golang using channels. The code tries to implement a parallelized map function. Made channels buffered too for easing constraints on blocking nature of channel. but code still leads to a deadlock.
func pmmap(inp []int, f func(int) int, p int) {
var wg sync.WaitGroup
var output []chan int
tot := len(inp)
out := make(chan int, tot)
slice := tot / p
for i := 0; i < p; i++ {
temp := make(chan int, slice)
output = append(output, temp)
start_ind := slice * i
end_ind := start_ind + slice
fmt.Println(start_ind, end_ind)
wg.Add(1)
go func(si, ei int, out chan int, wg *sync.WaitGroup) {
fmt.Println("goroutine started with ", si, ei)
for ind := si; ind < ei; ind++ {
out <- f(inp[ind])
}
wg.Done()
}(start_ind, end_ind, output[i], &wg)
}
wg.Add(1)
go func(wg *sync.WaitGroup) {
for i := 0; i < p; i++ {
for val := range output[i] {
out <- val
}
close(output[i])
}
wg.Done()
}(&wg)
wg.Add(1)
go func(wg *sync.WaitGroup) {
for i := range out {
fmt.Println(i)
}
wg.Done()
}(&wg)
time.Sleep(time.Second * 6)
wg.Wait()
close(out)
}
func add4(i int) int {
return i + 4
}
func main() {
temp := []int{}
for i := 1; i <= 20; i++ {
temp = append(temp, i)
}
pmmap(temp, add4, 2)
}
from the output of above code, I get the deadlock is because channel output[1] is never read. but I'm not sure why
0 10
10 20
goroutine started with 0 10
goroutine started with 10 20
5
6
7
8
9
10
11
12
13
14
fatal error: all goroutines are asleep - deadlock!
The problem is that range over a channel keeps trying to receive from the channel until the channel is closed, but you didn't close channels after finished sending.
Add close(out) before wg.Done in your code will fix it.
Playground: https://play.golang.org/p/NbKTx6Lke7X
Edit: fixed a bug of closing closed channel.
Related
I am encountering for the below code fatal error: all goroutines are asleep - deadlock!
Am I right in using a buffered channel? I would appreciate it if you can give me pointers. I am unfortunately at the end of my wits.
func main() {
valueChannel := make(chan int, 2)
defer close(valueChannel)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go doNothing(&wg, valueChannel)
}
for {
v, ok := <- valueChannel
if !ok {
break
}
fmt.Println(v)
}
wg.Wait()
}
func doNothing(wg *sync.WaitGroup, numChan chan int) {
defer wg.Done()
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
numChan <- 12
}
The main goroutine blocks on <- valueChannel after receiving all values. Close the channel to unblock the main goroutine.
func main() {
valueChannel := make(chan int, 2)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go doNothing(&wg, valueChannel)
}
// Close channel after goroutines complete.
go func() {
wg.Wait()
close(valueChannel)
}()
// Receive values until channel is closed.
// The for / range loop here does the same
// thing as the for loop in the question.
for v := range valueChannel {
fmt.Println(v)
}
}
Run the example on the playground.
The code above works independent of the number of values sent by the goroutines.
If the main() function can determine the number of values sent by the goroutines, then receive that number of values from main():
func main() {
const n = 10
valueChannel := make(chan int, 2)
for i := 0; i < n; i++ {
go doNothing(valueChannel)
}
// Each call to doNothing sends one value. Receive
// one value for each call to doNothing.
for i := 0; i < n; i++ {
fmt.Println(<-valueChannel)
}
}
func doNothing(numChan chan int) {
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
numChan <- 12
}
Run the example on the playground.
The main problem is on the for loop of channel receiving.
The comma ok idiom is slightly different on channels, ok indicates whether the received value was sent on the channel (true) or is a zero value returned because the channel is closed and empty (false).
In this case the channel is waiting a data to be sent and since it's already finished sending the value ten times : Deadlock.
So apart of the design of the code if I just need to do the less change possible here it is:
func main() {
valueChannel := make(chan int, 2)
defer close(valueChannel)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go doNothing(&wg, valueChannel)
}
for i := 0; i < 10; i++ {
v := <- valueChannel
fmt.Println(v)
}
wg.Wait()
}
func doNothing(wg *sync.WaitGroup, numChan chan int) {
defer wg.Done()
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
numChan <- 12
}
I want to make X number of goroutines to update CountValue using parallelism (numRoutines are as much as how many CPU you have).
Solution 1:
func count(numRoutines int) (countValue int) {
var mu sync.Mutex
k := func(i int) {
mu.Lock()
defer mu.Unlock()
countValue += 5
}
for i := 0; i < numRoutines; i++ {
go k(i)
}
It becomes a data race and the returned countValue = 0.
Solution 2:
func count(numRoutines int) (countValue int) {
k := func(i int, c chan int) {
c <- 5
}
c := make(chan int)
for i := 0; i < numRoutines; i++ {
go k(i, c)
}
for i := 0; i < numRoutines; i++ {
countValue += <- c
}
return
}
I did a benchmark test on it and doing a sequential addition would work faster than using goroutines. I think it's because I have two for loops here as when I put countValue += <- c inside the first for loop, the code runs faster.
Solution 3:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
go func() {
for i := range c {
countValue += i
}
}()
wg.Wait()
return
}
Still a race count :/
Is there any way better to do this?
There definitely is a better way to safely increment a variable: using sync/atomic:
import "sync/atomic"
var words int64
k := func() {
_ = atomic.AddInt64(&words, 5) // increment atomically
}
Using a channel basically eliminates the need for a mutex, or takes away the the risk of concurrent access to the variable itself, and a waitgroup here is just a bit overkill
Channels:
words := 0
done := make(chan struct{}) // or use context
ch := make(chan int, numRoutines) // buffer so each routine can write
go func () {
read := 0
for i := range ch {
words += 5 // or use i or something
read++
if read == numRoutines {
break // we've received data from all routines
}
}
close(done) // indicate this routine has terminated
}()
for i := 0; i < numRoutines; i++ {
ch <- i // write whatever value needs to be used in the counting routine on the channel
}
<- done // wait for our routine that increments words to return
close(ch) // this channel is no longer needed
fmt.Printf("Counted %d\n", words)
As you can tell, the numRoutines no longer is the number of routines, but rather the number of writes on the channel. You can move that to individual routines, still:
for i := 0; i < numRoutines; i++ {
go func(ch chan<- int, i int) {
// do stuff here
ch <- 5 * i // for example
}(ch, i)
}
WaitGroup:
Instead of using a context that you can cancel, or a channel, you can use a waitgroup + atomic to get the same result. The easiest way IMO to do so is to create a type:
type counter struct {
words int64
}
func (c *counter) doStuff(wg *sync.WaitGroup, i int) {
defer wg.Done()
_ = atomic.AddInt64(&c.words, i * 5) // whatever value you need to add
}
func main () {
cnt := counter{}
wg := sync.WaitGroup{}
wg.Add(numRoutines) // create the waitgroup
for i := 0; i < numRoutines; i++ {
go cnt.doStuff(&wg, i)
}
wg.Wait() // wait for all routines to finish
fmt.Println("Counted %d\n", cnt.words)
}
Fix for your third solution
As I mentioned in the comment: your third solution is still causing a race condition because the channel c is never closed, meaning the routine:
go func () {
for i := range c {
countValue += i
}
}()
Never returns. The waitgroup also only ensures that you've sent all values on the channel, but not that the countValue has been incremented to its final value. The fix would be to either close the channel after wg.Wait() returns so the routine can return, and add a done channel that you can close when this last routine returns, and add a <-done statement before returning.
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
done := make(chan struct{})
go func() {
for i := range c {
countValue += i
}
close(done)
}()
wg.Wait()
close(c)
<-done
return
}
This adds some clutter, though, and IMO is a bit messy. It might just be easier to just move the wg.Wait() call to a routine:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
// add wg as argument, makes it easier to move this function outside of this scope
k := func(wg *sync.WaitGroup, i int) {
defer wg.Done()
c <- 5
}
wg.Add(numShards) // increment the waitgroup once
for i := 0; i < numShards; i++ {
go k(&wg, i)
}
go func() {
wg.Wait()
close(c) // this ends the loop over the channel
}()
// just iterate over the channel until it is closed
for i := range c {
countValue += i
}
// we've added all values to countValue
return
}
I use simple code, but I get deadlock all the time.
Please explain to the beginner what I am doing wrong.
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
ok := make(chan int, 3)
for i := 0; i < 2; i++ {
wg.Add(1)
go func(i int, wg *sync.WaitGroup) {
for x := range ok {
fmt.Println(i, x)
if x >= 5 {
ok <- x - 1
}
}
wg.Done()
}(i, &wg)
}
ok <- 1
ok <- 3
ok <- 2
ok <- 5
ok <- 3
ok <- 9
wg.Wait()
}
fatal error: all goroutines are asleep - deadlock!
The background of the problem is that the range operator reads from the channel until the channel is closed. Hence the for-range loop keeps waiting for more input from the channel ok even though there are no more inputs to come at some point. At the same time wg.Wait() is waiting for the goroutines to finish. Hence a deadlock!
Either you'll have to close the ok channel at some point, where
there are no more incoming values to the channel ok.
Or you may use the worker pool just like,
package main
import (
"fmt"
"strconv"
"sync"
)
func worker(wg *sync.WaitGroup, ch chan string, i int) {
defer wg.Done()
ch <- "worker process value " + strconv.Itoa(i)
}
func workerMonitor(wg *sync.WaitGroup, ch chan string) {
wg.Wait()
close(ch)
}
func doWork(ch <-chan string, done chan<- bool) {
for i := range ch {
fmt.Println(i)
}
done <- true
}
func main() {
var wg sync.WaitGroup
ch := make(chan string)
var arr = [6]int{1, 3, 2, 5, 3, 9}
for i := 0; i < len(arr); i++ {
wg.Add(1)
if arr[i] >= 5 {
for j := arr[i]; j >= 5; j-- {
wg.Add(1)
go worker(&wg, ch, j-1)
}
}
go worker(&wg, ch, arr[i])
}
go workerMonitor(&wg, ch)
done := make(chan bool, 1)
go doWork(ch, done)
<-done
}
Play ground url
can someone give me some insight about this code, Why this get deadlock error in for x:=range c
func main() {
c:=make(chan int,10)
for i:=0;i<5;i++{
go func(chanel chan int,i int){
println("i",i)
chanel <- 1
}(c,i)
}
for x:=range c {
println(x)
}
println("Done!")
}
Because this:
for x:=range c {
println(x)
}
will loop until the channel c closes, which is never done here.
Here is one way you can fix it, using a WaitGroup:
package main
import "sync"
func main() {
var wg sync.WaitGroup
c := make(chan int, 10)
for i := 0; i < 5; i++ {
wg.Add(1)
go func(chanel chan int, i int) {
defer wg.Done()
println("i", i)
chanel <- 1
}(c, i)
}
go func() {
wg.Wait()
close(c)
}()
for x := range c {
println(x)
}
println("Done!")
}
Try it on Go Playground
You create five goroutines, each sending an integer value to the channel. Once all those five values are written, there's no other goroutine left that writes to the channel.
The main goroutine reads those five values from the channel. But there are no goroutines that can possibly write the sixth value or close the channel. So, you're deadlocked waiting data from the channel.
Close the channel once all the writes are completed. It should be an interesting exercise to figure out how you can do that with this code.
Channel needs to be closed to indicate task is complete.
Coordinate with a sync.WaitGroup:
c := make(chan int, 10)
var wg sync.WaitGroup // here
for i := 0; i < 5; i++ {
wg.Add(1) // here
go func(chanel chan int, i int) {
defer wg.Done()
println("i", i)
chanel <- 1
}(c, i)
}
go func() {
wg.Wait() // and here
close(c)
}()
for x := range c {
println(x)
}
println("Done!")
https://play.golang.org/p/VWcBC2YGLvM
I am experimenting with channel concept in Go. I wrote the below program playground to implement counter using channels. But I am not getting any output, although I am doing some printing in the goroutine body.
func main() {
wg := sync.WaitGroup{}
ch := make(chan int)
count := func(ch chan int) {
var last int
last = <-ch
last = last + 1
fmt.Println(last)
ch <- last
wg.Done()
}
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
}
I expect at least some output but I am getting none at all.
When the main() function (the main goroutine) ends, your program ends as well (it doesn't wait for other non-main goroutines to finish). You must add a wg.Wait() call to the end. See No output from goroutine in Go.
Once you do this, you'll hit a deadlock. This is because all goroutines start with attempting to receive a value from the channel, and only then would they send something.
So you should first send something on the channel to let at least one of the goroutines to proceed.
Once you do that, you'll see numbers printed 10 times, and again deadlock. This is because when the last goroutine tries to send its incremented number, there will be no one to receive that. An easy way to fix that is to give a buffer to the channel.
Final, working example:
wg := sync.WaitGroup{}
ch := make(chan int, 2)
count := func(ch chan int) {
var last int
last = <-ch
last = last + 1
fmt.Println(last)
ch <- last
wg.Done()
}
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
go func() {
ch <- 0
}()
wg.Wait()
Outputs (try it on the Go Playground):
1
2
3
4
5
6
7
8
9
10
Also note that since we made the channel buffered, it's not necessary to use another goroutine to send an initial value, we can do that in the main goroutine:
ch <- 0
wg.Wait()
This will output the same. Try it on the Go Playground.
func main() {
wg := sync.WaitGroup{}
ch := make(chan int)
count := func(ch chan int) {
var last int
last, ok := <-ch // 这里做一层保护
if !ok {
return
}
last = last + 1
fmt.Println(last)
go func(ch chan int, res int) {
ch <- res
}(ch, last)
wg.Done()
}
go func() {
ch <- 0
}()
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
wg.Wait()
fmt.Println("main finish")
close(ch)
}