I am currently learning concurrency in go. I write a simple Test program to practice goroutines. But whenever i execute this, i Get an error stating fatal error: all goroutines are asleep - deadlock! I dont know what i did wrong. Does anyone know how i can make this work as intended. Any help will be much appreciated
import (
"fmt"
"sync"
"time"
)
func slow(i int, cha chan int, wg *sync.WaitGroup) {
fmt.Println("HI", i)
time.Sleep(time.Second * 2)
cha <- i * 2
wg.Done()
}
func main() {
var wg sync.WaitGroup
values := make(chan int)
for i := 1; i < 12; i++ {
wg.Add(1)
go slow(i, values, &wg)
}
wg.Wait()
close(values)
}
EDIT: When i try to make the channel to a buffered channel, it works. I dont know how tho
Your current program is not working since there is no consumer for the unbuffered channel. Also, there is no need to use both WaitGroup and channel together.
Consider the below example:
func slow(i int, cha chan int) {
fmt.Println("HI", i)
time.Sleep(time.Second * 2)
cha <- i * 2
}
func main() {
values := make(chan int)
defer close(values)
for i := 1; i < 12; i++ {
go slow(i, values)
}
for i := 1; i < 12; i++ {
fmt.Println(<-values)
}
}
This way we can have a sender goroutine and the messages can be consumed in the main function.
If we just want to make sure that all the goroutines completes before main finishes the execution, we can remove the channels like below:
func slow(i int, wg *sync.WaitGroup) {
fmt.Println("HI", i)
time.Sleep(time.Second * 2)
wg.Done()
}
func main() {
var wg sync.WaitGroup
for i := 1; i < 12; i++ {
wg.Add(1)
go slow(i, &wg)
}
wg.Wait()
}
Using buffered channels worked, because pushing messages to channel is no longer blocking until the buffer is filled.
Related
I'm learning Go concurrency and wrote the obligatory worker pool example, where there are N pieces of work and M workers (N > M). I'm running into a deadlock (all goroutines are asleep), which I can't figure out; however, I'm also getting the expected output before the deadlock occurs, which has me even more confused. Can someone please point out the things I'm doing wrong?
My code is this:
package main
import (
"fmt"
"sync"
)
// A simple worker pool implementation using channels and WaitGroups.
// Our workers simply read from a channel of integers from an input
// channel and write their squares to an output channel.
func addJobs(jobsCh chan<- int, wg *sync.WaitGroup) {
// 100 numbers to crunch (jobs)
for i := 1; i < 101; i++ {
jobsCh <- i
}
wg.Done()
}
func worker(jobsCh <-chan int, resultsCh chan<- int, wg2 *sync.WaitGroup) {
for num := range jobsCh {
resultsCh <- num * num
}
wg2.Done()
}
func addWorkers(jobsCh <-chan int, resultsCh chan<- int, wg *sync.WaitGroup) {
var wg2 sync.WaitGroup
// 10 workers
for i := 0; i < 10; i++ {
wg2.Add(1)
go worker(jobsCh, resultsCh, &wg2)
}
wg.Done()
}
func readResults(resultsCh <-chan int, wg *sync.WaitGroup) {
for sq := range resultsCh {
fmt.Printf("%v ", sq)
}
wg.Done()
}
func main() {
var wg sync.WaitGroup
jobsCh := make(chan int)
resultsCh := make(chan int)
wg.Add(1)
go addJobs(jobsCh, &wg)
wg.Add(1)
go addWorkers(jobsCh, resultsCh, &wg)
wg.Add(1)
go readResults(resultsCh, &wg)
wg.Wait()
}
This prints the squares of the numbers (in random order), as expected, but also runs into a deadlock. Please see this playground link. :(
Close jobsCh to cause workers to exit:
func addJobs(jobsCh chan<- int, wg *sync.WaitGroup) {
// 100 numbers to crunch (jobs)
for i := 1; i < 101; i++ {
jobsCh <- i
}
close(jobsCh) // <-- add this line
wg.Done()
}
After workers are done, close resultsCh to cause results loop to exit:
func addWorkers(jobsCh <-chan int, resultsCh chan<- int, wg *sync.WaitGroup) {
var wg2 sync.WaitGroup
// 10 workers
for i := 0; i < 10; i++ {
wg2.Add(1)
go worker(jobsCh, resultsCh, &wg2)
}
wg2.Wait() // <-- add this line
close(resultsCh) // and this line
wg.Done()
}
I am encountering for the below code fatal error: all goroutines are asleep - deadlock!
Am I right in using a buffered channel? I would appreciate it if you can give me pointers. I am unfortunately at the end of my wits.
func main() {
valueChannel := make(chan int, 2)
defer close(valueChannel)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go doNothing(&wg, valueChannel)
}
for {
v, ok := <- valueChannel
if !ok {
break
}
fmt.Println(v)
}
wg.Wait()
}
func doNothing(wg *sync.WaitGroup, numChan chan int) {
defer wg.Done()
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
numChan <- 12
}
The main goroutine blocks on <- valueChannel after receiving all values. Close the channel to unblock the main goroutine.
func main() {
valueChannel := make(chan int, 2)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go doNothing(&wg, valueChannel)
}
// Close channel after goroutines complete.
go func() {
wg.Wait()
close(valueChannel)
}()
// Receive values until channel is closed.
// The for / range loop here does the same
// thing as the for loop in the question.
for v := range valueChannel {
fmt.Println(v)
}
}
Run the example on the playground.
The code above works independent of the number of values sent by the goroutines.
If the main() function can determine the number of values sent by the goroutines, then receive that number of values from main():
func main() {
const n = 10
valueChannel := make(chan int, 2)
for i := 0; i < n; i++ {
go doNothing(valueChannel)
}
// Each call to doNothing sends one value. Receive
// one value for each call to doNothing.
for i := 0; i < n; i++ {
fmt.Println(<-valueChannel)
}
}
func doNothing(numChan chan int) {
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
numChan <- 12
}
Run the example on the playground.
The main problem is on the for loop of channel receiving.
The comma ok idiom is slightly different on channels, ok indicates whether the received value was sent on the channel (true) or is a zero value returned because the channel is closed and empty (false).
In this case the channel is waiting a data to be sent and since it's already finished sending the value ten times : Deadlock.
So apart of the design of the code if I just need to do the less change possible here it is:
func main() {
valueChannel := make(chan int, 2)
defer close(valueChannel)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go doNothing(&wg, valueChannel)
}
for i := 0; i < 10; i++ {
v := <- valueChannel
fmt.Println(v)
}
wg.Wait()
}
func doNothing(wg *sync.WaitGroup, numChan chan int) {
defer wg.Done()
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
numChan <- 12
}
var wg sync.WaitGroup
wg.Add(len(work))
sem := make(chan struct{}, 10)
wgDone := make(chan bool)
for i < len(work)-1 {
go func() {
defer wg.Done()
sem <- struct{}{}
defer func() {
<-sem
}()
worker(work[i])
}()
i = i + 1
}
go func() {
wg.Wait()
close(wgDone)
}()
I only want 10 new goroutines at a time performing the work. This is my current solution, it blocks goroutines from continuing so there is only 10 at a time. How can I change this so it doesn't create an abundance of goroutines that are blocked waiting to work and instead only creates 10 that complete all the work?
Based on the use case one of these methods is useful:
Using max number of new goroutines and a channel as a queue (The Go playground):
package main
import (
"fmt"
"sync"
)
func main() {
const max = 10
queue := make(chan int, max)
wg := &sync.WaitGroup{}
for i := 0; i < max; i++ {
wg.Add(1)
go worker(wg, queue)
}
for i := 0; i < 100; i++ {
queue <- i
}
close(queue)
wg.Wait()
fmt.Println("Done")
}
func worker(wg *sync.WaitGroup, queue chan int) {
defer wg.Done()
for job := range queue {
fmt.Print(job, " ") // a job
}
}
Using a buffered channel as a semaphore to limits the new goroutines number to the max number (The Go playground):
package main
import (
"fmt"
"sync"
)
func main() {
const max = 10
semaphore := make(chan struct{}, max)
wg := &sync.WaitGroup{}
for i := 0; i < 1000; i++ {
semaphore <- struct{}{} // acquire
wg.Add(1)
go limited(i, wg, semaphore)
}
wg.Wait()
fmt.Println("Done")
}
func limited(i int, wg *sync.WaitGroup, semaphore chan struct{}) {
defer wg.Done()
fmt.Println("i =", i) // a job
<-semaphore // release
}
Using a buffered channel as a semaphore to limits the number of jobs to the max number - here number of goroutines are more than max number (The Go playground):
package main
import (
"fmt"
"sync"
)
func main() {
const max = 10
semaphore := make(chan struct{}, max)
wg := &sync.WaitGroup{}
for i := 0; i < 1000; i++ {
wg.Add(1)
go limited(i, wg, semaphore)
}
wg.Wait()
fmt.Println("Done")
}
func limited(i int, wg *sync.WaitGroup, semaphore chan struct{}) {
defer wg.Done()
semaphore <- struct{}{} // acquire
fmt.Println("i =", i) // a job
<-semaphore // release
}
So if you want only say 10 workers you should spawn 10 workers listening to a job Queue this can be a channel you can push the inputs to this channel and workers will pick it
Now it will only block jobs when the queue is full so you can decide the queue size based on your use case
package main
import (
"fmt"
"sync"
)
var jobQ chan int
var wg sync.WaitGroup
func main() {
jobQ = make(chan int, 100)
go func(){
wg.Add(1)
defer wg.Done()
//Spawn 10 workers
for i:=0;i<10;i++ {
fmt.Println("Spawn :", i)
wg.Add(1)
go worker(jobQ)
}
}()
for i := 0; i< 1000;i++ {
jobQ<- i
}
close(jobQ)
wg.Wait()
}
func worker(jobs chan int) {
defer wg.Done()
for job:=range jobs {
fmt.Println(job)
}
}
Now you can customize this and find other worker pool implementations; worker pools are used a lot and you will find different implementaions
Playground : https://play.golang.org/p/lzIMRUCvqR9
can someone give me some insight about this code, Why this get deadlock error in for x:=range c
func main() {
c:=make(chan int,10)
for i:=0;i<5;i++{
go func(chanel chan int,i int){
println("i",i)
chanel <- 1
}(c,i)
}
for x:=range c {
println(x)
}
println("Done!")
}
Because this:
for x:=range c {
println(x)
}
will loop until the channel c closes, which is never done here.
Here is one way you can fix it, using a WaitGroup:
package main
import "sync"
func main() {
var wg sync.WaitGroup
c := make(chan int, 10)
for i := 0; i < 5; i++ {
wg.Add(1)
go func(chanel chan int, i int) {
defer wg.Done()
println("i", i)
chanel <- 1
}(c, i)
}
go func() {
wg.Wait()
close(c)
}()
for x := range c {
println(x)
}
println("Done!")
}
Try it on Go Playground
You create five goroutines, each sending an integer value to the channel. Once all those five values are written, there's no other goroutine left that writes to the channel.
The main goroutine reads those five values from the channel. But there are no goroutines that can possibly write the sixth value or close the channel. So, you're deadlocked waiting data from the channel.
Close the channel once all the writes are completed. It should be an interesting exercise to figure out how you can do that with this code.
Channel needs to be closed to indicate task is complete.
Coordinate with a sync.WaitGroup:
c := make(chan int, 10)
var wg sync.WaitGroup // here
for i := 0; i < 5; i++ {
wg.Add(1) // here
go func(chanel chan int, i int) {
defer wg.Done()
println("i", i)
chanel <- 1
}(c, i)
}
go func() {
wg.Wait() // and here
close(c)
}()
for x := range c {
println(x)
}
println("Done!")
https://play.golang.org/p/VWcBC2YGLvM
I have the following code that goes into deadlock when sending on channel from a goroutine below:
package main
import (
"fmt"
"sync"
)
func main() {
for a := range getCh(10) {
fmt.Println("Got:", a)
}
}
func getCh(n int) <-chan int {
var wg sync.WaitGroup
ch := make(chan int)
defer func() {
fmt.Println("closing")
wg.Wait()
close(ch)
}()
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; i < n; i++ {
ch <- i
}
}()
wg.Add(1)
go func() {
defer wg.Done()
for i := n; i < 0; i-- {
ch <- i
}
}()
return ch
}
I know that it is legal to use wg.Wait() in defer. But I haven't been able to find a use in a function with channel as a return value.
I think the mistake you're making is that you think that the deferred function will run asynchronously too. But that is not the case, so the getCh() will block in its deferred part, waiting for the WaitGroup. But as there is no one reading from the channel, the goroutines which write into it can't return and thus the WaitGroup causes deadlock. Try something like this:
func getCh(n int) <-chan int {
ch := make(chan int)
go func() {
var wg sync.WaitGroup
wg.Add(1)
go func(n int) {
defer wg.Done()
for i := 0; i < n; i++ {
ch <- i
}
}(n)
wg.Add(1)
go func(n int) {
defer wg.Done()
for i := n; i > 0; i-- {
ch <- i
}
}(n)
wg.Wait()
fmt.Println("closing")
close(ch)
}()
return ch
}
It looks like your channels are blocking since you are not using any buffered channels. check out this quick example https://play.golang.org/p/zMnfA33qZk
ch := make(chan int, n)
Remember that channels block when they are filled. I am not sure what your goal is with your code, but it looks like you were aiming for using a buffered channel. This is a good piece from effective go https://golang.org/doc/effective_go.html#channels
Receivers always block until there is data to receive. If the channel is unbuffered, the sender blocks until the receiver has received the value. If the channel has a buffer, the sender blocks only until the value has been copied to the buffer; if the buffer is full, this means waiting until some receiver has retrieved a value.