Deadlock when running two go routine - go

Im studying Golang now on my freetime and I am trying sample exams online to test what i learned,
I came about this coding exam task but I cant seem to make it work/run without a crash,
im getting fatal error: all goroutines are asleep - deadlock! error, can anybody help what I am doing wrong here?
func executeParallel(ch chan<- int, done chan<- bool, functions ...func() int) {
ch <- functions[1]()
done <- true
}
func exampleFunction(counter int) int {
sum := 0
for i := 0; i < counter; i++ {
sum += 1
}
return sum
}
func main() {
expensiveFunction := func() int {
return exampleFunction(200000000)
}
cheapFunction := func() int {return exampleFunction(10000000)}
ch := make(chan int)
done := make(chan bool)
go executeParallel(ch, done, expensiveFunction, cheapFunction)
var isDone = <-done
for result := range ch {
fmt.Printf("Result: %d\n", result)
if isDone {
break;
}
}
}

Your executeParallel function will panic if less than 2 functions are provided - and will only run the 2nd function:
ch <- functions[1]() // runtime panic if less then 2 functions
I think it should look more like this: running all input functions in parallel and grabbing the first result.
for _, fn := range functions {
fn := fn // so each iteration/goroutine gets the proper value
go func() {
select {
case ch <- fn():
// first (fastest worker) wins
default:
// other workers results are discarded (if reader has not read results yet)
// this ensure we don't leak goroutines - since reader only reads one result from channel
}
}()
}
As such there's no need for a done channel - as we just need to read the one and only (quickest) result:
ch := make(chan int, 1) // big enough to capture one result - even if reader is not reading yet
executeParallel(ch, expensiveFunction, cheapFunction)
fmt.Printf("Result: %d\n", <-ch)
https://play.golang.org/p/skXc3gZZmRn

package main
import "fmt"
func executeParallel(ch chan<- int, done chan<- struct{}, functions ...func() int) {
// Only execute the second function [1], if available.
if len(functions) > 1 {
ch <- functions[1]()
}
// Close the done channel to signal the for-select to break and the main returns.
close(done)
}
// example returns the number of iterations for [0..counter-1]
func example(counter int) int {
sum := 0
for i := 0; i < counter; i++ {
sum += 1
}
return sum
// NOTE(SS): This function could just return "counter-1"
// to avoid the unnecessary calculation done above.
}
func main() {
var (
cheap = func() int { return example(10000000) }
expensive = func() int { return example(200000000) }
ch = make(chan int)
done = make(chan struct{})
)
// executeParallel takes ch, done channel followed by variable
// number of functions where on the second i.e., indexed 1
// function is executed on a separated goroutine which is then
// sent to ch channel which is then received by the for-select
// reciever below i.e., <-ch is the receiver.
go executeParallel(ch, done, expensive, cheap)
for {
select {
// Wait for something to be sent to done or the done channel
// to be closed.
case <-done:
return
// Keep receiving from ch (if something is sent to it)
case result := <-ch:
fmt.Println("Result:", result)
}
}
}
I have commented on the code so that it's understandable. As you didn't the actual question the logic could be still wrong.

Related

X number of goroutines to update the same variable

I want to make X number of goroutines to update CountValue using parallelism (numRoutines are as much as how many CPU you have).
Solution 1:
func count(numRoutines int) (countValue int) {
var mu sync.Mutex
k := func(i int) {
mu.Lock()
defer mu.Unlock()
countValue += 5
}
for i := 0; i < numRoutines; i++ {
go k(i)
}
It becomes a data race and the returned countValue = 0.
Solution 2:
func count(numRoutines int) (countValue int) {
k := func(i int, c chan int) {
c <- 5
}
c := make(chan int)
for i := 0; i < numRoutines; i++ {
go k(i, c)
}
for i := 0; i < numRoutines; i++ {
countValue += <- c
}
return
}
I did a benchmark test on it and doing a sequential addition would work faster than using goroutines. I think it's because I have two for loops here as when I put countValue += <- c inside the first for loop, the code runs faster.
Solution 3:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
go func() {
for i := range c {
countValue += i
}
}()
wg.Wait()
return
}
Still a race count :/
Is there any way better to do this?
There definitely is a better way to safely increment a variable: using sync/atomic:
import "sync/atomic"
var words int64
k := func() {
_ = atomic.AddInt64(&words, 5) // increment atomically
}
Using a channel basically eliminates the need for a mutex, or takes away the the risk of concurrent access to the variable itself, and a waitgroup here is just a bit overkill
Channels:
words := 0
done := make(chan struct{}) // or use context
ch := make(chan int, numRoutines) // buffer so each routine can write
go func () {
read := 0
for i := range ch {
words += 5 // or use i or something
read++
if read == numRoutines {
break // we've received data from all routines
}
}
close(done) // indicate this routine has terminated
}()
for i := 0; i < numRoutines; i++ {
ch <- i // write whatever value needs to be used in the counting routine on the channel
}
<- done // wait for our routine that increments words to return
close(ch) // this channel is no longer needed
fmt.Printf("Counted %d\n", words)
As you can tell, the numRoutines no longer is the number of routines, but rather the number of writes on the channel. You can move that to individual routines, still:
for i := 0; i < numRoutines; i++ {
go func(ch chan<- int, i int) {
// do stuff here
ch <- 5 * i // for example
}(ch, i)
}
WaitGroup:
Instead of using a context that you can cancel, or a channel, you can use a waitgroup + atomic to get the same result. The easiest way IMO to do so is to create a type:
type counter struct {
words int64
}
func (c *counter) doStuff(wg *sync.WaitGroup, i int) {
defer wg.Done()
_ = atomic.AddInt64(&c.words, i * 5) // whatever value you need to add
}
func main () {
cnt := counter{}
wg := sync.WaitGroup{}
wg.Add(numRoutines) // create the waitgroup
for i := 0; i < numRoutines; i++ {
go cnt.doStuff(&wg, i)
}
wg.Wait() // wait for all routines to finish
fmt.Println("Counted %d\n", cnt.words)
}
Fix for your third solution
As I mentioned in the comment: your third solution is still causing a race condition because the channel c is never closed, meaning the routine:
go func () {
for i := range c {
countValue += i
}
}()
Never returns. The waitgroup also only ensures that you've sent all values on the channel, but not that the countValue has been incremented to its final value. The fix would be to either close the channel after wg.Wait() returns so the routine can return, and add a done channel that you can close when this last routine returns, and add a <-done statement before returning.
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
done := make(chan struct{})
go func() {
for i := range c {
countValue += i
}
close(done)
}()
wg.Wait()
close(c)
<-done
return
}
This adds some clutter, though, and IMO is a bit messy. It might just be easier to just move the wg.Wait() call to a routine:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
// add wg as argument, makes it easier to move this function outside of this scope
k := func(wg *sync.WaitGroup, i int) {
defer wg.Done()
c <- 5
}
wg.Add(numShards) // increment the waitgroup once
for i := 0; i < numShards; i++ {
go k(&wg, i)
}
go func() {
wg.Wait()
close(c) // this ends the loop over the channel
}()
// just iterate over the channel until it is closed
for i := range c {
countValue += i
}
// we've added all values to countValue
return
}

Get the fastest goroutine

How to get only the result of the fastest goroutine. For example let's we have this function.
func Add(num int)int{
return num+5
}
And we have this function.
func compute(){
for i := 0; i < 5; i++ {
go Add(i)
}
}
I want to get the result of the goroutine that finishes first.
Use a buffered channel to get first result of many grouting execution
func Test(t *testing.T) {
ch := make(chan int, 1)
go func() {
for i := 0; i < 5; i++ {
go func(c chan<- int, i int) {
res := Add(i)
c <- res
}(ch, i)
}
}()
res := <-ch // blocking here, before get first result
//close(ch) - writing to close channel produce a panic
fmt.Println(res)
}
PLAYGROUND
It hasn't memory leak when leave a channel open and never close it. When the channel is no longer used, it will be garbage collected.

Why Filter get data before prime := <- ch

https://play.golang.org/p/AyKy5odhfZw
In my sight, prime := <- ch this line is before go Filter () , every time data put into ch will be got out by prime directly.
// A concurrent prime sieve
package main
import "fmt"
// Send the sequence 2, 3, 4, ... to channel 'ch'.
func Generate(ch chan<- int) {
for i := 2; ; i++ {
ch <- i // Send 'i' to channel 'ch'.
}
}
// Copy the values from channel 'in' to channel 'out',
// removing those divisible by 'prime'.
func Filter(in <-chan int, out chan<- int, prime int) {
for {
i := <-in // Receive value from 'in'.
if i%prime != 0 {
out <- i // Send 'i' to 'out'.
}
}
}
// The prime sieve: Daisy-chain Filter processes.
func main() {
ch := make(chan int) // Create a new channel.
go Generate(ch) // Launch Generate goroutine.
for i := 0; i < 10; i++ {
prime := <-ch
fmt.Println(prime)
ch1 := make(chan int)
go Filter(ch, ch1, prime)
ch = ch1
}
}
Filter does not receive data first.
The way the code is written means that the variable prime will always receive the first value on the output of the filter goroutine created in the previous loop (or the generator on the first loop).
The next time around the main loop the channel variable ch will have changed to output of the next filter due to the line ch=ch1 at the end of the loop. All further integers beyond the first output by a filter will pass to the next filter in the chain.
Placing a few print statements in the filter goroutine will let you see what is happening:
func Filter(in <-chan int, out chan<- int, prime int) {
for {
i := <-in // Receive value from 'in'.
if i%prime != 0 {
fmt.Printf("Prime filter (%d): passing %d\n", prime, i)
out <- i // Send 'i' to 'out'.
} else {
fmt.Printf("Prime filter (%d): filtered %d\n", prime, i)
}
}
}
It's a little messy that the goroutines in the pipeline aren't shut down elgantly by closing the generator but that's another story :)

Getting deadlock as I try to emulate fan in - fan out with factorial calculations

I am trying the fan in - fan out pattern with a factorial problem. But I am getting:
fatal error: all goroutines are asleep - deadlock!
and unable to identify the reason for deadlock.
I am trying to concurrently calculate factorial for 100 numbers using the fan-in fan-out pattern.
package main
import (
"fmt"
)
func main() {
_inChannel := _inListener(generator())
for val := range _inChannel {
fmt.Print(val, " -- ")
}
}
func generator() chan int { // NEED TO CALCULATE FACTORIAL FOR 100 NUMBERS
ch := make(chan int) // CREATE CHANNEL TO INPUT NUMBERS
go func() {
for i := 1; i <= 100; i++ {
ch <- i
}
close(ch) // CLOSE CHANNEL WHEN ALL NUMBERS HAVE BEEN WRITTEM
}()
return ch
}
func _inListener(ch chan int) chan int {
rec := make(chan int) // CHANNEL RECEIVED FROM GENERATOR
go func() {
for num := range ch { // RECEIVE THE INPUT NUMBERS FROM GENERATOR
result := factorial(num) // RESULT IS A NEW CHANNEL CREATED
rec <- <-result // MERGE INTO A SINGLE CHANNEL; rec
close(result)
}
close(rec)
}()
return rec // RETURN THE DEDICATED CHANNEL TO RECEIVE ALL OUTPUTS
}
func factorial(n int) chan int {
ch := make(chan int) // MAKE A NEW CHANNEL TO OUTPUT THE RESULT
// OF FACTORIAL
total := 1
for i := n; i > 0; i-- {
total *= i
}
ch <- total
return ch // RETURN THE CHANNEL HAVING THE FACTORIAL CALCULATED
}
I have put in comments, so that it becomes easier to follow the code.
I'm no expert in channels. I've taking on this to try and get more familiar with go.
Another issue is the int isn't large enough to take all factorials over 20 or so.
As you can see, I added a defer close as well as a logical channel called done in the generator func. The rest of the changes probably aren't needed. With channels you need to make sure something is ready to take off a value on the channel when you put something on a channel. Otherwise deadlock. Also, using
go run -race main.go
helps at least see which line(s) are causing problems.
I hope this helps and isn't removed for being off topic.
I was able to remove the deadlock by doing this:
package main
import (
"fmt"
)
func main() {
_gen := generator()
_inChannel := _inListener(_gen)
for val := range _inChannel {
fmt.Print(val, " -- \n")
}
}
func generator() chan int { // NEED TO CALCULATE FACTORIAL FOR 100 NUMBERS
ch := make(chan int) // CREATE CHANNEL TO INPUT NUMBERS
done := make(chan bool)
go func() {
defer close(ch)
for i := 1; i <= 100; i++ {
ch <- i
}
//close(ch) // CLOSE CHANNEL WHEN ALL NUMBERS HAVE BEEN WRITTEM
done <- true
}()
// this function will pull off the done for each function call above.
go func() {
for i := 1; i < 100; i++ {
<-done
}
}()
return ch
}
func _inListener(ch chan int) chan int {
rec := make(chan int) // CHANNEL RECEIVED FROM GENERATOR
go func() {
for num := range ch { // RECEIVE THE INPUT NUMBERS FROM GENERATOR
result := factorial(num) // RESULT IS A NEW CHANNEL CREATED
rec <- result // MERGE INTO A SINGLE CHANNEL; rec
}
close(rec)
}()
return rec // RETURN THE DEDICATED CHANNEL TO RECEIVE ALL OUTPUTS
}
func factorial(n int) int {
// OF FACTORIAL
total := 1
for i := n; i > 0; i-- {
total *= i
}
return total // RETURN THE CHANNEL HAVING THE FACTORIAL CALCULATED
}

Why happen here a deadlock

I am trying to understand, how golang channel works. I read a book about the go language and found the following example.
package main
import (
"fmt"
)
// Send the sequence 2, 3, 4, ... to returned channel
func generate() chan int {
ch := make(chan int)
go func() {
for i := 2; i <= 100 ; i++ {
ch <- i
}
}()
return ch
}
// Filter out input values divisible by 'prime', send rest to returned channel
func filter(in chan int, prime int) chan int {
out := make(chan int)
go func() {
for {
if i := <-in; i%prime != 0 {
out <- i
}
}
}()
return out
}
func sieve() chan int {
out := make(chan int)
go func() {
ch := generate()
for {
prime := <-ch
ch = filter(ch, prime)
out <- prime
}
}()
return out
}
func main() {
primes := sieve()
for {
fmt.Println(<-primes)
}
}
When I run this programm, I've got a deadlock, but when I change the generate function to
// Send the sequence 2, 3, 4, ... to returned channel
func generate() chan int {
ch := make(chan int)
go func() {
for i := 2; ; i++ {
ch <- i
}
}()
return ch
}
Then the programm will run the infinite loop, but not deadlock. Why do I get deadlock, when I remove the condition in for loop?
What do you mean with blocking principle?
You can see it illustrated in the blog post "The Nature Of Channels In Go "
for an unbuffered channel:
(Illustration from blog post "The Nature Of Channels In Go ", written by William Kennedy, Feb. 2014)
Unbuffered channels have no capacity and therefore require both goroutines to be ready to make any exchange.
When a goroutine attempts to write a resource to an unbuffered channel and there is no goroutine waiting to receive the resource, the channel will lock the goroutine and make it wait.
When a goroutine attempts to read from an unbuffered channel, and there is no goroutine waiting to send a resource, the channel will lock the goroutine and make it wait.
That is what happens in your case with your reader:
func main() {
primes := sieve()
for {
fmt.Println(<-primes)
}
}
since primes is never closed, main remains blocked.
It (main) is in step 3:
in step 3, the goroutine on the right places his hand into the channel or performs a read.
That goroutine is also locked in the channel until the exchange is complete.
The sender never calls close(primes).
Let's consider a simpler example:
func generate() chan int {
ch := make(chan int)
go func() {
for i := 2; /*i < 100*/; i++ {
ch <- i
}
}()
return ch
}
func main() {
for i := range generate() {
fmt.Println(i)
}
}
With the condition i < 100 uncommented, the goroutine spawned by generate stops after sending 98 numbers. However, it does not close the channel, so main has no way of knowing that no more numbers are going to be sent, and it just keeps blocking on the channel. Since main is now the only goroutine still in existence (the other one has returned), and it's blocking, you have a deadlock.

Resources