how to use channel inside paralleled for-loop - go

func parallelSum (c chan int){
sum := 0
for i :=1 ; i< 100;i++{
go func(i int){
sum += i
}(i)
}
time.Sleep(1*time.Second)
c <- sum
}
I'm trying to learn the parallel ability to speed up things like OpenMP. And here is an example of the intended summing up parallel loop in Go, this function runs as a goroutine.
Note that the variable sum is not a channel here, so does this mean the variable sum access inside the for loop is a blocked operation? Is it now efficient enough? Is there a better solution?
I knew the channel feature was designed for this, my obviously wrong implement below can compile, but with 100 runtime errors like following.
goroutine 4 [chan receive]:
main.parallelSumError(0xc0000180c0)
/home/tom/src/goland_newproject/main.go:58 +0xb4 //line 58 : temp := <-sum
created by main.main
/home/tom/src/goland_newproject/main.go:128 +0x2ca //line 128: go parallelSumError(pcr), the calling function
So what's the problem here? it seems summing is not a good example for paralleled for-loop, but actually I wish to know how to use channel inside paralleled for-loop.
func parallelSum (c chan int){
sum := make(chan int)
for i :=1 ; i< 100;i++{
go func(i int){
temp := <- sum //error here why?
temp += i
sum <- temp
}(i)
}
time.Sleep(1*time.Second)
temp := <-sum
c <- temp
}
both with the same main function
func main(){
pc := make(chan int)
go parallelSum(pc)
result = <- pc
fmt.Println("parallel result:", result)
}

I don't like the idea of summing numbers through channels. I'd rather use something classical like sync.Mutex or atomic.AddUint64. But, at least, I made your code working.
We aren't able to pass a value from one channel to another (I added temp variable). Also, there is sync.WaitGroup and other stuff.
But I still don't like the idea of the code.
package main
import (
"fmt"
"sync"
)
func main() {
pc := make(chan int)
go parallelSum(pc)
result := <- pc
fmt.Println("parallel result:", result)
}
func parallelSum (c chan int){
sum := make(chan int)
wg := sync.WaitGroup{}
wg.Add(100)
for i :=1 ; i <= 100;i++{
go func(i int){
temp := <- sum
temp += i
wg.Done()
sum <- temp
}(i)
}
sum <- 0
wg.Wait()
temp := <- sum
c <- temp
}

When using go routines (i.e. go foo()), it is preferable to use communication over memory-sharing. In this matter, as you mention, channels are the golang way to handle communication.
For your specific application, the paralleled sum similar to OpenMP, it would be preferable to detect the number of CPUs and generate as many routines as wished:
package main
import (
"fmt"
"runtime"
)
func main() {
numCPU := runtime.NumCPU()
sumc := make(chan int, numCPU)
valuec := make(chan int)
endc := make(chan interface{}, numCPU)
// generate go routine per cpu
for i := 0; i < numCPU; i++ {
go sumf(sumc, valuec, endc)
}
// generate values and pass it through the channels
for i := 0; i < 100; i++ {
valuec <- i
}
// tell go routines to end up when they are done
for i := 0; i < numCPU; i++ {
endc <- nil
}
// sum results
sum := 0
for i := 0; i < numCPU; i++ {
procSum := <-sumc
sum += procSum
}
fmt.Println(sum)
}
func sumf(sumc, valuec chan int, endc chan interface{}) {
sum := 0
for {
select {
case i := <-valuec:
sum += i
case <-endc:
sumc <- sum
return
}
}
}
Hopefully, this helps.

Related

X number of goroutines to update the same variable

I want to make X number of goroutines to update CountValue using parallelism (numRoutines are as much as how many CPU you have).
Solution 1:
func count(numRoutines int) (countValue int) {
var mu sync.Mutex
k := func(i int) {
mu.Lock()
defer mu.Unlock()
countValue += 5
}
for i := 0; i < numRoutines; i++ {
go k(i)
}
It becomes a data race and the returned countValue = 0.
Solution 2:
func count(numRoutines int) (countValue int) {
k := func(i int, c chan int) {
c <- 5
}
c := make(chan int)
for i := 0; i < numRoutines; i++ {
go k(i, c)
}
for i := 0; i < numRoutines; i++ {
countValue += <- c
}
return
}
I did a benchmark test on it and doing a sequential addition would work faster than using goroutines. I think it's because I have two for loops here as when I put countValue += <- c inside the first for loop, the code runs faster.
Solution 3:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
go func() {
for i := range c {
countValue += i
}
}()
wg.Wait()
return
}
Still a race count :/
Is there any way better to do this?
There definitely is a better way to safely increment a variable: using sync/atomic:
import "sync/atomic"
var words int64
k := func() {
_ = atomic.AddInt64(&words, 5) // increment atomically
}
Using a channel basically eliminates the need for a mutex, or takes away the the risk of concurrent access to the variable itself, and a waitgroup here is just a bit overkill
Channels:
words := 0
done := make(chan struct{}) // or use context
ch := make(chan int, numRoutines) // buffer so each routine can write
go func () {
read := 0
for i := range ch {
words += 5 // or use i or something
read++
if read == numRoutines {
break // we've received data from all routines
}
}
close(done) // indicate this routine has terminated
}()
for i := 0; i < numRoutines; i++ {
ch <- i // write whatever value needs to be used in the counting routine on the channel
}
<- done // wait for our routine that increments words to return
close(ch) // this channel is no longer needed
fmt.Printf("Counted %d\n", words)
As you can tell, the numRoutines no longer is the number of routines, but rather the number of writes on the channel. You can move that to individual routines, still:
for i := 0; i < numRoutines; i++ {
go func(ch chan<- int, i int) {
// do stuff here
ch <- 5 * i // for example
}(ch, i)
}
WaitGroup:
Instead of using a context that you can cancel, or a channel, you can use a waitgroup + atomic to get the same result. The easiest way IMO to do so is to create a type:
type counter struct {
words int64
}
func (c *counter) doStuff(wg *sync.WaitGroup, i int) {
defer wg.Done()
_ = atomic.AddInt64(&c.words, i * 5) // whatever value you need to add
}
func main () {
cnt := counter{}
wg := sync.WaitGroup{}
wg.Add(numRoutines) // create the waitgroup
for i := 0; i < numRoutines; i++ {
go cnt.doStuff(&wg, i)
}
wg.Wait() // wait for all routines to finish
fmt.Println("Counted %d\n", cnt.words)
}
Fix for your third solution
As I mentioned in the comment: your third solution is still causing a race condition because the channel c is never closed, meaning the routine:
go func () {
for i := range c {
countValue += i
}
}()
Never returns. The waitgroup also only ensures that you've sent all values on the channel, but not that the countValue has been incremented to its final value. The fix would be to either close the channel after wg.Wait() returns so the routine can return, and add a done channel that you can close when this last routine returns, and add a <-done statement before returning.
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
done := make(chan struct{})
go func() {
for i := range c {
countValue += i
}
close(done)
}()
wg.Wait()
close(c)
<-done
return
}
This adds some clutter, though, and IMO is a bit messy. It might just be easier to just move the wg.Wait() call to a routine:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
// add wg as argument, makes it easier to move this function outside of this scope
k := func(wg *sync.WaitGroup, i int) {
defer wg.Done()
c <- 5
}
wg.Add(numShards) // increment the waitgroup once
for i := 0; i < numShards; i++ {
go k(&wg, i)
}
go func() {
wg.Wait()
close(c) // this ends the loop over the channel
}()
// just iterate over the channel until it is closed
for i := range c {
countValue += i
}
// we've added all values to countValue
return
}

Getting deadlock as I try to emulate fan in - fan out with factorial calculations

I am trying the fan in - fan out pattern with a factorial problem. But I am getting:
fatal error: all goroutines are asleep - deadlock!
and unable to identify the reason for deadlock.
I am trying to concurrently calculate factorial for 100 numbers using the fan-in fan-out pattern.
package main
import (
"fmt"
)
func main() {
_inChannel := _inListener(generator())
for val := range _inChannel {
fmt.Print(val, " -- ")
}
}
func generator() chan int { // NEED TO CALCULATE FACTORIAL FOR 100 NUMBERS
ch := make(chan int) // CREATE CHANNEL TO INPUT NUMBERS
go func() {
for i := 1; i <= 100; i++ {
ch <- i
}
close(ch) // CLOSE CHANNEL WHEN ALL NUMBERS HAVE BEEN WRITTEM
}()
return ch
}
func _inListener(ch chan int) chan int {
rec := make(chan int) // CHANNEL RECEIVED FROM GENERATOR
go func() {
for num := range ch { // RECEIVE THE INPUT NUMBERS FROM GENERATOR
result := factorial(num) // RESULT IS A NEW CHANNEL CREATED
rec <- <-result // MERGE INTO A SINGLE CHANNEL; rec
close(result)
}
close(rec)
}()
return rec // RETURN THE DEDICATED CHANNEL TO RECEIVE ALL OUTPUTS
}
func factorial(n int) chan int {
ch := make(chan int) // MAKE A NEW CHANNEL TO OUTPUT THE RESULT
// OF FACTORIAL
total := 1
for i := n; i > 0; i-- {
total *= i
}
ch <- total
return ch // RETURN THE CHANNEL HAVING THE FACTORIAL CALCULATED
}
I have put in comments, so that it becomes easier to follow the code.
I'm no expert in channels. I've taking on this to try and get more familiar with go.
Another issue is the int isn't large enough to take all factorials over 20 or so.
As you can see, I added a defer close as well as a logical channel called done in the generator func. The rest of the changes probably aren't needed. With channels you need to make sure something is ready to take off a value on the channel when you put something on a channel. Otherwise deadlock. Also, using
go run -race main.go
helps at least see which line(s) are causing problems.
I hope this helps and isn't removed for being off topic.
I was able to remove the deadlock by doing this:
package main
import (
"fmt"
)
func main() {
_gen := generator()
_inChannel := _inListener(_gen)
for val := range _inChannel {
fmt.Print(val, " -- \n")
}
}
func generator() chan int { // NEED TO CALCULATE FACTORIAL FOR 100 NUMBERS
ch := make(chan int) // CREATE CHANNEL TO INPUT NUMBERS
done := make(chan bool)
go func() {
defer close(ch)
for i := 1; i <= 100; i++ {
ch <- i
}
//close(ch) // CLOSE CHANNEL WHEN ALL NUMBERS HAVE BEEN WRITTEM
done <- true
}()
// this function will pull off the done for each function call above.
go func() {
for i := 1; i < 100; i++ {
<-done
}
}()
return ch
}
func _inListener(ch chan int) chan int {
rec := make(chan int) // CHANNEL RECEIVED FROM GENERATOR
go func() {
for num := range ch { // RECEIVE THE INPUT NUMBERS FROM GENERATOR
result := factorial(num) // RESULT IS A NEW CHANNEL CREATED
rec <- result // MERGE INTO A SINGLE CHANNEL; rec
}
close(rec)
}()
return rec // RETURN THE DEDICATED CHANNEL TO RECEIVE ALL OUTPUTS
}
func factorial(n int) int {
// OF FACTORIAL
total := 1
for i := n; i > 0; i-- {
total *= i
}
return total // RETURN THE CHANNEL HAVING THE FACTORIAL CALCULATED
}

Go: transformed channel

Let's say I have an int channel in Go:
theint := make(chan int)
I want to wrap this channel in a new channel called incremented
incremented := make(chan int)
Such that:
go func() { theint <- 1 }
<- incremented // 2
appended can be assumed to be the only one that reads from the int.
It will work if a run a goroutine in the background
go func() {
for num := range theint {
incremented <- num + 1
}
}
However, I prefer to do it without a goroutine since I can't control it in my context.
Is there a simpler way to do it?
One thing that came to mind is python's yield:
for num in theint:
yield num + 1
Is something like this possible in go?
Generator pattern
What you are trying to implement is generator pattern. To use channels and goroutines for implementation of this pattern is totally common practice.
However, I prefer to do it without a goroutine since I can't control it in my context.
I believe the problem is deadlock
fatal error: all goroutines are asleep - deadlock!
To avoid deadlocks and orphaned (not closed) channels use sync.WaitGroup. This is an idiomatic way to control goroutines.
Playground
package main
import (
"fmt"
"sync"
)
func incGenerator(n []int) chan int {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(len(n))
for _, i := range n {
incremented := i + 1
go func() {
wg.Done()
ch <- incremented
}()
}
go func() {
wg.Wait()
close(ch)
}()
return ch
}
func main() {
n := []int{1, 2, 3, 4, 5}
for x := range incGenerator(n) {
fmt.Println(x)
}
}
One thing you can also consider is having a select on the int channel and an exit channel - in an infinite for loop. You can choose a variable increment value too. Please see code below:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
var accum int //accumulator of incremented values
var wg sync.WaitGroup
c1 := make(chan int)
exChan := make(chan bool)
wg.Add(1)
go func() {
time.Sleep(time.Second * 1)
c1 <- 1
wg.Done()
}()
wg.Add(1)
go func() {
time.Sleep(time.Second * 2)
c1 <- 2
wg.Done()
}()
wg.Add(1)
go func() {
time.Sleep(time.Second * 2)
c1 <- 5
wg.Done()
}()
go func() {
wg.Wait()
close(exChan)
}()
for {
var done bool
select {
case incBy := <-c1: //Increment by value in channel
accum += incBy
fmt.Println("Received value to increment:", incBy, "; Accumulated value is", accum)
case d := <-exChan:
done = !(d)
}
if done == true {
break
}
}
fmt.Println("Final accumulated value is", accum)
}
Playground: https://play.golang.org/p/HmdRmMCN7U
Exit channel not needed, if we are having non-zero increments always. I like #I159 's approach too!
Anyways, hope this helps.

Going mutex-less

Alright, Go "experts". How would you write this code in idiomatic Go, aka without a mutex in next?
package main
import (
"fmt"
)
func main() {
done := make(chan int)
x := 0
for i := 0; i < 10; i++ {
go func() {
y := next(&x)
fmt.Println(y)
done <- 0
}()
}
for i := 0; i < 10; i++ {
<-done
}
fmt.Println(x)
}
var mutex = make(chan int, 1)
func next(p *int) int {
mutex <- 0
// critical section BEGIN
x := *p
*p++
// critical section END
<-mutex
return x
}
Assume you can't have two goroutines in the critical section at the same time, or else bad things will happen.
My first guess is to have a separate goroutine to handle the state, but I can't figure out a way to match up inputs / outputs.
You would use an actual sync.Mutex:
var mutex sync.Mutex
func next(p *int) int {
mutex.Lock()
defer mutex.Unlock()
x := *p
*p++
return x
}
Though you would probably also group the next functionality, state and sync.Mutex into a single struct.
Though there's no reason to do so in this case, since a Mutex is better suited for mutual exclusion around a single resource, you can use goroutines and channels to achieve the same effect
http://play.golang.org/p/RR4TQXf2ct
x := 0
var wg sync.WaitGroup
send := make(chan *int)
recv := make(chan int)
go func() {
for i := range send {
x := *i
*i++
recv <- x
}
}()
for i := 0; i < 10; i++ {
wg.Add(1)
go func() {
defer wg.Done()
send <- &x
fmt.Println(<-recv)
}()
}
wg.Wait()
fmt.Println(x)
As #favoretti mentioned, sync/atomic is a way to do it.
But, you have to use int32 or int64 rather than int (since int can be different sizes on different platforms).
Here's an example on Playground
package main
import (
"fmt"
"sync/atomic"
)
func main() {
done := make(chan int)
x := int64(0)
for i := 0; i < 10; i++ {
go func() {
y := next(&x)
fmt.Println(y)
done <- 0
}()
}
for i := 0; i < 10; i++ {
<-done
}
fmt.Println(x)
}
func next(p *int64) int64 {
return atomic.AddInt64(p, 1) - 1
}

What is the correct way to write a distinct channel in Go?

I am a beginner in go.
I am trying to figure out an easy way to implement a channel that only output distinct values.
What I want to do is this:
package example
import (
"fmt"
"testing"
)
func TestShouldReturnDistinctValues(t *testing.T) {
var c := make([]chan int)
c <- 1
c <- 1
c <- 2
c <- 2
c <- 3
for e := range c {
// only print 1, 2 and 3.
fmt.println(e)
}
}
Should I be concern about memory leak here if I were to use a map to remember previous values?
You really can't do that, you'd have to keep a track of the values somehow, a map[int]struct{} is probably the most memory efficient way.
A simple example:
func UniqueGen(min, max int) <-chan int {
m := make(map[int]struct{}, max-min)
ch := make(chan int)
go func() {
for i := 0; i < 1000; i++ {
v := min + rand.Intn(max)
if _, ok := m[v]; !ok {
ch <- v
m[v] = struct{}{}
}
}
close(ch)
}()
return ch
}
I have done similar things before, except my problem was output inputs in ascending order. You can do this by adding a middle go routine. Here is an example:
package main
func main() {
input, output := distinct()
go func() {
input <- 1
input <- 1
input <- 2
input <- 2
input <- 3
close(input)
}()
for i := range output {
println(i)
}
}
func distinct() (input chan int, output chan int) {
input = make(chan int)
output = make(chan int)
go func() {
set := make(map[int]struct{})
for i := range input {
if _, ok := set[i]; !ok {
set[i] = struct{}{}
output <- i
}
}
close(output)
}()
return
}

Resources