Add items to channel in one goroutine and process in another - go

Really new to the world of go and trying to learn by doing. I want to see how I would use go routines to add items to a "queue" in one go routine while another go routine listens for the queue and processes things as they enter. In this case I have a func that adds a defined amount of items to a int[] while the other attempts to print them as they are added.. I assume a flag will need to be sent to signal that one go routine has stopped adding items to the queue. I apologize for the noob question but am struggling understanding the new terminology and syntax.
package main
import "fmt"
func printFromSlice(c chan []int){
i := <-c
// wait for item to be added to channel
fmt.Print(i)
}
func addToSlice (c chan []int, i int) {
for n := 0; n < i; n++{
c <- n //I know this isnt right
}
// How do we signal this is complete??
}
func main(){
c := make(chan []int)
//add things to []i int
go addToSlice(c, 25)
//do something from []int
go printFromSlice(c)
}
**UPDATE**
modified to use the following code however now it will only execute to a single print form the ~printFromSlice` function before it closes out...
package main
func printFromSlice(c chan int){
i := <-c
// wait for item to be added to channel
println("read")
println(i)
}
func addToSlice (c chan int) {
for n := 0; n < 100; n++{
println("add")
println(n)
c <- n
}
close(c)
}
func main(){
println("starting...")
c := make(chan int)
//add things to []i int
go addToSlice(c)
//do something from []int
go printFromSlice(c)
<-c
}

You need to add a sync.WaitGroup to block the main thread until the two goroutine could finish. You could refer to this Go by Example.
package main
import "sync"
func printFromSlice(c chan int, wg *sync.WaitGroup) {
defer wg.Done() // Decrement the waitgroup counter by 1 after `printFromSlice` returns
// wait for item to be added to channel
// i := <-c // this only waits / blocks 1 time
// For each message arriving at c, read and print
for i := range c { // this would read all messages until channel is closed
println("read")
println(i)
}
}
func addToSlice(c chan int, wg *sync.WaitGroup) {
defer wg.Done() // Decrement the waitgroup counter by 1 after `addToSlice` returns
for n := 0; n < 100; n++ {
println("add")
println(n)
c <- n
}
close(c)
}
func main() {
var wg sync.WaitGroup
println("starting...")
c := make(chan int)
wg.Add(2) // Adds 2 to the waitgroup counter
//add things to []i int
go addToSlice(c, &wg) // Pass the wait group as reference so they can call wg.Done()
//do something from []int
go printFromSlice(c, &wg) // Pass the wait group as reference so they can call wg.Done()
// <-c // No need for this to block the code
wg.Wait() // Waits / blocks until waitgroup counter is 0
}

Related

X number of goroutines to update the same variable

I want to make X number of goroutines to update CountValue using parallelism (numRoutines are as much as how many CPU you have).
Solution 1:
func count(numRoutines int) (countValue int) {
var mu sync.Mutex
k := func(i int) {
mu.Lock()
defer mu.Unlock()
countValue += 5
}
for i := 0; i < numRoutines; i++ {
go k(i)
}
It becomes a data race and the returned countValue = 0.
Solution 2:
func count(numRoutines int) (countValue int) {
k := func(i int, c chan int) {
c <- 5
}
c := make(chan int)
for i := 0; i < numRoutines; i++ {
go k(i, c)
}
for i := 0; i < numRoutines; i++ {
countValue += <- c
}
return
}
I did a benchmark test on it and doing a sequential addition would work faster than using goroutines. I think it's because I have two for loops here as when I put countValue += <- c inside the first for loop, the code runs faster.
Solution 3:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
go func() {
for i := range c {
countValue += i
}
}()
wg.Wait()
return
}
Still a race count :/
Is there any way better to do this?
There definitely is a better way to safely increment a variable: using sync/atomic:
import "sync/atomic"
var words int64
k := func() {
_ = atomic.AddInt64(&words, 5) // increment atomically
}
Using a channel basically eliminates the need for a mutex, or takes away the the risk of concurrent access to the variable itself, and a waitgroup here is just a bit overkill
Channels:
words := 0
done := make(chan struct{}) // or use context
ch := make(chan int, numRoutines) // buffer so each routine can write
go func () {
read := 0
for i := range ch {
words += 5 // or use i or something
read++
if read == numRoutines {
break // we've received data from all routines
}
}
close(done) // indicate this routine has terminated
}()
for i := 0; i < numRoutines; i++ {
ch <- i // write whatever value needs to be used in the counting routine on the channel
}
<- done // wait for our routine that increments words to return
close(ch) // this channel is no longer needed
fmt.Printf("Counted %d\n", words)
As you can tell, the numRoutines no longer is the number of routines, but rather the number of writes on the channel. You can move that to individual routines, still:
for i := 0; i < numRoutines; i++ {
go func(ch chan<- int, i int) {
// do stuff here
ch <- 5 * i // for example
}(ch, i)
}
WaitGroup:
Instead of using a context that you can cancel, or a channel, you can use a waitgroup + atomic to get the same result. The easiest way IMO to do so is to create a type:
type counter struct {
words int64
}
func (c *counter) doStuff(wg *sync.WaitGroup, i int) {
defer wg.Done()
_ = atomic.AddInt64(&c.words, i * 5) // whatever value you need to add
}
func main () {
cnt := counter{}
wg := sync.WaitGroup{}
wg.Add(numRoutines) // create the waitgroup
for i := 0; i < numRoutines; i++ {
go cnt.doStuff(&wg, i)
}
wg.Wait() // wait for all routines to finish
fmt.Println("Counted %d\n", cnt.words)
}
Fix for your third solution
As I mentioned in the comment: your third solution is still causing a race condition because the channel c is never closed, meaning the routine:
go func () {
for i := range c {
countValue += i
}
}()
Never returns. The waitgroup also only ensures that you've sent all values on the channel, but not that the countValue has been incremented to its final value. The fix would be to either close the channel after wg.Wait() returns so the routine can return, and add a done channel that you can close when this last routine returns, and add a <-done statement before returning.
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
done := make(chan struct{})
go func() {
for i := range c {
countValue += i
}
close(done)
}()
wg.Wait()
close(c)
<-done
return
}
This adds some clutter, though, and IMO is a bit messy. It might just be easier to just move the wg.Wait() call to a routine:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
// add wg as argument, makes it easier to move this function outside of this scope
k := func(wg *sync.WaitGroup, i int) {
defer wg.Done()
c <- 5
}
wg.Add(numShards) // increment the waitgroup once
for i := 0; i < numShards; i++ {
go k(&wg, i)
}
go func() {
wg.Wait()
close(c) // this ends the loop over the channel
}()
// just iterate over the channel until it is closed
for i := range c {
countValue += i
}
// we've added all values to countValue
return
}

Dead lock in goroutines

can someone give me some insight about this code, Why this get deadlock error in for x:=range c
func main() {
c:=make(chan int,10)
for i:=0;i<5;i++{
go func(chanel chan int,i int){
println("i",i)
chanel <- 1
}(c,i)
}
for x:=range c {
println(x)
}
println("Done!")
}
Because this:
for x:=range c {
println(x)
}
will loop until the channel c closes, which is never done here.
Here is one way you can fix it, using a WaitGroup:
package main
import "sync"
func main() {
var wg sync.WaitGroup
c := make(chan int, 10)
for i := 0; i < 5; i++ {
wg.Add(1)
go func(chanel chan int, i int) {
defer wg.Done()
println("i", i)
chanel <- 1
}(c, i)
}
go func() {
wg.Wait()
close(c)
}()
for x := range c {
println(x)
}
println("Done!")
}
Try it on Go Playground
You create five goroutines, each sending an integer value to the channel. Once all those five values are written, there's no other goroutine left that writes to the channel.
The main goroutine reads those five values from the channel. But there are no goroutines that can possibly write the sixth value or close the channel. So, you're deadlocked waiting data from the channel.
Close the channel once all the writes are completed. It should be an interesting exercise to figure out how you can do that with this code.
Channel needs to be closed to indicate task is complete.
Coordinate with a sync.WaitGroup:
c := make(chan int, 10)
var wg sync.WaitGroup // here
for i := 0; i < 5; i++ {
wg.Add(1) // here
go func(chanel chan int, i int) {
defer wg.Done()
println("i", i)
chanel <- 1
}(c, i)
}
go func() {
wg.Wait() // and here
close(c)
}()
for x := range c {
println(x)
}
println("Done!")
https://play.golang.org/p/VWcBC2YGLvM

Writing data into same chanel from different go routines is working fine without wait group

When writing data into same channel using multiple go routines with waitgroup after waiting wg.Wait() getting exception saying all go routines are asleep or deedlock.
package main
import (
"fmt"
"runtime"
"sync"
)
var wg sync.WaitGroup
func CreateMultipleRoutines() {
ch := make(chan int)
for i := 0; i < 10; i++ { // creates 10 go routines and adds to waitgroup
wg.Add(1)
go func() {
for j := 0; j < 10; j++ {
ch <- j
}
wg.Done() // indication of go routine is done to main routine
}()
}
fmt.Println(runtime.NumGoroutine())
wg.Wait() //wait for all go routines to complete
close(ch) // closing channel after completion of wait fo go routines
for v := range ch { // range can be used since channel is closed
fmt.Println(v)
}
fmt.Println("About to exit program ...")
}
When tried to implement this without waitgroup I am able to read data from channel by looping exact number of times data pushed to channel but i cant range since there will be panic when we close channel. here is the example code
package main
import (
"fmt"
"runtime"
)
func main() {
ch := make(chan int)
for i := 0; i < 10; i++ { // creates 10 go routines and adds to waitgroup
go func(i int) {
for j := 0; j < 10; j++ {
ch <- j * i
}
}(i)
}
fmt.Println(runtime.NumGoroutine())
for v := 0; v < 100; v++ {
fmt.Println(<-ch)
}
fmt.Println("About to exit program ...")
}
I want to understand why waitgroup in wait state is still waiting even though all go routines are signalled Done() which inturn makes number of go routines to zero
I think your original code has some problems.
You are closing the channel before reading from it.
You are not getting the advantage of using 10 goroutines because of your channel is 1 "sized". So one goroutine is producing one result per once.
My solution would be to spawn a new goroutine to monitor if the 10 goroutines finished its jobs. There you will use your WaitGroup.
Then the code would be like:
package main
import (
"fmt"
"runtime"
"sync"
)
var wg sync.WaitGroup
func main() {
ch := make(chan int, 10)
for i := 0; i < 10; i++ { // creates 10 go routines and adds to waitgroup
wg.Add(1)
go func() {
for j := 0; j < 10; j++ {
ch <- j
}
wg.Done() // indication of go routine is done to main routine
}()
}
go func(){
wg.Wait()
close(ch)
}()
fmt.Println(runtime.NumGoroutine())
for v := range ch { // range can be used since channel is closed
fmt.Println(v)
}
fmt.Println("About to exit program ...")
}
By default a chan holds no items, so all go routines are blocked on sending, until something reads from it. They never actually reach the wg.Done() statement.
A solution would be to close the channel in it's own go routine. Wrap your wg.Wait() and close(ch) lines like this:
go func() {
wg.Wait() //wait for all go routines to complete
close(ch) // closing channel after completion of wait fo go routines
}()
Then you can range over the channel, which will only close after all of the sending go routines have finished (and implicitly all values have been received).

All goroutines are asleep

I write some code which aims to synchronize using channel.
var counter int64 // shared resource
var wg sync.WaitGroup
func main() {
ch := make(chan int64)
wg.Add(2)
go incCounter(ch)
go incCounter(ch)
ch <- counter
wg.Wait()
fmt.Println("Final Counter:", counter) // expected value is 4
}
func incCounter(ch chan int64) {
defer wg.Done()
for count := 0; count < 2; count++ {
value := <-ch
value++
counter = value
ch <- counter
}
}
When I ran this program, an error happened: all goroutines are asleep - deadlock!. However I can't fix the problem and I don't know what is wrong. Could anyone help?
Channels make(chan int) has implicit size zero ( ref: https://golang.org/ref/spec#Making_slices_maps_and_channels)
A channel of size zero is unbuffered. A channel of specified size make(chan int, n) is buffered. See http://golang.org/ref/spec#Send_statements for a discussion on buffered vs. unbuffered channels. The example at http://play.golang.org/p/VZAiN1V8-P illustrates the difference.
Here, channel <-ch or ch <- will be blocked until someone processes it (concurrently). If you try the flow of this program in pen and paper, you will figure it out why it is blocked. Below figure shows possible data flow through channel ch:
So if you make your ch := make(chan int64) to ch := make(chan int64,1), it will work.
var counter int64 // shared resource
var wg sync.WaitGroup
func main() {
ch := make(chan int64, 1)
wg.Add(2)
go incCounter(ch)
go incCounter(ch)
ch <- counter
wg.Wait()
fmt.Println("Final Counter:", counter) // expected value is 4
}
func incCounter(ch chan int64) {
defer wg.Done()
for count := 0; count < 2; count++ {
value := <-ch
value++
counter = value
ch <- counter
}
}
If we analyse how program works when you are using ch := make(chan int64), we can see that one go routine is blocked in this program(another one is exited). With the help of time.Sleep(n) and receiving the last data from the channel in the blocked go routine, we can overcome the deadlock. See the code below:
var counter int64 // shared resource
var wg sync.WaitGroup
func main() {
ch := make(chan int64)
wg.Add(2)
go incCounter(ch)
go incCounter(ch)
ch <- counter
// to ensure one go routine 'incCounter' is completed and one go routine is blocked for unbuffered channel
time.Sleep(3*time.Second)
<-ch // to unblock the last go routine
wg.Wait()
fmt.Println("Final Counter:", counter) // expected value is 4
}
func incCounter(ch chan int64) {
defer wg.Done()
for count := 0; count < 2; count++ {
value := <-ch
value++
counter = value
ch <- counter
}
}

Go: transformed channel

Let's say I have an int channel in Go:
theint := make(chan int)
I want to wrap this channel in a new channel called incremented
incremented := make(chan int)
Such that:
go func() { theint <- 1 }
<- incremented // 2
appended can be assumed to be the only one that reads from the int.
It will work if a run a goroutine in the background
go func() {
for num := range theint {
incremented <- num + 1
}
}
However, I prefer to do it without a goroutine since I can't control it in my context.
Is there a simpler way to do it?
One thing that came to mind is python's yield:
for num in theint:
yield num + 1
Is something like this possible in go?
Generator pattern
What you are trying to implement is generator pattern. To use channels and goroutines for implementation of this pattern is totally common practice.
However, I prefer to do it without a goroutine since I can't control it in my context.
I believe the problem is deadlock
fatal error: all goroutines are asleep - deadlock!
To avoid deadlocks and orphaned (not closed) channels use sync.WaitGroup. This is an idiomatic way to control goroutines.
Playground
package main
import (
"fmt"
"sync"
)
func incGenerator(n []int) chan int {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(len(n))
for _, i := range n {
incremented := i + 1
go func() {
wg.Done()
ch <- incremented
}()
}
go func() {
wg.Wait()
close(ch)
}()
return ch
}
func main() {
n := []int{1, 2, 3, 4, 5}
for x := range incGenerator(n) {
fmt.Println(x)
}
}
One thing you can also consider is having a select on the int channel and an exit channel - in an infinite for loop. You can choose a variable increment value too. Please see code below:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
var accum int //accumulator of incremented values
var wg sync.WaitGroup
c1 := make(chan int)
exChan := make(chan bool)
wg.Add(1)
go func() {
time.Sleep(time.Second * 1)
c1 <- 1
wg.Done()
}()
wg.Add(1)
go func() {
time.Sleep(time.Second * 2)
c1 <- 2
wg.Done()
}()
wg.Add(1)
go func() {
time.Sleep(time.Second * 2)
c1 <- 5
wg.Done()
}()
go func() {
wg.Wait()
close(exChan)
}()
for {
var done bool
select {
case incBy := <-c1: //Increment by value in channel
accum += incBy
fmt.Println("Received value to increment:", incBy, "; Accumulated value is", accum)
case d := <-exChan:
done = !(d)
}
if done == true {
break
}
}
fmt.Println("Final accumulated value is", accum)
}
Playground: https://play.golang.org/p/HmdRmMCN7U
Exit channel not needed, if we are having non-zero increments always. I like #I159 's approach too!
Anyways, hope this helps.

Resources