Let's say I have an int channel in Go:
theint := make(chan int)
I want to wrap this channel in a new channel called incremented
incremented := make(chan int)
Such that:
go func() { theint <- 1 }
<- incremented // 2
appended can be assumed to be the only one that reads from the int.
It will work if a run a goroutine in the background
go func() {
for num := range theint {
incremented <- num + 1
}
}
However, I prefer to do it without a goroutine since I can't control it in my context.
Is there a simpler way to do it?
One thing that came to mind is python's yield:
for num in theint:
yield num + 1
Is something like this possible in go?
Generator pattern
What you are trying to implement is generator pattern. To use channels and goroutines for implementation of this pattern is totally common practice.
However, I prefer to do it without a goroutine since I can't control it in my context.
I believe the problem is deadlock
fatal error: all goroutines are asleep - deadlock!
To avoid deadlocks and orphaned (not closed) channels use sync.WaitGroup. This is an idiomatic way to control goroutines.
Playground
package main
import (
"fmt"
"sync"
)
func incGenerator(n []int) chan int {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(len(n))
for _, i := range n {
incremented := i + 1
go func() {
wg.Done()
ch <- incremented
}()
}
go func() {
wg.Wait()
close(ch)
}()
return ch
}
func main() {
n := []int{1, 2, 3, 4, 5}
for x := range incGenerator(n) {
fmt.Println(x)
}
}
One thing you can also consider is having a select on the int channel and an exit channel - in an infinite for loop. You can choose a variable increment value too. Please see code below:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
var accum int //accumulator of incremented values
var wg sync.WaitGroup
c1 := make(chan int)
exChan := make(chan bool)
wg.Add(1)
go func() {
time.Sleep(time.Second * 1)
c1 <- 1
wg.Done()
}()
wg.Add(1)
go func() {
time.Sleep(time.Second * 2)
c1 <- 2
wg.Done()
}()
wg.Add(1)
go func() {
time.Sleep(time.Second * 2)
c1 <- 5
wg.Done()
}()
go func() {
wg.Wait()
close(exChan)
}()
for {
var done bool
select {
case incBy := <-c1: //Increment by value in channel
accum += incBy
fmt.Println("Received value to increment:", incBy, "; Accumulated value is", accum)
case d := <-exChan:
done = !(d)
}
if done == true {
break
}
}
fmt.Println("Final accumulated value is", accum)
}
Playground: https://play.golang.org/p/HmdRmMCN7U
Exit channel not needed, if we are having non-zero increments always. I like #I159 's approach too!
Anyways, hope this helps.
Related
can someone give me some insight about this code, Why this get deadlock error in for x:=range c
func main() {
c:=make(chan int,10)
for i:=0;i<5;i++{
go func(chanel chan int,i int){
println("i",i)
chanel <- 1
}(c,i)
}
for x:=range c {
println(x)
}
println("Done!")
}
Because this:
for x:=range c {
println(x)
}
will loop until the channel c closes, which is never done here.
Here is one way you can fix it, using a WaitGroup:
package main
import "sync"
func main() {
var wg sync.WaitGroup
c := make(chan int, 10)
for i := 0; i < 5; i++ {
wg.Add(1)
go func(chanel chan int, i int) {
defer wg.Done()
println("i", i)
chanel <- 1
}(c, i)
}
go func() {
wg.Wait()
close(c)
}()
for x := range c {
println(x)
}
println("Done!")
}
Try it on Go Playground
You create five goroutines, each sending an integer value to the channel. Once all those five values are written, there's no other goroutine left that writes to the channel.
The main goroutine reads those five values from the channel. But there are no goroutines that can possibly write the sixth value or close the channel. So, you're deadlocked waiting data from the channel.
Close the channel once all the writes are completed. It should be an interesting exercise to figure out how you can do that with this code.
Channel needs to be closed to indicate task is complete.
Coordinate with a sync.WaitGroup:
c := make(chan int, 10)
var wg sync.WaitGroup // here
for i := 0; i < 5; i++ {
wg.Add(1) // here
go func(chanel chan int, i int) {
defer wg.Done()
println("i", i)
chanel <- 1
}(c, i)
}
go func() {
wg.Wait() // and here
close(c)
}()
for x := range c {
println(x)
}
println("Done!")
https://play.golang.org/p/VWcBC2YGLvM
I'm reading the "The Go Programming Language"
One way to limit the number of running go routines is to use a "counting semaphore".
The other way is Limiting number of go routines running
I am allowing 2 more go routines in the case. I'm getting deadlock error.
What causes the deadlock in my code?
package main
import (
"bytes"
//"context"
"fmt"
"runtime"
"strconv"
"sync"
"time"
)
func main() {
max := 2
var wg sync.WaitGroup
squares := make(chan int)
tokens := make(chan struct{}, max)
for i := 20; i >= 1; i-- {
tokens <- struct{}{}
wg.Add(1)
go func(n int) {
defer func() { <-tokens }()
defer wg.Done()
fmt.Println("run go routine ", getGID())
squares <- Square(n)
}(i)
}
go func() {
wg.Wait()
close(squares)
}()
for s := range squares {
fmt.Println("Get square: ", s)
}
}
func Square(num int) int {
time.Sleep(time.Second * time.Duration(num))
fmt.Println(num * num)
return num * num
}
func getGID() uint64 {
b := make([]byte, 64)
b = b[:runtime.Stack(b, false)]
b = bytes.TrimPrefix(b, []byte("goroutine "))
b = b[:bytes.IndexByte(b, ' ')]
n, _ := strconv.ParseUint(string(b), 10, 64)
return n
}
The goroutines block on sending to squares. Main is not receiving on squares because it blocks on starting the the goroutines.
Fix by moving the code that starts the goroutines to a goroutine. This allows main to continue executing to the receive on squares.
squares := make(chan int)
go func() {
max := 2
var wg sync.WaitGroup
tokens := make(chan struct{}, max)
for i := 20; i >= 1; i-- {
tokens <- struct{}{}
wg.Add(1)
go func(n int) {
defer func() { <-tokens }()
defer wg.Done()
fmt.Println("run go routine ", getGID())
squares <- Square(n)
}(i)
}
wg.Wait()
close(squares)
}()
fmt.Println("About to receive")
for s := range squares {
fmt.Println("Get square: ", s)
}
Run it on the playground.
I am experimenting with channel concept in Go. I wrote the below program playground to implement counter using channels. But I am not getting any output, although I am doing some printing in the goroutine body.
func main() {
wg := sync.WaitGroup{}
ch := make(chan int)
count := func(ch chan int) {
var last int
last = <-ch
last = last + 1
fmt.Println(last)
ch <- last
wg.Done()
}
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
}
I expect at least some output but I am getting none at all.
When the main() function (the main goroutine) ends, your program ends as well (it doesn't wait for other non-main goroutines to finish). You must add a wg.Wait() call to the end. See No output from goroutine in Go.
Once you do this, you'll hit a deadlock. This is because all goroutines start with attempting to receive a value from the channel, and only then would they send something.
So you should first send something on the channel to let at least one of the goroutines to proceed.
Once you do that, you'll see numbers printed 10 times, and again deadlock. This is because when the last goroutine tries to send its incremented number, there will be no one to receive that. An easy way to fix that is to give a buffer to the channel.
Final, working example:
wg := sync.WaitGroup{}
ch := make(chan int, 2)
count := func(ch chan int) {
var last int
last = <-ch
last = last + 1
fmt.Println(last)
ch <- last
wg.Done()
}
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
go func() {
ch <- 0
}()
wg.Wait()
Outputs (try it on the Go Playground):
1
2
3
4
5
6
7
8
9
10
Also note that since we made the channel buffered, it's not necessary to use another goroutine to send an initial value, we can do that in the main goroutine:
ch <- 0
wg.Wait()
This will output the same. Try it on the Go Playground.
func main() {
wg := sync.WaitGroup{}
ch := make(chan int)
count := func(ch chan int) {
var last int
last, ok := <-ch // 这里做一层保护
if !ok {
return
}
last = last + 1
fmt.Println(last)
go func(ch chan int, res int) {
ch <- res
}(ch, last)
wg.Done()
}
go func() {
ch <- 0
}()
wg.Add(10)
for i := 1; i <= 10; i++ {
go count(ch)
}
wg.Wait()
fmt.Println("main finish")
close(ch)
}
I have been working in Golang for a long time. But still I am facing this problem though I know the solution to my problem. But never figured out why is it happening.
For example If I have a pipeline situation for inbound and outbound channels like below:
package main
import (
"fmt"
)
func main() {
for n := range sq(sq(gen(3, 4))) {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n
}
close(out)
}()
return out
}
It does not give me a deadlock situation. But if I remove the go routine inside the outbound code as below:
func sq(in <-chan int) <-chan int {
out := make(chan int)
for n := range in {
out <- n * n
}
close(out)
return out
}
I received a deadlock error. Why is it so that looping over channels using range without go routine gives a deadlock.
This situation caused of output channel of sq function is not buffered. So sq is waiting until next function will read from output, but if sq is not async, it will not happen (Playground link):
package main
import (
"fmt"
"sync"
)
var wg sync.WaitGroup
func main() {
numsCh := gen(3, 4)
sqCh := sq(numsCh) // if there is no sq in body - we are locked here until input channel will be closed
result := sq(sqCh) // but if output channel is not buffered, so `sq` is locked, until next function will read from output channel
for n := range result {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int, 100)
for n := range in {
out <- n * n
}
close(out)
return out
}
Your function creates a channel, writes to it, then returns it. The writing will block until somebody can read the corresponding value, but that's impossible because nobody outside this function has the channel yet.
func sq(in <-chan int) <-chan int {
// Nobody else has this channel yet...
out := make(chan int)
for n := range in {
// ...but this line will block until somebody reads the value...
out <- n * n
}
close(out)
// ...and nobody else can possibly read it until after this return.
return out
}
If you wrap the loop in a goroutine then both the loop and the sq function are allowed to continue; even if the loop blocks, the return out statement can still go and eventually you'll be able to connect up a reader to the channel.
(There's nothing intrinsically bad about looping over channels outside of goroutines; your main function does it harmlessly and correctly.)
The reason of the deadlock is because the main is waiting for the sq return and finish, but the sq is waiting for someone read the chan then it can continue.
I simplified your code by removing layer of sq call, and split one sentence into 2 :
func main() {
result := sq(gen(3, 4)) // <-- block here, because sq doesn't return
for n := range result {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int)
for n := range in {
out <- n * n // <-- block here, because no one is reading from the chan
}
close(out)
return out
}
In sq method, if you put code in goroutine, then the sq will returned, and main func will not block, and consume the result queue, and the goroutine will continue, then there is no block any more.
func main() {
result := sq(gen(3, 4)) // will not blcok here, because the sq just start a goroutine and return
for n := range result {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n // will not block here, because main will continue and read the out chan
}
close(out)
}()
return out
}
The code is a bit complicated,
Let's simplify
First eq below, not has deadLock
func main() {
send := make(chan int)
receive := make(chan int)
go func() {
send<-3
send<-4
close(send)
}()
go func() {
receive<- <-send
receive<- <-send
close(receive)
}()
for v := range receive{
fmt.Println(v)
}
}
Second eq below,remove "go" has deadLock
func main() {
send := make(chan int)
receive := make(chan int)
go func() {
send<-3
send<-4
close(send)
}()
receive<- <-send
receive<- <-send
close(receive)
for v := range receive{
fmt.Println(v)
}
}
Let's simplify second code again
func main() {
ch := make(chan int)
ch <- 3
ch <- 4
close(ch)
for v := range ch{
fmt.Println(v)
}
}
The reason of the deadlock is no buffer channel waiting in main goroutine.
Two Solutions
// add more cap then "channel<-" time
func main() {
ch := make(chan int,2)
ch <- 3
ch <- 4
close(ch)
for v := range ch{
fmt.Println(v)
}
}
//async "<-channel"
func main() {
ch := make(chan int)
go func() {
for v := range ch {
fmt.Println(v)
}
}()
ch <- 3
ch <- 4
close(ch)
}
My understanding is
when the main thread is blocked for waiting for the chan to be writen or read, Go will detect if any other Go routine is running. If there is no any other Go routine running, it will have "fatal error: all goroutines are asleep - deadlock!"
I tested if by using the below simple case
func main() {
c := make(chan int)
go func() {
time.Sleep(10 * time.Second)
}()
c <- 1
}
The deadlock error is reported after 10 seconds.
I'm working on a concurrent Go library, and I stumbled upon two distinct patterns of synchronization between goroutines whose results are similar:
Waitgroup
package main
import (
"fmt"
"sync"
"time"
)
var wg sync.WaitGroup
func main() {
words := []string{"foo", "bar", "baz"}
for _, word := range words {
wg.Add(1)
go func(word string) {
time.Sleep(1 * time.Second)
defer wg.Done()
fmt.Println(word)
}(word)
}
// do concurrent things here
// blocks/waits for waitgroup
wg.Wait()
}
Channel
package main
import (
"fmt"
"time"
)
func main() {
words := []string{"foo", "bar", "baz"}
done := make(chan bool)
// defer close(done)
for _, word := range words {
// fmt.Println(len(done), cap(done))
go func(word string) {
time.Sleep(1 * time.Second)
fmt.Println(word)
done <- true
}(word)
}
// Do concurrent things here
// This blocks and waits for signal from channel
for range words {
<-done
}
}
I was advised that sync.WaitGroup is slightly more performant, and I have seen it being used commonly. However, I find channels more idiomatic. What is the real advantage of using sync.WaitGroup over channels and/or what might be the situation when it is better?
Independently of the correctness of your second example (as explained in the comments, you aren't doing what you think, but it's easily fixable), I tend to think that the first example is easier to grasp.
Now, I wouldn't even say that channels are more idiomatic. Channels being a signature feature of the Go language shouldn't mean that it is idiomatic to use them whenever possible. What is idiomatic in Go is to use the simplest and easiest to understand solution: here, the WaitGroup convey both the meaning (your main function is Waiting for workers to be done) and the mechanic (the workers notify when they are Done).
Unless you're in a very specific case, I don't recommend using the channel solution here.
For your simple example (signalling the completion of jobs), the WaitGroup is the obvious choice. And the Go compiler is very kind and won't blame you for using a channel for the simple signalling of the completion task, but some code reviewer do.
"A WaitGroup waits for a collection of goroutines to finish.
The main goroutine calls Add(n) to set the number of
goroutines to wait for. Then each of the goroutines
runs and calls Done() when finished. At the same time,
Wait can be used to block until all goroutines have finished."
words := []string{"foo", "bar", "baz"}
var wg sync.WaitGroup
for _, word := range words {
wg.Add(1)
go func(word string) {
defer wg.Done()
time.Sleep(100 * time.Millisecond) // a job
fmt.Println(word)
}(word)
}
wg.Wait()
The possibilities are limited only by your imagination:
Channels can be buffered:
words := []string{"foo", "bar", "baz"}
done := make(chan struct{}, len(words))
for _, word := range words {
go func(word string) {
time.Sleep(100 * time.Millisecond) // a job
fmt.Println(word)
done <- struct{}{} // not blocking
}(word)
}
for range words {
<-done
}
Channels can be unbuffered, and you may use just a signalling channel (e.g. chan struct{}):
words := []string{"foo", "bar", "baz"}
done := make(chan struct{})
for _, word := range words {
go func(word string) {
time.Sleep(100 * time.Millisecond) // a job
fmt.Println(word)
done <- struct{}{} // blocking
}(word)
}
for range words {
<-done
}
You may limit the number of concurrent jobs with buffered channel capacity:
t0 := time.Now()
var wg sync.WaitGroup
words := []string{"foo", "bar", "baz"}
done := make(chan struct{}, 1) // set the number of concurrent job here
for _, word := range words {
wg.Add(1)
go func(word string) {
done <- struct{}{}
time.Sleep(100 * time.Millisecond) // job
fmt.Println(word, time.Since(t0))
<-done
wg.Done()
}(word)
}
wg.Wait()
You may send a message using a channel:
done := make(chan string)
go func() {
for _, word := range []string{"foo", "bar", "baz"} {
done <- word
}
close(done)
}()
for word := range done {
fmt.Println(word)
}
Benchmark:
go test -benchmem -bench . -args -n 0
# BenchmarkEvenWaitgroup-8 1827517 652 ns/op 0 B/op 0 allocs/op
# BenchmarkEvenChannel-8 1000000 2373 ns/op 520 B/op 1 allocs/op
go test -benchmem -bench .
# BenchmarkEvenWaitgroup-8 1770260 678 ns/op 0 B/op 0 allocs/op
# BenchmarkEvenChannel-8 1560124 1249 ns/op 158 B/op 0 allocs/op
Code(main_test.go):
package main
import (
"flag"
"fmt"
"os"
"sync"
"testing"
)
func BenchmarkEvenWaitgroup(b *testing.B) {
evenWaitgroup(b.N)
}
func BenchmarkEvenChannel(b *testing.B) {
evenChannel(b.N)
}
func evenWaitgroup(n int) {
if n%2 == 1 { // make it even:
n++
}
for i := 0; i < n; i++ {
wg.Add(1)
go func(n int) {
select {
case ch <- n: // tx if channel is empty
case i := <-ch: // rx if channel is not empty
// fmt.Println(n, i)
_ = i
}
wg.Done()
}(i)
}
wg.Wait()
}
func evenChannel(n int) {
if n%2 == 1 { // make it even:
n++
}
for i := 0; i < n; i++ {
go func(n int) {
select {
case ch <- n: // tx if channel is empty
case i := <-ch: // rx if channel is not empty
// fmt.Println(n, i)
_ = i
}
done <- struct{}{}
}(i)
}
for i := 0; i < n; i++ {
<-done
}
}
func TestMain(m *testing.M) {
var n int // We use TestMain to set up the done channel.
flag.IntVar(&n, "n", 1_000_000, "chan cap")
flag.Parse()
done = make(chan struct{}, n)
fmt.Println("n=", n)
os.Exit(m.Run())
}
var (
done chan struct{}
ch = make(chan int)
wg sync.WaitGroup
)
It depends on the use case. If you are dispatching one-off jobs to be run in parallel without needing to know the results of each job, then you can use a WaitGroup. But if you need to collect the results from the goroutines then you should use a channel.
Since a channel works both ways, I almost always use a channel.
On another note, as pointed out in the comment your channel example isn't implemented correctly. You would need a separate channel to indicate there are no more jobs to do (one example is here). In your case, since you know the number of words in advance, you could just use one buffered channel and receive a fixed number of times to avoid declaring a close channel.
If you are particularly sticky about using only channels, then it needs to be done differently (if we use your example does, as #Not_a_Golfer points out, it'll produce incorrect results).
One way is to make a channel of type int. In the worker process send a number each time it completes the job (this can be the unique job id too, if you want you can track this in the receiver).
In the receiver main go routine (which will know the exact number of jobs submitted) - do a range loop over a channel, count on till the number of jobs submitted are not done, and break out of the loop when all jobs are completed. This is a good way if you want to track each of the jobs completion (and maybe do something if needed).
Here's the code for your reference. Decrementing totalJobsLeft will be safe as it'll ever be done only in the range loop of the channel!
//This is just an illustration of how to sync completion of multiple jobs using a channel
//A better way many a times might be to use wait groups
package main
import (
"fmt"
"math/rand"
"time"
)
func main() {
comChannel := make(chan int)
words := []string{"foo", "bar", "baz"}
totalJobsLeft := len(words)
//We know how many jobs are being sent
for j, word := range words {
jobId := j + 1
go func(word string, jobId int) {
fmt.Println("Job ID:", jobId, "Word:", word)
//Do some work here, maybe call functions that you need
//For emulating this - Sleep for a random time upto 5 seconds
randInt := rand.Intn(5)
//fmt.Println("Got random number", randInt)
time.Sleep(time.Duration(randInt) * time.Second)
comChannel <- jobId
}(word, jobId)
}
for j := range comChannel {
fmt.Println("Got job ID", j)
totalJobsLeft--
fmt.Println("Total jobs left", totalJobsLeft)
if totalJobsLeft == 0 {
break
}
}
fmt.Println("Closing communication channel. All jobs completed!")
close(comChannel)
}
I often use channels to collect error messages from goroutines that could produce an error. Here is a simple example:
func couldGoWrong() (err error) {
errorChannel := make(chan error, 3)
// start a go routine
go func() (err error) {
defer func() { errorChannel <- err }()
for c := 0; c < 10; c++ {
_, err = fmt.Println(c)
if err != nil {
return
}
}
return
}()
// start another go routine
go func() (err error) {
defer func() { errorChannel <- err }()
for c := 10; c < 100; c++ {
_, err = fmt.Println(c)
if err != nil {
return
}
}
return
}()
// start yet another go routine
go func() (err error) {
defer func() { errorChannel <- err }()
for c := 100; c < 1000; c++ {
_, err = fmt.Println(c)
if err != nil {
return
}
}
return
}()
// synchronize go routines and collect errors here
for c := 0; c < cap(errorChannel); c++ {
err = <-errorChannel
if err != nil {
return
}
}
return
}
Also suggest to use waitgroup but still you want to do it with channel then below i mention a simple use of channel
package main
import (
"fmt"
"time"
)
func main() {
c := make(chan string)
words := []string{"foo", "bar", "baz"}
go printWordrs(words, c)
for j := range c {
fmt.Println(j)
}
}
func printWordrs(words []string, c chan string) {
defer close(c)
for _, word := range words {
time.Sleep(1 * time.Second)
c <- word
}
}