I'm kind of lost here, I was trying to get a goroutine to add to an array and another goroutine to read from it, which I suspect is somewhat close to what I have below but I need to play around with the wait().
However, I am getting the error prog.go:19:14: too many variables in range, line 19 is for _, v := range c { I can't find an answer for that online, what am I doing or not doing here?
package main
import (
"fmt"
//"time"
"sync"
)
func hello(wg *sync.WaitGroup, s []int, c chan int) {
for _, v := range s {
c <- v
}
fmt.Println("Finished adding to channel")
wg.Done()
}
func hello2(wg *sync.WaitGroup, c chan int) {
fmt.Println("Channel",c)
for _, v := range c {
fmt.Println("Received",v)
}
fmt.Println("Finished taking from channel")
wg.Done()
}
func main() {
s := []int{1, 2, 3, 4, 5}
var c = make(chan int, 5)
var wg sync.WaitGroup
wg.Add(1)
go hello(&wg, s, c)
wg.Wait()
wg.Add(1)
go hello2(&wg, c)
wg.Wait()
//time.Sleep(1 * time.Second)
fmt.Println("main function")
}
When you range over a channel, iterations only produce a single value, the values that were sent on the channel. There is no index or key value like in case of slices or maps.
So you must use:
for v := range c {
fmt.Println("Received", v)
}
This is detailed in Spec: For statements:
If the range expression is a channel, at most one iteration variable is permitted, otherwise there may be up to two.
And:
For channels, the iteration values produced are the successive values sent on the channel until the channel is closed. If the channel is nil, the range expression blocks forever.
And also:
Function calls on the left are evaluated once per iteration. For each iteration, iteration values are produced as follows if the respective iteration variables are present:
Range expression 1st value 2nd value
array or slice a [n]E, *[n]E, or []E index i int a[i] E
string s string type index i int see below rune
map m map[K]V key k K m[k] V
channel c chan E, <-chan E element e E
I know you were not intending/wishing to use an iterator for your channel for range loop, but if you do wish to have an iterator it is still possible via defining the iterator yourself outside of the loop and incrementing it each time a value in the channel is read.
var wg sync.WaitGroup
var iterator = 0
for v := range c1 {
wg.Add(1)
iterator += 1
go func(v2, i2 int) {
c2 <- timeConsumingWork(v2)
fmt.Println("item ", i2, " done")
wg.Done()
}(v, iterator)
}
wg.Wait()
close(c2)
}
Here you can see the go func is accepting the channel value(v2) and the iterator value(i2), while incrementing it each time a value is read from the channel.
Additionally I see you used buffered Waitgroups to avoid blocking with the sequential calling of your two methods. Ideal Go code should avoid buffered channels whenever possible and instead call the sending and receiving functions in an interlocked fashion so that they do not need be seperated.
Related
This question already has answers here:
Why does Go handle closures differently in goroutines?
(2 answers)
Cannot assign variable to anonymous func in for loop
(1 answer)
Captured Closure (for Loop Variable) in Go
(1 answer)
go vet - loop variable i captured by func literal
(1 answer)
Closed 8 months ago.
I created the following simple program to test the fan-in-fan-out pattern using channel. What it does is generate a few go routines to calculate the square of a number coming from an input channel and send the square into an output channel. All output channels will then be merged into a single channel to print the square in main.
func calculateSquare(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for num := range in {
fmt.Printf("Receving num %v\n", num)
out <- num * num
fmt.Printf("Sending square %v\n", num * num)
}
fmt.Println("Closing out")
close(out)
}()
return out
}
func fanOut(in <-chan int, workerCount int) []<-chan int {
outs := make([]<-chan int, 0, workerCount)
for i := 0 ; i < workerCount ; i++ {
outs = append(outs, calculateSquare(in))
}
return outs
}
func fanIn(outs []<-chan int) <-chan int {
var wg sync.WaitGroup
merge := make(chan int)
for _, out := range outs {
wg.Add(1)
go func() {
for result := range out {
merge <- result
}
wg.Done()
}()
}
go func() {
wg.Wait()
fmt.Println("Closing merge")
close(merge)
}()
return merge
}
func main() {
in := make(chan int)
go func() {
for i := 0 ; i < 4 ; i++ {
fmt.Printf("Sending num %v\n", i)
in <- i
}
close(in)
}()
outs := fanOut(in, 5)
merge := fanIn(outs)
for num := range merge {
fmt.Printf("Final square %v\n", num)
}
}
In the main function, I'm sending in 4 numbers 0 -> 3 into the input channel and I expect to see 4 square printed in the console. However, when I ran the program, even though the output fluctuates a bit but I never ever see 4 square numbers printed in the console.
Below is a sample output I'm seeing.
Sending num 0
Sending num 1
Sending num 2
Sending num 3
Closing out
Receving num 0
Receving num 1
Receving num 2
Sending square 4
Closing out
Receving num 3
Final square 4
Closing merge
I'd be very grateful if someone could explain to me why Receving num 1 was printed but Sending square 1 is never coming. In addition, if Sending square 1 is not printed, how did the output channel get closed. I'm only seeing 2 Closing out, yet, the wait group where I was merging the result ended its Wait().
I must have done something wrong somewhere.
To fix:
for _, out := range outs {
wg.Add(1)
out := out // <- add this
Why?
https://golang.org/doc/effective_go is an excellent resource and covers the exact closure bug (that #JimB mentioned) towards the end of the channels section:
It may seem odd to write
req := req
but it's legal and idiomatic in Go to do this. You get a
fresh version of the variable with the same name, deliberately
shadowing the loop variable locally but unique to each goroutine.
your issue is in the code below, for loop in fanIn function.
for _, out := range outs {
wg.Add(1)
go func() {
for result := range out {
merge <- result
}
wg.Done()
}()
}
Reason for this is you using out iterator variable in gofunc, when gofunc going to use it, loop is gone to it's end.
This is describe in go/wiki/CommonMistakes under the sub topic Using goroutines on loop iterator variables
For more example - read this
corrected loop should be as below,
for _, out := range outs {
wg.Add(1)
go func(c <- chan int) {
for result := range c {
merge <- result
}
wg.Done()
}(out)
}
I have a loop throwing multiple go routines, they call a function that makes a http get petition and calculate and object.
I want to recover the result of all those routines.
I tried using channels, but hey are empty, even if I force wait for all the routines to be done.
This is the code that starts the routines:
func main() {
pairs := getPairs() //Returns an array of strings
c := make(chan result)
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
fmt.Println(len(c))
time.Sleep(5000 * time.Millisecond)
fmt.Println(len(c))
}
And at the end of getScore() I do this, c being the name of the channel in the function and res the result of the function
c <- res
The length of the channel is 0 in both prints.
What's the best way to get the result of the functions?
A channel is a synchronization prototype against a shared memory (in simple point of view). A buffered channel has a length but not a regular channel. Buffered channel is useful in little bit cases but not as a general approaches.
The simplest way to Just add a loop by range of pair or len of pairs:
// start processing
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
// getting a result
for i:=0; i<len(pairs); i ++ {
result := <-c
// process a result value
}
Or another way is collecting result in another grouting:
// result and sync variable
var (
wait sync.WaitGroup
result int32
)
// start processing
for _, product := range pairs {
wait.Add(1)
go getScore(product.Symbol, 1, c)
go func() {
defer wait.Done()
// simple accumulate or maybe more complicated actions
atomic.AddInt32(&result, <-c)
}()
}
// wait finishing
wait.Wait()
c := make(chan result)
Creates an unbuffered channel. Therefore send statements, such as
c <- res
cannot proceed until another goroutine is attempting a receive operation.
In other words, execute the number of receive operations in your main goroutine matching the number of sends that will be attempted from other goroutines. Like this:
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
for x := 0; x < len(pairs); x++ {
fmt.Println(<-c)
}
See the Go Tour section on channels, and the Effective Go section on channels for more information.
Below is a struct of type Stuff. It has three ints. A Number, its Double and its Power. Let's pretend that calculating the double and power of a given list of ints is an expensive computation.
type Stuff struct {
Number int
Double int
Power int
}
func main() {
nums := []int{2, 3, 4} // given numbers
stuff := []Stuff{} // struct of stuff with transformed ints
double := make(chan int)
power := make(chan int)
for _, i := range nums {
go doubleNumber(i, double)
go powerNumber(i, power)
}
// How do I get the values back in the right order?
fmt.Println(stuff)
}
func doubleNumber(i int, c chan int) {
c <- i + i
}
func powerNumber(i int, c chan int) {
c <- i * i
}
The result of fmt.Println(stuff) should be the same as if stuff was initialized like:
stuff := []Stuff{
{Number: 2, Double: 4, Power: 4}
{Number: 3, Double: 6, Power: 9}
{Number: 4, Double: 8, Power: 16}
}
I know I can use <- double and <- power to collect values from the channels, but I don't know what double / powers belong to what numbers.
Goroutines run concurrently, independently, so without explicit synchronization you can't predict execution and completion order. So as it is, you can't pair returned numbers with the input numbers.
You can either return more data (e.g. the input number and the output, wrapped in a struct for example), or pass pointers to the worker functions (launched as new goroutines), e.g. *Stuff and have the goroutines fill the calculated data in the Stuff itself.
Returning more data
I will use a channel type chan Pair where Pair is:
type Pair struct{ Number, Result int }
So calculation will look like this:
func doubleNumber(i int, c chan Pair) { c <- Pair{i, i + i} }
func powerNumber(i int, c chan Pair) { c <- Pair{i, i * i} }
And I will use a map[int]*Stuff because collectable data comes from multiple channels (double and power), and I want to find the appropriate Stuff easily and fast (pointer is required so I can also modify it "in the map").
So the main function:
nums := []int{2, 3, 4} // given numbers
stuffs := map[int]*Stuff{}
double := make(chan Pair)
power := make(chan Pair)
for _, i := range nums {
go doubleNumber(i, double)
go powerNumber(i, power)
}
// How do I get the values back in the right order?
for i := 0; i < len(nums)*2; i++ {
getStuff := func(number int) *Stuff {
s := stuffs[number]
if s == nil {
s = &Stuff{Number: number}
stuffs[number] = s
}
return s
}
select {
case p := <-double:
getStuff(p.Number).Double = p.Result
case p := <-power:
getStuff(p.Number).Power = p.Result
}
}
for _, v := range nums {
fmt.Printf("%+v\n", stuffs[v])
}
Output (try it on the Go Playground):
&{Number:2 Double:4 Power:4}
&{Number:3 Double:6 Power:9}
&{Number:4 Double:8 Power:16}
Using pointers
Since now we're passing *Stuff values, we can "pre-fill" the input number in the Stuff itself.
But care must be taken, you can only read/write values with proper synchronization. Easiest is to wait for all "worker" goroutines to finish their jobs.
var wg = &sync.WaitGroup{}
func main() {
nums := []int{2, 3, 4} // given numbers
stuffs := make([]Stuff, len(nums))
for i, n := range nums {
stuffs[i].Number = n
wg.Add(2)
go doubleNumber(&stuffs[i])
go powerNumber(&stuffs[i])
}
wg.Wait()
fmt.Printf("%+v", stuffs)
}
func doubleNumber(s *Stuff) {
defer wg.Done()
s.Double = s.Number + s.Number
}
func powerNumber(s *Stuff) {
defer wg.Done()
s.Power = s.Number * s.Number
}
Output (try it on the Go Playground):
[{Number:2 Double:4 Power:4} {Number:3 Double:6 Power:9} {Number:4 Double:8 Power:16}]
Writing different slice elements concurrently
Also note that since you can write different array or slice elements concurrently (for details see Can I concurrently write different slice elements), you can write the results directly in a slice without channels. See Refactor code to use a single channel in an idiomatic way how this can be done.
Personally, I would use a chan Stuff to pass the results back on, then spin up goroutines computing a full Stuff and pass it back. If you need the various part of a single Stuff computed concurrently, you can spawn goroutines from each goroutine, using dedicated channels. Once you've collected all the results, you can then (optionally) sort the slice with the accumulated values.
Example of what I mean below (you could, in principle, use a sync.WaitGroup to coordinate things, but if the input count is known, you don't strictly speaking need it).
type Stuff struct {
number int64
double int64
square int64
}
// Compute a Stuff with individual computations in-line, send it out
func computeStuff(n int64, out chan<- Stuff) {
rv := Stuff{number: n}
rv.double = n * 2
rv.square = n * n
out <- rv
}
// Compute a Stuff with individual computations concurrent
func computeStuffConcurrent(n int64, out chan<- Stuff) {
rv := Stuff{number: n}
dc := make(chan int64)
sc := make(chan int64)
defer close(dc)
defer close(sc)
go double(n, dc)
go square(n, sc)
rv.double = <-dc
rv.square = <-sc
out <- rv
}
func double(n int64, result chan<- int) {
result <- n * 2
}
func square(n int64, result chan<- int) {
result <- n * n
}
func main() {
inputs := []int64{1, 2, 3}
results := []Stuff{}
resultChannel := make(chan Stuff)
for _, input := range inputs {
go computeStuff(input, resultChannel)
// Or the concurrent version, if the extra performance is needed
}
for c := 0; c < len(inputs); c++ {
results = append(results, <- resultChannel)
}
// We now have all results, sort them if you need them sorted
}
I'm reading/working through Go Concurrency Patterns: Pipelines and cancellation, but i'm having trouble understanding the Stopping short section. We have the following functions:
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n
}
close(out)
}()
return out
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func merge(cs ...<-chan int) <-chan int {
var wg sync.WaitGroup
out := make(chan int, 1) // enough space for the unread inputs
// Start an output goroutine for each input channel in cs. output
// copies values from c to out until c is closed, then calls wg.Done.
output := func(c <-chan int) {
for n := range c {
out <- n
}
wg.Done()
}
wg.Add(len(cs))
for _, c := range cs {
go output(c)
}
// Start a goroutine to close out once all the output goroutines are
// done. This must start after the wg.Add call.
go func() {
wg.Wait()
close(out)
}()
return out
}
func main() {
in := gen(2, 3)
// Distribute the sq work across two goroutines that both read from in.
c1 := sq(in)
c2 := sq(in)
// Consume the first value from output.
out := merge(c1, c2)
fmt.Println(<-out) // 4 or 9
return
// Apparently if we had not set the merge out buffer size to 1
// then we would have a hanging go routine.
}
Now, if you notice line 2 in merge, it says we make the out chan with buffer size 1, because this is enough space for the unread inputs. However, I'm almost positive that we should allocate a chan with buffer size 2. In accordance with this code sample:
c := make(chan int, 2) // buffer size 2
c <- 1 // succeeds immediately
c <- 2 // succeeds immediately
c <- 3 // blocks until another goroutine does <-c and receives 1
Because this section implies that a chan of buffer size 3 would not block. Can anyone please clarify/assist my understanding?
The program sends two values to the channel out and reads one value from the channel out. One of the values is not received.
If the channel is unbuffered (capacity 0), then one of the sending goroutines will block until the program exits. This is a leak.
If the channel is created with a capacity of 1, then both goroutines can send to the channel and exit. The first value sent to the channel is received by main. The second value remains in the channel.
If the main function does not receive a value from the channel out, then a channel of capacity 2 is required to prevent the goroutines from blocking indefinitely.
In Go, is there a more idiomatic way to add all of the elements of an array/slice into a channel than the following?
ch := make(chan string)
values := []string{"lol", "cat", "lolcat"}
go func() {
for _, v := range values {
ch <- v
}
}()
I was looking for something like ch <- values... but that is rejected by the compiler.
Until iterators will come along, yes, the code you wrote is as idiomatic as it gets. I have it packaged for reuse as something like this in codebases I work on:
// ToChan returns a channel containing all elements in the slice s.
// The channel is closed when all elements are consumed from the channel.
func ToChan[T any](s []T) <-chan T {
ch := make(chan T, len(s))
for _, e := range s {
ch <- e
}
close(ch)
return ch
}
https://go.dev/play/p/c5v4df_M1IG
A for/range loop is the idiomatic way to send all of the elements of a slice to a channel:
for _, v := range values {
ch <- v
}
It is not necessary to run the for loop in a goroutine, as shown in the question.
You can declare a chan of string arrays, unless you absolutely want to keep a chan of strings :
package main
import "fmt"
func main() {
ch := make(chan []string)
values := []string{"lol", "cat", "lolcat"}
go func() {
ch <- values
}()
fmt.Printf("Values : %+v\n", <-ch)
}