Related
I'm kind of lost here, I was trying to get a goroutine to add to an array and another goroutine to read from it, which I suspect is somewhat close to what I have below but I need to play around with the wait().
However, I am getting the error prog.go:19:14: too many variables in range, line 19 is for _, v := range c { I can't find an answer for that online, what am I doing or not doing here?
package main
import (
"fmt"
//"time"
"sync"
)
func hello(wg *sync.WaitGroup, s []int, c chan int) {
for _, v := range s {
c <- v
}
fmt.Println("Finished adding to channel")
wg.Done()
}
func hello2(wg *sync.WaitGroup, c chan int) {
fmt.Println("Channel",c)
for _, v := range c {
fmt.Println("Received",v)
}
fmt.Println("Finished taking from channel")
wg.Done()
}
func main() {
s := []int{1, 2, 3, 4, 5}
var c = make(chan int, 5)
var wg sync.WaitGroup
wg.Add(1)
go hello(&wg, s, c)
wg.Wait()
wg.Add(1)
go hello2(&wg, c)
wg.Wait()
//time.Sleep(1 * time.Second)
fmt.Println("main function")
}
When you range over a channel, iterations only produce a single value, the values that were sent on the channel. There is no index or key value like in case of slices or maps.
So you must use:
for v := range c {
fmt.Println("Received", v)
}
This is detailed in Spec: For statements:
If the range expression is a channel, at most one iteration variable is permitted, otherwise there may be up to two.
And:
For channels, the iteration values produced are the successive values sent on the channel until the channel is closed. If the channel is nil, the range expression blocks forever.
And also:
Function calls on the left are evaluated once per iteration. For each iteration, iteration values are produced as follows if the respective iteration variables are present:
Range expression 1st value 2nd value
array or slice a [n]E, *[n]E, or []E index i int a[i] E
string s string type index i int see below rune
map m map[K]V key k K m[k] V
channel c chan E, <-chan E element e E
I know you were not intending/wishing to use an iterator for your channel for range loop, but if you do wish to have an iterator it is still possible via defining the iterator yourself outside of the loop and incrementing it each time a value in the channel is read.
var wg sync.WaitGroup
var iterator = 0
for v := range c1 {
wg.Add(1)
iterator += 1
go func(v2, i2 int) {
c2 <- timeConsumingWork(v2)
fmt.Println("item ", i2, " done")
wg.Done()
}(v, iterator)
}
wg.Wait()
close(c2)
}
Here you can see the go func is accepting the channel value(v2) and the iterator value(i2), while incrementing it each time a value is read from the channel.
Additionally I see you used buffered Waitgroups to avoid blocking with the sequential calling of your two methods. Ideal Go code should avoid buffered channels whenever possible and instead call the sending and receiving functions in an interlocked fashion so that they do not need be seperated.
to start an endless loop of executing two goroutines, I can use the code below:
after receiving the msg it will start a new goroutine and go on for ever.
c1 := make(chan string)
c2 := make(chan string)
go DoStuff(c1, 5)
go DoStuff(c2, 2)
for ; true; {
select {
case msg1 := <-c1:
fmt.Println("received ", msg1)
go DoStuff(c1, 1)
case msg2 := <-c2:
fmt.Println("received ", msg2)
go DoStuff(c2, 9)
}
}
I would now like to have the same behavior for N goroutines, but how will the select statement look in that case?
This is the code bit I have started with, but I am confused how to code the select statement
numChans := 2
//I keep the channels in this slice, and want to "loop" over them in the select statemnt
var chans = [] chan string{}
for i:=0;i<numChans;i++{
tmp := make(chan string);
chans = append(chans, tmp);
go DoStuff(tmp, i + 1)
//How shall the select statment be coded for this case?
for ; true; {
select {
case msg1 := <-c1:
fmt.Println("received ", msg1)
go DoStuff(c1, 1)
case msg2 := <-c2:
fmt.Println("received ", msg2)
go DoStuff(c2, 9)
}
}
You can do this using the Select function from the reflect package:
func Select(cases []SelectCase) (chosen int, recv Value, recvOK bool)
Select executes a select operation described by the list of cases. Like
the Go select statement, it blocks until at least one of the cases can
proceed, makes a uniform pseudo-random choice, and then executes that
case. It returns the index of the chosen case and, if that case was a
receive operation, the value received and a boolean indicating whether
the value corresponds to a send on the channel (as opposed to a zero
value received because the channel is closed).
You pass in an array of SelectCase structs that identify the channel to select on, the direction of the operation, and a value to send in the case of a send operation.
So you could do something like this:
cases := make([]reflect.SelectCase, len(chans))
for i, ch := range chans {
cases[i] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ch)}
}
chosen, value, ok := reflect.Select(cases)
// ok will be true if the channel has not been closed.
ch := chans[chosen]
msg := value.String()
You can experiment with a more fleshed out example here: http://play.golang.org/p/8zwvSk4kjx
You can accomplish this by wrapping each channel in a goroutine which "forwards" messages to a shared "aggregate" channel. For example:
agg := make(chan string)
for _, ch := range chans {
go func(c chan string) {
for msg := range c {
agg <- msg
}
}(ch)
}
select {
case msg <- agg:
fmt.Println("received ", msg)
}
If you need to know which channel the message originated from, you could wrap it in a struct with any extra information before forwarding it to the aggregate channel.
In my (limited) testing, this method greatly out performs using the reflect package:
$ go test dynamic_select_test.go -test.bench=.
...
BenchmarkReflectSelect 1 5265109013 ns/op
BenchmarkGoSelect 20 81911344 ns/op
ok command-line-arguments 9.463s
Benchmark code here
To expand on some comments on previous answers and to provide a clearer comparison here is an example of both approaches presented so far given the same input, a slice of channels to read from and a function to call for each value which also need to know which channel the value came from.
There are three main differences between the approaches:
Complexity. Although it may partially be a reader preference I find the channel approach more idiomatic, straight-forward, and readable.
Performance. On my Xeon amd64 system the goroutines+channels out performs the reflect solution by about two orders of magnitude (in general reflection in Go is often slower and should only be used when absolutely required). Of course, if there is any significant delay in either the function processing the results or in the writing of values to the input channels this performance difference can easily become insignificant.
Blocking/buffering semantics. The importantance of this depends on the use case. Most often it either won't matter or the slight extra buffering in the goroutine merging solution may be helpful for throughput. However, if it is desirable to have the semantics that only a single writer is unblocked and it's value fully handled before any other writer is unblocked, then that can only be achieved with the reflect solution.
Note, both approaches can be simplified if either the "id" of the sending channel isn't required or if the source channels will never be closed.
Goroutine merging channel:
// Process1 calls `fn` for each value received from any of the `chans`
// channels. The arguments to `fn` are the index of the channel the
// value came from and the string value. Process1 returns once all the
// channels are closed.
func Process1(chans []<-chan string, fn func(int, string)) {
// Setup
type item struct {
int // index of which channel this came from
string // the actual string item
}
merged := make(chan item)
var wg sync.WaitGroup
wg.Add(len(chans))
for i, c := range chans {
go func(i int, c <-chan string) {
// Reads and buffers a single item from `c` before
// we even know if we can write to `merged`.
//
// Go doesn't provide a way to do something like:
// merged <- (<-c)
// atomically, where we delay the read from `c`
// until we can write to `merged`. The read from
// `c` will always happen first (blocking as
// required) and then we block on `merged` (with
// either the above or the below syntax making
// no difference).
for s := range c {
merged <- item{i, s}
}
// If/when this input channel is closed we just stop
// writing to the merged channel and via the WaitGroup
// let it be known there is one fewer channel active.
wg.Done()
}(i, c)
}
// One extra goroutine to watch for all the merging goroutines to
// be finished and then close the merged channel.
go func() {
wg.Wait()
close(merged)
}()
// "select-like" loop
for i := range merged {
// Process each value
fn(i.int, i.string)
}
}
Reflection select:
// Process2 is identical to Process1 except that it uses the reflect
// package to select and read from the input channels which guarantees
// there is only one value "in-flight" (i.e. when `fn` is called only
// a single send on a single channel will have succeeded, the rest will
// be blocked). It is approximately two orders of magnitude slower than
// Process1 (which is still insignificant if their is a significant
// delay between incoming values or if `fn` runs for a significant
// time).
func Process2(chans []<-chan string, fn func(int, string)) {
// Setup
cases := make([]reflect.SelectCase, len(chans))
// `ids` maps the index within cases to the original `chans` index.
ids := make([]int, len(chans))
for i, c := range chans {
cases[i] = reflect.SelectCase{
Dir: reflect.SelectRecv,
Chan: reflect.ValueOf(c),
}
ids[i] = i
}
// Select loop
for len(cases) > 0 {
// A difference here from the merging goroutines is
// that `v` is the only value "in-flight" that any of
// the workers have sent. All other workers are blocked
// trying to send the single value they have calculated
// where-as the goroutine version reads/buffers a single
// extra value from each worker.
i, v, ok := reflect.Select(cases)
if !ok {
// Channel cases[i] has been closed, remove it
// from our slice of cases and update our ids
// mapping as well.
cases = append(cases[:i], cases[i+1:]...)
ids = append(ids[:i], ids[i+1:]...)
continue
}
// Process each value
fn(ids[i], v.String())
}
}
[Full code on the Go playground.]
We actually made some research about this subject and found the best solution. We used reflect.Select for a while and it is a great solution for the problem. It is much lighter than a goroutine per channel and simple to operate. But unfortunately, it doesn't really support a massive amount of channels which is our case so we found something interesting and wrote a blog post about it: https://cyolo.io/blog/how-we-enabled-dynamic-channel-selection-at-scale-in-go/
I'll summarize what is written there:
We statically created batches of select..case statements for every result of the power of two of exponent up to 32 along with a function that routes to the different cases and aggregates the results through an aggregate channel.
An example of such a batch:
func select4(ctx context.Context, chanz []chan interface{}, res chan *r, r *r, i int) {
select {
case r.v, r.ok = <-chanz[0]:
r.i = i + 0
res <- r
case r.v, r.ok = <-chanz[1]:
r.i = i + 1
res <- r
case r.v, r.ok = <-chanz[2]:
r.i = i + 2
res <- r
case r.v, r.ok = <-chanz[3]:
r.i = i + 3
res <- r
case <-ctx.Done():
break
}
}
And the logic of aggregating the first result from any number of channels using these kinds of select..case batches:
for i < len(channels) {
l = len(channels) - i
switch {
case l > 31 && maxBatchSize >= 32:
go select32(ctx, channels[i:i+32], agg, rPool.Get().(*r), i)
i += 32
case l > 15 && maxBatchSize >= 16:
go select16(ctx, channels[i:i+16], agg, rPool.Get().(*r), i)
i += 16
case l > 7 && maxBatchSize >= 8:
go select8(ctx, channels[i:i+8], agg, rPool.Get().(*r), i)
i += 8
case l > 3 && maxBatchSize >= 4:
go select4(ctx, channels[i:i+4], agg, rPool.Get().(*r), i)
i += 4
case l > 1 && maxBatchSize >= 2:
go select2(ctx, channels[i:i+2], agg, rPool.Get().(*r), i)
i += 2
case l > 0:
go select1(ctx, channels[i], agg, rPool.Get().(*r), i)
i += 1
}
}
Possibly simpler option:
Instead of having an array of channels, why not pass just one channel as a parameter to the functions being run on separate goroutines, and then listen to the channel in a consumer goroutine?
This allows you to select on just one channel in your listener, making for a simple select, and avoiding creation of new goroutines to aggregate messages from multiple channels?
Based on the answer of James Henstridge,
I made this generic (go >=1.18) Select function that takes a context and a slice of channels and returns the selected one:
func Select[T any](ctx context.Context, chs []chan T) (int, T, error) {
var zeroT T
cases := make([]reflect.SelectCase, len(chs)+1)
for i, ch := range chs {
cases[i] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ch)}
}
cases[len(chs)] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ctx.Done())}
// ok will be true if the channel has not been closed.
chosen, value, ok := reflect.Select(cases)
if !ok {
if ctx.Err() != nil {
return -1, zeroT, ctx.Err()
}
return chosen, zeroT, errors.New("channel closed")
}
if ret, ok := value.Interface().(T); ok {
return chosen, ret, nil
}
return chosen, zeroT, errors.New("failed to cast value")
}
Here is an example on how to use it:
func TestSelect(t *testing.T) {
c1 := make(chan int)
c2 := make(chan int)
c3 := make(chan int)
chs := []chan int{c1, c2, c3}
go func() {
time.Sleep(time.Second)
//close(c2)
c2 <- 42
}()
ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
chosen, val, err := Select(ctx, chs)
assert.Equal(t, 1, chosen)
assert.Equal(t, 42, val)
assert.NoError(t, err)
}
Why this approach wouldn't work assuming that somebody is sending events?
func main() {
numChans := 2
var chans = []chan string{}
for i := 0; i < numChans; i++ {
tmp := make(chan string)
chans = append(chans, tmp)
}
for true {
for i, c := range chans {
select {
case x = <-c:
fmt.Printf("received %d \n", i)
go DoShit(x, i)
default: continue
}
}
}
}
to start an endless loop of executing two goroutines, I can use the code below:
after receiving the msg it will start a new goroutine and go on for ever.
c1 := make(chan string)
c2 := make(chan string)
go DoStuff(c1, 5)
go DoStuff(c2, 2)
for ; true; {
select {
case msg1 := <-c1:
fmt.Println("received ", msg1)
go DoStuff(c1, 1)
case msg2 := <-c2:
fmt.Println("received ", msg2)
go DoStuff(c2, 9)
}
}
I would now like to have the same behavior for N goroutines, but how will the select statement look in that case?
This is the code bit I have started with, but I am confused how to code the select statement
numChans := 2
//I keep the channels in this slice, and want to "loop" over them in the select statemnt
var chans = [] chan string{}
for i:=0;i<numChans;i++{
tmp := make(chan string);
chans = append(chans, tmp);
go DoStuff(tmp, i + 1)
//How shall the select statment be coded for this case?
for ; true; {
select {
case msg1 := <-c1:
fmt.Println("received ", msg1)
go DoStuff(c1, 1)
case msg2 := <-c2:
fmt.Println("received ", msg2)
go DoStuff(c2, 9)
}
}
You can do this using the Select function from the reflect package:
func Select(cases []SelectCase) (chosen int, recv Value, recvOK bool)
Select executes a select operation described by the list of cases. Like
the Go select statement, it blocks until at least one of the cases can
proceed, makes a uniform pseudo-random choice, and then executes that
case. It returns the index of the chosen case and, if that case was a
receive operation, the value received and a boolean indicating whether
the value corresponds to a send on the channel (as opposed to a zero
value received because the channel is closed).
You pass in an array of SelectCase structs that identify the channel to select on, the direction of the operation, and a value to send in the case of a send operation.
So you could do something like this:
cases := make([]reflect.SelectCase, len(chans))
for i, ch := range chans {
cases[i] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ch)}
}
chosen, value, ok := reflect.Select(cases)
// ok will be true if the channel has not been closed.
ch := chans[chosen]
msg := value.String()
You can experiment with a more fleshed out example here: http://play.golang.org/p/8zwvSk4kjx
You can accomplish this by wrapping each channel in a goroutine which "forwards" messages to a shared "aggregate" channel. For example:
agg := make(chan string)
for _, ch := range chans {
go func(c chan string) {
for msg := range c {
agg <- msg
}
}(ch)
}
select {
case msg <- agg:
fmt.Println("received ", msg)
}
If you need to know which channel the message originated from, you could wrap it in a struct with any extra information before forwarding it to the aggregate channel.
In my (limited) testing, this method greatly out performs using the reflect package:
$ go test dynamic_select_test.go -test.bench=.
...
BenchmarkReflectSelect 1 5265109013 ns/op
BenchmarkGoSelect 20 81911344 ns/op
ok command-line-arguments 9.463s
Benchmark code here
To expand on some comments on previous answers and to provide a clearer comparison here is an example of both approaches presented so far given the same input, a slice of channels to read from and a function to call for each value which also need to know which channel the value came from.
There are three main differences between the approaches:
Complexity. Although it may partially be a reader preference I find the channel approach more idiomatic, straight-forward, and readable.
Performance. On my Xeon amd64 system the goroutines+channels out performs the reflect solution by about two orders of magnitude (in general reflection in Go is often slower and should only be used when absolutely required). Of course, if there is any significant delay in either the function processing the results or in the writing of values to the input channels this performance difference can easily become insignificant.
Blocking/buffering semantics. The importantance of this depends on the use case. Most often it either won't matter or the slight extra buffering in the goroutine merging solution may be helpful for throughput. However, if it is desirable to have the semantics that only a single writer is unblocked and it's value fully handled before any other writer is unblocked, then that can only be achieved with the reflect solution.
Note, both approaches can be simplified if either the "id" of the sending channel isn't required or if the source channels will never be closed.
Goroutine merging channel:
// Process1 calls `fn` for each value received from any of the `chans`
// channels. The arguments to `fn` are the index of the channel the
// value came from and the string value. Process1 returns once all the
// channels are closed.
func Process1(chans []<-chan string, fn func(int, string)) {
// Setup
type item struct {
int // index of which channel this came from
string // the actual string item
}
merged := make(chan item)
var wg sync.WaitGroup
wg.Add(len(chans))
for i, c := range chans {
go func(i int, c <-chan string) {
// Reads and buffers a single item from `c` before
// we even know if we can write to `merged`.
//
// Go doesn't provide a way to do something like:
// merged <- (<-c)
// atomically, where we delay the read from `c`
// until we can write to `merged`. The read from
// `c` will always happen first (blocking as
// required) and then we block on `merged` (with
// either the above or the below syntax making
// no difference).
for s := range c {
merged <- item{i, s}
}
// If/when this input channel is closed we just stop
// writing to the merged channel and via the WaitGroup
// let it be known there is one fewer channel active.
wg.Done()
}(i, c)
}
// One extra goroutine to watch for all the merging goroutines to
// be finished and then close the merged channel.
go func() {
wg.Wait()
close(merged)
}()
// "select-like" loop
for i := range merged {
// Process each value
fn(i.int, i.string)
}
}
Reflection select:
// Process2 is identical to Process1 except that it uses the reflect
// package to select and read from the input channels which guarantees
// there is only one value "in-flight" (i.e. when `fn` is called only
// a single send on a single channel will have succeeded, the rest will
// be blocked). It is approximately two orders of magnitude slower than
// Process1 (which is still insignificant if their is a significant
// delay between incoming values or if `fn` runs for a significant
// time).
func Process2(chans []<-chan string, fn func(int, string)) {
// Setup
cases := make([]reflect.SelectCase, len(chans))
// `ids` maps the index within cases to the original `chans` index.
ids := make([]int, len(chans))
for i, c := range chans {
cases[i] = reflect.SelectCase{
Dir: reflect.SelectRecv,
Chan: reflect.ValueOf(c),
}
ids[i] = i
}
// Select loop
for len(cases) > 0 {
// A difference here from the merging goroutines is
// that `v` is the only value "in-flight" that any of
// the workers have sent. All other workers are blocked
// trying to send the single value they have calculated
// where-as the goroutine version reads/buffers a single
// extra value from each worker.
i, v, ok := reflect.Select(cases)
if !ok {
// Channel cases[i] has been closed, remove it
// from our slice of cases and update our ids
// mapping as well.
cases = append(cases[:i], cases[i+1:]...)
ids = append(ids[:i], ids[i+1:]...)
continue
}
// Process each value
fn(ids[i], v.String())
}
}
[Full code on the Go playground.]
We actually made some research about this subject and found the best solution. We used reflect.Select for a while and it is a great solution for the problem. It is much lighter than a goroutine per channel and simple to operate. But unfortunately, it doesn't really support a massive amount of channels which is our case so we found something interesting and wrote a blog post about it: https://cyolo.io/blog/how-we-enabled-dynamic-channel-selection-at-scale-in-go/
I'll summarize what is written there:
We statically created batches of select..case statements for every result of the power of two of exponent up to 32 along with a function that routes to the different cases and aggregates the results through an aggregate channel.
An example of such a batch:
func select4(ctx context.Context, chanz []chan interface{}, res chan *r, r *r, i int) {
select {
case r.v, r.ok = <-chanz[0]:
r.i = i + 0
res <- r
case r.v, r.ok = <-chanz[1]:
r.i = i + 1
res <- r
case r.v, r.ok = <-chanz[2]:
r.i = i + 2
res <- r
case r.v, r.ok = <-chanz[3]:
r.i = i + 3
res <- r
case <-ctx.Done():
break
}
}
And the logic of aggregating the first result from any number of channels using these kinds of select..case batches:
for i < len(channels) {
l = len(channels) - i
switch {
case l > 31 && maxBatchSize >= 32:
go select32(ctx, channels[i:i+32], agg, rPool.Get().(*r), i)
i += 32
case l > 15 && maxBatchSize >= 16:
go select16(ctx, channels[i:i+16], agg, rPool.Get().(*r), i)
i += 16
case l > 7 && maxBatchSize >= 8:
go select8(ctx, channels[i:i+8], agg, rPool.Get().(*r), i)
i += 8
case l > 3 && maxBatchSize >= 4:
go select4(ctx, channels[i:i+4], agg, rPool.Get().(*r), i)
i += 4
case l > 1 && maxBatchSize >= 2:
go select2(ctx, channels[i:i+2], agg, rPool.Get().(*r), i)
i += 2
case l > 0:
go select1(ctx, channels[i], agg, rPool.Get().(*r), i)
i += 1
}
}
Possibly simpler option:
Instead of having an array of channels, why not pass just one channel as a parameter to the functions being run on separate goroutines, and then listen to the channel in a consumer goroutine?
This allows you to select on just one channel in your listener, making for a simple select, and avoiding creation of new goroutines to aggregate messages from multiple channels?
Based on the answer of James Henstridge,
I made this generic (go >=1.18) Select function that takes a context and a slice of channels and returns the selected one:
func Select[T any](ctx context.Context, chs []chan T) (int, T, error) {
var zeroT T
cases := make([]reflect.SelectCase, len(chs)+1)
for i, ch := range chs {
cases[i] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ch)}
}
cases[len(chs)] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ctx.Done())}
// ok will be true if the channel has not been closed.
chosen, value, ok := reflect.Select(cases)
if !ok {
if ctx.Err() != nil {
return -1, zeroT, ctx.Err()
}
return chosen, zeroT, errors.New("channel closed")
}
if ret, ok := value.Interface().(T); ok {
return chosen, ret, nil
}
return chosen, zeroT, errors.New("failed to cast value")
}
Here is an example on how to use it:
func TestSelect(t *testing.T) {
c1 := make(chan int)
c2 := make(chan int)
c3 := make(chan int)
chs := []chan int{c1, c2, c3}
go func() {
time.Sleep(time.Second)
//close(c2)
c2 <- 42
}()
ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
chosen, val, err := Select(ctx, chs)
assert.Equal(t, 1, chosen)
assert.Equal(t, 42, val)
assert.NoError(t, err)
}
Why this approach wouldn't work assuming that somebody is sending events?
func main() {
numChans := 2
var chans = []chan string{}
for i := 0; i < numChans; i++ {
tmp := make(chan string)
chans = append(chans, tmp)
}
for true {
for i, c := range chans {
select {
case x = <-c:
fmt.Printf("received %d \n", i)
go DoShit(x, i)
default: continue
}
}
}
}
to start an endless loop of executing two goroutines, I can use the code below:
after receiving the msg it will start a new goroutine and go on for ever.
c1 := make(chan string)
c2 := make(chan string)
go DoStuff(c1, 5)
go DoStuff(c2, 2)
for ; true; {
select {
case msg1 := <-c1:
fmt.Println("received ", msg1)
go DoStuff(c1, 1)
case msg2 := <-c2:
fmt.Println("received ", msg2)
go DoStuff(c2, 9)
}
}
I would now like to have the same behavior for N goroutines, but how will the select statement look in that case?
This is the code bit I have started with, but I am confused how to code the select statement
numChans := 2
//I keep the channels in this slice, and want to "loop" over them in the select statemnt
var chans = [] chan string{}
for i:=0;i<numChans;i++{
tmp := make(chan string);
chans = append(chans, tmp);
go DoStuff(tmp, i + 1)
//How shall the select statment be coded for this case?
for ; true; {
select {
case msg1 := <-c1:
fmt.Println("received ", msg1)
go DoStuff(c1, 1)
case msg2 := <-c2:
fmt.Println("received ", msg2)
go DoStuff(c2, 9)
}
}
You can do this using the Select function from the reflect package:
func Select(cases []SelectCase) (chosen int, recv Value, recvOK bool)
Select executes a select operation described by the list of cases. Like
the Go select statement, it blocks until at least one of the cases can
proceed, makes a uniform pseudo-random choice, and then executes that
case. It returns the index of the chosen case and, if that case was a
receive operation, the value received and a boolean indicating whether
the value corresponds to a send on the channel (as opposed to a zero
value received because the channel is closed).
You pass in an array of SelectCase structs that identify the channel to select on, the direction of the operation, and a value to send in the case of a send operation.
So you could do something like this:
cases := make([]reflect.SelectCase, len(chans))
for i, ch := range chans {
cases[i] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ch)}
}
chosen, value, ok := reflect.Select(cases)
// ok will be true if the channel has not been closed.
ch := chans[chosen]
msg := value.String()
You can experiment with a more fleshed out example here: http://play.golang.org/p/8zwvSk4kjx
You can accomplish this by wrapping each channel in a goroutine which "forwards" messages to a shared "aggregate" channel. For example:
agg := make(chan string)
for _, ch := range chans {
go func(c chan string) {
for msg := range c {
agg <- msg
}
}(ch)
}
select {
case msg <- agg:
fmt.Println("received ", msg)
}
If you need to know which channel the message originated from, you could wrap it in a struct with any extra information before forwarding it to the aggregate channel.
In my (limited) testing, this method greatly out performs using the reflect package:
$ go test dynamic_select_test.go -test.bench=.
...
BenchmarkReflectSelect 1 5265109013 ns/op
BenchmarkGoSelect 20 81911344 ns/op
ok command-line-arguments 9.463s
Benchmark code here
To expand on some comments on previous answers and to provide a clearer comparison here is an example of both approaches presented so far given the same input, a slice of channels to read from and a function to call for each value which also need to know which channel the value came from.
There are three main differences between the approaches:
Complexity. Although it may partially be a reader preference I find the channel approach more idiomatic, straight-forward, and readable.
Performance. On my Xeon amd64 system the goroutines+channels out performs the reflect solution by about two orders of magnitude (in general reflection in Go is often slower and should only be used when absolutely required). Of course, if there is any significant delay in either the function processing the results or in the writing of values to the input channels this performance difference can easily become insignificant.
Blocking/buffering semantics. The importantance of this depends on the use case. Most often it either won't matter or the slight extra buffering in the goroutine merging solution may be helpful for throughput. However, if it is desirable to have the semantics that only a single writer is unblocked and it's value fully handled before any other writer is unblocked, then that can only be achieved with the reflect solution.
Note, both approaches can be simplified if either the "id" of the sending channel isn't required or if the source channels will never be closed.
Goroutine merging channel:
// Process1 calls `fn` for each value received from any of the `chans`
// channels. The arguments to `fn` are the index of the channel the
// value came from and the string value. Process1 returns once all the
// channels are closed.
func Process1(chans []<-chan string, fn func(int, string)) {
// Setup
type item struct {
int // index of which channel this came from
string // the actual string item
}
merged := make(chan item)
var wg sync.WaitGroup
wg.Add(len(chans))
for i, c := range chans {
go func(i int, c <-chan string) {
// Reads and buffers a single item from `c` before
// we even know if we can write to `merged`.
//
// Go doesn't provide a way to do something like:
// merged <- (<-c)
// atomically, where we delay the read from `c`
// until we can write to `merged`. The read from
// `c` will always happen first (blocking as
// required) and then we block on `merged` (with
// either the above or the below syntax making
// no difference).
for s := range c {
merged <- item{i, s}
}
// If/when this input channel is closed we just stop
// writing to the merged channel and via the WaitGroup
// let it be known there is one fewer channel active.
wg.Done()
}(i, c)
}
// One extra goroutine to watch for all the merging goroutines to
// be finished and then close the merged channel.
go func() {
wg.Wait()
close(merged)
}()
// "select-like" loop
for i := range merged {
// Process each value
fn(i.int, i.string)
}
}
Reflection select:
// Process2 is identical to Process1 except that it uses the reflect
// package to select and read from the input channels which guarantees
// there is only one value "in-flight" (i.e. when `fn` is called only
// a single send on a single channel will have succeeded, the rest will
// be blocked). It is approximately two orders of magnitude slower than
// Process1 (which is still insignificant if their is a significant
// delay between incoming values or if `fn` runs for a significant
// time).
func Process2(chans []<-chan string, fn func(int, string)) {
// Setup
cases := make([]reflect.SelectCase, len(chans))
// `ids` maps the index within cases to the original `chans` index.
ids := make([]int, len(chans))
for i, c := range chans {
cases[i] = reflect.SelectCase{
Dir: reflect.SelectRecv,
Chan: reflect.ValueOf(c),
}
ids[i] = i
}
// Select loop
for len(cases) > 0 {
// A difference here from the merging goroutines is
// that `v` is the only value "in-flight" that any of
// the workers have sent. All other workers are blocked
// trying to send the single value they have calculated
// where-as the goroutine version reads/buffers a single
// extra value from each worker.
i, v, ok := reflect.Select(cases)
if !ok {
// Channel cases[i] has been closed, remove it
// from our slice of cases and update our ids
// mapping as well.
cases = append(cases[:i], cases[i+1:]...)
ids = append(ids[:i], ids[i+1:]...)
continue
}
// Process each value
fn(ids[i], v.String())
}
}
[Full code on the Go playground.]
We actually made some research about this subject and found the best solution. We used reflect.Select for a while and it is a great solution for the problem. It is much lighter than a goroutine per channel and simple to operate. But unfortunately, it doesn't really support a massive amount of channels which is our case so we found something interesting and wrote a blog post about it: https://cyolo.io/blog/how-we-enabled-dynamic-channel-selection-at-scale-in-go/
I'll summarize what is written there:
We statically created batches of select..case statements for every result of the power of two of exponent up to 32 along with a function that routes to the different cases and aggregates the results through an aggregate channel.
An example of such a batch:
func select4(ctx context.Context, chanz []chan interface{}, res chan *r, r *r, i int) {
select {
case r.v, r.ok = <-chanz[0]:
r.i = i + 0
res <- r
case r.v, r.ok = <-chanz[1]:
r.i = i + 1
res <- r
case r.v, r.ok = <-chanz[2]:
r.i = i + 2
res <- r
case r.v, r.ok = <-chanz[3]:
r.i = i + 3
res <- r
case <-ctx.Done():
break
}
}
And the logic of aggregating the first result from any number of channels using these kinds of select..case batches:
for i < len(channels) {
l = len(channels) - i
switch {
case l > 31 && maxBatchSize >= 32:
go select32(ctx, channels[i:i+32], agg, rPool.Get().(*r), i)
i += 32
case l > 15 && maxBatchSize >= 16:
go select16(ctx, channels[i:i+16], agg, rPool.Get().(*r), i)
i += 16
case l > 7 && maxBatchSize >= 8:
go select8(ctx, channels[i:i+8], agg, rPool.Get().(*r), i)
i += 8
case l > 3 && maxBatchSize >= 4:
go select4(ctx, channels[i:i+4], agg, rPool.Get().(*r), i)
i += 4
case l > 1 && maxBatchSize >= 2:
go select2(ctx, channels[i:i+2], agg, rPool.Get().(*r), i)
i += 2
case l > 0:
go select1(ctx, channels[i], agg, rPool.Get().(*r), i)
i += 1
}
}
Possibly simpler option:
Instead of having an array of channels, why not pass just one channel as a parameter to the functions being run on separate goroutines, and then listen to the channel in a consumer goroutine?
This allows you to select on just one channel in your listener, making for a simple select, and avoiding creation of new goroutines to aggregate messages from multiple channels?
Based on the answer of James Henstridge,
I made this generic (go >=1.18) Select function that takes a context and a slice of channels and returns the selected one:
func Select[T any](ctx context.Context, chs []chan T) (int, T, error) {
var zeroT T
cases := make([]reflect.SelectCase, len(chs)+1)
for i, ch := range chs {
cases[i] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ch)}
}
cases[len(chs)] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ctx.Done())}
// ok will be true if the channel has not been closed.
chosen, value, ok := reflect.Select(cases)
if !ok {
if ctx.Err() != nil {
return -1, zeroT, ctx.Err()
}
return chosen, zeroT, errors.New("channel closed")
}
if ret, ok := value.Interface().(T); ok {
return chosen, ret, nil
}
return chosen, zeroT, errors.New("failed to cast value")
}
Here is an example on how to use it:
func TestSelect(t *testing.T) {
c1 := make(chan int)
c2 := make(chan int)
c3 := make(chan int)
chs := []chan int{c1, c2, c3}
go func() {
time.Sleep(time.Second)
//close(c2)
c2 <- 42
}()
ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
chosen, val, err := Select(ctx, chs)
assert.Equal(t, 1, chosen)
assert.Equal(t, 42, val)
assert.NoError(t, err)
}
Why this approach wouldn't work assuming that somebody is sending events?
func main() {
numChans := 2
var chans = []chan string{}
for i := 0; i < numChans; i++ {
tmp := make(chan string)
chans = append(chans, tmp)
}
for true {
for i, c := range chans {
select {
case x = <-c:
fmt.Printf("received %d \n", i)
go DoShit(x, i)
default: continue
}
}
}
}
I am attempting to do some computation in parallel. The program is designed so that each worker goroutine sends "pieces" of a solved puzzle back to the controller goroutine that waits to receive and assembles everything sent from the worker routines.
What is the idomatic Go for closing the single channel? I cannot call close on the channel in each goroutine because then I could possibly send on a closed channel. Likewise, there is no way to predetermine which goroutine will finish first. Is a sync.WaitGroup necessary here?
Here is an example using the sync.WaitGroup to do what you are looking for,
This example accepts a lenghty list of integers, then sums them all up by handing N parallel workers an equal-sized chunk of the input data. It can be run on go playground:
package main
import (
"fmt"
"sync"
)
const WorkerCount = 10
func main() {
// Some input data to operate on.
// Each worker gets an equal share to work on.
data := make([]int, WorkerCount*10)
for i := range data {
data[i] = i
}
// Sum all the entries.
result := sum(data)
fmt.Printf("Sum: %d\n", result)
}
// sum adds up the numbers in the given list, by having the operation delegated
// to workers operating in parallel on sub-slices of the input data.
func sum(data []int) int {
var sum int
result := make(chan int)
defer close(result)
// Accumulate results from workers.
go func() {
for {
select {
case value := <-result:
sum += value
}
}
}()
// The WaitGroup will track completion of all our workers.
wg := new(sync.WaitGroup)
wg.Add(WorkerCount)
// Divide the work up over the number of workers.
chunkSize := len(data) / WorkerCount
// Spawn workers.
for i := 0; i < WorkerCount; i++ {
go func(i int) {
offset := i * chunkSize
worker(result, data[offset:offset+chunkSize])
wg.Done()
}(i)
}
// Wait for all workers to finish, before returning the result.
wg.Wait()
return sum
}
// worker sums up the numbers in the given list.
func worker(result chan int, data []int) {
var sum int
for _, v := range data {
sum += v
}
result <- sum
}
Yes, This is a perfect use case for sync.WaitGroup.
Your other option is to use 1 channel per goroutine and one multiplexer goroutine that feeds from each channel into a single channel. But that would get unwieldy fast so I'd just go with a sync.WaitGroup.