Go channel: consume data from channel although not push anything to channel - go

For example I have this code:
package main
import (
"fmt"
)
func main() {
c1 := make(chan interface{})
close(c1)
c2 := make(chan interface{})
close(c2)
var c1Count, c2Count int
for i := 1000; i >= 0; i-- {
select {
case <-c1:
c1Count++
case <-c2:
c2Count++
}
}
fmt.Printf("c1Count: %d\nc2Count: %d\n ", c1Count, c2Count)
}
When running, the output will be:
c1Count: 513
c2Count: 488
The thing I don't know is: we create c1 and c2 channel without doing anything. Why in select/case block, c1Count and c2Count can increase value ?
Thanks

The Go Programming Language Specification
Close
After calling close, and after any previously sent values have been
received, receive operations will return the zero value for the
channel's type without blocking. The multi-valued receive operation
returns a received value along with an indication of whether the
channel is closed.
You are counting zero values.
For example,
package main
import (
"fmt"
)
func main() {
c1 := make(chan interface{})
close(c1)
c2 := make(chan interface{})
close(c2)
var c1Count, c2Count int
var z1Count, z2Count int
for i := 1000; i >= 0; i-- {
select {
case z1 := <-c1:
c1Count++
if z1 == nil {
z1Count++
}
case z2 := <-c2:
c2Count++
if z2 == nil {
z2Count++
}
}
}
fmt.Printf("c1Count: %d\nc2Count: %d\n", c1Count, c2Count)
fmt.Printf("z1Count: %d\nz2Count: %d\n", z1Count, z2Count)
}
Playground: https://play.golang.org/p/tPRkqXrAFno
Output:
c1Count: 511
c2Count: 490
z1Count: 511
z2Count: 490
The Go Programming Language Specification
For statements
For statements with range clause
For channels, the iteration values produced are the successive values
sent on the channel until the channel is closed. If the channel is
nil, the range expression blocks forever.
Close is useful with a for statement with a range clause.

Related

Im trying to understand why this case is always going off

I'm trying to understand why in this select statement the first case always goes off and does not wait for the channel to be filled. For this program I'm trying to get the program to wait till all the channels have been filled and whenever a channel is filled by the method it is put in the first available space in the array of channels
I tried putting the line <-res[i] in the case statement but for some reason this case always goes off regardless of whether or not the channels have a value.
package main
import (
"fmt"
"math"
"math/rand"
"time"
)
func numbers(sz int) (res chan float64) {
res = make(chan float64)
go func() {
defer close(res)
num := 0.0
time.Sleep(time.Duration(rand.Intn(1000)) *time.Microsecond)
for i := 0; i < sz; i++ {
num += math.Sqrt(math.Abs(rand.Float64()))
}
num /= float64(sz)
res <- num
return
}()
return
}
func main() {
var nGo int
rand.Seed(42)
fmt.Print("Number of Go routines: ")
fmt.Scanf("%d \n", &nGo)
res := make([]chan float64, nGo)
j:=0
for i := 0; i < nGo; i++ {
res[i] =numbers(1000)
}
for true{
for i := 0; i < nGo; {
select {
case <-res[i]:{
res[j]=res[i]//this line
j++
}
default:
i++
}
}
if j==nGo{
break
}
}
fmt.Println(<-res[nGo-1])
}
The print line should print some float.
<-res[i] in the case statement but for some reason this case always goes off regarless of wether or not the channels has a value
It will only not choose this case if the channel's buffer is full (i.e. a value cannot be sent without blocking). Your channel has a buffer length equal to the number of values you're sending on it, so it will never block, giving it no reason to ever take the default case.

Select all values from multiple channels when channel ready

I'm new in golang and I faced with the problem.
I have several channels.
Some payload gets to this channels in different time.
How I can get all values one by one from channels in the time when channel ready to spit it.
For example I wrote this code:
package main
import (
"fmt"
"time"
"math/rand"
)
func main() {
arr1 := []int8{1,2,3,4,5}
arr2 := []int8{6,7,8,9,10}
c1 := make(chan int8)
c2 := make(chan int8)
go func() {
for _, val := range arr1 {
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
c1 <- val
}
}()
go func() {
for _, val := range arr2 {
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
c2 <- val
}
}()
select {
case res1 := <- c1:
fmt.Println(res1)
case res2 := <- c2:
fmt.Println(res2)
}
fmt.Println("Hello, test")
}
But in this case, I get only first value, from one of the channels.
Please, give me advise how to solve my issue.
Link to go-play https://play.golang.org/p/FOmkP57YCyR
You have to do couple of things.
1) Make sure you close channels once you are done with source.
2) Iterate over channels until it's closed.
Example:
package main
import (
"fmt"
"math/rand"
"time"
)
func main() {
arr1 := []int8{1, 2, 3, 4, 5}
arr2 := []int8{6, 7, 8, 9, 10}
c1 := make(chan int8)
c2 := make(chan int8)
go func() {
for _, val := range arr1 {
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
c1 <- val
}
close(c1)
}()
go func() {
for _, val := range arr2 {
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
c2 <- val
}
close(c2)
}()
_c1 := true
_c2 := true
var res1, res2 int8
for _c1 == true || _c2 == true {
select {
case res1, _c1 = <-c1:
if _c1 == true {
fmt.Println(res1)
}
case res2, _c2 = <-c2:
if _c2 == true {
fmt.Println(res2)
}
}
}
fmt.Println("Hello, test")
}
On execution, I got following output on screen.
6
1
7
2
3
4
8
5
9
10
Hello, test
You don't have to use 2 channels. Just use 1 channel and store values into it from multiple goroutines. Channel is lightweight thread connector that is fast and can be instantiated multiple times to store values from multiple goroutines.
The problem with your code is that it has no loop to loop over values from goroutines channel. You only print it once using select. select makes other goroutines wait until it executes one of possible cases it has. If all of the cases are possible, then it selects randomly to executes.
The reason you only got one value from your channel is because when your goroutines are working, they store values from the array to channel sequentially. while this happen, you call select statement in your main thread with the cases to get the values from the channel in your goroutines. Since you don't loop over the channels, you'll only get one value from the channel which is the value it firstly received. In this case, you loop array sequentially into channel in your goroutines, hence you'll get the first index of the array since it's the value that will be firstly sent to select statement in main thread. All of your select cases are possible to be executed, hence it will execute one of the case randomly and you'll get first index in one of those arrays.
To fix this, you need to loop over the channel to get the values stored in it one by one. Beside, you also need to synchronize all the threads in order to avoid deadlock conditions, which is happened when your main thread doesn't know when to stop invoking channel from goroutines since they work asynchronously. Your channel is ready to spit the value right after it gets its value inside goroutines and invoked into main thread in loop. Here is the code:
package main
import (
"fmt"
"time"
"math/rand"
"sync"
)
// writer set numbers from array to channel
func writer(ch chan int, arr []int ,wgwrite *sync.WaitGroup) {
defer wgwrite.Done()
for _, val := range arr {
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
ch <- val
}
}
// reader receive input from writer channels and print them all
func reader(ch chan int, wgread *sync.WaitGroup) {
defer wgread.Done()
for i:= range ch {
fmt.Println(i)
}
fmt.Println("Hello, test")
}
func main() {
arr1 := []int{1,2,3,4,5}
arr2 := []int{6,7,8,9,10}
ch := make(chan int)
wgwrite := &sync.WaitGroup{}
wgread := &sync.WaitGroup{}
wgwrite.Add(2)
go writer(ch, arr1, wgwrite)
go writer(ch, arr2, wgwrite)
wgread.Add(1)
go reader(ch, wgread)
wgwrite.Wait()
close(ch)
wgread.Wait()
}
https://play.golang.org/p/32Fgetq_Zu7
Hope it helps.
Select does not wait for go routines. To achieve that you should wrap it in for statement. In this way select will be running until one of the cases returns and breaks out of for statement.
for {
select {
...
You can also use buffered channels which are non-blocking and wait groups. Like this:
arr1 := []int8{1,2,3,4,5}
arr2 := []int8{6,7,8,9,10}
c1 := make(chan int8, len(arr1))
c2 := make(chan int8, len(arr2))
var wg sync.WaitGroup
wg.Add(1) // First wait group
go func() {
for _, val := range arr1 {
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
c1 <- val
}
wg.Done()
}()
wg.Add(1) // Second wait group
go func() {
for _, val := range arr2 {
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
c2 <- val
}
wg.Done()
}()
// executed after wg.Done() is called 2 times since we have 2 wait groups
wg.Wait()
// We are not writing to channels anymore so we can close them.
close(c1)
close(c2)
for value := range c1 {
fmt.Println(value)
}
for value := range c2 {
fmt.Println(value)
}
fmt.Println("Hello, test")

Getting deadlock as I try to emulate fan in - fan out with factorial calculations

I am trying the fan in - fan out pattern with a factorial problem. But I am getting:
fatal error: all goroutines are asleep - deadlock!
and unable to identify the reason for deadlock.
I am trying to concurrently calculate factorial for 100 numbers using the fan-in fan-out pattern.
package main
import (
"fmt"
)
func main() {
_inChannel := _inListener(generator())
for val := range _inChannel {
fmt.Print(val, " -- ")
}
}
func generator() chan int { // NEED TO CALCULATE FACTORIAL FOR 100 NUMBERS
ch := make(chan int) // CREATE CHANNEL TO INPUT NUMBERS
go func() {
for i := 1; i <= 100; i++ {
ch <- i
}
close(ch) // CLOSE CHANNEL WHEN ALL NUMBERS HAVE BEEN WRITTEM
}()
return ch
}
func _inListener(ch chan int) chan int {
rec := make(chan int) // CHANNEL RECEIVED FROM GENERATOR
go func() {
for num := range ch { // RECEIVE THE INPUT NUMBERS FROM GENERATOR
result := factorial(num) // RESULT IS A NEW CHANNEL CREATED
rec <- <-result // MERGE INTO A SINGLE CHANNEL; rec
close(result)
}
close(rec)
}()
return rec // RETURN THE DEDICATED CHANNEL TO RECEIVE ALL OUTPUTS
}
func factorial(n int) chan int {
ch := make(chan int) // MAKE A NEW CHANNEL TO OUTPUT THE RESULT
// OF FACTORIAL
total := 1
for i := n; i > 0; i-- {
total *= i
}
ch <- total
return ch // RETURN THE CHANNEL HAVING THE FACTORIAL CALCULATED
}
I have put in comments, so that it becomes easier to follow the code.
I'm no expert in channels. I've taking on this to try and get more familiar with go.
Another issue is the int isn't large enough to take all factorials over 20 or so.
As you can see, I added a defer close as well as a logical channel called done in the generator func. The rest of the changes probably aren't needed. With channels you need to make sure something is ready to take off a value on the channel when you put something on a channel. Otherwise deadlock. Also, using
go run -race main.go
helps at least see which line(s) are causing problems.
I hope this helps and isn't removed for being off topic.
I was able to remove the deadlock by doing this:
package main
import (
"fmt"
)
func main() {
_gen := generator()
_inChannel := _inListener(_gen)
for val := range _inChannel {
fmt.Print(val, " -- \n")
}
}
func generator() chan int { // NEED TO CALCULATE FACTORIAL FOR 100 NUMBERS
ch := make(chan int) // CREATE CHANNEL TO INPUT NUMBERS
done := make(chan bool)
go func() {
defer close(ch)
for i := 1; i <= 100; i++ {
ch <- i
}
//close(ch) // CLOSE CHANNEL WHEN ALL NUMBERS HAVE BEEN WRITTEM
done <- true
}()
// this function will pull off the done for each function call above.
go func() {
for i := 1; i < 100; i++ {
<-done
}
}()
return ch
}
func _inListener(ch chan int) chan int {
rec := make(chan int) // CHANNEL RECEIVED FROM GENERATOR
go func() {
for num := range ch { // RECEIVE THE INPUT NUMBERS FROM GENERATOR
result := factorial(num) // RESULT IS A NEW CHANNEL CREATED
rec <- result // MERGE INTO A SINGLE CHANNEL; rec
}
close(rec)
}()
return rec // RETURN THE DEDICATED CHANNEL TO RECEIVE ALL OUTPUTS
}
func factorial(n int) int {
// OF FACTORIAL
total := 1
for i := n; i > 0; i-- {
total *= i
}
return total // RETURN THE CHANNEL HAVING THE FACTORIAL CALCULATED
}

How to break out of select gracefuly in golang

I have a program in golang that counts SHA1s and prints ones that start with two zeros. I want to use goroutines and channels. My problem is that I don't know how to gracefully exit select clause if I don't know how many results it will produce.
Many tutorials know that in advance and exit when counter hits. Other suggest using WaitGroups, but I don't want to do that: I want to print results in main thread as soon it appears in channel. Some suggest to close a channel when goroutines are finished, but I want to close it after asynchronous for finishes, so I don't know how.
Please help me to achieve my requirements:
package main
import (
"crypto/sha1"
"fmt"
"time"
"runtime"
"math/rand"
)
type Hash struct {
message string
hash [sha1.Size]byte
}
var counter int = 0
var max int = 100000
var channel = make(chan Hash)
var source = rand.NewSource(time.Now().UnixNano())
var generator = rand.New(source)
func main() {
nCPU := runtime.NumCPU()
runtime.GOMAXPROCS(nCPU)
fmt.Println("Number of CPUs: ", nCPU)
start := time.Now()
for i := 0 ; i < max ; i++ {
go func(j int) {
count(j)
}(i)
}
// close channel here? I can't because asynchronous producers work now
for {
select {
// how to stop receiving if there are no producers left?
case hash := <- channel:
fmt.Printf("Hash is %v\n ", hash)
}
}
fmt.Printf("Count of %v sha1 took %v\n", max, time.Since(start))
}
func count(i int) {
random := fmt.Sprintf("This is a test %v", generator.Int())
hash := sha1.Sum([]byte(random))
if (hash[0] == 0 && hash[1] == 0) {
channel <- Hash{random, hash}
}
}
Firstly: if you don't know when your computation ends, how could you even model it? Make sure you know exactly when and under what circumstances your program terminates. If you're done you know how to write it in code.
You're basically dealing with a producer-consumer problem. A standard case. I would model
that this way (on play):
Producer
func producer(max int, out chan<- Hash, wg *sync.WaitGroup) {
defer wg.Done()
for i := 0; i < max; i++ {
random := fmt.Sprintf("This is a test %v", rand.Int())
hash := sha1.Sum([]byte(random))
if hash[0] == 0 && hash[1] == 0 {
out <- Hash{random, hash}
}
}
close(out)
}
Obviously you're brute-forcing hashes, so the end is reached when the loop is finished.
We can close the channel here and signal the other goroutines that there is nothing more to listen for.
Consumer
func consumer(max int, in <-chan Hash, wg *sync.WaitGroup) {
defer wg.Done()
for {
hash, ok := <-in
if !ok {
break
}
fmt.Printf("Hash is %v\n ", hash)
}
}
The consumer takes all the incoming messages from the in channel and checks if it was closed (ok).
If it is closed, we're done. Otherwise print the received hashes.
Main
To start this all up we can write:
wg := &sync.WaitGroup{}
c := make(chan Hash)
wg.Add(1)
go producer(max, c, wg)
wg.Add(1)
go consumer(max, c, wg)
wg.Wait()
The WaitGroup's purpose is to wait until the spawned goroutines finished, signalled by
the call of wg.Done in the goroutines.
Sidenote
Also note that the Rand you're using is not safe for concurrent access. Use the one initialized
globally in math/rand. Example:
rand.Seed(time.Now().UnixNano())
rand.Int()
The structure of your program should probably be re-examined.
Here is a working example of what I presume you are looking for.
It can be run on the Go playground
package main
import (
"crypto/sha1"
"fmt"
"math/rand"
"runtime"
"time"
)
type Hash struct {
message string
hash [sha1.Size]byte
}
const Max int = 100000
func main() {
nCPU := runtime.NumCPU()
runtime.GOMAXPROCS(nCPU)
fmt.Println("Number of CPUs: ", nCPU)
hashes := Generate()
start := time.Now()
for hash := range hashes {
fmt.Printf("Hash is %v\n ", hash)
}
fmt.Printf("Count of %v sha1 took %v\n", Max, time.Since(start))
}
func Generate() <-chan Hash {
c := make(chan Hash, 1)
go func() {
defer close(c)
source := rand.NewSource(time.Now().UnixNano())
generator := rand.New(source)
for i := 0; i < Max; i++ {
random := fmt.Sprintf("This is a test %v", generator.Int())
hash := sha1.Sum([]byte(random))
if hash[0] == 0 && hash[1] == 0 {
c <- Hash{random, hash}
}
}
}()
return c
}
Edit: This does not fire up a separate routine for each Hash computation,
but to be honest, I fail to see the value on doing so. The scheduling of all those routines will likely cost you far more than running the code in a single routine.
If need be, you can split it up into chunks of N routines, but a 1:1 mapping is not the way to go with this.

how to listen to N channels? (dynamic select statement)

to start an endless loop of executing two goroutines, I can use the code below:
after receiving the msg it will start a new goroutine and go on for ever.
c1 := make(chan string)
c2 := make(chan string)
go DoStuff(c1, 5)
go DoStuff(c2, 2)
for ; true; {
select {
case msg1 := <-c1:
fmt.Println("received ", msg1)
go DoStuff(c1, 1)
case msg2 := <-c2:
fmt.Println("received ", msg2)
go DoStuff(c2, 9)
}
}
I would now like to have the same behavior for N goroutines, but how will the select statement look in that case?
This is the code bit I have started with, but I am confused how to code the select statement
numChans := 2
//I keep the channels in this slice, and want to "loop" over them in the select statemnt
var chans = [] chan string{}
for i:=0;i<numChans;i++{
tmp := make(chan string);
chans = append(chans, tmp);
go DoStuff(tmp, i + 1)
//How shall the select statment be coded for this case?
for ; true; {
select {
case msg1 := <-c1:
fmt.Println("received ", msg1)
go DoStuff(c1, 1)
case msg2 := <-c2:
fmt.Println("received ", msg2)
go DoStuff(c2, 9)
}
}
You can do this using the Select function from the reflect package:
func Select(cases []SelectCase) (chosen int, recv Value, recvOK bool)
Select executes a select operation described by the list of cases. Like
the Go select statement, it blocks until at least one of the cases can
proceed, makes a uniform pseudo-random choice, and then executes that
case. It returns the index of the chosen case and, if that case was a
receive operation, the value received and a boolean indicating whether
the value corresponds to a send on the channel (as opposed to a zero
value received because the channel is closed).
You pass in an array of SelectCase structs that identify the channel to select on, the direction of the operation, and a value to send in the case of a send operation.
So you could do something like this:
cases := make([]reflect.SelectCase, len(chans))
for i, ch := range chans {
cases[i] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ch)}
}
chosen, value, ok := reflect.Select(cases)
// ok will be true if the channel has not been closed.
ch := chans[chosen]
msg := value.String()
You can experiment with a more fleshed out example here: http://play.golang.org/p/8zwvSk4kjx
You can accomplish this by wrapping each channel in a goroutine which "forwards" messages to a shared "aggregate" channel. For example:
agg := make(chan string)
for _, ch := range chans {
go func(c chan string) {
for msg := range c {
agg <- msg
}
}(ch)
}
select {
case msg <- agg:
fmt.Println("received ", msg)
}
If you need to know which channel the message originated from, you could wrap it in a struct with any extra information before forwarding it to the aggregate channel.
In my (limited) testing, this method greatly out performs using the reflect package:
$ go test dynamic_select_test.go -test.bench=.
...
BenchmarkReflectSelect 1 5265109013 ns/op
BenchmarkGoSelect 20 81911344 ns/op
ok command-line-arguments 9.463s
Benchmark code here
To expand on some comments on previous answers and to provide a clearer comparison here is an example of both approaches presented so far given the same input, a slice of channels to read from and a function to call for each value which also need to know which channel the value came from.
There are three main differences between the approaches:
Complexity. Although it may partially be a reader preference I find the channel approach more idiomatic, straight-forward, and readable.
Performance. On my Xeon amd64 system the goroutines+channels out performs the reflect solution by about two orders of magnitude (in general reflection in Go is often slower and should only be used when absolutely required). Of course, if there is any significant delay in either the function processing the results or in the writing of values to the input channels this performance difference can easily become insignificant.
Blocking/buffering semantics. The importantance of this depends on the use case. Most often it either won't matter or the slight extra buffering in the goroutine merging solution may be helpful for throughput. However, if it is desirable to have the semantics that only a single writer is unblocked and it's value fully handled before any other writer is unblocked, then that can only be achieved with the reflect solution.
Note, both approaches can be simplified if either the "id" of the sending channel isn't required or if the source channels will never be closed.
Goroutine merging channel:
// Process1 calls `fn` for each value received from any of the `chans`
// channels. The arguments to `fn` are the index of the channel the
// value came from and the string value. Process1 returns once all the
// channels are closed.
func Process1(chans []<-chan string, fn func(int, string)) {
// Setup
type item struct {
int // index of which channel this came from
string // the actual string item
}
merged := make(chan item)
var wg sync.WaitGroup
wg.Add(len(chans))
for i, c := range chans {
go func(i int, c <-chan string) {
// Reads and buffers a single item from `c` before
// we even know if we can write to `merged`.
//
// Go doesn't provide a way to do something like:
// merged <- (<-c)
// atomically, where we delay the read from `c`
// until we can write to `merged`. The read from
// `c` will always happen first (blocking as
// required) and then we block on `merged` (with
// either the above or the below syntax making
// no difference).
for s := range c {
merged <- item{i, s}
}
// If/when this input channel is closed we just stop
// writing to the merged channel and via the WaitGroup
// let it be known there is one fewer channel active.
wg.Done()
}(i, c)
}
// One extra goroutine to watch for all the merging goroutines to
// be finished and then close the merged channel.
go func() {
wg.Wait()
close(merged)
}()
// "select-like" loop
for i := range merged {
// Process each value
fn(i.int, i.string)
}
}
Reflection select:
// Process2 is identical to Process1 except that it uses the reflect
// package to select and read from the input channels which guarantees
// there is only one value "in-flight" (i.e. when `fn` is called only
// a single send on a single channel will have succeeded, the rest will
// be blocked). It is approximately two orders of magnitude slower than
// Process1 (which is still insignificant if their is a significant
// delay between incoming values or if `fn` runs for a significant
// time).
func Process2(chans []<-chan string, fn func(int, string)) {
// Setup
cases := make([]reflect.SelectCase, len(chans))
// `ids` maps the index within cases to the original `chans` index.
ids := make([]int, len(chans))
for i, c := range chans {
cases[i] = reflect.SelectCase{
Dir: reflect.SelectRecv,
Chan: reflect.ValueOf(c),
}
ids[i] = i
}
// Select loop
for len(cases) > 0 {
// A difference here from the merging goroutines is
// that `v` is the only value "in-flight" that any of
// the workers have sent. All other workers are blocked
// trying to send the single value they have calculated
// where-as the goroutine version reads/buffers a single
// extra value from each worker.
i, v, ok := reflect.Select(cases)
if !ok {
// Channel cases[i] has been closed, remove it
// from our slice of cases and update our ids
// mapping as well.
cases = append(cases[:i], cases[i+1:]...)
ids = append(ids[:i], ids[i+1:]...)
continue
}
// Process each value
fn(ids[i], v.String())
}
}
[Full code on the Go playground.]
We actually made some research about this subject and found the best solution. We used reflect.Select for a while and it is a great solution for the problem. It is much lighter than a goroutine per channel and simple to operate. But unfortunately, it doesn't really support a massive amount of channels which is our case so we found something interesting and wrote a blog post about it: https://cyolo.io/blog/how-we-enabled-dynamic-channel-selection-at-scale-in-go/
I'll summarize what is written there:
We statically created batches of select..case statements for every result of the power of two of exponent up to 32 along with a function that routes to the different cases and aggregates the results through an aggregate channel.
An example of such a batch:
func select4(ctx context.Context, chanz []chan interface{}, res chan *r, r *r, i int) {
select {
case r.v, r.ok = <-chanz[0]:
r.i = i + 0
res <- r
case r.v, r.ok = <-chanz[1]:
r.i = i + 1
res <- r
case r.v, r.ok = <-chanz[2]:
r.i = i + 2
res <- r
case r.v, r.ok = <-chanz[3]:
r.i = i + 3
res <- r
case <-ctx.Done():
break
}
}
And the logic of aggregating the first result from any number of channels using these kinds of select..case batches:
for i < len(channels) {
l = len(channels) - i
switch {
case l > 31 && maxBatchSize >= 32:
go select32(ctx, channels[i:i+32], agg, rPool.Get().(*r), i)
i += 32
case l > 15 && maxBatchSize >= 16:
go select16(ctx, channels[i:i+16], agg, rPool.Get().(*r), i)
i += 16
case l > 7 && maxBatchSize >= 8:
go select8(ctx, channels[i:i+8], agg, rPool.Get().(*r), i)
i += 8
case l > 3 && maxBatchSize >= 4:
go select4(ctx, channels[i:i+4], agg, rPool.Get().(*r), i)
i += 4
case l > 1 && maxBatchSize >= 2:
go select2(ctx, channels[i:i+2], agg, rPool.Get().(*r), i)
i += 2
case l > 0:
go select1(ctx, channels[i], agg, rPool.Get().(*r), i)
i += 1
}
}
Possibly simpler option:
Instead of having an array of channels, why not pass just one channel as a parameter to the functions being run on separate goroutines, and then listen to the channel in a consumer goroutine?
This allows you to select on just one channel in your listener, making for a simple select, and avoiding creation of new goroutines to aggregate messages from multiple channels?
Based on the answer of James Henstridge,
I made this generic (go >=1.18) Select function that takes a context and a slice of channels and returns the selected one:
func Select[T any](ctx context.Context, chs []chan T) (int, T, error) {
var zeroT T
cases := make([]reflect.SelectCase, len(chs)+1)
for i, ch := range chs {
cases[i] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ch)}
}
cases[len(chs)] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ctx.Done())}
// ok will be true if the channel has not been closed.
chosen, value, ok := reflect.Select(cases)
if !ok {
if ctx.Err() != nil {
return -1, zeroT, ctx.Err()
}
return chosen, zeroT, errors.New("channel closed")
}
if ret, ok := value.Interface().(T); ok {
return chosen, ret, nil
}
return chosen, zeroT, errors.New("failed to cast value")
}
Here is an example on how to use it:
func TestSelect(t *testing.T) {
c1 := make(chan int)
c2 := make(chan int)
c3 := make(chan int)
chs := []chan int{c1, c2, c3}
go func() {
time.Sleep(time.Second)
//close(c2)
c2 <- 42
}()
ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
chosen, val, err := Select(ctx, chs)
assert.Equal(t, 1, chosen)
assert.Equal(t, 42, val)
assert.NoError(t, err)
}
Why this approach wouldn't work assuming that somebody is sending events?
func main() {
numChans := 2
var chans = []chan string{}
for i := 0; i < numChans; i++ {
tmp := make(chan string)
chans = append(chans, tmp)
}
for true {
for i, c := range chans {
select {
case x = <-c:
fmt.Printf("received %d \n", i)
go DoShit(x, i)
default: continue
}
}
}
}

Resources