Reading multiple elements from a channel in Go - go

Question

I'm reading values from a channel in a loop like this:
for {
capturedFrame := <-capturedFrameChan
remoteCopy(capturedFrame)
}
To make it more efficient, I would like to read these values in a batch, with something like this (pseudo-code):
for {
capturedFrames := <-capturedFrameChan
multipleRemoteCopy(capturedFrames)
}
But I'm not sure how to do that. If I call capturedFrames := <-capturedFrameChan multiple times it's going to block.
Basically, what I would like is to read all the available values in captureFrameChan and, if none is available, it blocks as usual.
What would be the way to accomplish this in Go?

Answer 1

Something like this should work:
for {
// we initialize our slice. You may want to add a larger cap to avoid multiple memory allocations on `append`
capturedFrames := make([]Frame, 1)
// We block waiting for a first frame
capturedFrames[0] = <-capturedFrameChan
forLoop:
for {
select {
case buf := <-capturedFrameChan:
// if there is more frame immediately available, we add them to our slice
capturedFrames = append(capturedFrames, buf)
default:
// else we move on without blocking
break forLoop
}
}
multipleRemoteCopy(capturedFrames)
}

Answer 2

Try this (for channel ch with type T):
for firstItem := range ch { // For ensure that any batch could not be empty
var itemsBatch []T
itemsBatch = append(itemsBatch, firstItem)
Remaining:
for len(itemsBatch) < BATCHSIZE { // For control maximum size of batch
select {
case item := <-ch:
itemsBatch = append(itemsBatch, item)
default:
break Remaining
}
}
// Consume itemsBatch here...
}
But, if BATCHSIZE is constant, this code would be more efficient:
var i int
itemsBatch := [BATCHSIZE]T{}
for firstItem := range ch { // For ensure that any batch could not be empty
itemsBatch[0] = firstItem
Remaining:
for i = 1; i < BATCHSIZE; i++ { // For control maximum size of batch
select {
case itemsBatch[i] = <-ch:
default:
break Remaining
}
}
// Now you have itemsBatch with length i <= BATCHSIZE;
// Consume that here...
}

Answer 3

By using len(capturedFrames), you can do it like below:
for {
select {
case frame := <-capturedFrames:
frames := []Frame{frame}
for i := 0; i < len(capturedFrames); i++ {
frames = append(frames, <-capturedFrames)
}
multipleRemoteCopy(frames)
}
}

Answer 4

Seems you can also benchmark just
for {
capturedFrame := <-capturedFrameChan
go remoteCopy(capturedFrame)
}
without any codebase refactoring to see if it increase efficiency.

Answer 5

I've ended up doing it as below. Basically I've used len(capturedFrames) to know how many frames are available, then retrieved them in a loop:
for {
var paths []string
itemCount := len(capturedFrames)
if itemCount <= 0 {
time.Sleep(50 * time.Millisecond)
continue
}
for i := 0; i < itemCount; i++ {
f := <-capturedFrames
paths = append(paths, f)
}
err := multipleRemoteCopy(paths, opts)
if err != nil {
fmt.Printf("Error: could not remote copy \"%s\": %s", paths, err)
}
}