Reading from a stream of go channels - go

I was trying to understand the following piece of code that reads from a channel of channels. I am having some difficulties wrapping my head around the idea.
bridge := func(done <-chan interface{}, chanStream <-chan <-chan interface{}) <-chan interface{} {
outStream := make(chan interface{})
go func() {
defer close(outStream)
for {
var stream <-chan interface{}
select {
case <-done:
return
case maybeSteram, ok := <-chanStream:
if ok == false {
return
}
stream = maybeSteram
}
for c := range orDone(done, stream) {
select {
case outStream <- c:
case <-done: // Why we are selection from the done channel here?
}
}
}
}()
return outStream
}
The orDone function:
orDone := func(done <-chan interface{}, inStream <-chan interface{}) <-chan interface{} {
outStream := make(chan interface{})
go func() {
defer close(outStream)
for {
select {
case <-done:
return
case v, ok := <-inStream:
if ok == false {
return
}
select {
case outStream <- v:
case <-done: // Again why we are only reading from this channel? Shouldn't we return from here?
// Why we are not retuening from here?
}
}
}
}()
return outStream
}
As mentioned in the comment, I need some help to understand why we are selecting in the for c := range orDone(donem, stream). Can anyone explain what is going on here?
Thanks in advance.
Edit
I took the code from the book concurrency in go. The full code can be found here: https://github.com/kat-co/concurrency-in-go-src/blob/master/concurrency-patterns-in-go/the-bridge-channel/fig-bridge-channel.go

In both cases, the select is done to avoid blocking — if the reader isn't reading from our output channel, the write might block (maybe even forever), but we want the goroutine to terminate when the done channel is closed, without waiting for anything else. By using the select, it will wait until either thing happens, and then continue, instead of waiting indefinitely for the write to complete before checking done.
As for the other question, "why are we not returning here?": well, we could. But we don't have to, because a closed channel remains readable forever (producing an unlimited number of zero values) once it's been closed. So it's okay to do nothing in those "bottom" selects; if done was in fact closed we will go back up to the top of the loop and hit the case <-done: return there. I suppose it's a matter of style. I probably would have written the return myself, but the author of this sample may have wanted to avoid handling the same condition in two places. As long as it's just return it doesn't really matter, but if you wanted to do some additional action on done, that behavior would have to be updated in two places if the bottom select returns, but only in one place if it doesn't.

Fist I will explain the usage of done channel. It's a common pattern followed in many concurrency related package implementation in Go. Done channels purpose is to denote the end of computation or more like a stop signal. Usually done channel will be listened by many go-routines or multiple places in code flow. One such example is Done channel in Go's builtin "context" package. Since Go doesn't have anything like broadcast ie., to signal all listeners of a channel (expecting this feature is not a good idea too), people just close the channel and all listeners will receive nil value. In your case, since the second select statement is in the end of for block, the code owner might have decided to just continue the loop so that on next iteration, the first select statement which listens on done channel will return from the function.

Related

How does a channel for a signal inside another channel work together?

I have recently started my study of go and am trying to understand how channels work. I'm reading a book there they give such an example:
At times you will be working with channels from disparate parts of your system.
Unlike with pipelines, you can’t make any assertions about how a channel will behave
when code you’re working with is canceled via its done channel. That is to say, you
don’t know if the fact that your goroutine was canceled means the channel you’re
reading from will have been canceled. For this reason we need to wrap our read from the channel with a select statement that also selects from a done channel. This is perfectly fine, but
doing so takes code that’s easily read like this
func orDone(done, c <-chan interface{}) <-chan interface{} {
valStream := make(chan interface{})
go func() {
defer close(valStream)
for {
select {
case <-done:
return
case v, ok := <-c:
if ok == false {
return
}
select {
case valStream <- v:
case <-done:
fmt.Println("inside", count)
}
}
}
}()
return valStream
}
And I don't quite understand how closing a channel through a done channel works.
Let's say I have this code:
func main() {
done1 := make(chan interface{})
done2 := make(chan interface{})
inf := infGenerator(done1)
for v := range orDone(done2, inf) {
if count == 3 {
close(done1)
}
count++
fmt.Println(v)
}
}
If , for example , the first channel is closed , then everything is clear , it will be executed
if ok == false {
return
}
But I don 't quite understand how the program works when we close the second channel instead of 1.
First, the program goes to
case <-done:
fmt.Println("inside", count)
}
And then it gets into the upper one. And I don't understand why it works this way.Shouldn't the nested case <-done: get the closure message and pick it up? Or if we close the channel, then the signal about it is not taken from the channel? And if there is a chance that another case can be executed when closing ?
Assuming infGenerator closes inf after receiving the done signal:
Closing done1 results in closing inf, which causes orDone to return as you said.
If you close done instead, the function orDone executes the nested done case, then the outer done case, simply because the nested done case did not return from the function. It reads the done signal, continues looping, and the loop reads the done signal again and returns.
An unbuffered channel does not hold any data. Thus, if you close an unbuffered channel, any goroutines blocked reading from it will be enabled with false. Any further read attempts will also immediately return false. Because of this, closing a channel is a good way to broadcast completion to all goroutines.

Multiple Go routines reading from the same channel

Hi I'm having a problem with a control channel (of sorts).
The essence of my program:
I do not know how many go routines I will be running at runtime
I will need to restart these go routines at set times, however, they could also potentially error out (and then restarted), so their timing will not be predictable.
These go routines will be putting messages onto a single channel.
So What I've done is created a simple random message generator to put messages onto a channel.
When the timer is up (random duration for testing) I put a message onto a control channel which is a struct payload, so I know there was a close signal and which go routine it was; in reality I'd then do some other stuff I'd need to do before starting the go routines again.
My problem is:
I receive the control message within my reflect.Select loop
I do not (or unable to) receive it in my randmsgs() loop
Therefore I can not stop my randmsgs() go routine.
I believe I'm right in understanding that multiple go routines can read from a single channel, therefore I think I'm misunderstanding how reflect.SelectCases fit into all of this.
My code:
package main
import (
"fmt"
"math/rand"
"reflect"
"time"
)
type testing struct {
control bool
market string
}
func main() {
rand.Seed(time.Now().UnixNano())
// explicitly define chanids for tests.
var chanids []string = []string{"GR I", "GR II", "GR III", "GR IV"}
stream := make(chan string)
control := make([]chan testing, len(chanids))
reflectCases := make([]reflect.SelectCase, len(chanids)+1)
// MAKE REFLECT SELECTS FOR 4 CONTROL CHANS AND 1 DATA CHANNEL
for i := range chanids {
control[i] = make(chan testing)
reflectCases[i] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(control[i])}
}
reflectCases[4] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(stream)}
// START GO ROUTINES
for i, val := range chanids {
runningFunc(control[i], val, stream, 1+rand.Intn(30-1))
}
// READ DATA
for {
o, recieved, ok := reflect.Select(reflectCases)
if !ok {
fmt.Println("You really buggered this one up...")
}
ty, err := recieved.Interface().(testing)
if err == true {
fmt.Printf("Read from: %v, and recieved close signal from: %s\n", o, ty.market)
// close control & stream here.
} else {
ty := recieved.Interface().(string)
fmt.Printf("Read from: %v, and recieved value from: %s\n", o, ty)
}
}
}
// THE GO ROUTINES - TIMER AND RANDMSGS
func runningFunc(q chan testing, chanid string, stream chan string, dur int) {
go timer(q, dur, chanid)
go randmsgs(q, chanid, stream)
}
func timer(q chan testing, t int, message string) {
for t > 0 {
time.Sleep(time.Second)
t--
}
q <- testing{true, message}
}
func randmsgs(q chan testing, chanid string, stream chan string) {
for {
select {
case <-q:
fmt.Println("Just sitting by the mailbox. :(")
return
default:
secondsToWait := 1 + rand.Intn(5-1)
time.Sleep(time.Second * time.Duration(secondsToWait))
stream <- fmt.Sprintf("%s: %d", chanid, secondsToWait)
}
}
}
I apologise for the wall of text, but I'm all out of ideas :(!
K/Regards,
C.
Your channels q in the second half are the same as control[0...3] in the first.
Your reflect.Select that you are running also reads from all of these channels, with no delay.
The problem I think comes down to that your reflect.Select is simply running too fast and "stealing" all the channel output right away. This is why randmsgs is never able to read the messages.
You'll notice that if you remove the default case from randmsgs, the function is able to (potentially) read some of the messages from q.
select {
case <-q:
fmt.Println("Just sitting by the mailbox. :(")
return
}
This is because now that it is running without delay, it is always waiting for a message on q and thus has the chance to beat the reflect.Select in the race.
If you read from the same channel in multiple goroutines, then the data passed will simply go to whatever goroutine reads it first.
This program appears to just be an experiment / learning experience, but I'll offer some criticism that may help.
Again, generally you don't have multiple goroutines reading from the same channel if both goroutines are doing different tasks. You're creating a mostly non-deterministic race as to which goroutine fetches the data first.
Second, this is a common beginner's anti-pattern with select that you should avoid:
for {
select {
case v := <-myChan:
doSomething(v)
default:
// Oh no, there wasn't anything! Guess we have to wait and try again.
time.Sleep(time.Second)
}
This code is redundant because select already behaves in such a way that if no case is initially ready, it will wait until any case is ready and then proceed with that one. This default: sleep is effectively making your select loop slower and yet spending less time actually waiting on the channel (because 99.999...% of the time is spent on time.Sleep).

Select on blocked call and channel

I am pretty sure I've seen a question on this before but can't find it now.
Essentially I want to select on a blocked call as well as a channel.
I know I can push the blocked call into a goroutine and wait on the result via a channel, however that feels like the wrong solution.
Is there an idiomatic way to write this that I'm missing?
Optimally there would be something like:
select {
case a <- c:
...
case ans := connection.Read():
...
}
If you have a channel and a function of which you want to select, using a goroutine and a channel is the idiomatic solution. Note though that if a value is received from the channel, that will not affect the function and it will continue to run. You may use context.Context to signal its result is no longer needed and it may terminate early.
If you're allowed to refactor though, you can "make" the function send on the same channel, so you only need to receive from a single channel.
Another refactoring idea would be for the function to monitor the same channel and return early, so you may just do a single call without select.
Note that if you need to do this in many places, you may create a helper function to launch it asychronously:
func launch(f func()) <-chan struct{} {
done := make(chan struct{})
go func() {
defer close(done)
f()
}()
return done
}
Example function:
func test() {
time.Sleep(time.Second)
}
And then using it:
select {
case a := <-c:
fmt.Println("received from channel:", a)
case <-launch(test):
fmt.Println("test() finished")
}
Try it on the Go Playground.

What is the proper way to read from the errors channel in Sarama?

I am using the Sarama library written in Go to read from an error channel when I produce a message. The overall code looks like this which is enclosed within a function:
producer.AsyncProducer.Input() <- &sarama.ProducerMessage{Topic: topic, Key: nil, Value: sarama.ByteEncoder(message)}
go func() {
for err := range saramaProducer.Errors() {
if producer.callbacks.OnError != nil {
producer.callbacks.OnError(err)
}
}
}()
As my understanding of go routines goes, my go routine would keep iterating over the Errors() channel until it receives one. Is there a way to make it stop listening for errors once my function is done executing?
You can use another channel and a select to make the loop return.
var quit chan struct{}
go func() {
for {
select {
case err:=<-saramaProducer.Errors():
//handle errors
case <-quit:
return
}
}
}
defer func() { quit<-struct{}{} }()
The orignal for ... range loop does not iterate the channel until it gets one. Instead, it blocks until it gets an error, handle it, and waits for a new error again, until the channel close or the main returns.
There is a little problem about the above code, that owhen both quit and error channel is ready, the select picks one randomly, thus may cause a single error loss. If this is worth handling, just put another switch with default to get that error and then return.

How do I iterate over a go time.Tick channel?

I'm having difficulty using time.Tick. I expect this code to print "hi" 10 times then quit after 1 second, but instead it hangs:
ticker := time.NewTicker(100 * time.Millisecond)
time.AfterFunc(time.Second, func () {
ticker.Stop()
})
for _ = range ticker.C {
go fmt.Println("hi")
}
https://play.golang.org/p/1p6-ViSvma
Looking at the source, I see that the channel isn't closed when Stop() is called. In that case, what is the idiomatic way to iterate over the ticker channel?
You're right, ticker's channel is not being closed on stop, that's stated in a documentation:
Stop turns off a ticker. After Stop, no more ticks will be sent. Stop does not close the channel, to prevent a read from the channel succeeding incorrectly.
I believe ticker is more about fire and forget and even if you want to stop it, you could even leave the routine hanging forever (depends on your application of course).
If you really need a finite ticker, you can do tricks and provide a separate channel (per ThunderCat's answer), but what I would do is providing my own implementation of ticker. This should be relatively easy and will give you flexibility with its behaviour, things like what to pass on the channel or deciding what to do with missing ticks (i.e. when reader is falling behind).
My example:
func finiteTicker(n int, d time.Duration) <-chan time.Time {
ch := make(chan time.Time, 1)
go func() {
for i := 0; i < n; i++ {
time.Sleep(d)
ch <- time.Now()
}
close(ch)
}()
return ch
}
func main() {
for range finiteTicker(10, 100*time.Millisecond) {
fmt.Println("hi")
}
}
http://play.golang.org/p/ZOwJlM8rDm
I asked on IRC as well, at got some useful insight from #Tv`.
Despite timer.Ticker looking like it should be part of a go pipeline, it does not actually play well with the pipeline idioms:
Here are the guidelines for pipeline construction:
stages close their outbound channels when all the send operations are done.
stages keep receiving values from inbound channels until those channels are closed or the senders are unblocked.
Pipelines unblock senders either by ensuring there's enough buffer for all the values that are sent or by explicitly signalling senders when the receiver may abandon the channel.
The reason for this inconsistency appears to be primarily to support the following idiom:
for {
select {
case <-ticker.C:
// do something
case <-done:
return
}
}
I don't know why this is the case, and why the pipelining idiom wasn't used:
for {
select {
case _, ok := <-ticker.C:
if ok {
// do something
} else {
return
}
}
}
(or more cleanly)
for _ = range ticker.C {
// do something
}
But this is the way go is :(

Resources