I am going to show a simple compilable code snipped where I get weird behaviour: after I intentionally cause a panic in processData (because I pass nil-pointer) the sending to channel l.done is blocked!
package main
import (
"context"
"fmt"
"time"
)
type Loop struct {
done chan struct{}
}
type Result struct {
ID string
}
func NewLoop() *Loop {
return &Loop{
done: make(chan struct{}),
}
}
func (l *Loop) StartAsync(ctx context.Context) {
go func() {
defer func() {
l.done <- struct{}{} // BLOCKED! But I allocated it in NewLoop ctor
fmt.Sprintf("done")
}()
for {
/**/
var data *Result
l.processData(ctx, data) // passed nil
}
}()
}
func (l *Loop) processData(ctx context.Context, data *Result) {
_ = fmt.Sprintf("%s", data.ID) // intentional panic - OK
}
func main() {
l := NewLoop()
l.StartAsync(context.Background())
time.Sleep(10 * time.Second)
}
I can recover() a panic before sending to channel and I get correct error message.
What does happen with channel? It it was closed, I would get panic on sending to closed channel
It's blocking because there isn't anything receiving from the channel. Both the receive & the send operations on an initialized and unbuffered channel will block indefinitely if the opposite operation is missing. I.e. a send to a channel will block until another goroutine receives from that channel, and, likewise, a receive from a channel will block until another goroutine sends to that channel.
So basically
done := new(chan struct{})
done<-struct{}{}
will block forever unless there is another goroutine that receives from the channel, i.e. <-done. That's how channels are supposed to behave. That's how the languages specification defines their behaviour.
about the possible fixes :
given the name of your channel, you may want to run close(l.done) rather than l.done <- struct{}{}.
using a buffered channel and l.done <- struct{}{} on completion : only one call to <-l.done will be unblocked.
Suppose you have some code looking like :
l := NewLoop()
go func(){
<-l.done
closeLoggers()
}()
go func(){
<-l.done
closeDatabase()
}()
sending one item on the done channel will make that only one consumer will receive it, and in the above example only one of the two actions will be triggered when the loop completes.
using close(l.done) : once a channel is closed all calls to receive from it will proceed.
In the above example, all actions will proceed.
As a side note: there is no need for a buffer if you use a channel only for its "open/closed" state.
Related
In a project the program receives data via websocket. This data needs to be processed by n algorithms. The amount of algorithms can change dynamically.
My attempt is to create some pub/sub pattern where subscriptions can be started and canceled on the fly. Turns out that this is a bit more challenging than expected.
Here's what I came up with (which is based on https://eli.thegreenplace.net/2020/pubsub-using-channels-in-go/):
package pubsub
import (
"context"
"sync"
"time"
)
type Pubsub struct {
sync.RWMutex
subs []*Subsciption
closed bool
}
func New() *Pubsub {
ps := &Pubsub{}
ps.subs = []*Subsciption{}
return ps
}
func (ps *Pubsub) Publish(msg interface{}) {
ps.RLock()
defer ps.RUnlock()
if ps.closed {
return
}
for _, sub := range ps.subs {
// ISSUE1: These goroutines apparently do not exit properly...
go func(ch chan interface{}) {
ch <- msg
}(sub.Data)
}
}
func (ps *Pubsub) Subscribe() (context.Context, *Subsciption, error) {
ps.Lock()
defer ps.Unlock()
// prep channel
ctx, cancel := context.WithCancel(context.Background())
sub := &Subsciption{
Data: make(chan interface{}, 1),
cancel: cancel,
ps: ps,
}
// prep subsciption
ps.subs = append(ps.subs, sub)
return ctx, sub, nil
}
func (ps *Pubsub) unsubscribe(s *Subsciption) bool {
ps.Lock()
defer ps.Unlock()
found := false
index := 0
for i, sub := range ps.subs {
if sub == s {
index = i
found = true
}
}
if found {
s.cancel()
ps.subs[index] = ps.subs[len(ps.subs)-1]
ps.subs = ps.subs[:len(ps.subs)-1]
// ISSUE2: close the channel async with a delay to ensure
// nothing will be written to the channel anymore
// via a pending goroutine from Publish()
go func(ch chan interface{}) {
time.Sleep(500 * time.Millisecond)
close(ch)
}(s.Data)
}
return found
}
func (ps *Pubsub) Close() {
ps.Lock()
defer ps.Unlock()
if !ps.closed {
ps.closed = true
for _, sub := range ps.subs {
sub.cancel()
// ISSUE2: close the channel async with a delay to ensure
// nothing will be written to the channel anymore
// via a pending goroutine from Publish()
go func(ch chan interface{}) {
time.Sleep(500 * time.Millisecond)
close(ch)
}(sub.Data)
}
}
}
type Subsciption struct {
Data chan interface{}
cancel func()
ps *Pubsub
}
func (s *Subsciption) Unsubscribe() {
s.ps.unsubscribe(s)
}
As mentioned in the comments there are (at least) two issues with this:
ISSUE1:
After a while of running in implementation of this I get a few errors of this kind:
goroutine 120624 [runnable]:
bm/internal/pubsub.(*Pubsub).Publish.func1(0x8586c0, 0xc00b44e880, 0xc008617740)
/home/X/Projects/bm/internal/pubsub/pubsub.go:30
created by bookmaker/internal/pubsub.(*Pubsub).Publish
/home/X/Projects/bm/internal/pubsub/pubsub.go:30 +0xbb
Without really understanding this it appears to me that the goroutines created in Publish() do accumulate/leak. Is this correct and what am I doing wrong here?
ISSUE2:
When I end a subscription via Unsubscribe() it occurs that Publish() tried to write to a closed channel and panics. To mitigate this I have created a goroutine to close the channel with a delay. This feel really off-best-practice but I was not able to find a proper solution to this. What would be a deterministic way to do this?
Heres a little playground for you to test with: https://play.golang.org/p/K-L8vLjt7_9
Before diving into your solution and its issues, let me recommend again another Broker approach presented in this answer: How to broadcast message using channel
Now on to your solution.
Whenever you launch a goroutine, always think of how it will end and make sure it does if the goroutine is not ought to run for the lifetime of your app.
// ISSUE1: These goroutines apparently do not exit properly...
go func(ch chan interface{}) {
ch <- msg
}(sub.Data)
This goroutine tries to send a value on ch. This may be a blocking operation: it will block if ch's buffer is full and there is no ready receiver on ch. This is out of the control of the launched goroutine, and also out of the control of the pubsub package. This may be fine in some cases, but this already places a burden on the users of the package. Try to avoid these. Try to create APIs that are easy to use and hard to misuse.
Also, launching a goroutine just to send a value on a channel is a waste of resources (goroutines are cheap and light, but you shouldn't spam them whenever you can).
You do it because you don't want to get blocked. To avoid blocking, you may use a buffered channel with a "reasonable" high buffer. Yes, this doesn't solve the blocking issue, in only helps with "slow" clients receiving from the channel.
To "truly" avoid blocking without launching a goroutine, you may use non-blocking send:
select {
case ch <- msg:
default:
// ch's buffer is full, we cannot deliver now
}
If send on ch can proceed, it will happen. If not, the default branch is chosen immediately. You have to decide what to do then. Is it acceptable to "lose" a message? Is it acceptable to wait for some time until "giving up"? Or is it acceptable to launch a goroutine to do this (but then you'll be back at what we're trying to fix here)? Or is it acceptable to get blocked until the client can receive from the channel...
Choosing a reasonable high buffer, if you encounter a situation when it still gets full, it may be acceptable to block until the client can advance and receive from the message. If it can't, then your whole app might be in an unacceptable state, and it might be acceptable to "hang" or "crash".
// ISSUE2: close the channel async with a delay to ensure
// nothing will be written to the channel anymore
// via a pending goroutine from Publish()
go func(ch chan interface{}) {
time.Sleep(500 * time.Millisecond)
close(ch)
}(s.Data)
Closing a channel is a signal to the receiver(s) that no more values will be sent on the channel. So always it should be the sender's job (and responsibility) to close the channel. Launching a goroutine to close the channel, you "hand" that job and responsibility to another "entity" (a goroutine) that will not be synchronized to the sender. This may easily end up in a panic (sending on a closed channel is a runtime panic, for other axioms see How does a non initialized channel behave?). Don't do that.
Yes, this was necessary because you launched goroutines to send. If you don't do that, then you may close "in-place", without launching a goroutine, because then the sender and closer will be the same entity: the Pubsub itself, whose sending and closing operations are protected by a mutex. So solving the first issue solves the second naturally.
In general if there are multiple senders for a channel, then closing the channel must be coordinated. There must be a single entity (often not any of the senders) that waits for all senders to finish, practically using a sync.WaitGroup, and then that single entity can close the channel, safely. See Closing channel of unknown length.
Why is it not sending on the channel and blocking the execution? How can I make this constellation work so that I can send a signal into MoneyDive() and continue execution?
package main
import (
"fmt"
)
type Quack func(ch chan bool)
type DagobertDuck struct {
quack Quack
}
func (self *DagobertDuck) MoneyDive() {
ch := make(chan bool)
self.quack(ch)
b := <-ch
if b {
fmt.Println("true")
} else {
fmt.Println("false")
}
}
func mockQuack(ch chan bool) {
fmt.Println("mockQuack start")
ch <- true
fmt.Println("mockQuack done")
}
func main() {
dd := DagobertDuck{quack: mockQuack}
dd.MoneyDive()
}
https://play.golang.org/p/1omlb7u6-A
Because you have an unbuffered channel, and you can only send a value on an unbuffered channel without blocking if there is another goroutine which is ready to receive from it.
Since you only have 1 goroutine, it gets blocked. Solution is simple: launch your Quack.quack() method in a new goroutine:
go self.quack(ch)
Then the output (try it on the Go Playground):
mockQuack start
mockQuack done
true
Another option is to not launch a new goroutine but make a buffered channel, so it can hold some values without any receivers ready to receive from it:
ch := make(chan bool, 1) // buffered channel, buffer for 1 value
This creates a channel which is capable of "storing" one value without any receivers ready to receive it. A second send on the channel would also block, unless the value is received from it first (or a receiver ready to receive a value from it).
Try this buffered channel version on the Go Playground.
Relevant section from the spec: Send statements:
Both the channel and the value expression are evaluated before communication begins. Communication blocks until the send can proceed. A send on an unbuffered channel can proceed if a receiver is ready. A send on a buffered channel can proceed if there is room in the buffer. A send on a closed channel proceeds by causing a run-time panic. A send on a nil channel blocks forever.
Notes:
Based on the received value you print true or false. This can be done with a single line, without the if statement:
fmt.Println(b)
You can even get rid of the b local variable, and print the received value right away:
fmt.Println(<-ch)
Also I assume you used channels because you wanted to play with them, but in your case mockQuack() could simply return the bool value, without the use of channels.
I am trying to understand why my code does not work, so I managed to recreate the problem in a simpler example.
I expected this code to output the string "broadcasted", but it does not output anything.
package main
import (
"fmt"
"time"
)
type hub struct {
handle chan []byte
broadcast chan []byte
}
func (h *hub) init() {
for {
select {
case m := <-h.handle:
handler(m)
case _ = <-h.broadcast:
fmt.Println("broadcasted")
}
}
}
var socketHub = hub{
handle: make(chan []byte),
broadcast: make(chan []byte),
}
func main() {
go socketHub.init()
m := []byte("this is the message")
socketHub.handle <- m
time.Sleep(time.Second * 2)
}
func handler(m []byte) {
// Do some stuff.
socketHub.broadcast <- m
}
Why does this not work?
Your broadcast channel is unbuffered. This means that:
You send the message to the handle channel: main goroutine blocks until...
In the goroutine, the select case gets the message...
And calls (in the goroutine) handler, which sends the message to the broadcast channel, blocking until...
And that's it: your child goroutine is blocking, waiting on itself to pick the message. Concurrently, your main goroutine sleeps, then reaches the end of main, exits and terminates the program.
You can 'solve' it in several ways:
Make your broadcast channel buffered: this way, goroutine sends the message, which succeeds immediately, and returns in the for loop, selecting it back and printing as intended
Make your send a (new) goroutine, either in handler, or by calling go handler(m) in your loop
Have two different goroutines listen to handler and broadcast
Which one you choose depends on the exact problem you try to solve: in this tiny example, it's hard to find a 'best' one.
Here is an example:
func main() {
c := make(chan int)
i := 0
go goroutine(c)
c <- i
time.Sleep(10 * time.Second)
}
func goroutine(c chan int) {
for {
num := <- c
fmt.Println(num)
num++
time.Sleep(1 * time.Second)
c <- num
}
}
What I'm trying to do inside of goroutine is to receive number from channel, print it, increment and after one second send it back to the channel. After this I want to repeat the action.
But as a result, operation is only done once.
Output:
0
Am I doing something wrong?
By default the goroutine communication is synchronous and unbuffered: sends do not complete until there is a receiver to accept the value. There must be a receiver ready to receive data from the channel and then the sender can hand it over directly to the receiver.
So channel send/receive operations block until the other side is ready:
1. A send operation on a channel blocks until a receiver is available for the same channel: if there’s no recipient for the value on ch, no other value can be put in the channel. And the other way around: no new value can be sent in ch when the channel is not empty! So the send operation will wait until ch becomes available again.
2. A receive operation for a channel blocks until a sender is available for the same channel: if there is no value in the channel, the receiver blocks.
This is illustrated in the below example:
package main
import "fmt"
func main() {
ch1 := make(chan int)
go pump(ch1) // pump hangs
fmt.Println(<-ch1) // prints only 0
}
func pump(ch chan int) {
for i:= 0; ; i++ {
ch <- i
}
}
Because there is no receiver the goroutine hangs and print only the first number.
To workaround this we need to define a new goroutine which reads from the channel in an infinite loop.
func receive(ch chan int) {
for {
fmt.Println(<- ch)
}
}
Then in main():
func main() {
ch := make(chan int)
go pump(ch)
go receive(ch)
}
Go Playground
You create an unbuffered channel c with
c := make(chan int)
On unbuffered channels, operations are symmentrical, i.e. every send on a channel needs exactly one receive, every receive needs exactly one send. You send your i into the channel, the goroutine receives it into num. After that, the goroutine sends the incremented num into the channel, but nobody is there to receive it.
In short: The statement
c <- num
will block.
You could use a 1-buffered channel, that should work.
There is another problem with your code that you solved by waiting ten seconds in main: You don't know when your goroutine finishes. Typically, a sync.WaitGroup is used in these situations. But: Your goroutine does not finish. It's idiomatic to introduce a chan struct{} that you close in your main goroutine after a while and select over both channels in the worker goroutine.
You use unbuffered channel, so your goroutine hang on c <- num
You should use buffered channel, like this: c := make(chan int, 1)
Try it on the Go playground
I made two attempts to pass a channel to a function as a parameter, but they both fail (deadlock):
Attempt 1:
func done(signal *chan bool) {
*signal <- true
}
func main() {
signal := make(chan bool)
done(&signal)
<-signal
fmt.Println("completed")
}
Attempt 2:
func done(signal chan bool) {
signal <- true
}
func main() {
signal := make(chan bool)
done(signal)
<-signal
fmt.Println("completed")
}
Well I am out of ideas. What should be the proper way to pass the channel to the function?
A channel is unbuffered by default, so when you send on a channel, that send will block
until someone receives the value. This is clearly the case in your code here, at the point where you you're doing signal <- true , there's noone that can or ever will receive on the same channel.
You can create a goroutine that does the sending, that way execution in main() continues and there will be someone that's receiving the value:
go done(signal)
Or you can create a buffered channel, so sending on a channel doesn't block if there's room for the value in the channel:
signal := make(chan bool, 1)
done(&signal) is called synchronously. Maybe what you wanted to do is to call it asynchronously?
to do so, put the keyword go in front of the function call
go done(&signal)
The main thread will block until the done function writes to the channel. And the done method will block on writing to the channel until the main thread reads the channel.