Is there a way to update unread data that been sent to a channel with more up to date data?
I have a goroutine (producer) with a channel that's provides progress updates to another goroutine (consumer). In some scenarios, the progress can update much faster than the consumer consumes the update messages.
This causes me issues as I can either:
Block on sending data to the channel. This means that if the consumer is slow to read data, the progress updating goroutine totally blocks - which it shouldn't.
Don't block on sending and skip over progress updates when the channel is full. This means the consumer is always reading old, out of data data.
As an example, I might have something like this:
Progress reporting goroutine: Posts "1%" to channel
Progress reporting goroutine: Posts "2%" to channel
Progress reporting goroutine: Posts "3%" to channel
Progress consuming goroutine: Reads "1%", "2%" and "3%" from channel. "1% and "2%" are outdated information.
Is there any way to update unread channel data? Or is there a better way of going about this issue?
You can store some value in a global variable protected with RWMutex It keeps progress. Generator updates it. Consumer reads and shows.
Also you can make a non-blocking writing to channel with length of 1:
var c = make(chan struct{}, 1)
select {
case c <- struct{}{}:
default:
}
This way sender either adds one element to the channel either do nothing if it’s full.
Reader treats this empty struct as a signal - it should take updated value in the global variable.
Another way: Updatable channel
var c = make(chan int, 1)
select {
case c <- value: // channel was empty - ok
default: // channel if full - we have to delete a value from it with some precautions to not get locked in our own channel
select {
case <- c: // read stale value and put a fresh one
c <- value
default: // consumer have read it - so skip and not get locked
}
}
package main
import "fmt"
func main() {
// channel buffer must be 1 in this case
var ch = make(chan int, 1)
// when channel was not consumed and you want to update value at channel
produceToChannel(ch, 1)
produceToChannel(ch, 2)
fmt.Println(consumeFromChannel(ch)) // prints 2
// when channel was already consumed and you are producing new value
produceToChannel(ch, 3)
consumeFromChannel(ch)
produceToChannel(ch, 4)
fmt.Println(consumeFromChannel(ch)) // prints 4
}
func produceToChannel(ch chan int, v int) {
select {
case <-ch:
default:
}
ch <- v
}
func consumeFromChannel(ch chan int) int {
return <- ch
}
Just clear the channel each time before sending value to the channel.
In other words, when you send 3%, 3% becomes the only value in channel.
You can make your channel with buffer length of 1, so simply use <-ch to clear the channel.
Edit: use a select with default to clear the channel with <-ch and so do not block in the case the previous value is already read.
How about a concurrent map that stores versions for all your inbound objects, with 0 say being the default version
import "sync"
var Versions sync.Map = sync.Map{}
type Data struct {
Payload interface{}
Version int
ID int
}
func produce(c chan Data) {
for {
data := produceData()
if hasNewVersion(data) {
Versions.Store(data.ID, data.Version)
}
c <- data
}
}
func consume(c chan Data) {
for {
select {
case val:= <- c:
if ver, ok := Versions.Load(val.ID); ok {
if ver.(int) == val.Version {
// process
}
}
}
}
}
Related
type Reader struct {
sync.RWMutex
logger *zerolog.Logger
wg *sync.WaitGroup
ticker *time.Ticker
messages chan *TransactionData
}
type TransactionData struct {
ID string
messages []*Message
}
func NewReader(logger zerolog.Logger) *Reader {
return &Reader{
wg: &sync.WaitGroup{},
logger: &l,
ticker: time.NewTicker(1 * time.Second),
messages: make(chan *TransactionData),
}
}
func (r *Reader) Write(ID string, messages []*Message) error {
r.wg.Add(1)
....
go func(ID string, messages []*domain.Message) {
defer r.wg.Done()
r.add(&TransactionData{
chatID: ID,
messages: messages,
})
}(ID, messages)
return nil
}
func (r *Reader) add(data *TransactionData) {
r.Lock()
defer r.Unlock()
r.messages <- data
}
func (r *Reader) get() {
r.Lock()
defer func() {
r.Unlock()
}()
for {
select {
case r.ticker.C:
// how to select all the data from the channel?
}
}
}
I'm exploring channels right now. And with the help of channels I'm trying to solve one problem.
Task: user reads chat history, selects all messages that user read, put these messages into channel, second channel selects data and creates transaction, through goroutine I add data into channel using Write method. I want to create a second goroutine which every second (using ticker.C) will select all the data from the channel and form a transaction. But I can't figure out how I can select all the data from the channel that I've managed to add there, example:
user-1: Write(&TransactionData{...}),
user-2: Write(&TransactionData{...}),
....
user-n: Write(&TransactionData{...})
How do I select absolutely all data and do I need to select all data or do I need to select one at a time?
Brief description of the solution:
Read from the channel in an infinite loop using goroutine
Put the subtracted tasks into the sync.Map thread-safe place
Run a second goroutine with time.Ticker.C and when it is triggered, select data for the transaction.
First question: How do I select all the data from the channel so that I collect one transaction and can update more data, provided that, I don't use any keys?(I attached an outline of the read method)
Second question: Should I get ALL the data from the channel or should I get only one TransactionData at a time and form one transaction?
There are several problems with your approach and implementation.
First: you don't need to lock a mutex to send to a channel.
Second: you don't need to lock a mutex to receive from a channel.
As for the implementation: since the channel is unbuffered, channel send will only work when there is a listening goroutine. If you receive from a channel when a ticker hits, then all the sends will happen after that ticker hits. Send operations will block until then.
So, you should get rid of that ticker, and read from the channel in a for-loop. When all the writes are completed, you should close the channel, which will terminate the for-loop, so you can write. Closing the channel is the idiomatic way of signaling to the receiver that all the data is sent.
I'm a fresh to golang, here is my purpose, I wanna get 2 routines running concurrently with a common channel, consumer should be start after channel created and always get the data until channel closed, my code template is as below:
var userChannel chan string
for index := 0; index < *clientNums; index++ {
wg.Add(1)
go run1()
go run2()
}
wg.Wait()
}
func run1() {
defer wg.Done()
// ...some logic
userChannel = make(chan string, *readUserNums)
for index := 0; index < *readUserNums; index++ {
//...some logic
userChannel <- userId
//...some logic
}
close(userChannel)
}
func run2() {
for sendId := range userChannel {
//...some logic
}
}
in my code, if run2 run first it will be panic as channel haven't been create and no data in channel. how can I achieve my purpose? thank you
Create the channel first, and pass it into your goroutines, instead of storing it in a global and creating it on the fly.
Even if no data is present on userChannel initially, it won't be a issue provided userChannel is created, here it is failing because the channel is not created.
It is always a best practice to create the channel before calling run1 or run2 and pass the channel.
FYI: If channel is created by if no data is present on the channel and if we try to read data from the channel it will be a blocking call and will wait until a data is received into the waiting channel.
I'm trying to create message hub in golang. Messages are getting through different channels that persist in map[uint32]chan []float64. I do an endless loop over map and check if a channel has a message. If it has, I write it to the common client's write channel together with an incoming channel's id. It works fine, but uses all CPU, and other processes are throttled.
UPD: Items in map adding and removing dynamically by another function.
I thinking to limit CPU for this app through Docker, but maybe there is more elegant path?
My code :
func (c *Client) accumHandler() {
for !c.stop {
c.channels.Range(func(key, value interface{}) bool {
select {
case message := <-value.(chan []float64):
mess := map[uint32]interface{}{key.(uint32): message}
select {
case c.send <- mess:
}
default:
}
return !c.stop
})
}
}
If I'm reading the cards correctly, it seems like you are trying to pass along an array of floats to a common channel along with a channel identifier. I assume that you are doing this to pass multiple channels out to different publishers, but so that you only have to track a single channel for your consumer.
It turns out that you don't need to loop over channels to see when it's outputting a value. You can chain channels together inside of goroutines. For this reason, no busy wait is necessary. Something like this will suit your purposes (again, if I'm reading the cards correctly). Look for the all caps comment for the way around your busy loop. Link to playground.
var num_publishes = 3
func main() {
num_publishers := 10
single_consumer := make(chan []float64)
for i:=0;i<num_publishers;i+=1 {
c := make(chan []float64)
// connect channel to your single consumer channel
go func() { for { single_consumer <- <-c } }() // THIS IS PROBABLY WHAT YOU DIDN'T KNOW ABOUT
// send the channel to the publisher
go publisher(c, i*100)
}
// dumb consumer example
for i:=0;i<num_publishers*num_publishes;i+=1 {
fmt.Println(<-single_consumer)
}
}
func publisher(c chan []float64, publisher_id int) {
dummy := []float64{
float64(publisher_id+1),
float64(publisher_id+2),
float64(publisher_id+3),
}
for i:=0;i<num_publishes;i+=1 {
time.Sleep(time.Duration(rand.Intn(10000)) * time.Millisecond)
c <- dummy
}
}
It is eating all of your CPU because you are continuously cycling around the dictionary checking for messages, so even when there are no messages to process the CPU, or at least a thread or core, is running flat out. You need blocking sends and receives on the channels!
I assume you are doing this because you don't know how many channels there will be and therefor can't just select on all of the input channels. A better pattern would be to start a separate goroutine for each input channel you are currently storing in the dictionary. Each goroutine should have a loop in which it blocks waiting the input channel and on receiving a message does a blocking send to a channel to the client which is shared by all.
The question isn't complete so can't give exact code, but you're going to have goroutines that look something like this:
type Message struct {
id uint32,
message []float64
}
func receiverGoroutine(id uint32, input chan []float64, output chan Message) {
for {
message := <- input
output <- Message{id: id, message: message}
}
}
func clientGoroutine(c *Client, input chan Message) {
for {
message := <- input
// do stuff
}
}
(You'll need to add some "done" channels as well though)
Elsewhere you will start them with code like this:
clientChan := make(chan Message)
go clientGoroutine(client, clientChan)
for i:=0; i<max; i++ {
go receiverGoroutine( (uint32)i, make(chan []float64, clientChan)
}
Or you can just start the client routine and then add the others as they are needed rather than in a loop up front - depends on your use case.
I am new to go and I am trying to create a simple chat server where clients can broadcast messages to all connected clients.
In my server, I have a goroutine (infinite for loop) that accepts connection and all the connections are received by a channel.
go func() {
for {
conn, _ := listener.Accept()
ch <- conn
}
}()
Then, I start a handler (goroutine) for every connected client. Inside the handler, I try to broadcast to all connections by iterating through the channel.
for c := range ch {
conn.Write(msg)
}
However, I cannot broadcast because (I think from reading the docs) the channel needs to be closed before iterating. I am not sure when I should close the channel because I want to continuously accept new connections and closing the channel won't let me do that. If anyone can help me, or provide a better way to broadcast messages to all connected clients, it would be appreciated.
What you are doing is a fan out pattern, that is to say, multiple endpoints are listening to a single input source. The result of this pattern is, only one of these listeners will be able to get the message whenever there's a message in the input source. The only exception is a close of channel. This close will be recognized by all of the listeners, and thus a "broadcast".
But what you want to do is broadcasting a message read from connection, so we could do something like this:
When the number of listeners is known
Let each worker listen to dedicated broadcast channel, and dispatch the message from the main channel to each dedicated broadcast channel.
type worker struct {
source chan interface{}
quit chan struct{}
}
func (w *worker) Start() {
w.source = make(chan interface{}, 10) // some buffer size to avoid blocking
go func() {
for {
select {
case msg := <-w.source
// do something with msg
case <-quit: // will explain this in the last section
return
}
}
}()
}
And then we could have a bunch of workers:
workers := []*worker{&worker{}, &worker{}}
for _, worker := range workers { worker.Start() }
Then start our listener:
go func() {
for {
conn, _ := listener.Accept()
ch <- conn
}
}()
And a dispatcher:
go func() {
for {
msg := <- ch
for _, worker := workers {
worker.source <- msg
}
}
}()
When the number of listeners is not known
In this case, the solution given above still works. The only difference is, whenever you need a new worker, you need to create a new worker, start it up, and then push it into workers slice. But this method requires a thread-safe slice, which need a lock around it. One of the implementation may look like as follows:
type threadSafeSlice struct {
sync.Mutex
workers []*worker
}
func (slice *threadSafeSlice) Push(w *worker) {
slice.Lock()
defer slice.Unlock()
workers = append(workers, w)
}
func (slice *threadSafeSlice) Iter(routine func(*worker)) {
slice.Lock()
defer slice.Unlock()
for _, worker := range workers {
routine(worker)
}
}
Whenever you want to start a worker:
w := &worker{}
w.Start()
threadSafeSlice.Push(w)
And your dispatcher will be changed to:
go func() {
for {
msg := <- ch
threadSafeSlice.Iter(func(w *worker) { w.source <- msg })
}
}()
Last words: never leave a dangling goroutine
One of the good practices is: never leave a dangling goroutine. So when you finished listening, you need to close all of the goroutines you fired. This will be done via quit channel in worker:
First we need to create a global quit signalling channel:
globalQuit := make(chan struct{})
And whenever we create a worker, we assign the globalQuit channel to it as its quit signal:
worker.quit = globalQuit
Then when we want to shutdown all workers, we simply do:
close(globalQuit)
Since close will be recognized by all listening goroutines (this is the point you understood), all goroutines will be returned. Remember to close your dispatcher routine as well, but I will leave it to you :)
A more elegant solution is a "broker", where clients may subscribe and unsubscribe to messages.
To also handle subscribing and unsubscribing elegantly, we may utilize channels for this, so the main loop of the broker which receives and distributes the messages can incorporate all these using a single select statement, and synchronization is given from the solution's nature.
Another trick is to store the subscribers in a map, mapping from the channel we use to distribute messages to them. So use the channel as the key in the map, and then adding and removing the clients is "dead" simple. This is made possible because channel values are comparable, and their comparison is very efficient as channel values are simple pointers to channel descriptors.
Without further ado, here's a simple broker implementation:
type Broker[T any] struct {
stopCh chan struct{}
publishCh chan T
subCh chan chan T
unsubCh chan chan T
}
func NewBroker[T any]() *Broker[T] {
return &Broker[T]{
stopCh: make(chan struct{}),
publishCh: make(chan T, 1),
subCh: make(chan chan T, 1),
unsubCh: make(chan chan T, 1),
}
}
func (b *Broker[T]) Start() {
subs := map[chan T]struct{}{}
for {
select {
case <-b.stopCh:
return
case msgCh := <-b.subCh:
subs[msgCh] = struct{}{}
case msgCh := <-b.unsubCh:
delete(subs, msgCh)
case msg := <-b.publishCh:
for msgCh := range subs {
// msgCh is buffered, use non-blocking send to protect the broker:
select {
case msgCh <- msg:
default:
}
}
}
}
}
func (b *Broker[T]) Stop() {
close(b.stopCh)
}
func (b *Broker[T]) Subscribe() chan T {
msgCh := make(chan T, 5)
b.subCh <- msgCh
return msgCh
}
func (b *Broker[T]) Unsubscribe(msgCh chan T) {
b.unsubCh <- msgCh
}
func (b *Broker[T]) Publish(msg T) {
b.publishCh <- msg
}
Example using it:
func main() {
// Create and start a broker:
b := NewBroker[string]()
go b.Start()
// Create and subscribe 3 clients:
clientFunc := func(id int) {
msgCh := b.Subscribe()
for {
fmt.Printf("Client %d got message: %v\n", id, <-msgCh)
}
}
for i := 0; i < 3; i++ {
go clientFunc(i)
}
// Start publishing messages:
go func() {
for msgId := 0; ; msgId++ {
b.Publish(fmt.Sprintf("msg#%d", msgId))
time.Sleep(300 * time.Millisecond)
}
}()
time.Sleep(time.Second)
}
Output of the above will be (try it on the Go Playground):
Client 2 got message: msg#0
Client 0 got message: msg#0
Client 1 got message: msg#0
Client 2 got message: msg#1
Client 0 got message: msg#1
Client 1 got message: msg#1
Client 1 got message: msg#2
Client 2 got message: msg#2
Client 0 got message: msg#2
Client 2 got message: msg#3
Client 0 got message: msg#3
Client 1 got message: msg#3
Improvements
You may consider the following improvements. These may or may not be useful depending on how / to what you use the broker.
Broker.Unsubscribe() may close the message channel, signalling that no more messages will be sent on it:
func (b *Broker[T]) Unsubscribe(msgCh chan T) {
b.unsubCh <- msgCh
close(msgCh)
}
This would allow clients to range over the message channel, like this:
msgCh := b.Subscribe()
for msg := range msgCh {
fmt.Printf("Client %d got message: %v\n", id, msg)
}
Then if someone unsubscribes this msgCh like this:
b.Unsubscribe(msgCh)
The above range loop will terminate after processing all messages that were sent before the call to Unsubscribe().
If you want your clients to rely on the message channel being closed, and the broker's lifetime is narrower than your app's lifetime, then you could also close all subscribed clients when the broker is stopped, in the Start() method like this:
case <-b.stopCh:
for msgCh := range subs {
close(msgCh)
}
return
Broadcast to a slice of channel and use sync.Mutex to manage channel add and remove may be the easiest way in your case.
Here is what you can do to broadcast in golang:
You can broadcast a share status change with sync.Cond. This way do not have any alloc once setup, but you can not add timeout functional or work with another channel.
You can broadcast a share status change with a close old channel and create new channel and sync.Mutex. This way have one alloc per status change, but you can add timeout functional and work with another channel.
You can broadcast to a slice of function callback and use sync.Mutex to manage them. The caller can do channel stuff. This way have more than one alloc per caller, and work with another channel.
You can broadcast to a slice of channel and use sync.Mutex to manage them. This way have more than one alloc per caller, and work with another channel.
You can broadcast to a slice of sync.WaitGroup and use sync.Mutex to manage them.
This is a late answer but I think it may appease some curious readers.
Go channels are widely welcomed to be used when it comes to concurrency.
Go community is rigid to follow this saying:
Do not communicate by sharing memory; instead, share memory by communicating.
I am completely neutral toward this and I think other options rather than well-defined channels should be considered when it comes to broadcasting.
Here is my take: Cond from sync packages are widely overlooked. Implementing braodcaster as suggested by Bronze man in very same context worths noting.
I was delighted witch icza suggestion to use channels and broadcast messages over them. I follow the same methods and use sync's conditional variable:
// Broadcaster is the struct which encompasses broadcasting
type Broadcaster struct {
cond *sync.Cond
subscribers map[interface{}]func(interface{})
message interface{}
running bool
}
this is the main struct that our whole broadcasting concept relies on.
Below, I define some behaviours for this struct. In a nutshell, subscribers should be able to be added, removed and whole the process should be revokable.
// SetupBroadcaster gives the broadcaster object to be used further in messaging
func SetupBroadcaster() *Broadcaster {
return &Broadcaster{
cond: sync.NewCond(&sync.RWMutex{}),
subscribers: map[interface{}]func(interface{}){},
}
}
// Subscribe let others enroll in broadcast event!
func (b *Broadcaster) Subscribe(id interface{}, f func(input interface{})) {
b.subscribers[id] = f
}
// Unsubscribe stop receiving broadcasting
func (b *Broadcaster) Unsubscribe(id interface{}) {
b.cond.L.Lock()
delete(b.subscribers, id)
b.cond.L.Unlock()
}
// Publish publishes the message
func (b *Broadcaster) Publish(message interface{}) {
go func() {
b.cond.L.Lock()
b.message = message
b.cond.Broadcast()
b.cond.L.Unlock()
}()
}
// Start the main broadcasting event
func (b *Broadcaster) Start() {
b.running = true
for b.running {
b.cond.L.Lock()
b.cond.Wait()
go func() {
for _, f := range b.subscribers {
f(b.message) // publishes the message
}
}()
b.cond.L.Unlock()
}
}
// Stop broadcasting event
func (b *Broadcaster) Stop() {
b.running = false
}
Next, I can use it quite easily:
messageToaster := func(message interface{}) {
fmt.Printf("[New Message]: %v\n", message)
}
unwillingReceiver := func(message interface{}) {
fmt.Println("Do not disturb!")
}
broadcaster := SetupBroadcaster()
broadcaster.Subscribe(1, messageToaster)
broadcaster.Subscribe(2, messageToaster)
broadcaster.Subscribe(3, unwillingReceiver)
go broadcaster.Start()
broadcaster.Publish("Hello!")
time.Sleep(time.Second)
broadcaster.Unsubscribe(3)
broadcaster.Publish("Goodbye!")
It should print something like this in any order:
[New Message]: Hello!
Do not disturb!
[New Message]: Hello!
[New Message]: Goodbye!
[New Message]: Goodbye!
See this on go playground
another one simple example:
https://play.golang.org
type Broadcaster struct {
mu sync.Mutex
clients map[int64]chan struct{}
}
func NewBroadcaster() *Broadcaster {
return &Broadcaster{
clients: make(map[int64]chan struct{}),
}
}
func (b *Broadcaster) Subscribe(id int64) (<-chan struct{}, error) {
defer b.mu.Unlock()
b.mu.Lock()
s := make(chan struct{}, 1)
if _, ok := b.clients[id]; ok {
return nil, fmt.Errorf("signal %d already exist", id)
}
b.clients[id] = s
return b.clients[id], nil
}
func (b *Broadcaster) Unsubscribe(id int64) {
defer b.mu.Unlock()
b.mu.Lock()
if _, ok := b.clients[id]; ok {
close(b.clients[id])
}
delete(b.clients, id)
}
func (b *Broadcaster) broadcast() {
defer b.mu.Unlock()
b.mu.Lock()
for k := range b.clients {
if len(b.clients[k]) == 0 {
b.clients[k] <- struct{}{}
}
}
}
type testClient struct {
name string
signal <-chan struct{}
signalID int64
brd *Broadcaster
}
func (c *testClient) doWork() {
i := 0
for range c.signal {
fmt.Println(c.name, "do work", i)
if i > 2 {
c.brd.Unsubscribe(c.signalID)
fmt.Println(c.name, "unsubscribed")
}
i++
}
fmt.Println(c.name, "done")
}
func main() {
var err error
brd := NewBroadcaster()
clients := make([]*testClient, 0)
for i := 0; i < 3; i++ {
c := &testClient{
name: fmt.Sprint("client:", i),
signalID: time.Now().UnixNano()+int64(i), // +int64(i) for play.golang.org
brd: brd,
}
c.signal, err = brd.Subscribe(c.signalID)
if err != nil {
log.Fatal(err)
}
clients = append(clients, c)
}
for i := 0; i < len(clients); i++ {
go clients[i].doWork()
}
for i := 0; i < 6; i++ {
brd.broadcast()
time.Sleep(time.Second)
}
}
output:
client:0 do work 0
client:2 do work 0
client:1 do work 0
client:2 do work 1
client:0 do work 1
client:1 do work 1
client:2 do work 2
client:0 do work 2
client:1 do work 2
client:2 do work 3
client:2 unsubscribed
client:2 done
client:0 do work 3
client:0 unsubscribed
client:0 done
client:1 do work 3
client:1 unsubscribed
client:1 done
Because Go channels follow the Communicating Sequential Processes (CSP) pattern, channels are a point-to-point communication entity. There is always one writer and one reader involved in each exchange.
However, each channel end can be shared amongst multiple goroutines. This is safe to do - there is no dangerous race condition.
So there can be multiple writers sharing the writing end. And/or there can be multiple readers sharing the reading end. I wrote more on this in a different answer, which includes examples.
If you really need a broadcast, you cannot do this directly, but it is not hard to implement an intermediate goroutine that copies a value out to each of a group of output channels.
The canonical (and idiomatic go) way to do this is via a slice of channels, as recommended above by Nevets and icza.
You should specifically not use a slice of callbacks. In some languages, you do typically register observers by passing a callback, but in those cases, you have to wrap their invocation in a fair amount of defensive code to protect the sender, and ideally you should have the generator of the message (the "Subject" in classic Observer pattern discussion) segregated from the observers by an intermediate message transport layer. This is where you typically use a pub-sub mesh (JMS brokers, gnats, MQ, whatever) when you're crossing process boundaries, but you should adhere to the same pattern if both subject and observers are internal to the same process (and most languages have available implementations of such mechanisms, so you shouldn't need to roll your own).
The reasons not to use callbacks include:
Unless you build in your own message transport layer, your subject is no longer both naive (it doesn't know the nature or cardinality of the observers) and disinterested (it doesn't care what they do with the message, only that it is made available to any interested parties);
If you want true broadcasting, then you need to act as if the order of receipt does not matter - ideally, everyone can see the message at the same time, even though in practice sending is iterative, even when using channels. But sending to recipient n+1 should absolutely not depend on confirmation of receipt by recipient n. That isn't broadcasting, it's serialized assignment. I say assignment because, if you are asking for a callback, then in executing the callback, you are enforcing (even if only minimally) some behavior to be taken by the recipient. You've basically turned your sender into an orchestrator, which is a very different sort of pattern with a different set of use cases.
Absent a defensive boundary (wrapping each callback invocation in a separate goroutine with a timeout context, e.g.), you are vulnerable to being blocked by a recipient - this is antithetical to broadcasting. Receipt (and optionally, taking any action at all based on) a broadcast message must be entirely asynchronous with respect to the original sending.
Is it doable to provide pseudo-broadcasting by using callbacks in go? Sure, but you have to invest in so much additional complexity to keep things clean - and why would you do that when go provides an easy and rather robust way to do it? The examples of channel-driven broadcasting above are good ones and how you should do it pretty much every time.
The specific exception when you absolutely should use callbacks is when you are not disinterested - you really do care that, on the basis of the sent message, the recipients take some action (and usually something specified by contract). For example, "I am about to unmount this filesystem, so flush and close your filehandles, let me know once you're done." (I know that's a pretty old-fashioned example, but it's the first one that comes to mind.)
Short story:
I'm having an issue where a map that previously had data but should now be empty is reporting a len() of > 0 even though it appears to be empty, and I have no idea why.
Longer story:
I need to process a number of devices at a time. Each device can have a number of messages. The concurrency of Go seemed like an obvious place to begin, so I wrote up some code to handle it and it seems to be going mostly very well. However...
I started a single goroutine for each device. In the main() function I have a map that contains each of the devices. When a message comes in I check to see whether the device already exists and if not I create it, store it in the map, and then pass the message into the device's receiving buffered channel.
This works great, and each device is being processed nicely. However, I need the device (and its goroutine) to terminate when it doesn't receive any messages for a preset amount of time. I've done this by checking in the goroutine itself how much time has passed since the last message was received, and if the goroutine is considered stale then the receiving channel is closed. But how to remove from the map?
So I passed in a pointer to the map, and I have the goroutine delete the device from the map and close the receiving channel before returning. The problem though is that at the end I'm finding that the len() function returns a value > 0, but when I output the map itself I see that it's empty.
I've written up a toy example to try to replicate the fault, and indeed I'm seeing that len() is reporting > 0 when the map is apparently empty. The last time I tried it I saw 10. The time before that 14. The time before that one, 53.
So I can replicate the fault, but I'm not sure whether the fault is with me or with Go. How is len() reporting > 0 when there are apparently no items in it?
Here's an example of how I've been able to replicate. I'm using Go v1.5.1 windows/amd64
There are two things here, as far as I'm concerned:
Am I managing the goroutines properly (probably not) and
Why does len(m) report > 0 when there are no items in it?
Thanks all
Example Code:
package main
import (
"log"
"os"
"time"
)
const (
chBuffSize = 100 // How large the thing's channel buffer should be
thingIdleLifetime = time.Second * 5 // How long things can live for when idle
thingsToMake = 1000 // How many things and associated goroutines to make
thingMessageCount = 10 // How many messages to send to the thing
)
// The thing that we'll be passing into a goroutine to process -----------------
type thing struct {
id string
ch chan bool
}
// Go go gadget map test -------------------------------------------------------
func main() {
// Make all of the things!
things := make(map[string]thing)
for i := 0; i < thingsToMake; i++ {
t := thing{
id: string(i),
ch: make(chan bool, chBuffSize),
}
things[t.id] = t
// Pass the thing into it's own goroutine
go doSomething(t, &things)
// Send (thingMessageCount) messages to the thing
go func(t thing) {
for x := 0; x < thingMessageCount; x++ {
t.ch <- true
}
}(t)
}
// Check the map of things to see whether we're empty or not
size := 0
for {
if size == len(things) && size != thingsToMake {
log.Println("Same number of items in map as last time")
log.Println(things)
os.Exit(1)
}
size = len(things)
log.Printf("Map size: %d\n", size)
time.Sleep(time.Second)
}
}
// Func for each goroutine to run ----------------------------------------------
//
// Takes two arguments:
// 1) the thing that it is working with
// 2) a pointer to the map of things
//
// When this goroutine is ready to terminate, it should remove the associated
// thing from the map of things to clean up after itself
func doSomething(t thing, things *map[string]thing) {
lastAccessed := time.Now()
for {
select {
case <-t.ch:
// We received a message, so extend the lastAccessed time
lastAccessed = time.Now()
default:
// We haven't received a message, so check if we're allowed to continue
n := time.Now()
d := n.Sub(lastAccessed)
if d > thingIdleLifetime {
// We've run for >thingIdleLifetime, so close the channel, delete the
// associated thing from the map and return, terminating the goroutine
close(t.ch)
delete(*things, string(t.id))
return
}
}
// Just sleep for a second in each loop to prevent the CPU being eaten up
time.Sleep(time.Second)
}
}
Just to add; in my original code this is looping forever. The program is designed to listen for TCP connections and receive and process the data, so the function that is checking the map count is running in it's own goroutine. However, this example has exactly the same symptom even though the map len() check is in the main() function and it is designed to handle an initial burst of data and then break out of the loop.
UPDATE 2015/11/23 15:56 UTC
I've refactored my example below. I'm not sure if I've misunderstood #RobNapier or not but this works much better. However, if I change thingsToMake to a larger number, say 100000, then I get lots of errors like this:
goroutine 199734 [select]:
main.doSomething(0xc0d62e7680, 0x4, 0xc0d64efba0, 0xc082016240)
C:/Users/anttheknee/go/src/maptest/maptest.go:83 +0x144
created by main.main
C:/Users/anttheknee/go/src/maptest/maptest.go:46 +0x463
I'm not sure if the problem is that I'm asking Go to do too much, or if I've made a hash of understanding the solution. Any thoughts?
package main
import (
"log"
"os"
"time"
)
const (
chBuffSize = 100 // How large the thing's channel buffer should be
thingIdleLifetime = time.Second * 5 // How long things can live for when idle
thingsToMake = 10000 // How many things and associated goroutines to make
thingMessageCount = 10 // How many messages to send to the thing
)
// The thing that we'll be passing into a goroutine to process -----------------
type thing struct {
id string
ch chan bool
done chan string
}
// Go go gadget map test -------------------------------------------------------
func main() {
// Make all of the things!
things := make(map[string]thing)
// Make a channel to receive completion notification on
doneCh := make(chan string, chBuffSize)
log.Printf("Making %d things\n", thingsToMake)
for i := 0; i < thingsToMake; i++ {
t := thing{
id: string(i),
ch: make(chan bool, chBuffSize),
done: doneCh,
}
things[t.id] = t
// Pass the thing into it's own goroutine
go doSomething(t)
// Send (thingMessageCount) messages to the thing
go func(t thing) {
for x := 0; x < thingMessageCount; x++ {
t.ch <- true
time.Sleep(time.Millisecond * 10)
}
}(t)
}
log.Printf("All %d things made\n", thingsToMake)
// Receive on doneCh when the goroutine is complete and clean the map up
for {
id := <-doneCh
close(things[id].ch)
delete(things, id)
if len(things) == 0 {
log.Printf("Map: %v", things)
log.Println("All done. Exiting")
os.Exit(0)
}
}
}
// Func for each goroutine to run ----------------------------------------------
//
// Takes two arguments:
// 1) the thing that it is working with
// 2) the channel to report that we're done through
//
// When this goroutine is ready to terminate, it should remove the associated
// thing from the map of things to clean up after itself
func doSomething(t thing) {
timer := time.NewTimer(thingIdleLifetime)
for {
select {
case <-t.ch:
// We received a message, so extend the timer
timer.Reset(thingIdleLifetime)
case <-timer.C:
// Timer returned so we need to exit now
t.done <- t.id
return
}
}
}
UPDATE 2015/11/23 16:41 UTC
The completed code that appears to be working properly. Do feel free to let me know if there are any improvements that could be made, but this works (sleeps are deliberate to see progress as it's otherwise too fast!)
package main
import (
"log"
"os"
"strconv"
"time"
)
const (
chBuffSize = 100 // How large the thing's channel buffer should be
thingIdleLifetime = time.Second * 5 // How long things can live for when idle
thingsToMake = 100000 // How many things and associated goroutines to make
thingMessageCount = 10 // How many messages to send to the thing
)
// The thing that we'll be passing into a goroutine to process -----------------
type thing struct {
id string
receiver chan bool
done chan string
}
// Go go gadget map test -------------------------------------------------------
func main() {
// Make all of the things!
things := make(map[string]thing)
// Make a channel to receive completion notification on
doneCh := make(chan string, chBuffSize)
log.Printf("Making %d things\n", thingsToMake)
for i := 0; i < thingsToMake; i++ {
t := thing{
id: strconv.Itoa(i),
receiver: make(chan bool, chBuffSize),
done: doneCh,
}
things[t.id] = t
// Pass the thing into it's own goroutine
go doSomething(t)
// Send (thingMessageCount) messages to the thing
go func(t thing) {
for x := 0; x < thingMessageCount; x++ {
t.receiver <- true
time.Sleep(time.Millisecond * 100)
}
}(t)
}
log.Printf("All %d things made\n", thingsToMake)
// Check the `len()` of things every second and exit when empty
go func() {
for {
time.Sleep(time.Second)
m := things
log.Printf("Map length: %v", len(m))
if len(m) == 0 {
log.Printf("Confirming empty map: %v", things)
log.Println("All done. Exiting")
os.Exit(0)
}
}
}()
// Receive on doneCh when the goroutine is complete and clean the map up
for {
id := <-doneCh
close(things[id].receiver)
delete(things, id)
}
}
// Func for each goroutine to run ----------------------------------------------
//
// When this goroutine is ready to terminate it should respond through t.done to
// notify the caller that it has finished and can be cleaned up. It will wait
// for `thingIdleLifetime` until it times out and terminates on it's own
func doSomething(t thing) {
timer := time.NewTimer(thingIdleLifetime)
for {
select {
case <-t.receiver:
// We received a message, so extend the timer
timer.Reset(thingIdleLifetime)
case <-timer.C:
// Timer expired so we need to exit now
t.done <- t.id
return
}
}
}
map is not thread-safe. You cannot access a map on multiple goroutines safely. You can corrupt the map, as you're seeing in this case.
Rather than allow the goroutine to modify the map, the goroutine should write their identifier to a channel before returning. The main loop should watch that channel, and when an identifier comes back, should remove that element from the map.
You'll probably want to read up on Go concurrency patterns. In particular, you may want to look at Fan-out/Fan-in. Look at the links at the bottom. The Go blog has a lot of information on concurrency.
Note that your goroutine is busy waiting to check for timeout. There's no reason for that. The fact that you "sleep(1 second)") should be a clue that there's a mistake. Instead, look at time.Timer which will give you a chan that will receive a value after some time, which you can reset.
Your problem is how you're converting numbers to strings:
id: string(i),
That creates a string using i as a rune (int32). For example string(65) is A. Some unequal Runes resolve to equal strings. You get a collision and close the same channel twice. See http://play.golang.org/p/__KpnfQc1V
You meant this:
id: strconv.Itoa(i),