I wrote a little worker queue using buffered channels.
I want to have the ability to "restart" this worker.
But when I do so I get a panic saying "panic: close of closed channel".
Actually I don't understand why its a closed channel because it shouldn't be closed any more after the make.
Here is the example code (http://play.golang.org/p/nLvNiMaOoA):
package main
import (
"fmt"
"time"
)
type T struct {
ch chan int
}
func (s T) reset() {
close(s.ch)
s.ch = make(chan int, 2)
}
func (s T) wrk() {
for i := range s.ch {
fmt.Println(i)
}
fmt.Println("close")
}
func main() {
t := T{make(chan int, 2)}
for {
go t.wrk()
time.Sleep(time.Second)
t.reset()
}
}
Can you tell me what I'm doing wrong there?
The problem is that you have a value receiver in your reset function which means that s will be copied and you don't see the effects on your t variable in the loop.
To fix that, make it a pointer receiver:
func (s *T) reset() {
close(s.ch)
s.ch = make(chan int, 2)
}
For more info on this topic see Effective Go.
Related
I wrote a Queue class
type Queue struct {
data []interface{}
cond *sync.Cond
}
func New() Queue {
return Queue{
data: []interface{}{},
cond: sync.NewCond(&sync.Mutex{}),
chanStop: make(chan interface{}),
}
}
func (q *Queue) Push(val interface{}) {
q.cond.L.Lock()
q.data = append(q.data, val)
q.cond.Signal()
q.cond.L.Unlock()
}
func (q *Queue) Pop() (interface{}, bool) {
q.cond.L.Lock()
for len(q.data) == 0 {
q.cond.Wait()
}
retVal := q.data[0]
q.data = q.data[1:]
q.cond.L.Unlock()
return retVal, true
}
func (q *Queue) Close() {
}
If the queue is empty Pop() callers will be blocked. Is there any way to stop waiting all routines that are blocked with Pop() by any Cond calls?
Of course I can do something like
type Queue struct {
data []interface{}
cond *sync.Cond
chanStop chan interface{}
}
func (q *Queue) Pop() (interface{}, bool) {
var retVal interface{}
retFlag := false
select {
case <-q.chanStop:
case <-func() <-chan interface{} {
out := make(chan interface{})
go func() {
defer close(out)
q.cond.L.Lock()
for len(q.data) == 0 {
q.cond.Wait()
}
retVal = q.data[0]
retFlag = true
q.data = q.data[1:]
q.cond.L.Unlock()
}()
return out
}():
}
return retVal, retFlag
}
func (q *Queue) Close() {
close(q.chanStop)
}
But maybe there's some way to stop waiting without all this select verbosity?
UPDATE:
Actually it might be done so:
package queue
import "sync"
type Queue struct {
data []interface{}
cond *sync.Cond
stop bool
}
func New() Queue {
return Queue{
data: []interface{}{},
cond: sync.NewCond(&sync.Mutex{}),
stop: false,
}
}
func (q *Queue) Push(val interface{}) {
q.cond.L.Lock()
q.data = append(q.data, val)
q.cond.Signal()
q.cond.L.Unlock()
}
func (q *Queue) Pop() (interface{}, bool) {
q.cond.L.Lock()
for len(q.data) == 0 && !q.stop {
q.cond.Wait()
}
if q.stop {
q.cond.L.Unlock()
return nil, false
}
retVal := q.data[0]
q.data = q.data[1:]
q.cond.L.Unlock()
return retVal, true
}
func (q *Queue) Close() {
q.cond.L.Lock()
q.stop = true
q.cond.Broadcast()
q.cond.L.Unlock()
}
And yes, sync.Cond is sooo weird.
You could wake all clients waiting in Pop() with Cond.Broadcast(), but then you would also have to handle if q.data is empty and there's nothing to return.
Moreover, if clients keep calling Pop() after the queue has been closed, you would also need to check if the queue had been closed before and not enter the waiting state but return early.
Generally sync.Cond is under-documented, it is incompatible with other Go synchronization patterns (e.g. select), and many don't consider it to be a useful synchronization primitive in Go, and potentially it will be removed in Go 2, see details.
Channels may be used instead of sync.Cond, e.g. closing the channel corresponds to Cond.Broadcast(), sending a value on the channel corresponds to Cond.Signal().
Back to your example. The simplest concurrency-safe queue is a buffered channel itself. Push operation is a send on the channel, pop operation is a receive from the channel. Channels are safe for concurrent use.
One thing that a buffered channel "doesn't know" is that it has a fixed buffer size, and once created, the buffer size cannot be changed. Still, I think it's a small price to pay to allocate a big buffer prior and not worry about anything later. Sends on a channel whose buffer is full would not panic "just" block until someone receives from the channel.
Suppose I have the following struct:
package manager
type Manager struct {
strings []string
}
func (m *Manager) AddString(s string) {
m.strings = append(m.strings, s)
}
func (m *Manager) RemoveString(s string) {
for i, str := range m.strings {
if str == s {
m.strings = append(m.strings[:i], m.strings[i+1:]...)
}
}
}
This pattern is not thread safe, so the following test fails due to some race condition (array index out of bounds):
func TestManagerConcurrently(t *testing.T) {
m := &manager.Manager{}
wg := sync.WaitGroup{}
for i:=0; i<100; i++ {
wg.Add(1)
go func () {
m.AddString("a")
m.AddString("b")
m.AddString("c")
m.RemoveString("b")
wg.Done()
} ()
}
wg.Wait()
fmt.Println(m)
}
I'm new to Go, and from googling around I suppose I should use channels (?). So one way to make this concurrent would be like this:
type ManagerA struct {
Manager
addStringChan chan string
removeStringChan chan string
}
func NewManagerA() *ManagerA {
ma := &ManagerA{
addStringChan: make(chan string),
removeStringChan: make(chan string),
}
go func () {
for {
select {
case msg := <-ma.addStringChan:
ma.AddString(msg)
case msg := <-ma.removeStringChan:
ma.RemoveString(msg)
}
}
}()
return ma
}
func (m* ManagerA) AddStringA(s string) {
m.addStringChan <- s
}
func (m* ManagerA) RemoveStringA(s string) {
m.removeStringChan <- s
}
I would like to expose an API similar to the non-concurrent example, hence AddStringA, RemoveStringA.
This seems to work as expected concurrently (although I guess the inner goroutine should also exit at some point). My problem with this is that there is a lot of extra boilerplate:
need to define & initialize channels
define inner goroutine loop with select
map functions to channel calls
It seems a bit much to me. Is there a way to simplify this (refactor / syntax / library)?
I think the best way to implement this would be to use a Mutex instead? But is it still possible to simplify this sort of boilerplate?
Using a mutex would be perfectly idiomatic like this:
type Manager struct {
mu sync.Mutex
strings []string
}
func (m *Manager) AddString(s string) {
m.mu.Lock()
m.strings = append(m.strings, s)
m.mu.Unlock()
}
func (m *Manager) RemoveString(s string) {
m.mu.Lock()
for i, str := range m.strings {
if str == s {
m.strings = append(m.strings[:i], m.strings[i+1:]...)
}
}
m.mu.Unlock()
}
You could do this with channels, but as you noted it is a lot of extra work for not much gain. Just use a mutex is my advice!
If you simply need to make the access to the struct thread-safe, use mutex:
type Manager struct {
sync.Mutex
data []string
}
func (m *Manager) AddString(s string) {
m.Lock()
m.strings = append(m.strings, s)
m.Unlock()
}
I tried to implement a locking version of reading/writing from a map in golang, but it doesn't return the desired result.
package main
import (
"sync"
"fmt"
)
var m = map[int]string{}
var lock = sync.RWMutex{}
func StoreUrl(id int, url string) {
for {
lock.Lock()
defer lock.Unlock()
m[id] = url
}
}
func LoadUrl(id int, ch chan string) {
for {
lock.RLock()
defer lock.RUnlock()
r := m[id]
ch <- r
}
}
func main() {
go StoreUrl(125, "www.google.com")
chb := make(chan string)
go LoadUrl(125, chb);
C := <-chb
fmt.Println("Result:", C)
}
The output is:
Result:
Meaning the value is not returned via the channel, which I don't get. Without the locking/goroutines it seems to work fine. What did I do wrong?
The code can also be found here:
https://play.golang.org/p/-WmRcMty5B
Infinite loops without sleep or some kind of IO are always bad idea.
In your code if you put a print statement at the start of StoreUrl, you will find that it never gets printed i.e the go routine was never started, the go call is setting putting the info about this new go routine in some run queue of the go scheduler but the scheduler hasn't ran yet to schedule that task. How do you run the scheduler? Do sleep/IO/channel reading/writing.
Another problem is that your infinite loop is taking lock and trying to take the lock again, which will cause it to deadlock. Defer only run after function exit and that function will never exit because of infinite loop.
Below is modified code that uses sleep to make sure every execution thread gets time to do its job.
package main
import (
"sync"
"fmt"
"time"
)
var m = map[int]string{}
var lock = sync.RWMutex{}
func StoreUrl(id int, url string) {
for {
lock.Lock()
m[id] = url
lock.Unlock()
time.Sleep(1)
}
}
func LoadUrl(id int, ch chan string) {
for {
lock.RLock()
r := m[id]
lock.RUnlock()
ch <- r
}
}
func main() {
go StoreUrl(125, "www.google.com")
time.Sleep(1)
chb := make(chan string)
go LoadUrl(125, chb);
C := <-chb
fmt.Println("Result:", C)
}
Edit: As #Jaun mentioned in the comment, you can also use runtime.Gosched() instead of sleep.
Usage of defer incorrect, defer execute at end of function, not for statement.
func StoreUrl(id int, url string) {
for {
func() {
lock.Lock()
defer lock.Unlock()
m[id] = url
}()
}
}
or
func StoreUrl(id int, url string) {
for {
lock.Lock()
m[id] = url
lock.Unlock()
}
}
We can't control the order of go routine, so add time.Sleep() to control the order.
code here:
https://play.golang.org/p/Bu8Lo46SA2
Hello i learn about go routine and channel.
I do some experiment with channel, i send a data over channel and try to catch it in 2 functions. But my second function not run
Here is my code :
package main
import (
"fmt"
"os"
"time"
)
func timeout(duration int, ch chan<- bool) {
time.AfterFunc(time.Duration(duration)*time.Second, func() {
ch <- true
})
}
func watcher(duration int, ch <-chan bool) {
<-ch
fmt.Println("\nTimeout! no Answer after", duration, "seconds")
os.Exit(0)
}
func watcher2(duration int, ch <-chan bool) {
<-ch
fmt.Println("This is watcher 2 as a second receiver")
}
func main() {
var data = make(chan bool)
var duration = 5
go timeout(duration, data)
go watcher(duration, data)
go watcher2(duration, data)
var input string
fmt.Print("What is 725/25 ? ")
fmt.Scan(&input)
if input == "29" {
fmt.Println("Correct")
} else {
fmt.Println("Wrong!")
}
}
Can you tell me some explanation about it?
Thank you
As #Andy Schweig mentioned, you can pull from Go channel only once. If you still want to receive message twice, you can use Observer design pattern:
import "fmt"
type Observer interface {
Notify(message string)
}
type Watcher struct {
name string
}
func (w Watcher) Notify(message string) {
fmt.Printf("Watcher %s got message %s\n", w.name, message)
}
var watchers = [...]Watcher {{name: "Watcher 1"}, {name: "Watcher 2"}}
var c = make(chan string)
func notifier() {
var message string
for {
// Messaged pulled only once
message = <- c
// But all watchers still receive it
for _, w := range watchers {
w.Notify(message)
}
}
}
func main() {
go notifier()
c <- "hello"
c <- "how are you?"
}
The channel you declared can only deal with one receiver. By default channels are unbuffered, meaning that they will only accept sends if there is a corresponding receiver to receive the sent value. Whereas a buffered channel accept a limited number of values without a corresponding receiver for those values. If you are looking to inject multiple input and its subsequent receive then you need declare your channel as buffered channel.
ch := make(chan bool, n) //n being the number of items to buffer
I would like to broadcast data received from a channel to a list of channel. The list of channel is dynamic an can be modified during the run phase.
As a new developper in Go, I wrote this code. I found it quite heavy for what I want. Is-there a better way to do this?
package utils
import "sync"
// StringChannelBroadcaster broadcasts string data from a channel to multiple channels
type StringChannelBroadcaster struct {
Source chan string
Subscribers map[string]*StringChannelSubscriber
stopChannel chan bool
mutex sync.Mutex
capacity uint64
}
// NewStringChannelBroadcaster creates a StringChannelBroadcaster
func NewStringChannelBroadcaster(capacity uint64) (b *StringChannelBroadcaster) {
return &StringChannelBroadcaster{
Source: make(chan string, capacity),
Subscribers: make(map[string]*StringChannelSubscriber),
capacity: capacity,
}
}
// Dispatch starts dispatching message
func (b *StringChannelBroadcaster) Dispatch() {
b.stopChannel = make(chan bool)
for {
select {
case val, ok := <-b.Source:
if ok {
b.mutex.Lock()
for _, value := range b.Subscribers {
value.Channel <- val
}
b.mutex.Unlock()
}
case <-b.stopChannel:
return
}
}
}
// Stop stops the Broadcaster
func (b *StringChannelBroadcaster) Stop() {
close(b.stopChannel)
}
// StringChannelSubscriber defines a subscriber to a StringChannelBroadcaster
type StringChannelSubscriber struct {
Key string
Channel chan string
}
// NewSubscriber returns a new subsriber to the StringChannelBroadcaster
func (b *StringChannelBroadcaster) NewSubscriber() *StringChannelSubscriber {
key := RandString(20)
newSubscriber := StringChannelSubscriber{
Key: key,
Channel: make(chan string, b.capacity),
}
b.mutex.Lock()
b.Subscribers[key] = &newSubscriber
b.mutex.Unlock()
return &newSubscriber
}
// RemoveSubscriber removes a subscrber from the StringChannelBroadcaster
func (b *StringChannelBroadcaster) RemoveSubscriber(subscriber *StringChannelSubscriber) {
b.mutex.Lock()
delete(b.Subscribers, subscriber.Key)
b.mutex.Unlock()
}
Thank you,
Julien
I think you can simplify it a bit: get rid of stopChannel and the Stop method. You can just close Source instead of calling Stop, and detect that in Dispatch (ok will be false) to quit (you can just range over the source channel actually).
You can get rid of Dispatch, and just start a goroutine in NewStringChannelBroadcaster with the for cycle, so external code doesn't have to start the dispatch cycle separately.
You can use a channel type as the map key, so your map can become map[chan string]struct{} (empty struct because you don't need the map value). So your NewSubscriber can take a channel type parameter (or create a new channel and return it), and insert that into the map, you don't need the random string or the StringChannelSubscriber type.
I also made some improvements, like closing the subscriber channels:
package main
import "sync"
import (
"fmt"
"time"
)
// StringChannelBroadcaster broadcasts string data from a channel to multiple channels
type StringChannelBroadcaster struct {
Source chan string
Subscribers map[chan string]struct{}
mutex sync.Mutex
capacity uint64
}
// NewStringChannelBroadcaster creates a StringChannelBroadcaster
func NewStringChannelBroadcaster(capacity uint64) *StringChannelBroadcaster {
b := &StringChannelBroadcaster{
Source: make(chan string, capacity),
Subscribers: make(map[chan string]struct{}),
capacity: capacity,
}
go b.dispatch()
return b
}
// Dispatch starts dispatching message
func (b *StringChannelBroadcaster) dispatch() {
// for iterates until the channel is closed
for val := range b.Source {
b.mutex.Lock()
for ch := range b.Subscribers {
ch <- val
}
b.mutex.Unlock()
}
b.mutex.Lock()
for ch := range b.Subscribers {
close(ch)
// you shouldn't be calling RemoveSubscriber after closing b.Source
// but it's better to be safe than sorry
delete(b.Subscribers, ch)
}
b.Subscribers = nil
b.mutex.Unlock()
}
func (b *StringChannelBroadcaster) NewSubscriber() chan string {
ch := make(chan string, b.capacity)
b.mutex.Lock()
if b.Subscribers == nil {
panic(fmt.Errorf("NewSubscriber called on closed broadcaster"))
}
b.Subscribers[ch] = struct{}{}
b.mutex.Unlock()
return ch
}
// RemoveSubscriber removes a subscrber from the StringChannelBroadcaster
func (b *StringChannelBroadcaster) RemoveSubscriber(ch chan string) {
b.mutex.Lock()
if _, ok := b.Subscribers[ch]; ok {
close(ch) // this line does have to be inside the if to prevent close of closed channel, in case RemoveSubscriber is called twice on the same channel
delete(b.Subscribers, ch) // this line doesn't need to be inside the if
}
b.mutex.Unlock()
}
func main() {
b := NewStringChannelBroadcaster(0)
var toberemoved chan string
for i := 0; i < 3; i++ {
i := i
ch := b.NewSubscriber()
if i == 1 {
toberemoved = ch
}
go func() {
for v := range ch {
fmt.Printf("receive %v: %v\n", i, v)
}
fmt.Printf("Exit %v\n", i)
}()
}
b.Source <- "Test 1"
b.Source <- "Test 2"
// This is a race condition: the second reader may or may not receive the first two messages.
b.RemoveSubscriber(toberemoved)
b.Source <- "Test 3"
// let the reader goroutines receive the last message
time.Sleep(2 * time.Second)
close(b.Source)
// let the reader goroutines write close message
time.Sleep(1 * time.Second)
}
https://play.golang.org/p/X-NcikvbDM
Edit: I've added your edit to fix the panic when calling RemoveSubscriber after closing Source, but you shouldn't be doing that, you should let the struct and everything in it be garbage collected after the channel is closed.
I've also added a panic to NewSubscriber if it's called after closing Source. Previously you could do that and it'd leak the created channel and presumably the goroutine that will block forever on that channel.
If you can call NewSubscriber (or RemoveSubscriber) on an already closed broadcaster, that probably means there's an error in your code somewhere, since you're holding on to a broadcaster that you shouldn't be.