Channel concurrency guarantees - caching

I'm writing a concurrency-safe memo:
package mu
import (
"sync"
)
// Func represents a memoizable function, operating on a string key, to use with a Mu
type Func func(key string) interface{}
// Mu is a cache that memoizes results of an expensive computation
//
// It has a traditional implementation using mutexes.
type Mu struct {
// guards done
mu sync.RWMutex
done map[string]chan bool
memo map[string]interface{}
f Func
}
// Get a string key if it exists, otherwise computes the value and caches it.
//
// Returns the value and whether or not the key existed.
func (c *Mu) Get(key string) (interface{}, bool) {
c.mu.RLock()
_, ok := c.done[key]
c.mu.RUnlock()
if ok {
return c.get(key), true
}
c.mu.Lock()
_, ok = c.done[key]
if ok {
c.mu.Unlock()
} else {
c.done[key] = make(chan bool)
c.mu.Unlock()
v := c.f(key)
c.memo[key] = v
close(c.done[key])
}
return c.get(key), ok
}
// get returns the value of key, blocking on an existing computation
func (c *Mu) get(key string) interface{} {
<-c.done[key]
v, _ := c.memo[key]
return v
}
As you can see, there's a mutex guarding the done field, which is used
to signal to other goroutines that a computation for a key is pending or done. This avoids duplicate computations (calls to c.f(key)) for the same key.
My question is around the guarantees of this code; by ensuring that the computing goroutine closes the channel after it writes to c.memo, does this guarantee that other goroutines that access c.memo[key] after a blocking call to <-c.done[key] are guaranteed to see the result of the computation?

The short answer is yes.
We can simplify some of the code to get to the essence of why. Consider your Mu struct:
type Mu struct {
memo int
done chan bool
}
We can now define 2 functions, compute and read
func compute(r *Mu) {
time.Sleep(2 * time.Second)
r.memo = 42
close(r.done)
}
func read(r *Mu) {
<-r.done
fmt.Println("Read value: ", r.memo)
}
Here, compute is a computationally heavy task (which we can simulate by sleeping for some time)
Now, in the main function, we start a new compute go routine, along with starting some read go routines at regular intervals:
func main() {
r := &Mu{}
r.done = make(chan bool)
go compute(r)
// this one starts immediately
go read(r)
time.Sleep(time.Second)
// this one starts in the middle of computation
go read(r)
time.Sleep(2*time.Second)
// this one starts after the computation is complete
go read(r)
// This is to prevent the program from terminating immediately
time.Sleep(3 * time.Second)
}
In all three cases, we print out the result of the compute task.
Working code here
When you "close" a channel in go, all statements which wait for the result of the channel (including statements that are executed after it's closed) will block. So provided that the only place that the channel is being closed from is the place where the memo value is computed, you will have that guarantee.
The only place where you should be careful, is to make sure that this channel isn't closed anywhere else in your code.

Related

How to coordinate shutdown with many goroutines

Say I have a function
type Foo struct {}
func (a *Foo) Bar() {
// some expensive work - does some calls to redis
}
which gets executed within a goroutine at some point in my app. Lots of these may be executing at any given point. Prior to application termination, I would like to ensure all remaining goroutines have finished their work.
Can I do something like this:
type Foo struct {
wg sync.WaitGroup
}
func (a *Foo) Close() {
a.wg.Wait()
}
func (a *Foo) Bar() {
a.wg.Add(1)
defer a.wg.Done()
// some expensive work - does some calls to redis
}
Assuming here that Bar gets executed within a goroutine and many of these may be running at a given time and that Bar should not be called once Close is called and Close is called upon a sigterm or sigint.
Does this make sense?
Usually I would see the Bar function look like this:
func (a *Foo) Bar() {
a.wg.Add(1)
go func() {
defer a.wg.Done()
// some expensive work - does some calls to redis
}()
}
Yes, WaitGroup is the right answer. You can use WaitGroup.Add at anytime that the counter is greater than zero, as per doc.
Note that calls with a positive delta that occur when the counter is zero must happen before a Wait. Calls with a negative delta, or calls with a positive delta that start when the counter is greater than zero, may happen at any time. Typically this means the calls to Add should execute before the statement creating the goroutine or other event to be waited for. If a WaitGroup is reused to wait for several independent sets of events, new Add calls must happen after all previous Wait calls have returned. See the WaitGroup example.
But one trick is that, you should always keep the counter greater than zero, before Close is called. That usually means you should call wg.Add in NewFoo (or something like that) and wg.Done in Close. And to prevent multiple calls to Done ruining the wait group, you should wrap Close into sync.Once. You may also want to prevent new Bar() from being called.
WaitGroup is one way, however, the Go team introduced the errgroup for your use case exactly. The most inconvenient part of leaf bebop's answer, is the disregard for error handling. Error handling is the reason errgroup exists. And idiomatic go code should never swallow errors.
However, keeping the signatures of your Foo struct, (except a cosmetic workerNumber)—and no error handling—my proposal looks like this:
package main
import (
"fmt"
"math/rand"
"time"
"golang.org/x/sync/errgroup"
)
type Foo struct {
errg errgroup.Group
}
func NewFoo() *Foo {
foo := &Foo{
errg: errgroup.Group{},
}
return foo
}
func (a *Foo) Bar(workerNumber int) {
a.errg.Go(func() error {
select {
// simulates the long running clals
case <-time.After(time.Second * time.Duration(rand.Intn(10))):
fmt.Println(fmt.Sprintf("worker %d completed its work", workerNumber))
return nil
}
})
}
func (a *Foo) Close() {
a.errg.Wait()
}
func main() {
foo := NewFoo()
for i := 0; i < 10; i++ {
foo.Bar(i)
}
<-time.After(time.Second * 5)
fmt.Println("Waiting for workers to complete...")
foo.Close()
fmt.Println("Done.")
}
The benefit here, is that if you introduce error handling in your code (you should), you only need to slightly modify this code: In short, errg.Wait() would return the first redis error, and Close() could propagate this up through the stack (to main, in this case).
Utilizing the context.Context package as well, you would also be able to immediately cancel any running redis call, if one fails. There are examples of this in the errgroup documentation.
I think waiting indefinitely for all the go routines to finish is not the right way.
If one of the go routines get blocked or say it hangs due to some reason and never terminates successfully, what should happen kill the process or wait for go routines to finish ?
Instead you should wait with some timeout and kill the app irrespective of whether all the routines have finished or not.
Edit: Original ans
Thanks #leaf bebop for pointing it out. I misunderstood the question.
Context package can be used to signal all the go routines to handle kill signal.
appCtx, cancel := context.WithCancel(context.Background())
Here appCtx will have to be passed to all the go routines.
On exit signal call cancel().
functions running as go routines can handle how to handle cancel context.
Using context cancellation in Go
A pattern i use a lot is: https://play.golang.org/p/ibMz36TS62z
package main
import (
"fmt"
"sync"
"time"
)
type response struct {
message string
}
func task(i int, done chan response) {
time.Sleep(1 * time.Second)
done <- response{fmt.Sprintf("%d done", i)}
}
func main() {
responses := GetResponses(10)
fmt.Println("all done", len(responses))
}
func GetResponses(n int) []response {
donequeue := make(chan response)
wg := sync.WaitGroup{}
for i := 0; i < n; i++ {
wg.Add(1)
go func(value int) {
defer wg.Done()
task(value, donequeue)
}(i)
}
go func() {
wg.Wait()
close(donequeue)
}()
responses := []response{}
for result := range donequeue {
responses = append(responses, result)
}
return responses
}
this makes it easy to throttle as well: https://play.golang.org/p/a4MKwJKj634
package main
import (
"fmt"
"sync"
"time"
)
type response struct {
message string
}
func task(i int, done chan response) {
time.Sleep(1 * time.Second)
done <- response{fmt.Sprintf("%d done", i)}
}
func main() {
responses := GetResponses(10, 2)
fmt.Println("all done", len(responses))
}
func GetResponses(n, concurrent int) []response {
throttle := make(chan int, concurrent)
for i := 0; i < concurrent; i++ {
throttle <- i
}
donequeue := make(chan response)
wg := sync.WaitGroup{}
for i := 0; i < n; i++ {
wg.Add(1)
<-throttle
go func(value int) {
defer wg.Done()
throttle <- 1
task(value, donequeue)
}(i)
}
go func() {
wg.Wait()
close(donequeue)
}()
responses := []response{}
for result := range donequeue {
responses = append(responses, result)
}
return responses
}

how to efficiently call a func after X hours?

I know I can do like this:
func randomFunc() {
// do stuff
go destroyObjectAfterXHours(4, "idofobject")
// do other stuff
}
func destroyObjectAfterXHours(hours int, id string) {
time.Sleep(hours * time.Hour)
destroyObject(id)
}
but if we imagine destroyObjectAfterXHours is called a million times within a few minutes, this solution will be very bad.
I was hoping someone could share a more efficient solution to this problem.
I've been thinking about a potential solution where destruction time and object id was stored somewhere, and then there would be one func that would traverse through the list every X minutes, destroy the objects that had to be destroyed and remove their id and time info from wherever that info was stored. Would this be a good solution?
I worry it would also be bad solution since you will then have to traverse through a list with millions of items all the time, and then have to efficiently remove some of the items, etc.
The time.AfterFunc function is designed for this use case:
func randomFunc() {
// do stuff
time.AfterFunc(4*time.Hour, func() { destroyObject("idofobject") })
// do other stuff
}
time.AfterFunc is efficient and simple to use.
As the documentation states, the function is called in a goroutine after the duration elapses. The goroutine is not created up front as in the question.
So I'd agree with your solution #2 than number 1.
Traversing through a list of a million numbers is much easier than having a million separate Go Routines
Go Routines are expensive (compared to loops) and take memory and processing time. For eg. a million Go Routines take about 4GB of RAM.
Traversing through a list on the other hand takes very little space and is done in O(n) time.
A good example of this exact function is Go Cache which deletes its expired elements in a Go Routine it runs periodically
https://github.com/patrickmn/go-cache/blob/master/cache.go#L931
This is a more detailed example of how they did it:
type Item struct {
Object interface{}
Expiration int64
}
func (item Item) Expired() bool {
if item.Expiration == 0 {
return false
}
return time.Now().UnixNano() > item.Expiration
}
func RemoveItem(s []Item, index int) []int {
return append(s[:index], s[index+1:]...)
}
func deleteExpired(items []Item){
var deletedItems []int
for k, v := range items {
if v.Expired(){
deletedItems = append(deletedItems, k)
}
}
for key, deletedIndex := range deleteditems{
items = RemoveItem(items, deletedIndex)
}
}
The above implementation could definitely be improved with a linked list instead of an array but this is the general idea
This is an interesting question. I come to a solution where it use a heap to maintain the queue of items to be destroyed and sleep for exactly time until the next item is up for destruction. I think it is more efficient, but the gain might be slim on some cases. Nonetheless, you can see the code here:
package main
import (
"container/heap"
"fmt"
"time"
)
type Item struct {
Expiration time.Time
Object interface{} // It would make more sence to be *interface{}, but not as convinient
}
//MINIT is the minimal interval for delete to run. In most cases, it is better to be set as 0
const MININT = 1 * time.Second
func deleteExpired(addCh chan Item) (quitCh chan bool) {
quitCh = make(chan bool)
go func() {
h := make(ExpHeap, 0)
var t *time.Timer
item := <-addCh
heap.Push(&h, &item)
t = time.NewTimer(time.Until(h[0].Expiration))
for {
//Check unfinished incoming first
for incoming := true; incoming; {
select {
case item := <-addCh:
heap.Push(&h, &item)
default:
incoming = false
}
}
if delta := time.Until(h[0].Expiration); delta >= MININT {
t.Reset(delta)
} else {
t.Reset(MININT)
}
select {
case <-quitCh:
return
//New Item incoming, break the timer
case item := <-addCh:
heap.Push(&h, &item)
if item.Expiration.After(h[0].Expiration) {
continue
}
if delta := time.Until(item.Expiration); delta >= MININT {
t.Reset(delta)
} else {
t.Reset(MININT)
}
//Wait until next item to be deleted
case <-t.C:
for !h[0].Expiration.After(time.Now()) {
item := heap.Pop(&h).(*Item)
destroy(item.Object)
}
if delta := time.Until(h[0].Expiration); delta >= MININT {
t.Reset(delta)
} else {
t.Reset(MININT)
}
}
}
}()
return quitCh
}
type ExpHeap []*Item
func (h ExpHeap) Len() int {
return len(h)
}
func (h ExpHeap) Swap(i, j int) {
h[i], h[j] = h[j], h[i]
}
func (h ExpHeap) Less(i, j int) bool {
return h[i].Expiration.Before(h[j].Expiration)
}
func (h *ExpHeap) Push(x interface{}) {
item := x.(*Item)
*h = append(*h, item)
}
func (h *ExpHeap) Pop() interface{} {
old, n := *h, len(*h)
item := old[n-1]
*h = old[:n-1]
return item
}
//Auctural destroy code.
func destroy(x interface{}) {
fmt.Printf("%v # %v\n", x, time.Now())
}
func main() {
addCh := make(chan Item)
quitCh := deleteExpired(addCh)
for i := 30; i > 0; i-- {
t := time.Now().Add(time.Duration(i) * time.Second / 2)
addCh <- Item{t, t}
}
time.Sleep(7 * time.Second)
quitCh <- true
}
playground: https://play.golang.org/p/JNV_6VJ_yfK
By the way, there are packages like cron for job management, but I am not familiar with them so I cannot speak for their efficiency.
Edit:
Still I haven't enough reputation to comment :(
About performance: this code basically has less CPU usage as it only wake it self when necessary and only traverse items that is up for destruction instead of the whole list. Based on personal (actually ACM experience), roughly a mordern CPU can process a loop of 10^9 in 1.2 seconds or so, which means on a scale of 10^6, traversing the whole list takes about over 1 millisecond excluding auctual destruction code AND data copy (which will cost a lot on average on more than thousands of runs, to a scale of 100 millisecond or so). My code's approach is O(lg N) which on 10^6 scale is at least a thousand times faster (considering constant). Please note again all these calculation is based on experience instead of benchmarks (there was but I am not able to provide them).
Edit 2:
With a second thought, I think the plain solution can use a simple optimization:
func deleteExpired(items []Item){
tail = len(items)
for index, v := range items { //better naming
if v.Expired(){
tail--
items[tail],items[index] = v,items[tail]
}
}
deleteditems := items[tail:]
items:=items[:tail]
}
With this change, it no longer copy data unefficiently and will not allocate extra space.
Edit 3:
changing code from here
I tested the memoryuse of afterfunc. On my laptop it is 250 bytes per call, while on palyground it is 69 (I am curious at the reason). With my code, A pointer + a time.Time is 28 byte. At a scale of a million, the difference is slim. Using After Func is a much better option.
If it is a one-shot, this can be easily achieved with
// Make the destruction cancelable
cancel := make(chan bool)
go func(t time.Duration, id int){
expired := time.NewTimer(t).C
select {
// destroy the object when the timer is expired
case <-expired:
destroyObject(id)
// or cancel the destruction in case we get a cancel signal
// before it is destroyed
case <-cancel:
fmt.Println("Cancelled destruction of",id)
return
}
}(time.Hours * 4, id)
if weather == weather.SUNNY {
cancel <- true
}
If you want to do it every 4 h:
// Same as above, though since id may have already been destroyed
// once, I name the channel different
done := make(chan bool)
go func(t time.Duration,id int){
// Sends to the channel every t
tick := time.NewTicker(t).C
// Wrap, otherwise select will only execute the first tick
for{
select {
// t has passed, so id can be destroyed
case <-tick:
destroyObject(id)
// We are finished destroying stuff
case <-done:
fmt.Println("Ok, ok, I quit destroying...")
return
}
}
}()
if weather == weather.RAINY {
done <- true
}
The idea behind it is to run a single goroutine per destruction job which can be cancelled. Say, you have a session and the user did something, so you want to keep the session alive. Since goroutines are extremely cheap, you can simply fire off another goroutine.

Go goroutine lock and unlock

I been reading about goroutines and the sync package and my question is... Do I always need to lock unlock when reading writting to data on different goroutines?
For example I have a variable on my server
config := make(map[string]string)
Then on different goroutines I want to read from config. Is it safe to read without using sync or it is not?
I guess writting needs to be done using the sync package. but I am not sure about reading
For example I have a simple in-memory cache system
type Cache interface {
Get(key string) interface{}
Put(key string, expires int64, value interface{})
}
// MemoryCache represents a memory type of cache
type MemoryCache struct {
c map[string]*MemoryCacheValue
rw sync.RWMutex
}
// MemoryCacheValue represents a memory cache value
type MemoryCacheValue struct {
value interface{}
expires int64
}
// NewMemoryCache creates a new memory cache
func NewMemoryCache() Cache {
return &MemoryCache{
c: make(map[string]*MemoryCacheValue),
}
}
// Get stores something into the cache
func (m *MemoryCache) Get(key string) interface{} {
if v, ok := m.c[key]; ok {
return v
}
return nil
}
// Put retrieves something from the cache
func (m *MemoryCache) Put(key string, expires int64, value interface{}) {
m.rw.Lock()
m.c[key] = &MemoryCacheValue{
value,
time.Now().Unix() + expires,
}
m.rw.Unlock()
}
I am acting safe here or I still need to lock unlock when I want to only read?
You're diving into the world of race conditions. The basic rule of thumb is that if ANY routine writes to or changes a piece of data that can be or is read by (or also written to) by any number of other coroutines/threads, you need to have some sort of synchronization system in place.
For example, lets say you have that map. It has ["Joe"] = "Smith" and ["Sean"] = "Howard" in it. One goroutine wants to read the value of ["Joe"]. Another routine is updating ["Joe"] to "Cooper". Which value does the first goroutine read? Depends on which goroutine gets to the data first. That's the race condition, the behavior is undefined and unpredictable.
The easiest method to control that access is with a sync.Mutex. In your case, since some routines only need to read and not write, you can instead use a sync.RWMutex (main difference is that a RWMutex allows any number of threads to read, as long as none are trying to write). You would bake this into the map using a structure like this:
type MutexMap struct {
m map[string]string
*sync.RWMutex
}
Then, in routines that need to read from the map, you would do:
func ReadSomething(o MutexMap, key string) string {
o.RLock() // lock for reading, blocks until the Mutex is ready
defer o.RUnlock() // make SURE you do this, else it will be locked permanently
return o.m[key]
}
And to write:
func WriteSomething(o MutexMap, key, value string) {
o.Lock() // lock for writing, blocks until the Mutex is ready
defer o.Unlock() // again, make SURE you do this, else it will be locked permanently
o.m[key] = value
}
Note that both of these could be written as methods of the struct, rather than functions, if desired.
You can also approach this using channels. You make a controller structure that runs in a goroutine, and you make requests to it over channels. Example:
package main
import "fmt"
type MapCtrl struct {
m map[string]string
ReadCh chan chan map[string]string
WriteCh chan map[string]string
QuitCh chan struct{}
}
func NewMapController() *MapCtrl {
return &MapCtrl{
m: make(map[string]string),
ReadCh: make(chan chan map[string]string),
WriteCh: make(chan map[string]string),
QuitCh: make(chan struct{}),
}
}
func (ctrl *MapCtrl) Control() {
for {
select {
case r := <-ctrl.ReadCh:
fmt.Println("Read request received")
retmap := make(map[string]string)
for k, v := range ctrl.m { // copy map, so it doesn't change in place after return
retmap[k] = v
}
r <- retmap
case w := <-ctrl.WriteCh:
fmt.Println("Write request received with", w)
for k, v := range w {
ctrl.m[k] = v
}
case <-ctrl.QuitCh:
fmt.Println("Quit request received")
return
}
}
}
func main() {
ctrl := NewMapController()
defer close(ctrl.QuitCh)
go ctrl.Control()
m := make(map[string]string)
m["Joe"] = "Smith"
m["Sean"] = "Howard"
ctrl.WriteCh <- m
r := make(chan map[string]string, 1)
ctrl.ReadCh <- r
fmt.Println(<-r)
}
Runnable version

Data race when GOMAXPROCS=1

I'm trying to understand one of the Golang typical data races where access to an unprotected global variable from multiple goroutines may cause a race condition:
var service map[string]net.Addr
func RegisterService(name string, addr net.Addr) {
service[name] = addr
}
func LookupService(name string) net.Addr {
return service[name]
}
It goes on to say that we can solve this by protecting it with a mutex:
var (
service map[string]net.Addr
serviceMu sync.Mutex
)
func RegisterService(name string, addr net.Addr) {
serviceMu.Lock()
defer serviceMu.Unlock()
service[name] = addr
}
func LookupService(name string) net.Addr {
serviceMu.Lock()
defer serviceMu.Unlock()
return service[name]
}
So far, so good. What's confusing me is this:
The accepted answer to this question suggests that a CPU-bound goroutine will any starve other goroutines that have been multiplexed onto the same OS thread (unless we explicitly yield with runtime.Gosched()). Which makes sense.
To me, the RegisterService() and LookupService() functions above look to be CPU bound, as there's no IO and no yield. Is this correct?
If it is, and if GOMAXPROCS is set to 1, then is a mutex in the above example still strictly necessary? Doesn't the fact that the goroutines are CPU-bound at the point where race conditions might occur take care of it?
Even if it does, I presume that in real life using a mutex is here is still a good idea as we may not be able to guarantee what GOMAXPROCS is set to. Are there another reasons?
As FUZxxl and Nick Craig-Wood noted, current behavior of goroutines is implementation-specific. So, maybe, read or write to map can yield. Considering that maps are not thread safe, mutex or another syncronization is required for correct concurrent access.
As alternative to mutex, you can spawn a goroutine, that performs all operations on your map and talk to it with channels:
type writereq struct {
key string
value net.Addr
reply chan struct{}
}
type readreq struct {
key string
reply chan net.Addr
}
var service map[string]net.Addr
var reads = make(chan readreq)
var writes = make(chan writereq)
func RegisterService(name string, addr net.Addr) {
w := writereq{name, addr, make(chan struct{})}
writes <- w
return <-w.reply // return after registration confirmation
}
func LookupService(name string) net.Addr {
r := readreq{name, make(chan net.Addr)}
reads <- r
return <-r.reply
}
func serveRegistry() {
for {
select {
case r := <-reads:
r.reply <- service[r.name]
case w := <-writes:
service[w.name] = w.addr
w.reply <- struct{}
}
}
}

How to check a channel is closed or not without reading it?

This is a good example of workers & controller mode in Go written by #Jimt, in answer to
"Is there some elegant way to pause & resume any other goroutine in golang?"
package main
import (
"fmt"
"runtime"
"sync"
"time"
)
// Possible worker states.
const (
Stopped = 0
Paused = 1
Running = 2
)
// Maximum number of workers.
const WorkerCount = 1000
func main() {
// Launch workers.
var wg sync.WaitGroup
wg.Add(WorkerCount + 1)
workers := make([]chan int, WorkerCount)
for i := range workers {
workers[i] = make(chan int)
go func(i int) {
worker(i, workers[i])
wg.Done()
}(i)
}
// Launch controller routine.
go func() {
controller(workers)
wg.Done()
}()
// Wait for all goroutines to finish.
wg.Wait()
}
func worker(id int, ws <-chan int) {
state := Paused // Begin in the paused state.
for {
select {
case state = <-ws:
switch state {
case Stopped:
fmt.Printf("Worker %d: Stopped\n", id)
return
case Running:
fmt.Printf("Worker %d: Running\n", id)
case Paused:
fmt.Printf("Worker %d: Paused\n", id)
}
default:
// We use runtime.Gosched() to prevent a deadlock in this case.
// It will not be needed of work is performed here which yields
// to the scheduler.
runtime.Gosched()
if state == Paused {
break
}
// Do actual work here.
}
}
}
// controller handles the current state of all workers. They can be
// instructed to be either running, paused or stopped entirely.
func controller(workers []chan int) {
// Start workers
for i := range workers {
workers[i] <- Running
}
// Pause workers.
<-time.After(1e9)
for i := range workers {
workers[i] <- Paused
}
// Unpause workers.
<-time.After(1e9)
for i := range workers {
workers[i] <- Running
}
// Shutdown workers.
<-time.After(1e9)
for i := range workers {
close(workers[i])
}
}
But this code also has an issue: If you want to remove a worker channel in workers when worker() exits, dead lock happens.
If you close(workers[i]), next time controller writes into it will cause a panic since go can't write into a closed channel. If you use some mutex to protect it, then it will be stuck on workers[i] <- Running since the worker is not reading anything from the channel and write will be blocked, and mutex will cause a dead lock. You can also give a bigger buffer to channel as a work-around, but it's not good enough.
So I think the best way to solve this is worker() close channel when exits, if the controller finds a channel closed, it will jump over it and do nothing. But I can't find how to check a channel is already closed or not in this situation. If I try to read the channel in controller, the controller might be blocked. So I'm very confused for now.
PS: Recovering the raised panic is what I have tried, but it will close goroutine which raised panic. In this case it will be controller so it's no use.
Still, I think it's useful for Go team to implement this function in next version of Go.
There's no way to write a safe application where you need to know whether a channel is open without interacting with it.
The best way to do what you're wanting to do is with two channels -- one for the work and one to indicate a desire to change state (as well as the completion of that state change if that's important).
Channels are cheap. Complex design overloading semantics isn't.
[also]
<-time.After(1e9)
is a really confusing and non-obvious way to write
time.Sleep(time.Second)
Keep things simple and everyone (including you) can understand them.
In a hacky way it can be done for channels which one attempts to write to by recovering the raised panic. But you cannot check if a read channel is closed without reading from it.
Either you will
eventually read the "true" value from it (v <- c)
read the "true" value and 'not closed' indicator (v, ok <- c)
read a zero value and the 'closed' indicator (v, ok <- c) (example)
will block in the channel read forever (v <- c)
Only the last one technically doesn't read from the channel, but that's of little use.
I know this answer is so late, I have wrote this solution, Hacking Go run-time, It's not safety, It may crashes:
import (
"unsafe"
"reflect"
)
func isChanClosed(ch interface{}) bool {
if reflect.TypeOf(ch).Kind() != reflect.Chan {
panic("only channels!")
}
// get interface value pointer, from cgo_export
// typedef struct { void *t; void *v; } GoInterface;
// then get channel real pointer
cptr := *(*uintptr)(unsafe.Pointer(
unsafe.Pointer(uintptr(unsafe.Pointer(&ch)) + unsafe.Sizeof(uint(0))),
))
// this function will return true if chan.closed > 0
// see hchan on https://github.com/golang/go/blob/master/src/runtime/chan.go
// type hchan struct {
// qcount uint // total data in the queue
// dataqsiz uint // size of the circular queue
// buf unsafe.Pointer // points to an array of dataqsiz elements
// elemsize uint16
// closed uint32
// **
cptr += unsafe.Sizeof(uint(0))*2
cptr += unsafe.Sizeof(unsafe.Pointer(uintptr(0)))
cptr += unsafe.Sizeof(uint16(0))
return *(*uint32)(unsafe.Pointer(cptr)) > 0
}
Well, you can use default branch to detect it, for a closed channel will be selected, for example: the following code will select default, channel, channel, the first select is not blocked.
func main() {
ch := make(chan int)
go func() {
select {
case <-ch:
log.Printf("1.channel")
default:
log.Printf("1.default")
}
select {
case <-ch:
log.Printf("2.channel")
}
close(ch)
select {
case <-ch:
log.Printf("3.channel")
default:
log.Printf("3.default")
}
}()
time.Sleep(time.Second)
ch <- 1
time.Sleep(time.Second)
}
Prints
2018/05/24 08:00:00 1.default
2018/05/24 08:00:01 2.channel
2018/05/24 08:00:01 3.channel
Note, refer to comment by #Angad under this answer:
It doesn't work if you're using a Buffered Channel and it contains
unread data
I have had this problem frequently with multiple concurrent goroutines.
It may or may not be a good pattern, but I define a a struct for my workers with a quit channel and field for the worker state:
type Worker struct {
data chan struct
quit chan bool
stopped bool
}
Then you can have a controller call a stop function for the worker:
func (w *Worker) Stop() {
w.quit <- true
w.stopped = true
}
func (w *Worker) eventloop() {
for {
if w.Stopped {
return
}
select {
case d := <-w.data:
//DO something
if w.Stopped {
return
}
case <-w.quit:
return
}
}
}
This gives you a pretty good way to get a clean stop on your workers without anything hanging or generating errors, which is especially good when running in a container.
You could set your channel to nil in addition to closing it. That way you can check if it is nil.
example in the playground:
https://play.golang.org/p/v0f3d4DisCz
edit:
This is actually a bad solution as demonstrated in the next example,
because setting the channel to nil in a function would break it:
https://play.golang.org/p/YVE2-LV9TOp
ch1 := make(chan int)
ch2 := make(chan int)
go func(){
for i:=0; i<10; i++{
ch1 <- i
}
close(ch1)
}()
go func(){
for i:=10; i<15; i++{
ch2 <- i
}
close(ch2)
}()
ok1, ok2 := false, false
v := 0
for{
ok1, ok2 = true, true
select{
case v,ok1 = <-ch1:
if ok1 {fmt.Println(v)}
default:
}
select{
case v,ok2 = <-ch2:
if ok2 {fmt.Println(v)}
default:
}
if !ok1 && !ok2{return}
}
}
From the documentation:
A channel may be closed with the built-in function close. The multi-valued assignment form of the receive operator reports whether a received value was sent before the channel was closed.
https://golang.org/ref/spec#Receive_operator
Example by Golang in Action shows this case:
// This sample program demonstrates how to use an unbuffered
// channel to simulate a game of tennis between two goroutines.
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
// wg is used to wait for the program to finish.
var wg sync.WaitGroup
func init() {
rand.Seed(time.Now().UnixNano())
}
// main is the entry point for all Go programs.
func main() {
// Create an unbuffered channel.
court := make(chan int)
// Add a count of two, one for each goroutine.
wg.Add(2)
// Launch two players.
go player("Nadal", court)
go player("Djokovic", court)
// Start the set.
court <- 1
// Wait for the game to finish.
wg.Wait()
}
// player simulates a person playing the game of tennis.
func player(name string, court chan int) {
// Schedule the call to Done to tell main we are done.
defer wg.Done()
for {
// Wait for the ball to be hit back to us.
ball, ok := <-court
fmt.Printf("ok %t\n", ok)
if !ok {
// If the channel was closed we won.
fmt.Printf("Player %s Won\n", name)
return
}
// Pick a random number and see if we miss the ball.
n := rand.Intn(100)
if n%13 == 0 {
fmt.Printf("Player %s Missed\n", name)
// Close the channel to signal we lost.
close(court)
return
}
// Display and then increment the hit count by one.
fmt.Printf("Player %s Hit %d\n", name, ball)
ball++
// Hit the ball back to the opposing player.
court <- ball
}
}
it's easier to check first if the channel has elements, that would ensure the channel is alive.
func isChanClosed(ch chan interface{}) bool {
if len(ch) == 0 {
select {
case _, ok := <-ch:
return !ok
}
}
return false
}
If you listen this channel you always can findout that channel was closed.
case state, opened := <-ws:
if !opened {
// channel was closed
// return or made some final work
}
switch state {
case Stopped:
But remember, you can not close one channel two times. This will raise panic.

Resources