Channel won't receive message sent from inside Go Routine - go

I'm trying to get a better understanding of channel in GO.
Want 5 routines to be running at all times. At specific times during the first routine, I want to try to start another routine. Assuming 5 routines are already running, I want to queue up the next routine and run it as soon as one of the other routines has been completed.
My logic was call pass a message to spawner, check to see if there are 5 processes already running, if so, keep waiting until there isn't, and start up. From what I can tell, is is p.complete <- struct{}{} isn't working as expecting and removing a process. It works fine outside of the go routine.
package main
import (
"fmt"
"os"
"os/signal"
"syscall"
"time"
)
type ProcessManager struct {
spawner chan struct{}
complete chan struct{}
process int
}
func NewProcessManager() *ProcessManager {
return &ProcessManager{
spawner: make(chan struct{}),
complete: make(chan struct{}),
process: 0,
}
}
func (p *ProcessManager) Run(limit int) {
for {
select {
case <-p.spawner:
for {
if p.process <= limit {
fmt.Println("breaking for new process")
break
}
time.Sleep(time.Second * 10)
}
p.process++
go func() {
fmt.Println("+ Starting goroutine")
p.spawner <- struct{}{}
time.Sleep(time.Second * 2)
fmt.Println("- Stopping goroutine")
p.complete <- struct{}{}
}()
case <-p.complete:
fmt.Println("complete")
p.process--
}
}
}
func main() {
interruptChannel := make(chan os.Signal, 1)
signal.Notify(interruptChannel, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
pm := NewProcessManager()
go pm.Run(5)
pm.spawner <- struct{}{}
<-interruptChannel
}

Managed to solve this by switching process to process: make(chan int, 4)
Then I just used this to block instead of the for loop p.process <- 1
and then use this to mark a routine as completed <-p.process
The length of the channel (4) will determine the max number of processed allowed to run at any given time. Updated test I provided below:
func NewProcessManager() *ProcessManager {
return &ProcessManager{
spawner: make(chan struct{}),
complete: make(chan struct{}),
process: make(chan int, 4),
}
}
func (p *ProcessManager) Run() {
for {
select {
case <-p.spawner:
p.process <- 1
go func() {
fmt.Println("+ Starting goroutine")
// do stuff before starting next routine
p.spawner <- struct{}{}
// do stuff rest of stuff
fmt.Println("- Stopping goroutine")
<-p.process
}()
}
}

The whole purpose of the ProcessManager in my eyes is to serialize access to the metadata keeping track of running processes. As such, it runs sequential.
case <-p.spawner:
for {
if p.process <= limit {
fmt.Println("breaking for new process")
break
}
time.Sleep(time.Second * 10)
}
In this section of code, the goroutine is sleeping. While sleeping, the complete case in the select statement cannot run.
Recall the following property of unbuffered channels:
By default, sends and receives block until the other side is ready.
Because the channels are unbuffered, this line of code must block until that complete case is triggered:
p.complete <- struct{}{}

Related

Will Go leak memory if a goroutine is trying to send a message on a thread that no one is listening on?

Synopsis
I have a timer utility that I wrote—its exact purpose isn't important, but that's the motivating context—which spawns one process to sleep for a certain amount of time, and another process to listen on a channel to know when to run a function.
Here's the catch: I need to be able to restart that timer. I've decided to make it so that I can swap out which channel the function-running process is listening on, effectively making it so that it waits for a different process to send a message.
Code
Here's the code I've written:
type Timer struct {
mu sync.Mutex
Alarm func() // Function to run on timeout
lastInterval time.Duration // Last time used (for "reset")
lastWakeCh chan int // Where is the alarm listening?
alarmUpdate chan (chan int) // Update what channel the alarm runs on
}
func (t *Timer) start(timeout time.Duration) {
t.mu.Lock()
defer t.mu.Unlock()
t.lastInterval = timeout
t.lastWakeCh = make(chan int)
t.alarmUpdate = make(chan (chan int))
go sleepAndNotify(timeout, t.lastWakeCh)
go alarmThread(t.alarmUpdate, t.Alarm)
t.alarmUpdate <- t.lastWakeCh
}
func (t *Timer) restart() {
t.mu.Lock()
defer t.mu.Unlock()
t.lastWakeCh = make(chan int)
t.alarmUpdate <- t.lastWakeCh
go sleepAndNotify(t.lastInterval, t.lastWakeCh)
}
func sleepAndNotify(timeout time.Duration, ch chan int) {
time.Sleep(timeout)
ch <- 1
}
func alarmThread(chUpdate chan (chan int), alarm func()) {
ch := <-chUpdate
for {
select {
case newCh := <-chUpdate:
ch = newCh
case <-ch:
go alarm()
}
}
}
You can use it like so:
t1 := Timer{Alarm: func() { ... }}
t1.start(100 * time.Millisecond)
// some time passes
t1.restart()
Note that every time restart is called, a new goroutine running sleepAndNotify will be created, and will block while trying to send a message down the wake channel ch.
Will this cause a memory/process leak? Or is Go smart enough to see that there are no more references to this channel (or something like that) and so no other process could possible read from this, and just abort that goroutine?
If that is not the case, is there a cheap and easy fix for this?

GO routine never exits based on stop condition - unable to find the reason

In this example, we have a worker. The idea here is simulate clean shutdown of all go routines based on a condition.
In this case, go routines get spun - based on workers count. Each go routine reads the channel, does some work and sends output to the outputChannel.
The main go routine reads this output and prints it. To simulate a stop condition, the doneChannel is closed. Expected outcome is that select inside each go routine will pick this up and execute return which in turn will call the defer println. The actual output is that it never gets called and main exits.
Not sure what's the reason behind this.
package main
import (
"log"
"time"
)
const jobs = 100
const workers = 1
var timeout = time.After(5 * time.Second)
func main() {
doneChannel := make(chan interface{})
outputChannel := make(chan int)
numberStream := generator()
for i := 1; i <= workers; i++ {
go worker(doneChannel, numberStream, outputChannel)
}
// listen for output
loop:
for {
select {
case i := <-outputChannel:
log.Println(i)
case <-timeout:
// before you timeout cleanup go routines
break loop
}
}
close(doneChannel)
time.Sleep(5 * time.Second)
log.Println("main exited")
}
func generator() <-chan int {
defer log.Println("generator completed !")
c := make(chan int)
go func() {
for i := 1; i <= jobs; i++ {
c <- i
}
defer close(c)
}()
return c
}
func worker(done <-chan interface{}, c <-chan int, output chan<- int) {
// this will be a go routine
// Do some work and send results to output Channel.
// Incase if the done channel is called kill the go routine.
defer log.Println("go routines exited")
for {
select {
case <-done:
log.Println("here")
return
case i := <-c:
time.Sleep(1 * time.Second) // worker delay
output <- i * 100
}
}
}
When your main loop finishes during the timeout, you continue your program and
Close done channel
Print message
Exit
There is no reason to wait for any goroutine to process the signal of this channel.
If you add a small sleep you will see some messages
In real scenarios we use a waitgroup to be sure all goroutine finish properly

send multiple messages to one channel in go for collating

I am trying to wrap my mind around go. I want to make a simple program that basically
starts a bunch of go routines
process messages
sends the processed result to a channel
have the main thread collect these results
shut down.
Seems simple. I started with no logic at all. I just send a number and try to get that number back.
issue: I'm deadlocking and I'm not sure why. I think I might be misusing wait groups with channels, because they work individually, but I'm not sure how to get the main thread to block on an arbitrary number of initiated go routines.
package main
import (
"fmt"
"sync"
"time"
)
func main() {
queue := make(chan int)
start := time.Now()
var wg sync.WaitGroup
for i := 0; i < 10; i += 1 {
wg.Add(1)
go count(i, queue, &wg)
}
wg.Wait()
for value := range queue {
println(value)
}
close(queue)
fmt.Println(time.Now().Sub(start))
// fmt.Println(summation)
}
func count(number int, queue chan int, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("Starting ", number)
queue <- number
fmt.Println("ending")
}
Your goroutines block on queue <- number because queue is an unbuffered channel and nobody is reading from it, as main blocks on wg.Wait.
Declare queue as a buffered channel instead. For example: queue := make(chan int, 10)
From the Go Tour (concurrency) and subsequent page:
By default, sends and receives block until the other side is ready. This allows goroutines to synchronize without explicit locks or condition variables.
Sends to a buffered channel block only when the buffer is full. Receives block when the buffer is empty.
Alternatively, move wg.Wait after the for v := range queue loop.
This should help.
package main
import (
"fmt"
"sync"
"time"
)
type event struct {
data chan string
numWorker int
}
func (e event) Send() {
var wg sync.WaitGroup
// Spaw numWorker goroutines that sends message to
// the same channel.
for i := 0; i < e.numWorker; i++ {
wg.Add(1)
go func(id int) {
// Do some fake work
time.Sleep(1 * time.Second)
e.data <- fmt.Sprintf("message from go #%d", id)
wg.Done()
}(i)
}
// Wait for goroutines to finish their work.
wg.Wait()
// Close the channel to signal Recv to stop ranging
// over the channel.
close(e.data)
}
func (e event) Recv() {
// Range over the data channel to receive message(s).
for msg := range e.data {
fmt.Println(msg)
}
}
func main() {
e := event{
numWorker: 10, // Number of worker goroutine(s)
data: make(chan string, 5 /* Buffer Size */),
}
// Spawn a goroutine for Send
go e.Send()
// Recv receives data from Send
e.Recv()
}
To avoid deadlocking you can manage the channel and wait groups in separate goroutine. Try change that:
wg.Wait()
for value := range queue {
println(value)
}
close(queue)
with this:
go func() {
wg.Wait()
close(queue)
}()
for value := range queue {
println(value)
}

Send value through channel to multiple go routines

I want to send a value in a channel to go routines from main function. What happens is which go routine will receive the value from the channel first.
package main
import (
"fmt"
"math/rand"
//"runtime"
"strconv"
"time"
)
func main() {
var ch chan int
ch = make(chan int)
ch <- 1
receive(ch)
}
func receive(ch chan int){
for i := 0; i < 4; i++ {
// Create some threads
go func(i int) {
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
fmt.Println(<-ch)
}(i)
}
}
My current implementation is giving an error.
fatal error: all goroutines are asleep - deadlock!
How can I know that which go routine will receive the value from the channel first. And what happens to other go routine If those will run or throw an error since there is no channel to receive the value. As it is already received by one of them.
If a create a buffered channel my code works. So I don't get it what has happened behind the scene which is making it work when creating a buffered channel like below:
func main() {
var ch chan int
ch = make(chan int, 10)
ch <- 1
receive(ch)
}
If we look at below code. I can see that we can send values through channels directly there is no need of creating a go routine to send a value thorugh a channel to another go routines.
package main
import "fmt"
func main() {
// We'll iterate over 2 values in the `queue` channel.
queue := make(chan string, 2)
queue <- "one"
queue <- "two"
close(queue)
for elem := range queue {
fmt.Println(elem)
}
}
Then what is wrong with my code. Why is it creating a deadlock.
If all you need is to start several workers and send a task to any of them, then you'd better run workers before sending a value to a channel, because as #mkopriva said above, writing to a channel is a blocking operation. You always have to have a consumer, or the execution will freeze.
func main() {
var ch chan int
ch = make(chan int)
receive(ch)
ch <- 1
}
func receive(ch chan int) {
for i := 0; i < 4; i++ {
// Create some threads
go func(i int) {
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
fmt.Printf("Worker no %d is processing the value %d\n", i, <-ch)
}(i)
}
}
Short answer for the question "Which go routine will receive it?" - Whatever. :) Any of them, you can't say for sure.
However I have no idea what is time.Sleep(...) for there, kept it as is.
An unbuffered channel (without a length) blocks until the value has been received. This means the program that wrote to the channel will stop after writing to the channel until it has been read from. If that happens in the main thread, before your call to receive, it causes a deadlock.
There are two more issues: you need to use a WaitGroup to pause the completion until finished, and a channel behaves like a concurrent queue. In particular, it has push and pop operations, which are both performed using <-. For example:
//Push to channel, channel contains 1 unless other things were there
c <- 1
//Pop from channel, channel is empty
x := <-c
Here is a working example:
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
func main() {
var ch chan int
ch = make(chan int)
go func() {
ch <- 1
ch <- 1
ch <- 1
ch <- 1
}()
receive(ch)
}
func receive(ch chan int) {
wg := &sync.WaitGroup{}
for i := 0; i < 4; i++ {
// Create some threads
wg.Add(1)
go func(i int) {
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
fmt.Println(<-ch)
wg.Done()
}(i)
}
wg.Wait()
fmt.Println("done waiting")
}
Playground Link
As you can see the WaitGroup is quite simple as well. You declare it at a higher scope. It's essentially a fancy counter, with three primary methods. When you call wg.Add(1) the counter is increased, when you call wg.Done() the counter is decreased, and when you call wg.Wait(), the execution is halted until the counter reaches 0.

How to check a channel is closed or not without reading it?

This is a good example of workers & controller mode in Go written by #Jimt, in answer to
"Is there some elegant way to pause & resume any other goroutine in golang?"
package main
import (
"fmt"
"runtime"
"sync"
"time"
)
// Possible worker states.
const (
Stopped = 0
Paused = 1
Running = 2
)
// Maximum number of workers.
const WorkerCount = 1000
func main() {
// Launch workers.
var wg sync.WaitGroup
wg.Add(WorkerCount + 1)
workers := make([]chan int, WorkerCount)
for i := range workers {
workers[i] = make(chan int)
go func(i int) {
worker(i, workers[i])
wg.Done()
}(i)
}
// Launch controller routine.
go func() {
controller(workers)
wg.Done()
}()
// Wait for all goroutines to finish.
wg.Wait()
}
func worker(id int, ws <-chan int) {
state := Paused // Begin in the paused state.
for {
select {
case state = <-ws:
switch state {
case Stopped:
fmt.Printf("Worker %d: Stopped\n", id)
return
case Running:
fmt.Printf("Worker %d: Running\n", id)
case Paused:
fmt.Printf("Worker %d: Paused\n", id)
}
default:
// We use runtime.Gosched() to prevent a deadlock in this case.
// It will not be needed of work is performed here which yields
// to the scheduler.
runtime.Gosched()
if state == Paused {
break
}
// Do actual work here.
}
}
}
// controller handles the current state of all workers. They can be
// instructed to be either running, paused or stopped entirely.
func controller(workers []chan int) {
// Start workers
for i := range workers {
workers[i] <- Running
}
// Pause workers.
<-time.After(1e9)
for i := range workers {
workers[i] <- Paused
}
// Unpause workers.
<-time.After(1e9)
for i := range workers {
workers[i] <- Running
}
// Shutdown workers.
<-time.After(1e9)
for i := range workers {
close(workers[i])
}
}
But this code also has an issue: If you want to remove a worker channel in workers when worker() exits, dead lock happens.
If you close(workers[i]), next time controller writes into it will cause a panic since go can't write into a closed channel. If you use some mutex to protect it, then it will be stuck on workers[i] <- Running since the worker is not reading anything from the channel and write will be blocked, and mutex will cause a dead lock. You can also give a bigger buffer to channel as a work-around, but it's not good enough.
So I think the best way to solve this is worker() close channel when exits, if the controller finds a channel closed, it will jump over it and do nothing. But I can't find how to check a channel is already closed or not in this situation. If I try to read the channel in controller, the controller might be blocked. So I'm very confused for now.
PS: Recovering the raised panic is what I have tried, but it will close goroutine which raised panic. In this case it will be controller so it's no use.
Still, I think it's useful for Go team to implement this function in next version of Go.
There's no way to write a safe application where you need to know whether a channel is open without interacting with it.
The best way to do what you're wanting to do is with two channels -- one for the work and one to indicate a desire to change state (as well as the completion of that state change if that's important).
Channels are cheap. Complex design overloading semantics isn't.
[also]
<-time.After(1e9)
is a really confusing and non-obvious way to write
time.Sleep(time.Second)
Keep things simple and everyone (including you) can understand them.
In a hacky way it can be done for channels which one attempts to write to by recovering the raised panic. But you cannot check if a read channel is closed without reading from it.
Either you will
eventually read the "true" value from it (v <- c)
read the "true" value and 'not closed' indicator (v, ok <- c)
read a zero value and the 'closed' indicator (v, ok <- c) (example)
will block in the channel read forever (v <- c)
Only the last one technically doesn't read from the channel, but that's of little use.
I know this answer is so late, I have wrote this solution, Hacking Go run-time, It's not safety, It may crashes:
import (
"unsafe"
"reflect"
)
func isChanClosed(ch interface{}) bool {
if reflect.TypeOf(ch).Kind() != reflect.Chan {
panic("only channels!")
}
// get interface value pointer, from cgo_export
// typedef struct { void *t; void *v; } GoInterface;
// then get channel real pointer
cptr := *(*uintptr)(unsafe.Pointer(
unsafe.Pointer(uintptr(unsafe.Pointer(&ch)) + unsafe.Sizeof(uint(0))),
))
// this function will return true if chan.closed > 0
// see hchan on https://github.com/golang/go/blob/master/src/runtime/chan.go
// type hchan struct {
// qcount uint // total data in the queue
// dataqsiz uint // size of the circular queue
// buf unsafe.Pointer // points to an array of dataqsiz elements
// elemsize uint16
// closed uint32
// **
cptr += unsafe.Sizeof(uint(0))*2
cptr += unsafe.Sizeof(unsafe.Pointer(uintptr(0)))
cptr += unsafe.Sizeof(uint16(0))
return *(*uint32)(unsafe.Pointer(cptr)) > 0
}
Well, you can use default branch to detect it, for a closed channel will be selected, for example: the following code will select default, channel, channel, the first select is not blocked.
func main() {
ch := make(chan int)
go func() {
select {
case <-ch:
log.Printf("1.channel")
default:
log.Printf("1.default")
}
select {
case <-ch:
log.Printf("2.channel")
}
close(ch)
select {
case <-ch:
log.Printf("3.channel")
default:
log.Printf("3.default")
}
}()
time.Sleep(time.Second)
ch <- 1
time.Sleep(time.Second)
}
Prints
2018/05/24 08:00:00 1.default
2018/05/24 08:00:01 2.channel
2018/05/24 08:00:01 3.channel
Note, refer to comment by #Angad under this answer:
It doesn't work if you're using a Buffered Channel and it contains
unread data
I have had this problem frequently with multiple concurrent goroutines.
It may or may not be a good pattern, but I define a a struct for my workers with a quit channel and field for the worker state:
type Worker struct {
data chan struct
quit chan bool
stopped bool
}
Then you can have a controller call a stop function for the worker:
func (w *Worker) Stop() {
w.quit <- true
w.stopped = true
}
func (w *Worker) eventloop() {
for {
if w.Stopped {
return
}
select {
case d := <-w.data:
//DO something
if w.Stopped {
return
}
case <-w.quit:
return
}
}
}
This gives you a pretty good way to get a clean stop on your workers without anything hanging or generating errors, which is especially good when running in a container.
You could set your channel to nil in addition to closing it. That way you can check if it is nil.
example in the playground:
https://play.golang.org/p/v0f3d4DisCz
edit:
This is actually a bad solution as demonstrated in the next example,
because setting the channel to nil in a function would break it:
https://play.golang.org/p/YVE2-LV9TOp
ch1 := make(chan int)
ch2 := make(chan int)
go func(){
for i:=0; i<10; i++{
ch1 <- i
}
close(ch1)
}()
go func(){
for i:=10; i<15; i++{
ch2 <- i
}
close(ch2)
}()
ok1, ok2 := false, false
v := 0
for{
ok1, ok2 = true, true
select{
case v,ok1 = <-ch1:
if ok1 {fmt.Println(v)}
default:
}
select{
case v,ok2 = <-ch2:
if ok2 {fmt.Println(v)}
default:
}
if !ok1 && !ok2{return}
}
}
From the documentation:
A channel may be closed with the built-in function close. The multi-valued assignment form of the receive operator reports whether a received value was sent before the channel was closed.
https://golang.org/ref/spec#Receive_operator
Example by Golang in Action shows this case:
// This sample program demonstrates how to use an unbuffered
// channel to simulate a game of tennis between two goroutines.
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
// wg is used to wait for the program to finish.
var wg sync.WaitGroup
func init() {
rand.Seed(time.Now().UnixNano())
}
// main is the entry point for all Go programs.
func main() {
// Create an unbuffered channel.
court := make(chan int)
// Add a count of two, one for each goroutine.
wg.Add(2)
// Launch two players.
go player("Nadal", court)
go player("Djokovic", court)
// Start the set.
court <- 1
// Wait for the game to finish.
wg.Wait()
}
// player simulates a person playing the game of tennis.
func player(name string, court chan int) {
// Schedule the call to Done to tell main we are done.
defer wg.Done()
for {
// Wait for the ball to be hit back to us.
ball, ok := <-court
fmt.Printf("ok %t\n", ok)
if !ok {
// If the channel was closed we won.
fmt.Printf("Player %s Won\n", name)
return
}
// Pick a random number and see if we miss the ball.
n := rand.Intn(100)
if n%13 == 0 {
fmt.Printf("Player %s Missed\n", name)
// Close the channel to signal we lost.
close(court)
return
}
// Display and then increment the hit count by one.
fmt.Printf("Player %s Hit %d\n", name, ball)
ball++
// Hit the ball back to the opposing player.
court <- ball
}
}
it's easier to check first if the channel has elements, that would ensure the channel is alive.
func isChanClosed(ch chan interface{}) bool {
if len(ch) == 0 {
select {
case _, ok := <-ch:
return !ok
}
}
return false
}
If you listen this channel you always can findout that channel was closed.
case state, opened := <-ws:
if !opened {
// channel was closed
// return or made some final work
}
switch state {
case Stopped:
But remember, you can not close one channel two times. This will raise panic.

Resources