Go: Act on Signal while not Blocking Operations - go

I want to fire the ps command continuously in a goroutine to monitor mem and cpu usages. I didn't use top because top doesn't allow me to select columns as ps does. This goroutine needs to receive a stop signal to stop the command, but I don't know how not to block running the command while waiting the signal. For top I can do:
top := exec.Command("top")
<-stop // blocking
top.Process.Signal(os.Kill)
But for ps if I do:
ps := exec.Command("ps")
for {
ps.Run()
<-stop
}
The above code will block on stop. I want to keep firing ps.Run(), while being able to stop when a stop signal is ready. Thanks.

One way that you could achieve this is by utilizing the for/select timeout idiom, there are a couple of similar methods of doing this. Take this trivial example:
package main
import (
"fmt"
"time"
)
func main() {
abort := make(chan struct{})
go func() {
for {
select {
case <-abort:
return
case <-time.After(1 * time.Second):
// replace fmt.Println() with the command you wish to run
fmt.Println("tick")
}
}
}()
// replace time.Sleep() with code waiting for 'abort' command input
time.Sleep(10 * time.Second)
abort <- struct{}{}
}
To modify this example to work in your circumstance place the code that you want to run in the <-time.After(): case, which (in this instance) will run once per second, if no other case is available to receive for that duration. And instead of time.Sleep() which I placed at the end, put the code that will interrupt the <-time.After(): case, sending <- struct{}{} on the abort channel (or whatever you name it).
NOTE: In an earlier version of this answer I had abort as a chan bool, as I like the clarity of <-abort true and don't consider chan struct{} to be as clear, I opted to change it in this example however, as <- struct{}{} isn't unclear, especially once you've gotten used to the pattern.
Also, if you want the command to execute on each iteration of the for loop and not wait for a timeout then you can make that case default:, remove <-time.After() and it will run on each iteration of the loop where another channel is not ready to receive.
You can play with this example in the playground if you'd like, although it will not allow syscalls, or the default: case example to be run in that environment.

To run it over and over, without sleep:
func run(stop <-chan struct{}) {
ps := exec.Command("ps")
for {
select {
case <-stop:
return
default:
ps.Run()
//processing the output
}
}
}
And then go run(stop). And to run it every 3 seconds (for example):
func run(stop <-chan struct{}) {
ps := exec.Command("ps")
tk := time.Tick(time.Second * 3)
for {
select {
case <-stop:
return
case <-tk:
ps.Run()
//processing the output
}
}
}

Related

Concurrency in Go loop polling case branch not hit

I am implementing a very simple concurrent program in Go. There are 2 channels todo and done that are used for signaling which task is done. There are 5 routines that are executed and each one require its own time to complete. I would like to see every 100ms the status of what is happening.
However I tried but the polling branch case <-time.After(100 * time.Millisecond): seems that is never been called. It is called sometimes (not in a consisted way) if I reduce the time to something less than 100ms.
My understanding is that go func executes the method in a separate Go scheduler thread. I do not understand therefore why the case of the polling is never hit. I tried to move the specific case branch before/after the other but nothing changed.
Any suggestions?
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
func concurrent(id int, done chan int, todo chan int) {
for {
// doing a task
t := randInt(50, 100)
time.Sleep(time.Duration(t) * time.Millisecond)
done <- id
// redo again this task
t = randInt(50, 100)
time.Sleep(time.Duration(t) * time.Millisecond)
todo <- id
}
}
func randInt(min int, max int) int {
return (min + rand.Intn(max-min))
}
func seedRandom() {
rand.Seed(time.Now().UTC().UnixNano())
}
func main() {
seedRandom()
todo := make(chan int, 5)
done := make(chan int, 5)
for i := 0; i < 5; i++ {
todo <- i
}
timeout := make(chan bool)
go func() {
time.Sleep(1 * time.Second)
timeout <- true
}()
var mu sync.Mutex
var output []int
loop:
for {
select {
case <-time.After(100 * time.Millisecond):
//this branch is never hit?
fmt.Printf("\nPolling status: %v\n", output)
case <-timeout:
fmt.Printf("\nDing ding, time is up!\n")
break loop
case id := <-done:
mu.Lock()
output = append(output, id)
fmt.Printf(".")
mu.Unlock()
case id := <-todo:
go concurrent(id, done, todo)
}
}
}
Update After following the answers I created this version in Go Playgound: https://play.golang.org/p/f08t984BdPt. That works as expected
you are creating 5 goroutines (func concurrent) and in your select case using the todo channel and this channel is being used in concurrent function so you end up creating a lot of goroutines
func concurrent(id int, done chan int, todo chan int) {
for {
// doing a task
t := randInt(50, 100)
time.Sleep(time.Duration(t) * time.Millisecond)
done <- id
// redo again this task
t = randInt(50, 100)
time.Sleep(time.Duration(t) * time.Millisecond)
by doing this call you are re-crating the go-routime
todo <- id
}
}
when I ran your code I got "runtime.NumGoroutine()"
"number of goRoutines still running 347"
as you are implementing the time.After(100 * time.Millisecond) inside the for loop it gets reset every time some other case gets hit and in your case
case id := <-todo: && id := <-done: will always get hit within 100 Milliseconds that's why you didn't get the expected output (from how your code is now i would say that the number of go-routines would increase exponentially and each of em would be waiting to send value to done and few on todo channel so your loop wont get enough time(100 ms) to wait on time.After)
loop:
for {
select {
case <-time.After(100 * time.Millisecond): ->this will always get reset ( we can use time.Ticker as it will create a single object that will signal for each and every 100ms https://golang.org/pkg/time/#NewTicker
//this branch is never hit?
fmt.Printf("\nPolling status: %v\n", output)
case <-timeout:
fmt.Printf("\nDing ding, time is up!\n")
break loop
case id := <-done: -> **this will get called**
//the mutex call is actually not very usefull as this only get called once per loop and is prefectly thread safe in this code
mu.Lock()
output = append(output, id)
fmt.Printf(".")
mu.Unlock()
case id := <-todo: -> **this will get called**
go concurrent(id, done, todo)
}
}
}
https://play.golang.org/p/SmlSIUIF5jn -> I have made some modifications to make your code work as expected..
try referring this to get a better understanding of golang channels and goroutine
https://tour.golang.org/concurrency/1
time.After(100*time.Millisecond) creates a brand new channel, with a brand new timer, which starts at the moment that function is called.
So, in your loop :
for {
select {
// this statement resets the 100ms timer each time you execute the loop :
case <-time.After(100*time.Millisecond):
...
Your branch never gets hit because, with 5 goroutines sending signals within less than 100ms on one of the other cases, this time.After(100ms) never reaches completion.
You need to choose a way to keep the same timer between iterations.
Here is one way to adapt your time.After(...) call :
// store the timer in a variable *outside* the loop :
statusTimer := time.After(100*time.Millisecond)
for {
select {
case <-statusTimer:
fmt.Printf("\nPolling status: %v\n", output)
// reset the timer :
statusTimer = time.After(100*time.Millisecond)
case <-timeout:
...
Another way is, as #blackgreen suggests, to use a time.Ticker :
statusTicker := time.NewTicker(100*time.Millisecond)
for {
select {
case <-statusTicker.C:
fmt.Printf("\nPolling status: %v\n", output)
case <-timeout:
...
side notes
a. if the output slice is not shared with other goroutines, you don't need a mutex around its access :
for {
select {
case <-statusTicker.C:
fmt.Printf("\nPolling status: %v\n", output)
...
case i <-done:
// no race condition here : all happens within the same goroutine,
// the 'select' statement makes sure that 'case's are executed
// one at a time
output = append(output, id)
fmt.Printf(".")
b. For your timeout channel :
Another generic way to "signal" that some event occurred with a channel is to close the channel instead of sending a value on it :
// if you don't actually care about the value you send over this channel :
// you can make it unbuffered, and use the empty 'struct{}' type
timeout := make(chan struct{})
go func(){
// wait for some condition ...
<-time.After(1*time.Second)
close(timeout)
}()
select {
case <-statusTimer:
...
case <-timeout: // this branch will also be taken once timeout is closed
fmt.Printf("\nDing ding, time is up!\n")
break loop
case ...
The bug you will avoid is the following : suppose you want to use that timeout channel in two goroutines
if you send a value over the timeout channel, only one goroutine will get signaled - it will "eat up" the value from the channel, and the other goroutine will only have a blocking channel,
if you close the channel, both goroutines will correctly "receive" the signal
In absence of a default case, when multiple cases are ready, it executes one of them at random. It's not deterministic.
To make sure the case runs, you should run it in a separate goroutine. (In that case, you must synchronize accesses to the output variable).
Moreover you say "I would like to see every 100ms", but time.After sends on the channel only once.
To execute the case periodically, use <-time.NewTicker(100 * time.Millis).C instead.
var mu sync.Mutex
var output []int
go func() {
ticker := time.NewTicker(100 * time.Millisecond)
defer ticker.Stop()
for {
select {
case <-ticker.C:
// TODO: must synchronize access
fmt.Printf("\nPolling status: %v\n", output)
case <-timeout:
return
}
}
}()
loop:
for {
select {
// other cases
}
}

Manage a background operation (create / kill)

I'm posting here for the first time because I couldn't find a clean solution on the internet.
My goal is simple, I need to create a background operation (goroutine or process or whatever...) that I can kill properly (not leave in the background).
I've tried many things like using chan or context.
But I never could find the right way to avoid leaks.
Here is an example:
package main
import (
"log"
"strconv"
"runtime"
"time"
"math/rand"
)
func main() {
log.Println("goroutines: " + strconv.Itoa(runtime.NumGoroutine()))
func1()
leak := ""
if runtime.NumGoroutine() > 1 {
leak = " there is one LEAK !!"
}
log.Println("goroutines: " + strconv.Itoa(runtime.NumGoroutine()) + leak)
}
func func1() {
done := make(chan struct{})
quit := make(chan struct{})
go func() {
log.Println("goroutines: " + strconv.Itoa(runtime.NumGoroutine()))
select {
case <-quit:
log.Println("USEFUL ???")
return
default:
func2()
done<-struct{}{}
}
}()
select {
case <-time.After(4 * time.Second):
quit<-struct{}{}
log.Println("TIMEOUT")
case <-done:
log.Println("NO TIMEOUT")
}
}
func func2() {
log.Println("JOB START")
rand.Seed(time.Now().UnixNano())
val := rand.Intn(10)
log.Println("JOB DURATION: " + strconv.Itoa(val))
time.Sleep(time.Duration(val) * time.Second) // fake a long process with an unknown duration
log.Println("JOB DONE")
}
In this example, if the job is done before the 4 seconds timeout, everything is fine and the final number of goroutines will be 1, but otherwise it will be 2 like every example that I could find.
But it's just an example, maybe it's not possible with goroutines maybe it's even not possible in Go..
Your problem is here:
quit := make(chan struct{})
go func() {
for {
select {
case <-quit:
log.Println("USEFUL ???")
return
default:
func2()
quit<-struct{}{}
return
}
}
}()
This goroutine is signalling itself on an unbuffered channel. That means when it gets to the quit<-struct{}{}, that send will block forever, because it's waiting on itself to receive. It's not entirely clear how this is intended to work, though; there are a few odd things happening here:
A goroutine is signalling itself via channel, which seems wrong - it shouldn't need to communicate to itself
The channel and loop seem unnecessary; quit<-struct{}{} could be replaced with log.Println("USEFUL ???") and the whole for/select/channel business could be deleted
The function returns in each case of the select, so putting it in a loop is pointless anyway - there is no scenario where this code can ever execute a second iteration of the loop

What is the best way to call N concurrent functions periodically in Go?

I've been struggling with a problem for the past day or so in figuring out the best way to create N concurrent functions which are called periodically at the same interval in Go. I want to be able to specify an arbitrary number of functions, have them all run periodically simultaneously, and end them all after a specified amount of time.
Right now I have a solution which works but a new ticker has to be created for each concurrent function. I'm also not sure how to use sync.WaitGroup properly, as my current implementation results in the program never ending (just gets stuck on wg.Wait() at the end)
I briefly looked at a ticker wrapper called Multitick, but I'm not sure how to implement it. Maybe Multitick could be the solution here?
func main() {
N := 10
var wg sync.WaitGroup
wg.Add(N)
quit := make(chan struct{})
for i := 0; i < N; i++ {
tick := time.NewTicker(500 * time.Millisecond)
go func(t *time.Ticker) {
for a := range tick.C {
select {
case <-quit:
break
default:
fmt.Println(a) // do something on tick
}
}
wg.Done()
}(tick)
}
time.Sleep(10 * time.Second)
close(quit)
wg.Wait()
}
Go Playground Demo
So this solution works, executing all of the tickers concurrently at the proper intervals and finishing after 10 seconds, but it doesn't actually exit the program, hanging up on the wg.Wait() line at the end. Additionally, each concurrent function call uses its own ticker- is there any way I can have one "master" ticker that all of the functions operate from?
Thanks in advance! This is my first time really delving in to concurrency in Go.
The reason that your program never exits is a strange quirk of the Go language: the break statement for case <-quit quits the select statement, instead of the loop. (Not sure why this behaviour would ever be useful.) To fix your program, you need to explicitly break the loop:
tickLoop:
for a := range tick.C {
select {
case <-quit:
break tickLoop
default:
fmt.Println(a, "function #", id) // do something on tick
}
}
As the code is written, it always waits until the next tick before quitting. You can fix this, by reading tick.C in the select statement, too:
tickLoop:
for {
select {
case <-quit:
break tickLoop
case a := <-tick.C:
fmt.Println(a, "function #", id) // do something on tick
}
}
Finally, if you want to restructure your program to use only one ticker, you can start an extra goroutine, which listens on the ticker and on the quit channel, and then starts N sub-goroutines on every tick.

"Select" statement inside of goroutine with for loop

Could somebody please explain, why if goroutine has endless for loop and select inside of the loop, a code in the loop is run only one time?
package main
import (
"time"
)
func f1(quit chan bool){
go func() {
for {
println("f1 is working...")
time.Sleep(1 * time.Second)
select{
case <-quit:
println("stopping f1")
break
}
}
}()
}
func main() {
quit := make(chan bool)
f1(quit)
time.Sleep(4 * time.Second)
}
Output:
f1 is working...
Program exited.
But if "select" is commented out:
package main
import (
"time"
)
func f1(quit chan bool){
go func() {
for {
println("f1 is working...")
time.Sleep(1 * time.Second)
//select{
//case <-quit:
//println("stopping f1")
//break
//}
}
}()
}
func main() {
quit := make(chan bool)
f1(quit)
time.Sleep(4 * time.Second)
}
Output:
f1 is working...
f1 is working...
f1 is working...
f1 is working...
f1 is working...
https://play.golang.org/p/MxKy2XqQlt8
A select statement without a default case is blocking until a read or write in at least one of the case statements can be executed. Accordingly, your select will block until a read from the quit channel is possible (either of a value or the zero value if the channel is closed). The language spec provides a concrete description of this behavior, specifically:
If one or more of the communications [expressed in the case statements] can proceed, a single one that can proceed is chosen via a uniform pseudo-random selection. Otherwise, if there is a default case, that case is chosen. If there is no default case, the "select" statement blocks until at least one of the communications can proceed.
Other code issues
Caution! break applies to the select statement
However, even if you did close the quit channel to signal shutdown of your program, your implementation would likely not have the desired effect. A (unlabelled) call to break will terminate execution of the inner-most for, select or switch statement within the function. In this case, the select statement will break and the for loop will run through again. If quit was closed, it will run forever until something else stops the program, otherwise it will block again in select (Playground example)
Closing quit will (probably) not immediately stop the program
As noted in the comments, the call to time.Sleep blocks for a second on each iteration of the loop, so any attempt to stop the program by closing quit will be delayed for up to approximately a second before the goroutine checks quit and escapes. It is unlikely this sleep period must complete in its entirety before the program stops.
More idiomatic Go would block in the select statement on receiving from two channels:
The quit channel
The channel returned by time.After – this call is an abstraction around a Timer which sleeps for a period of time and then writes a value to the provided channel.
Resolution
Solution with minimal changes
A resolution to fix your immediate issue with minimal changes to your code would be:
Make the read from quit non-blocking by adding a default case to the select statement.
Ensuring the goroutine actually returns when the read from quit succeeds:
Label the for loop and use a labelled call to break; or
return from the f1 function when it is time to quit (preferred)
Depending on your circumstances, you may find it more idiomatic Go to use a context.Context to signal termination, and to use a sync.WaitGroup to wait for the goroutine to finish before returning from main.
package main
import (
"fmt"
"time"
)
func f1(quit chan bool) {
go func() {
for {
println("f1 is working...")
time.Sleep(1 * time.Second)
select {
case <-quit:
fmt.Println("stopping")
return
default:
}
}
}()
}
func main() {
quit := make(chan bool)
f1(quit)
time.Sleep(4 * time.Second)
close(quit)
time.Sleep(4 * time.Second)
}
Working example
(Note: I have added an additional time.Sleep call in your main method to avoid this returning immediately after the call to close and terminating the program.)
Fixing the sleep blocking issue
To fix the additional issue regarding the blocking sleep preventing immediate quit, move the sleep to a timer in the select block. Modifying your for loop as per this playground example from the comments does exactly this:
for {
println("f1 is working...")
select {
case <-quit:
println("stopping f1")
return
case <-time.After(1 * time.Second):
// repeats loop
}
}
Related literature
Go-by-example: Non-blocking channel operations
Dave Cheney's Channel axioms, specifically the fourth axiom "A receive from a closed channel returns the zero value immediately."

What is correct way to use goroutine?

I need to apply some tests to each request and fire responce based on result of tests. If one of the test fail, I need to send responce imediatelly, otherwise I wait when all tests are done succesfully. I want to do that tests with concurrency.
Now, I do that like this (simplified):
func handler_request_checker(w http.ResponseWriter, r *http.Request) {
done := make(chan bool)
quit := make(chan bool)
counter := 0
go TestOne(r,done,quit)
go TestTwo(r,done,quit)
..............
go TestTen(r,done,quit)
for {
select {
case <- quit:
fmt.Println("got quit signal")
return
case <- done:
counter++
if counter == 10 {
fmt.Println("All checks passed succesfully")
return
}
}
}
func main() {
http.HandleFunc("/", handler_request_checker)
http.ListenAndServe()
}
Example of one of goroutine:
func TestOne(r *http.Request, done,quit chan bool) {
ip,_,ok := net.SplitHostPort(r.RemoteAddr)
if ok == nil {
for _,item := range BAD_IP_LIST {
if strings.Contains(ip,item) {
quit <- true
return
}
}
done <- true
return
} else {
quit <- true
return
}
}
Problem is that goroutines doesnt' free memory after I go t quit signal. I suppose that happens because there is something in done chanel.
I'm completely new in GO, so maybe I use them wrong way?
For example, when I start load check http_load -parallel 10 -seconds 10, that goroutine easily eats 100+ MB of RAM and dont'g give it back to the system. On next check, it eat 100+ MB more, and so on.
If I do that tests without go (step-by-step), program takes no more than 10-15mb with any load checks.
Your guess is right. You are using synchronous channels, which means that both the sender and the receiver must be available in order to transmit a value.
Once your handler_request_checker function gets a quit signal, it stops retrieving any values of the done and the quit channel. Further TestOne, TestTwo, etc. goroutines will be blocked when they try to send their result. Even worse, they will stay in memory forever because they are still running since they haven't transmitted their result yet.
One way to solve the problem would be to use buffered (asynchronous) channels for done and quit. For example:
done := make(chan bool, 10)
quit := make(chan bool, 10)
If you use a buffer size of 10 for your 10 tests, then all goroutines are able to send their results, even when there is no reader available anymore. All goroutines will exit cleanly and the channels (that might contain some unread results) will get garbage collected once all goroutines have quit.

Resources