should I use a channel or a sync.Mutex lock()? - go

While doing a go test -race, I found that a call to os.Process.Kill, was made before the command started cmd.Start(), I came with to posible solutions, one to use a channel:
package main
import "os/exec"
func main() {
cmd := exec.Command("sleep", "10")
started := make(chan struct{}, 1)
go func() {
<-started
cmd.Process.Kill()
}()
if err := cmd.Start(); err != nil {
panic(err)
}
started <- struct{}{}
cmd.Wait()
}
or to use a lock:
package main
import (
"os/exec"
"sync"
)
func main() {
var lock sync.Mutex
cmd := exec.Command("sleep", "10")
lock.Lock()
if err := cmd.Start(); err != nil {
panic(err)
}
lock.Unlock()
go func() {
cmd.Process.Kill()
}()
cmd.Wait()
}
Both options work but wondering what could be the most idiomatic or better approach, while the main goal is just to prevent killing a process that hasn't been started.

I would suggest you use a channel, but let me point out something about your code.
I noticed you used a buffered channel, and then sent data on that channel first, before calling the goroutine that consumes the channel. In my opinion, it would be better to:
1) use an unbuffered channel for signalling, especially in this case.
2) Have the goroutine be responsible for starting the process and calling wait, while signaling to the main that it has started.
Like this:
package main
import "os/exec"
func main() {
cmd := exec.Command("sleep", "10")
started := make(chan struct{})
go func(cmd *exec.Cmd, signal chan struct{}) {
if err := cmd.Start(); err != nil {
panic(err)
}
started <- struct{}{}
cmd.Wait()
}(cmd, started)
<-started
cmd.Process.Kill()
}

Related

goroutine not seeing context cancel?

I have two goroutines running at the same time.
At some point, I want my program to exit gracefully so I use the cancel() func to notify my goroutines that they need to be stopped, but only one of the two receive the message.
here is my main (simplified):
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
done := make(chan os.Signal, 1)
signal.Notify(done, os.Interrupt, syscall.SIGINT, syscall.SIGTERM)
wg := &sync.WaitGroup{}
wg.Add(2)
go func() {
err := eng.Watcher(ctx, wg)
if err != nil {
cancel()
}
}()
go func() {
err := eng.Suspender(ctx, wg)
if err != nil {
cancel()
}
}()
<-done // wait for SIGINT / SIGTERM
log.Print("receive shutdown")
cancel()
wg.Wait()
log.Print("controller exited properly")
The Suspender goroutine exist successfully (here is the code):
package main
import (
"context"
"sync"
"time"
log "github.com/sirupsen/logrus"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/util/retry"
)
func (eng *Engine) Suspender(ctx context.Context, wg *sync.WaitGroup) error {
contextLogger := eng.logger.WithFields(log.Fields{
"go-routine": "Suspender",
})
contextLogger.Info("starting Suspender goroutine")
now := time.Now().In(eng.loc)
for {
select {
case n := <-eng.Wl:
//dostuff
case <-ctx.Done():
// The context is over, stop processing results
contextLogger.Infof("goroutine Suspender canceled by context")
return nil
}
}
}
and here is the func that is not receiving the context cancellation:
package main
import (
"context"
"sync"
"time"
log "github.com/sirupsen/logrus"
)
func (eng *Engine) Watcher(ctx context.Context, wg *sync.WaitGroup) error {
contextLogger := eng.logger.WithFields(log.Fields{
"go-routine": "Watcher",
"uptime-schedule": eng.upTimeSchedule,
})
contextLogger.Info("starting Watcher goroutine")
ticker := time.NewTicker(time.Second * 30)
for {
select {
case <-ctx.Done():
contextLogger.Infof("goroutine watcher canceled by context")
log.Printf("toto")
return nil
case <-ticker.C:
//dostuff
}
}
}
}
Can you please help me ?
Thanks :)
Did you try it with an errgroup? It has context cancellation baked in:
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
defer cancel()
done := make(chan os.Signal, 1)
signal.Notify(done, os.Interrupt, syscall.SIGINT, syscall.SIGTERM)
// "golang.org/x/sync/errgroup"
wg, ctx := errgroup.WithContext(ctx)
wg.Go(func() error {
return eng.Watcher(ctx, wg)
})
wg.Go(func() error {
return eng.Suspender(ctx, wg)
})
wg.Go(func() error {
defer cancel()
<-done
return nil
})
err := wg.Wait()
if err != nil {
log.Print(err)
}
log.Print("receive shutdown")
log.Print("controller exited properly")
On the surface the code looks good. The only thing I can think is that it's busy in "dostuff". It can be tricky to step through timing related code in the debugger so try adding some logging:
case <-ticker.C:
log.Println("doing stuff")
//dostuff
log.Println("done stuff")
(I also assume you are calling wg.Done() in your go-routines somewhere though if they are missing that would not be the cause of the problem you describe.)
The code in Suspender and in Watcher doesn't decrement the waitgroup counter through the Done() method call - the reason behind the infinite execution.
And to be honest it's quite normal to forget such small things. That's why as a standard general practice in Go, it is suggested to use defer and handle things that are critical (and should be handled inside the function/method ) at the very beginning.
The updated implementation might look like
func (eng *Engine) Suspender(ctx context.Context, wg *sync.WaitGroup) error {
defer wg.Done()
// ------------------------------------
func (eng *Engine) Watcher(ctx context.Context, wg *sync.WaitGroup) error {
defer wg.Done()
contextLogger := eng.logger.WithFields(log.Fields{
Also, another suggestion, looking at the main routine, it is always suggested to pass context by value to any go-routine or method calls (lambda) that are being invoked.
This approach saves developers from a lot of program-related bugs that can't be noticed very easily.
go func(ctx context.Context) {
err := eng.Watcher(ctx, wg)
if err != nil {
cancel()
}
}(ctx)
Edit-1: (the exact solution)
Try passing the context using the value in the go routines as I mentioned earlier. Otherwise, both of the go routine will use a single context (because you are referencing it) and only one ctx.Done() will be fired.
By passing ctx as a value 2 separate child contexts are created in Go. And while closing parent with cancel() - both children independently fires ctx.Done().

Handling multiple websocket connections

I'm trying to create a program that will connect to several servers though gorilla web-sockets. I currently have a program that will iterate over a list of server addresses and create a new goroutine that will create its own Websocket.conn and handle reading and writing.
The problem is that every time a new goroutine is created the previous goroutines are blocked and only the last one can continue. I believe this is because the gorilla websocket library is blocking each gorotutine, but I might be mistaken.
I have tried putting a timer in the server list iterator and each goroutine will work perfectly but then the moment a new goroutine is made with another address the previous goroutine is blocked.
The relevant bits of my code:
In my main.go
for _, server := range servers {
go control(ctx, server, port)
}
In control()
func control(ctx context.Context, server, port string) {
url := url.URL{
Scheme: "ws",
Host: server + ":" + port,
Path: "",
}
conn, _, err := websocket.DefaultDialer.Dial(url.String(), nil)
if err != nil {
panic(err)
}
defer conn.Close()
go sendHandler(ctx, conn)
go readHandler(ctx, conn)
}
readHandler(ctx context.Context, conn *websocket.Con) {
for {
_, p, err := conn.ReadMessage(); if err != nil {
panic(err)
}
select {
case <-ctx.Done():
goto TERM
default:
// do nothing
}
}
TERM:
// do termination
}
sendHandler(ctx context.Context, conn *websocket.Con) {
for _, msg := range msges {
err = conn.WriteMessage(websocket.TextMessage, msg)
if err != nil {
panic(err)
}
}
<-ctx.Done()
}
I removed the parts where I add waitgroups and other unnecessary pieces of code.
So what I expect is for there to be 3n goroutines running (where n is the number of servers) without blocking but right now I see only 3 goroutines running which are the ones called by the last iteration of the server list.
Thanks!
EDIT 14/06/2019:
I spent some time making a small working example and in the example the bug did not occur - none of the threads blocked each other. I'm still unsure what was causing it but here is my small working example:
main.go
package main
import (
"context"
"fmt"
"os"
"time"
"os/signal"
"syscall"
"sync"
"net/url"
"github.com/gorilla/websocket"
)
func main() {
servers := []string{"5555","5556", "5557"}
comms := make(chan os.Signal, 1)
signal.Notify(comms, os.Interrupt, syscall.SIGTERM)
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
var wg sync.WaitGroup
for _, server := range servers {
wg.Add(1)
go control(server,
ctx,
&wg)
}
<-comms
cancel()
wg.Wait()
}
func control(server string, ctx context.Context, wg *sync.WaitGroup) {
fmt.Printf("Started control for %s\n", server)
url := url.URL {
Scheme: "ws",
Host: "0.0.0.0" + ":" + server,
Path: "",
}
conn, _, err := websocket.DefaultDialer.Dial(url.String(), nil)
if err != nil {
panic(err)
}
defer conn.Close()
var localwg sync.WaitGroup
localwg.Add(1)
go sendHandler(ctx, conn, &localwg, server)
localwg.Add(1)
go readHandler(ctx, conn, &localwg, server)
<- ctx.Done()
localwg.Wait()
wg.Done()
return
}
func sendHandler(ctx context.Context, conn *websocket.Conn, wg *sync.WaitGroup, server string) {
for i := 0; i < 50; i++ {
err := conn.WriteMessage(websocket.TextMessage, []byte("ping"))
if err != nil {
panic(err)
}
fmt.Printf("sent msg to %s\n", server)
time.Sleep(1 * time.Second)
}
<- ctx.Done()
wg.Done()
}
func readHandler(ctx context.Context, conn *websocket.Conn, wg *sync.WaitGroup, server string) {
for {
select {
case <- ctx.Done():
wg.Done()
return
default:
_, p, err := conn.ReadMessage()
if err != nil {
wg.Done()
fmt.Println("done")
}
fmt.Printf("Got [%s] from %s\n", string(p), server)
}
}
}
I tested it with dpallot's simple-websocket-server by a server on 5555, 5556 and 5557 respectively.
This part of your code is causing the problem:
conn, _, err := websocket.DefaultDialer.Dial(url.String(), nil)
if err != nil {
panic(err)
}
defer conn.Close()
go sendHandler(ctx, conn)
go readHandler(ctx, conn)
You create the connection, defer the close of it, start two other goroutines and then end the function. The function end closes the socket due to your defer.

close(channel) used to implement the observer pattern

I need to stop an HTTP server on demand besides calling other functions as well when receiving the "quit" signal in no specific order.
In my try to implement something like the observer pattern, I found "handy" to create a channel (quit := make(chan struct{}), let's say the "subject" and then on each of the goroutines "observers" listen on that channel <-quit waiting until a change for then to continue.
The way I trigger all the functions at once is by closing the channel close(quit) not by writing into it, I have tried this and so far working, but wondering if there are some cons with this approach or if there are better/idiomatic ways of implementing similar behavior/pattern.
package main
import (
"log"
"net/http"
"sync"
"time"
)
func main() {
var wg sync.WaitGroup
srv := &http.Server{Addr: ":8080"}
wg.Add(1)
go func() {
log.Println(srv.ListenAndServe())
wg.Done()
}()
quit := make(chan struct{})
go func() {
<-quit
if err := srv.Close(); err != nil {
log.Printf("HTTP server Shutdown: %v", err)
}
}()
wg.Add(1)
go func() {
<-quit
log.Println("just waiting 1")
wg.Done()
}()
wg.Add(1)
go func() {
<-quit
log.Println("just waiting 2")
wg.Done()
}()
<-time.After(2 * time.Second)
close(quit)
wg.Wait()
}
https://play.golang.org/p/uIfMJfN6xQy
I would say your way is good enough but lacks some elegance.
You could implement required behavior using sync.Cond:
https://golang.org/pkg/sync/#Cond
How to correctly use sync.Cond?

Why don't the GO routines start?

I wrote a simple GO program that should start 3 GO routines. However, the GO routines don't start.
Please note that the situation is not identical to the one describer within this post :
Why is my goroutine not executed?
The program should wait for the GO routines to end their execution... Thus, the program should wait forever (since the routines never stop).
package main
import (
"fmt"
"net"
"os"
"time"
"sync"
)
func main() {
wg := sync.WaitGroup{}
fmt.Print("Starting 3 clients\n")
for i:=0; i<3; i++ {
client := func(inName string) {
fmt.Printf("Client <%s> started\n", inName)
wg.Add(1)
conn, err := net.Dial("tcp", ":8000")
if err != nil {
fmt.Printf("[%s] Error while connecting to the server: %s", inName, err.Error())
os.Exit(1)
}
n := 0
sleepDuration, _ := time.ParseDuration("2s")
for {
message := fmt.Sprintf("[%s] > message %d\n", inName, n)
fmt.Printf("%s", message)
_, err := conn.Write([]byte(message))
if nil != err {
fmt.Sprintf("[%s] Error while writing data to the socket: %s", inName, err.Error())
os.Exit(1)
}
time.Sleep(sleepDuration)
n++
}
}
name := fmt.Sprintf("Client%d", i)
fmt.Printf("Starting client <%s>...\n", name)
go client(name)
fmt.Print("Done\n")
}
wg.Wait()
}
Result:
Starting 3 clients
Starting client <Client0>...
Done
Starting client <Client1>...
Done
Starting client <Client2>...
Done
As this solution says:
It is important that the wg.Add() happens before the go statement to
prevent race conditions. The following would also be correct:
So, you need to take out wg.Add(1) off inside of go routine itself and call it just before starting go routines. Also, call defer wg.Done() at the start of the function, so that, it will decrements the WaitGroup counter by one. Take a look at the code below.
package main
import (
"fmt"
"net"
"os"
"time"
"sync"
)
func main() {
wg := sync.WaitGroup{}
fmt.Print("Starting 3 clients\n")
for i:=0; i<3; i++ {
client := func(inName string) {
defer wg.Done() // added this line
fmt.Printf("Client <%s> started\n", inName)
conn, err := net.Dial("tcp", ":8000")
if err != nil {
fmt.Printf("[%s] Error while connecting to the server: %s", inName, err.Error())
os.Exit(1)
}
n := 0
sleepDuration, _ := time.ParseDuration("2s")
for {
message := fmt.Sprintf("[%s] > message %d\n", inName, n)
fmt.Printf("%s", message)
_, err := conn.Write([]byte(message))
if nil != err {
fmt.Sprintf("[%s] Error while writing data to the socket: %s", inName, err.Error())
os.Exit(1)
}
time.Sleep(sleepDuration)
n++
}
}
name := fmt.Sprintf("Client%d", i)
fmt.Printf("Starting client <%s>...\n", name)
wg.Add(1) // moved this line
go client(name)
fmt.Print("Done\n")
}
wg.Wait()
}

Golang app using sync.WaitGroup & channels never exits

I use sync.WaitGroup, defer wg.Close() and wg.Wait() to wait for my goroutines to complete.
The program do wait, but it never exits.
This is my program (runnable):
package main
import (
"fmt"
"io"
"log"
"net/http"
"os"
"sync"
)
var symbols = []string{
"ASSA-B.ST",
"ELUX-B.ST",
"HM-B.ST",
}
func main() {
fmt.Println("fetching quotes...")
fetchedSymbols := make(chan string)
var wg sync.WaitGroup
wg.Add(len(symbols))
for _, symbol := range symbols {
go fetchSymbol(symbol, &wg, fetchedSymbols)
}
for response := range fetchedSymbols {
fmt.Println("fetched " + response)
}
wg.Wait()
fmt.Println("done")
}
func fetchSymbol(symbol string, wg *sync.WaitGroup, c chan<- string) {
defer wg.Done()
resp, err := http.Get("http://ichart.yahoo.com/table.csv?s=" + symbol + "&a=0&b=1&c=2000")
defer resp.Body.Close()
if err != nil {
log.Fatal(err)
}
out, err := os.Create("./stock-quotes/" + symbol + ".csv")
defer out.Close()
if err != nil {
log.Fatal(err)
}
io.Copy(out, resp.Body)
c <- symbol
}
Shouldn't this program exit when all the quotes have been downloaded? (FYI: I just started learning GO)
You're never closing the fetchedSymbols channel, so that range loop will never exit.
One way to handle this is to use use the WaitGroup you already have to signal when to close the channel. Ranging over fetchedSymbols is enough to block the progress in main, and you don't need another channel or WaitGroup.
...
go func() {
wg.Wait()
close(fetchedSymbols)
}()
for response := range fetchedSymbols {
fmt.Println("fetched " + response)
}
...

Resources