I'm referring to this article: Graceful Shutdowns on Cloud Run
The example outlines how to do this in Node.js.
How would one do this in Golang? Any issues with simply adding this to the func init() method?
func shutdownGracefully() {
c := make(chan os.Signal)
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
go func() {
<-c
// Do some cleanup stuff here...
os.Exit(0)
}()
}
We have sample Go code for proper signal handling at https://github.com/GoogleCloudPlatform/golang-samples/pull/1902. It's not merged yet but essentially highlights how to do this properly on Go.
Beware that when running locally, sending Ctrl+C to your application is a SIGINT, however on Cloud Run you will be getting SIGTERM. It is also important to pass cancellations properly and handling server shutdown gracefully while not dropping the ongoing requests (though, net/http.Server handles a decent portion of this for you as you’ll see in sample code).
How would one do this in Golang?
An idiomatic way to handle graceful shutdowns in go is having a select statement blocking the main goroutine listening for any signal. From there you can trigger all the proper shutdowns when necessary.
For instance:
select {
case err := <-anyOtherError:
// ...
case signal := <- c:
// Handle shutdown e.g. like http Server Shutdown
}
I like to set it inside a run function with other startup tasks like starting the server, configs, etc.
Related
I'm having trouble to estimate if there will be side effects of running http.ListenAndServe in goroutine.
To make it possible for prometheus to collect stats data from a /metrics endpoint of a service running a kafkaclient(running a kafka consumer in an infinite for-loop)
var addr = flag.String("listen-address", ":8070", "The address to listen on for HTTP requests.")
func main() {
flag.Parse()
http.Handle("/metrics", promhttp.Handler())
go http.ListenAndServe(*addr, nil)
for {....}
What would be the best practices to start the monitoring endpoint and run the infinite loop?
Best practice is to check for and handle errors. The error from http.ListenAndServe is ignored.
If return from http.ListenAndServe is fatal to the application, then use the following code or some variation on it to handle the error.
go func() {
log.Fatal(http.ListenAndServe(*addr, nil))
}()
The call to log.Fatal logs the error and exits the application.
I'm writing a server application in Go, and using a wrapper to run it as a Windows service.
There's a need to gracefully shut down the server (to close resources and connections properly), and in UNIX it would be handled through the SIGTERM signal. No big deal.
Though on Windows things seem very different. I see on this guide that signals actually exist on windows (?), and the SIGTERM is defined, though other pages indicate they don't, or to use other mechanisms like WM_CLOSE.
What is the preferable way to tell a headless process to gracefully terminate? How should it be implemented in Go?
The server is designed to be multiplatform, so the most standard way of doing it is preferable.
The go way to initiate canceling a task/service, is to use the context.Context package.
So if you want a signal handler to trigger the closing of a task's context.Context:
func registerSigHandler() context.Context {
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
rootCtx := context.Background()
taskCtx, cancelFn := context.WithCancel(rootCtx)
go func() {
sig := <-sigCh
log.Println("received signal:", sig)
// let sub-task know to wrap up: cancel taskCtx
cancelFn()
}()
return taskCtx
}
and then pass the returned taskCtx on to your worker task for it to listen on.
select {
case <-taskCtx.Done():
// context was canceled
default: // poll - rather than block in this example
}
Playground source-code.
Output:
2019/03/10 19:18:51 Worker PID: 33186
2019/03/10 19:18:51 Will terminate on these signals: interrupt terminated
2019/03/10 19:18:51 2019-03-10 19:18:51.018962 -0400 EDT m=+0.001727305
2019/03/10 19:18:52 2019-03-10 19:18:52.022782 -0400 EDT m=+1.005517010
2019/03/10 19:18:53 2019-03-10 19:18:53.019925 -0400 EDT m=+2.002630457
$ kill -INT 33186
2019/03/10 19:18:53 received signal: interrupt
2019/03/10 19:18:53 task context terminated reason: context canceled
2019/03/10 19:18:53 wrapping up task...
2019/03/10 19:18:53 workerTask() exited cleanly
2019/03/10 19:18:53 main exited cleanly
EDIT:
I tested this on Windows 10 and the clean-up is triggered when a Ctrl-C is issued from the same console.
Not sure how to send signals externally on Windows - which may be the OP's original issue. Using say killtask /F /PID 33186 would indeed kill the process without any signal handler being triggered.
Signaling is implemented on Windows but Unix signals are unavailable.
There is an example in the signal package of golang for Windows to send a Ctrl-Break. It is refactored for interactive use here.
So, after a while, I just wanted to share how I solved it. It's ugly, but the only real way I found. Windows golang programs do listen to a CTRL+C, which is not a signal or anything like it, but it does trigger a graceful shutdown in the go program using the same signal UNIX has.
Here's the relevant code:
// Create shutdown channel
exitChan := make(chan int)
interruptChan := make(chan os.Signal, 1)
signal.Notify(interruptChan, os.Interrupt)
select {
case exitCode := <-exitChan:
os.Exit(exitCode)
case <-interruptChan:
// This works in Windows if ctrl-c is invoked. It sucks though
svr.Stop()
os.Exit(-1)
}
To trigger the CTRL+C given the process, I use a small program called windows-kill.exe
// Windows is "special" and we need to use a library for killing processes gracefully
// Unfortunately setting up the C++ library with go is a hassle, so we resort to a CLI tool
ex, _ := os.Executable()
cmdPath := filepath.Join(filepath.Dir(ex), "windows-kill.exe")
cmd := exec.CommandContext(context.Background(), cmdPath, "-SIGINT", strconv.Itoa(l.proc.Pid))
err := cmd.Start()
if err != nil {
panic(err)
}
Here's the repo of the library and tool: https://github.com/alirdn/windows-kill
The idea of using *nix constructs and functions to Windows is generally bad and prone to weird tricks that may or may not work.
The proper way to cleanup on shutdown a GUI application is the handling of WM_QUERYENDSESSION and WM_ENDSESSION.
Also, you have flexibility on generic shut-down mechanism, check here.
The proper way to get notified if you are a service is the service handler of your console application (SERVICE_STOPPED). For more, see Writing a ServiceMain.
If you are running some form of web-services, may be better to use the web-service itself to initiate a shutdown, as tracking PIDS for signals can get messy.
To stop a http web-service, simply add a route like this:
http.HandleFunc("/shutdown",
func(w http.ResponseWriter, r *http.Request) {
// stops server - may want to add admin credentials check here!
srv.Shutdown(context.Background())
})
Playground source-code example.
2019/03/10 16:58:15 Listening on: :8080
$ curl 'localhost:8080/shutdown'
2019/03/10 17:04:17 Listening on: :8080
2019/03/10 17:04:19 exited cleanly
I currently have a MQTT code that can subscribe to a topic, print out the messages received, then publish further instructions to a new topic. The subscribing/printing is completed in one Goroutine, and the publishing is done in another Goroutine. Here is my code:
var wg, pg sync.WaitGroup
// All messages are handled here - printing published messages and publishing new messages
var f MQTT.MessageHandler = func(client MQTT.Client, msg MQTT.Message) {
wg.Add(1)
pg.Add(1)
go func() {
defer wg.Done()
fmt.Printf("%s\n", msg.Payload())
//fmt.Println(os.Getpid())
}()
go func(){
defer pg.Done()
message := ""
//Changing configurations
if strings.Contains(string(msg.Payload()), "arduinoLED") == true {
message = fmt.Sprintf("change configuration")
}
if strings.Contains(string(msg.Payload()), "NAME CHANGED") == true{
message = fmt.Sprintf("change back")
}
// Publish further instructions to "sensor/instruction"
token := client.Publish("sensor/instruction", 0, false, message)
//fmt.Println(os.Getpid())
token.Wait()
}()
}
func main() {
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
opts := MQTT.NewClientOptions().AddBroker("tcp://test.mosquitto.org:1883")
opts.SetDefaultPublishHandler(f)
// Topic to subscribe to for sensor data
topic := "sensor/data"
opts.OnConnect = func(c MQTT.Client) {
if token := c.Subscribe(topic, 0, f); token.Wait() && token.Error() != nil {
panic(token.Error())
}
}
// Creating new client
client := MQTT.NewClient(opts)
if token := client.Connect(); token.Wait() && token.Error() != nil {
panic(token.Error())
} else {
fmt.Printf("Connected to server\n")
}
wg.Wait()
pg.Wait()
<-c
}
The commented out os.Getpid() line is to check which process I am running that Goroutine on. Right now they both display the same number (which means both are running on the same process?).
My question is: How can I run the two Goroutines on separate processes? Is there a way?
Edit: If this cannot be done, I want to write this code using channels. Here is the code I have for that:
var f MQTT.MessageHandler = func(client MQTT.Client, msg MQTT.Message) {
sensorData := make(chan []byte)
wg.Add(1)
pg.Add(1)
go func() {
defer wg.Done()
//fmt.Printf("%s\n", msg.Payload())
sensorData <- string(msg.Payload())
fmt.Println(<-sensorData) //currently not printing anything
}()
go func(){
defer pg.Done()
message := ""
//Changing configurations
if strings.Contains(<-sensorData, "arduinoLED") == true{
message = fmt.Sprintf("change configuration")
}
if strings.Contains(<-sensorData, "NAME CHANGED") == true{
message = fmt.Sprintf("change back")
}
// Publish further instructions to "sensor/instruction"
token := client.Publish("sensor/instruction", 0, false, message)
token.Wait()
}()
}
However, I am not able to print out any data using channels. What am I doing wrong?
You might be coming from Python, right? ;-)
It has the module named
multiprocessing
in its stdlib, and this might well explain why you have used
this name in the title of your question and why you apparently
are having trouble interpreting what #JimB meant by saying
If you need a separate process, you need to exec it yourself
"Multiprocessing" in Python
The thing is, Python's multiprocessing is a quite high-level
thing which hides under its hood a whole lot of stuff.
When you spawn a multiprocessing.Process and make it run
a function, what really happens is this:
The Python interpreter creates another operating system's
process (using
fork(2) on Unix-like systems
or CreateProcess on Windows) and arranges
for it to execute a Python interpter, too.
The crucial point is that you will now have two processes
running two Python interpters.
It is arranged for that Python interpterer in the
child process to have a way to communicate with the Python
interpreter in the parent process.
This "communication link" necessarily involves some form
of IPC #JimB referred to.
There is simply no other way to communicate data and actions
between separate processes exactly because a commodity
contemporary OS provides strict process separation.
When you exchange Python objects between the processes, the two communicating Python
interpreters serialize and deserialize them behind your back
before sending them over their IPC link and after receiving
them from there, correspondingly.
This is implemented using the pickle module.
Back to Go
Go does not have any direct solution which would closely
match Python's multiprocessing, and I really doubt it could
have been sensibly implemented.
The chief reason for this mostly stems from the fact Go
is quite more lower level than Python, and hence it does not
have the Python's luxury of making sheer assumptions about
the types of values it manages, and it also strives to have
as few hidden costs in its constructs as possible.
Go also strives to steer clear of "framework-style" approaches
to solve problems, and use "library-style" solutions when
possible. (A good rundown of the "framework vs library"
is given, for instance, here.)
Go has everything in its standard library to implement
something akin to Python's multiprocessing but there is no
ready-made frakework-y solution for this.
So what you could do for this is to roll along these lines:
Use os/exec to run another copy of your own process.
Make sure the spawned process "knows" it's started
in the special "slave" mode—to act accordingly.
Use any form of IPC to communicate with the new process.
Exchanging data via the standard I/O streams
of the child process is supposedly
the simplest way to roll (except when you need to exchange
opened files but this is a harder topic, so let's not digress).
Use any suitable package in the encoding/ hierarchy — such as binary, gob, xml — to serialize
and deserialize data when exchanging.
The "go-to" solution is supposedly encoding/gob
but encoding/json will also do just fine.
Invent and implement a simple protocol to tell the
child process what to do, and with which data,
and how to communicate the results back to master.
Does it really worth the trouble?
I would say that no, it doesn't—for a number of reasons:
Go has nothing like the dreaded GIL,
so there's no need to sidestep it to achieve real parallelism
when it is naturally possible.
Memory safety is all in your hands, and achieving it is
not really that hard when you dutifully obey the principle
that what is sent over a channel is now owned by
the receiver. In other words, sending values over a channel
is also the transfer of ownership of those values.
The Go toolchain has integrated race detector, so you
may run your test suite with the -race flag and create evaluation
builds of your program using go build -race for the same
purpose: when a program instrumented in such a way runs,
the race detector crashes it as soon as it detects any
unsynchronized read/write memory access.
The printout resulting from that crash includes
explanatory messages on what, and where went wrong,
with stack traces.
IPC is slow, so the gains may well be offset by the losses.
All-in-all, I see no real reason to separate processes unless
you're writing something like an e-mail processing server
where this concept comes naturally.
Channel is used for communicating between goroutines, you shouldn't use it in same goroutine like this code:
sensorData <- string(msg.Payload())
fmt.Println(<-sensorData) //currently not printing anything
If you like to test printing by channel, you can use buffered channel in same goroutine to avoid blocking, like this:
sensorData := make(chan []byte, 1)
Cheers
func main() {
messages := make(chan string)
go func() { messages <- "hello" }()
go func() { messages <- "ping" }()
msg := <-messages
msg2 := <-messages
fmt.Println(msg)
fmt.Println(msg2)
The above code consistently prints "ping" and then "hello" on my terminal.
I am confused about the order in which this prints, so I was wondering if I could get some clarification on my thinking.
I understand that unbuffered channels are blocking while waiting for both a sender and a receiver. So in the above case, when these 2 go routines are executed, there isn't, in both cases,a receiver yet. So I am guessing that both routines block until a receiver is available on the channel.
Now... I would assume that first "hello" is tried into the channel, but has to wait... at the same time, "ping" tries, but again has to wait. Then
msg := <- messages
shows up, so I would assume that at that stage, the program will arbitrarily pick one of the waiting goroutines and allow it to send its message over into the channel, since msg is ready to receive.
However, it seems that no matter how many times I run the program, it always is msg that gets assigned "ping" and msg2 that gets assigned "hello", which gives the impression that "ping" always gets priority to send first (to msg). Why is that?
It’s not about order of reading a channel but about order of goroutines execution which is not guaranteed.
Try to ‘Println’ from the function where you are writing to the channel (before and after writing) and I think it should be in same order as reading from the channel.
In Golang Spec Channels order is described as:-
Channels act as first-in-first-out queues. For example, if one
goroutine sends values on a channel and a second goroutine receives
them, the values are received in the order sent.
It will prints which value is available first to be received on other end.
If you wants to synchronize them use different channels or add wait Groups.
package main
import (
"fmt"
)
func main() {
messages1 := make(chan string)
messages2 := make(chan string)
go func(<-chan string) {
messages2 <- "ping"
}(messages2)
go func(<-chan string) {
messages1 <- "hello"
}(messages1)
fmt.Println(<-messages1)
fmt.Println(<-messages2)
}
If you see you can easily receive any value you want according to your choice using different channels.
Go playground
I just went through this same thing. See my post here: Golang channels, order of execution
Like you, I saw a pattern that was counter-intuitive. In a place where there actually shouldn't be a pattern. Once you launch a go process, you have launched a thread of execution, and basically all bets are off at that point regarding the order that the threads will execute their steps. But if there was going to be an order, logic tells us the first one called would be executed first.
In actual fact, if you recompile that program each time, the results will vary. That's what I found when I started compiling/running it on my local computer. In order to make the results random, I had to "dirty" the file, by adding and removing a space for instance. Then the compiler would re-compile the program, and then I would get a random order of execution. But when compiled in the go sandbox, the result was always the same.
When you use the sandbox, the results are apparently cached. I couldn't get the order to change in the sandbox by using insignificant changes. The only way I got it to change was to issue a time.Sleep(1) command between the launching of the go statements. Then, the first one launched would be the first one executed every time. I still don't think I'd bet my life on that continuing to happen though because they are separate threads of execution and there are no guarantees.
The bottom line is that I was seeing a deterministic result where there should be no determinism. That's what stuck me. I was fully cleared up when I found that the results really are random in a normal environment. The sandbox is a great tool to have. But it's not a normal environment. Compile and run your code locally and you will see the varying results you would expect.
I was confused when I first met this. but now I am clear about this. the reason cause this is not about channel, but for goroutine.
As The Go Memory Model mention, there's no guaranteed to goroutine's running and exit, so when you create two goroutine, you cannot make sure that they are running in order.
So if you want printing follow FIFO rule, you can change your code like this:
func main() {
messages := make(chan string)
go func() {
messages <- "hello"
messages <- "ping"
}()
//go func() { messages <- "ping" }()
msg := <-messages
msg2 := <-messages
fmt.Println(msg)
fmt.Println(msg2)
}
I want to listen for graceful server termination in grpc in my handler. When the server is being stopped gracefully, I want to add some logic in my code to close open ports, files, flush results etc. How can I do that?
How is it different in case of unary and streaming handler?
You can have shutdown hook by listening to the signals something like this
In your main function or where you start your server create channel for signals that you want to listen to
c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
// call your cleanup method with this channel as a routine
go cleanup(c)
In your clean up method
func cleanup(c chan os.Signal) {
// Wait for termination
<- c
// Do your cleanups here
}
Create the signal channel and call the cleanup function as a go routine before you start the gRPC server. Whenever the application (gRPC server) stopped or interrupted this channel will get the signal in the cleanup function where you can do the necessary cleanups
For grpc servers u can do this
func waitForGracefulShutdown(srv *grpc.Server) {
fmt.Println("Grpc messaging server started ...")
interruptChan := make(chan os.Signal, 1)
signal.Notify(interruptChan, os.Interrupt, syscall.SIGINT, syscall.SIGTERM)
// Block until we receive our signal.
<-interruptChan
// Create a deadline to wait for.
_, cancel := context.WithTimeout(context.Background(), time.Second*10)
defer cancel()
srv.GracefulStop()
publisher, messagingError := messaging.GetPublisherInstance()
if messagingError.Error == nil {
publisher.Close()
}
log.Println("Shutting down grpc messaging server.")
os.Exit(0)
}
Currently there's no mechanism to signal a service handler about Graceful stop. Cleanups and other such global functions that must happen upon server exiting wouldn't usually happen inside of a service handler.
That said, if you think your design is better off with such cleanups happening inside of the service handler and a signal from graceful close is critical, we would love to hear more about your use case. Perhaps open an issue on our github repo and we can discuss it there.
Best,
Mak