I'm writing a server application in Go, and using a wrapper to run it as a Windows service.
There's a need to gracefully shut down the server (to close resources and connections properly), and in UNIX it would be handled through the SIGTERM signal. No big deal.
Though on Windows things seem very different. I see on this guide that signals actually exist on windows (?), and the SIGTERM is defined, though other pages indicate they don't, or to use other mechanisms like WM_CLOSE.
What is the preferable way to tell a headless process to gracefully terminate? How should it be implemented in Go?
The server is designed to be multiplatform, so the most standard way of doing it is preferable.
The go way to initiate canceling a task/service, is to use the context.Context package.
So if you want a signal handler to trigger the closing of a task's context.Context:
func registerSigHandler() context.Context {
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
rootCtx := context.Background()
taskCtx, cancelFn := context.WithCancel(rootCtx)
go func() {
sig := <-sigCh
log.Println("received signal:", sig)
// let sub-task know to wrap up: cancel taskCtx
cancelFn()
}()
return taskCtx
}
and then pass the returned taskCtx on to your worker task for it to listen on.
select {
case <-taskCtx.Done():
// context was canceled
default: // poll - rather than block in this example
}
Playground source-code.
Output:
2019/03/10 19:18:51 Worker PID: 33186
2019/03/10 19:18:51 Will terminate on these signals: interrupt terminated
2019/03/10 19:18:51 2019-03-10 19:18:51.018962 -0400 EDT m=+0.001727305
2019/03/10 19:18:52 2019-03-10 19:18:52.022782 -0400 EDT m=+1.005517010
2019/03/10 19:18:53 2019-03-10 19:18:53.019925 -0400 EDT m=+2.002630457
$ kill -INT 33186
2019/03/10 19:18:53 received signal: interrupt
2019/03/10 19:18:53 task context terminated reason: context canceled
2019/03/10 19:18:53 wrapping up task...
2019/03/10 19:18:53 workerTask() exited cleanly
2019/03/10 19:18:53 main exited cleanly
EDIT:
I tested this on Windows 10 and the clean-up is triggered when a Ctrl-C is issued from the same console.
Not sure how to send signals externally on Windows - which may be the OP's original issue. Using say killtask /F /PID 33186 would indeed kill the process without any signal handler being triggered.
Signaling is implemented on Windows but Unix signals are unavailable.
There is an example in the signal package of golang for Windows to send a Ctrl-Break. It is refactored for interactive use here.
So, after a while, I just wanted to share how I solved it. It's ugly, but the only real way I found. Windows golang programs do listen to a CTRL+C, which is not a signal or anything like it, but it does trigger a graceful shutdown in the go program using the same signal UNIX has.
Here's the relevant code:
// Create shutdown channel
exitChan := make(chan int)
interruptChan := make(chan os.Signal, 1)
signal.Notify(interruptChan, os.Interrupt)
select {
case exitCode := <-exitChan:
os.Exit(exitCode)
case <-interruptChan:
// This works in Windows if ctrl-c is invoked. It sucks though
svr.Stop()
os.Exit(-1)
}
To trigger the CTRL+C given the process, I use a small program called windows-kill.exe
// Windows is "special" and we need to use a library for killing processes gracefully
// Unfortunately setting up the C++ library with go is a hassle, so we resort to a CLI tool
ex, _ := os.Executable()
cmdPath := filepath.Join(filepath.Dir(ex), "windows-kill.exe")
cmd := exec.CommandContext(context.Background(), cmdPath, "-SIGINT", strconv.Itoa(l.proc.Pid))
err := cmd.Start()
if err != nil {
panic(err)
}
Here's the repo of the library and tool: https://github.com/alirdn/windows-kill
The idea of using *nix constructs and functions to Windows is generally bad and prone to weird tricks that may or may not work.
The proper way to cleanup on shutdown a GUI application is the handling of WM_QUERYENDSESSION and WM_ENDSESSION.
Also, you have flexibility on generic shut-down mechanism, check here.
The proper way to get notified if you are a service is the service handler of your console application (SERVICE_STOPPED). For more, see Writing a ServiceMain.
If you are running some form of web-services, may be better to use the web-service itself to initiate a shutdown, as tracking PIDS for signals can get messy.
To stop a http web-service, simply add a route like this:
http.HandleFunc("/shutdown",
func(w http.ResponseWriter, r *http.Request) {
// stops server - may want to add admin credentials check here!
srv.Shutdown(context.Background())
})
Playground source-code example.
2019/03/10 16:58:15 Listening on: :8080
$ curl 'localhost:8080/shutdown'
2019/03/10 17:04:17 Listening on: :8080
2019/03/10 17:04:19 exited cleanly
Related
I'm referring to this article: Graceful Shutdowns on Cloud Run
The example outlines how to do this in Node.js.
How would one do this in Golang? Any issues with simply adding this to the func init() method?
func shutdownGracefully() {
c := make(chan os.Signal)
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
go func() {
<-c
// Do some cleanup stuff here...
os.Exit(0)
}()
}
We have sample Go code for proper signal handling at https://github.com/GoogleCloudPlatform/golang-samples/pull/1902. It's not merged yet but essentially highlights how to do this properly on Go.
Beware that when running locally, sending Ctrl+C to your application is a SIGINT, however on Cloud Run you will be getting SIGTERM. It is also important to pass cancellations properly and handling server shutdown gracefully while not dropping the ongoing requests (though, net/http.Server handles a decent portion of this for you as you’ll see in sample code).
How would one do this in Golang?
An idiomatic way to handle graceful shutdowns in go is having a select statement blocking the main goroutine listening for any signal. From there you can trigger all the proper shutdowns when necessary.
For instance:
select {
case err := <-anyOtherError:
// ...
case signal := <- c:
// Handle shutdown e.g. like http Server Shutdown
}
I like to set it inside a run function with other startup tasks like starting the server, configs, etc.
I currently have a MQTT code that can subscribe to a topic, print out the messages received, then publish further instructions to a new topic. The subscribing/printing is completed in one Goroutine, and the publishing is done in another Goroutine. Here is my code:
var wg, pg sync.WaitGroup
// All messages are handled here - printing published messages and publishing new messages
var f MQTT.MessageHandler = func(client MQTT.Client, msg MQTT.Message) {
wg.Add(1)
pg.Add(1)
go func() {
defer wg.Done()
fmt.Printf("%s\n", msg.Payload())
//fmt.Println(os.Getpid())
}()
go func(){
defer pg.Done()
message := ""
//Changing configurations
if strings.Contains(string(msg.Payload()), "arduinoLED") == true {
message = fmt.Sprintf("change configuration")
}
if strings.Contains(string(msg.Payload()), "NAME CHANGED") == true{
message = fmt.Sprintf("change back")
}
// Publish further instructions to "sensor/instruction"
token := client.Publish("sensor/instruction", 0, false, message)
//fmt.Println(os.Getpid())
token.Wait()
}()
}
func main() {
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
opts := MQTT.NewClientOptions().AddBroker("tcp://test.mosquitto.org:1883")
opts.SetDefaultPublishHandler(f)
// Topic to subscribe to for sensor data
topic := "sensor/data"
opts.OnConnect = func(c MQTT.Client) {
if token := c.Subscribe(topic, 0, f); token.Wait() && token.Error() != nil {
panic(token.Error())
}
}
// Creating new client
client := MQTT.NewClient(opts)
if token := client.Connect(); token.Wait() && token.Error() != nil {
panic(token.Error())
} else {
fmt.Printf("Connected to server\n")
}
wg.Wait()
pg.Wait()
<-c
}
The commented out os.Getpid() line is to check which process I am running that Goroutine on. Right now they both display the same number (which means both are running on the same process?).
My question is: How can I run the two Goroutines on separate processes? Is there a way?
Edit: If this cannot be done, I want to write this code using channels. Here is the code I have for that:
var f MQTT.MessageHandler = func(client MQTT.Client, msg MQTT.Message) {
sensorData := make(chan []byte)
wg.Add(1)
pg.Add(1)
go func() {
defer wg.Done()
//fmt.Printf("%s\n", msg.Payload())
sensorData <- string(msg.Payload())
fmt.Println(<-sensorData) //currently not printing anything
}()
go func(){
defer pg.Done()
message := ""
//Changing configurations
if strings.Contains(<-sensorData, "arduinoLED") == true{
message = fmt.Sprintf("change configuration")
}
if strings.Contains(<-sensorData, "NAME CHANGED") == true{
message = fmt.Sprintf("change back")
}
// Publish further instructions to "sensor/instruction"
token := client.Publish("sensor/instruction", 0, false, message)
token.Wait()
}()
}
However, I am not able to print out any data using channels. What am I doing wrong?
You might be coming from Python, right? ;-)
It has the module named
multiprocessing
in its stdlib, and this might well explain why you have used
this name in the title of your question and why you apparently
are having trouble interpreting what #JimB meant by saying
If you need a separate process, you need to exec it yourself
"Multiprocessing" in Python
The thing is, Python's multiprocessing is a quite high-level
thing which hides under its hood a whole lot of stuff.
When you spawn a multiprocessing.Process and make it run
a function, what really happens is this:
The Python interpreter creates another operating system's
process (using
fork(2) on Unix-like systems
or CreateProcess on Windows) and arranges
for it to execute a Python interpter, too.
The crucial point is that you will now have two processes
running two Python interpters.
It is arranged for that Python interpterer in the
child process to have a way to communicate with the Python
interpreter in the parent process.
This "communication link" necessarily involves some form
of IPC #JimB referred to.
There is simply no other way to communicate data and actions
between separate processes exactly because a commodity
contemporary OS provides strict process separation.
When you exchange Python objects between the processes, the two communicating Python
interpreters serialize and deserialize them behind your back
before sending them over their IPC link and after receiving
them from there, correspondingly.
This is implemented using the pickle module.
Back to Go
Go does not have any direct solution which would closely
match Python's multiprocessing, and I really doubt it could
have been sensibly implemented.
The chief reason for this mostly stems from the fact Go
is quite more lower level than Python, and hence it does not
have the Python's luxury of making sheer assumptions about
the types of values it manages, and it also strives to have
as few hidden costs in its constructs as possible.
Go also strives to steer clear of "framework-style" approaches
to solve problems, and use "library-style" solutions when
possible. (A good rundown of the "framework vs library"
is given, for instance, here.)
Go has everything in its standard library to implement
something akin to Python's multiprocessing but there is no
ready-made frakework-y solution for this.
So what you could do for this is to roll along these lines:
Use os/exec to run another copy of your own process.
Make sure the spawned process "knows" it's started
in the special "slave" mode—to act accordingly.
Use any form of IPC to communicate with the new process.
Exchanging data via the standard I/O streams
of the child process is supposedly
the simplest way to roll (except when you need to exchange
opened files but this is a harder topic, so let's not digress).
Use any suitable package in the encoding/ hierarchy — such as binary, gob, xml — to serialize
and deserialize data when exchanging.
The "go-to" solution is supposedly encoding/gob
but encoding/json will also do just fine.
Invent and implement a simple protocol to tell the
child process what to do, and with which data,
and how to communicate the results back to master.
Does it really worth the trouble?
I would say that no, it doesn't—for a number of reasons:
Go has nothing like the dreaded GIL,
so there's no need to sidestep it to achieve real parallelism
when it is naturally possible.
Memory safety is all in your hands, and achieving it is
not really that hard when you dutifully obey the principle
that what is sent over a channel is now owned by
the receiver. In other words, sending values over a channel
is also the transfer of ownership of those values.
The Go toolchain has integrated race detector, so you
may run your test suite with the -race flag and create evaluation
builds of your program using go build -race for the same
purpose: when a program instrumented in such a way runs,
the race detector crashes it as soon as it detects any
unsynchronized read/write memory access.
The printout resulting from that crash includes
explanatory messages on what, and where went wrong,
with stack traces.
IPC is slow, so the gains may well be offset by the losses.
All-in-all, I see no real reason to separate processes unless
you're writing something like an e-mail processing server
where this concept comes naturally.
Channel is used for communicating between goroutines, you shouldn't use it in same goroutine like this code:
sensorData <- string(msg.Payload())
fmt.Println(<-sensorData) //currently not printing anything
If you like to test printing by channel, you can use buffered channel in same goroutine to avoid blocking, like this:
sensorData := make(chan []byte, 1)
Cheers
I want to listen for graceful server termination in grpc in my handler. When the server is being stopped gracefully, I want to add some logic in my code to close open ports, files, flush results etc. How can I do that?
How is it different in case of unary and streaming handler?
You can have shutdown hook by listening to the signals something like this
In your main function or where you start your server create channel for signals that you want to listen to
c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
// call your cleanup method with this channel as a routine
go cleanup(c)
In your clean up method
func cleanup(c chan os.Signal) {
// Wait for termination
<- c
// Do your cleanups here
}
Create the signal channel and call the cleanup function as a go routine before you start the gRPC server. Whenever the application (gRPC server) stopped or interrupted this channel will get the signal in the cleanup function where you can do the necessary cleanups
For grpc servers u can do this
func waitForGracefulShutdown(srv *grpc.Server) {
fmt.Println("Grpc messaging server started ...")
interruptChan := make(chan os.Signal, 1)
signal.Notify(interruptChan, os.Interrupt, syscall.SIGINT, syscall.SIGTERM)
// Block until we receive our signal.
<-interruptChan
// Create a deadline to wait for.
_, cancel := context.WithTimeout(context.Background(), time.Second*10)
defer cancel()
srv.GracefulStop()
publisher, messagingError := messaging.GetPublisherInstance()
if messagingError.Error == nil {
publisher.Close()
}
log.Println("Shutting down grpc messaging server.")
os.Exit(0)
}
Currently there's no mechanism to signal a service handler about Graceful stop. Cleanups and other such global functions that must happen upon server exiting wouldn't usually happen inside of a service handler.
That said, if you think your design is better off with such cleanups happening inside of the service handler and a signal from graceful close is critical, we would love to hear more about your use case. Perhaps open an issue on our github repo and we can discuss it there.
Best,
Mak
I am launching a simple Java application through Go, with the goal of proving that Go can send a signal like SIGQUIT or SIGTERM, and the Java can catch that and handle it appropriately (i.e. shutdown gracefully). When I run the Java program on the command line and send it a CTRL+C, the Java program correctly catches the signal.
However, when I launch the Java program through Go and attempt to send the process a signal, the Java process is neither terminated nor does it handle the signal. The only one that works is SIGKILL, which of course is not caught, and simply kills the process.
Here are the relevant parts of my Go code:
Start:
startChan := make(chan bool)
go func(startChan chan<- bool) {
a.Cmd = exec.Command("java", "-jar", "C:\\Code\\sigterm\\TestApp.jar")
a.Cmd.Stdout = os.Stdout
a.Cmd.Stderr = os.Stderr
launchErr := a.Cmd.Start()
if launchErr != nil {
fmt.Println("Unable to start app:" + launchErr.Error())
}
startChan <- true
}(startChan)
Stop:
func (a *App) Stop(timeout time.Duration) {
a.Cmd.Process.Signal(syscall.SIGQUIT) //Does not work
a.Cmd.Process.Signal(syscall.SIGTERM) //Does not work
a.Cmd.Process.Signal(syscall.SIGKILL) //Works
}
This is on Windows, by the way. I tried it by launching another program (notepad) and got the same results; that is, only SIGKILL worked. I have confirmed that Go is getting the correct PID and so forth, so the signal should be going to the right place.
Any ideas about how I can get this to work?
This does not seem to be supported as stated in the documentation:
https://godoc.org/os#Process.Signal
Sending Interrupt on Windows is not implemented.
Also, see this issue discussion to get more context on why it could not be done:
https://github.com/golang/go/issues/6720
syscall package is tailored to the OS (https://golang.org/pkg/syscall/#pkg-overview).
Although interrupt is not implemented on Windows, signaling is usable.
There is an example in the signal package itself for Windows.
An interactive version (go run) is available here.
If you look at Nginx it calls "nginx reload" to reload itself. Is there any way to send a signal from the command line to a running process? Even if the main process starts child processes how can I send commands to the main to notify its children?
ex:
myapp start -debug // starts a server
myapp reload -gracefull // stops the app gracefully
Now i need to send os signals to notify my server to perform a graceful shutdown
kill -QUIT pid
kill -USR2 pid
I hope my question is clear enough
Thnx
Receive signals
Take a look at the os/signal package.
Package signal implements access to incoming signals.
There is even an example in the documentation :
// Set up channel on which to send signal notifications.
// We must use a buffered channel or risk missing the signal
// if we're not ready to receive when the signal is sent.
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt, os.Kill)
// Block until a signal is received.
s := <-c
fmt.Println("Got signal:", s)
Send signals
To see how to send signals take a look at signal_test.go, it uses syscall. For example :
// Send this process a SIGHUP
t.Logf("sighup...")
syscall.Kill(syscall.Getpid(), syscall.SIGHUP)
waitSig(t, c, syscall.SIGHUP)
I figured out that in go i we can pass the environment to syscall.Exec
err := syscall.Exec(argv0. os.Args. os.Environ())
simply copies the current env to the child process.