Panic in other goroutine not stopping child process - go

I need to run a long-running child process and kill it if I quit (for any reason) out of parent application.
Here is the code:
cmd := exec.Command("./long-process")
defer cmd.Process.Kill()
if err != nil {
log.Fatal(err)
}
var fail io.ReadCloser
fail.Close()
The fail here produces obvious
panic: runtime error: invalid memory address or nil pointer dereference
It works as expected - the child process is killed.
But this happens in a goroutine:
cmd := exec.Command("./long-process")
defer cmd.Process.Kill()
if err != nil {
log.Fatal(err)
}
go func() {
var fail io.ReadCloser
fail.Close()
}()
The panic still happens, but then it seems defer is not called and the child process is not killed.
Any way to go around this?
UPDATE I need a cross-platform solution (at least for Linux and FreeBSD)
Minimal example:
infinite-loop.sh
#!/bin/bash
while true; do
sleep 1
done
Don't forget to
chmod +x infinite-loop.sh
test1.go (error checking left out for brevity):
package main
import (
"time"
"io"
"os/exec"
"runtime"
)
func main() {
cmd := exec.Command("./infinite-loop.sh")
cmd.Start()
defer cmd.Process.Kill()
go func() {
time.Sleep(100 * time.Millisecond)
var fail io.ReadCloser
fail.Close()
}()
for {
runtime.Gosched()
}
}
Let's run
ps aux | grep infinite-loop.sh | grep -v grep | wc -l; \
go run test1.go; \
ps aux | grep infinite-loop.sh | grep -v grep | wc -l
0 <--- !!
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x2130]
goroutine 5 [running]:
main.main.func1()
.../multiline/test1.go:19 +0x30
created by main.main
.../multiline/test1.go:20 +0x9a
goroutine 1 [runnable]:
runtime.Gosched()
/usr/local/Cellar/go/1.5.1/libexec/src/runtime/proc.go:166 +0x14
main.main()
.../multiline/test1.go:23 +0x9f
exit status 2
1 <--- !!
0 processes before and 1 after exit.
If you comment out goroutine code - it works fine.
Now we can kill it:
kill $(ps aux | grep infinite-loop.sh | grep -v grep | awk {'print $2'})

There's no cross-platform solution to automatically kill a child process.
On Linux, you can use the pdeathsig functionality:
cmd := exec.Command("./long-process")
cmd.SysProcAttr = &syscall.SysProcAttr{
Pdeathsig: syscall.SIGTERM,
}
On other platforms, the child needs to determine when to exit on its own. One way is to monitor a pipe or socket FD given to it from the parent. You could also have a process manager of some sort monitor the processes and cleanup if something goes wrong.
In general though, panics should be rare and get fixed. If you do have areas of code that are prone to panic'ing, you can recover locally and call for the cleanup of child processes before exiting.

Related

how to keep subprocess running after program exit in golang?

i noticed that subprocesses created using Start() will be terminated after program exit, for example:
package main
import "os/exec"
func main() {
cmd := exec.Command("sh", "test.sh")
cmd.Start()
}
when main() exits, test.sh will stop running
The subprocess should continue to run after your process ends, as long as it ends cleanly, which won't happen if you hit ^C.
What you can do is intercept the signals sent to your process so you can end cleanly.
sigchan := make(chan os.Signal, 1)
signal.Notify(sigchan,
syscall.SIGINT,
syscall.SIGKILL,
syscall.SIGTERM,
syscall.SIGQUIT)
go func() {
s := <-sigchan
// do anything you need to end program cleanly
}()
A subprocess (if not waited on within the go program) will continue to run once the go program has finished (unless the subprocess naturally finishes before the parent go program).
The problem the original poster is likely encountering is that they are probably terminating their go program early (e.g. using <Ctrl-c>), and because the go program is not exiting cleanly the subprocess it spawned is also terminated.
Below is a reduced test case that helps validate this behaviour...
First I create a bash shell script I want to run (e.g. test.sh, don't forget to chmod +x ./test.sh so the script is considered 'executable'). The script is very simple. It sleeps for 10 seconds and then either creates a new file called testfile (if it doesn't exist) or if the file already exists it will update the 'last modified' timestamp. This is important because this is how I confirm the bash script is still running once my go program finishes (which I expect to finish long before the bash script finishes due to the 10 second sleep).
#!/usr/local/bin/bash
sleep 10
touch testfile
Next, I have a simple go program, which spawns a subprocess that runs the bash script above but importantly doesn't wait for it to complete. You'll see I've also added a 2 second sleep to my go program which gives me some time to press <Ctrl-c>. Now, even though I have a 2 second sleep, this program (if left to run without me pressing <Ctrl-c>) will finish before the subprocess bash script does (which is sleeping for 10 seconds):
package main
import (
"fmt"
"log"
"os/exec"
"time"
)
func main() {
cmd := exec.Command("./test.sh")
err := cmd.Start()
if err != nil {
log.Fatal(err)
}
time.Sleep(2 * time.Second)
fmt.Println("program finished, but what about the subprocess?")
}
If I run the go program and just let it finish naturally, I can ls -l testfile and check the timestamp on it. I'll then wait 10 seconds and run the ls -l testfile again and I will see the timestamp update (which shows the subprocess finished successfully).
Now if I was to re-run the go program and this time press <Ctrl-c> before the program finishes (this is why I add the 2 second sleep), then not only will the go program exit early, but the subprocess will be terminated also. So I can wait 10 seconds or 10 hours or longer, doesn't matter. The timestamp on the testfile will not update, proving the subprocess was terminated.
Try modding you program a to use Run instead of start. In that way the Go program will wait for the sh script to finish before exiting.
package main
import (
"log"
"os/exec"
)
func main() {
cmd := exec.Command("sh", "test.sh")
err := cmd.Run()
if err != nil {
log.Fatalln(err)
}
}
Likewise, you could always use a wait group but I think that's overkill here.
You could also just a go routine with or without a wait group. Depends on if you want go to wait for the program the sh program to complete
package main
import (
"os/exec"
)
func runOffMainProgram() {
cmd := exec.Command("sh", "test.sh")
cmd.Start()
}
func main() {
// This will start a go routine, but without a waitgroup this program will exit as soon as it runs
// regardless the sh program will be running in the background. Until the sh program completes
go runOffMainProgram()
}
The accepted answer is vague about where the signal should be handled. I think some more sophisticated techniques must be used to prevent sending interrupts to children, if at all possible.
TLDR;
So the only way to deal with ctrl-c is to anticipate the SIGINT and process that signal in the children.
I did some experimentation of my own.
go build -o ctrl-c ctrl-c.go
If the program is sent to the background, The only way to kill the main process is with kill -9 (SIGKILL).
SIGTERM (15) will not do.
$ ./ctrl-c & cmd=$! ; sleep 1 && echo kill $cmd && kill $cmd
[1] 1165918
1165918
bashed 1165926
bashed 1165927
bashed 1165928
main()
go SIGN 23 urgent I/O condition
go SIGN 23 urgent I/O condition
main()
kill 1165918
go SIGN 15 terminated
main()
$ main()
main()
main()
main()
main()
main() done.
Bash _ 1165926 EXITs
Bash q 1165927 EXITs
Bash c 1165928 EXITs
[1]+ Done ./ctrl-c
SIGINT (2) will not do.
$ ./ctrl-c & cmd=$! ; sleep 1 && echo kill $cmd && kill -INT $cmd
[1] 1167675
1167675
bashed 1167683
bashed 1167684
bashed 1167685
main()
main()
kill 1167675
go SIGN 2 interrupt
main()
balmora: ~/src/my/go/doodles/sub-process [master]
$ main()
main()
main()
main()
main()
main() done.
Bash _ 1167683 EXITs
Bash q 1167684 EXITs
Bash c 1167685 EXITs
SIGKILL kills the main process but not the bash sub-commands.
$ ./ctrl-c & cmd=$! ; sleep 1 && echo kill $cmd && kill -KILL $cmd
[1] 1170721
1170721
bashed 1170729
bashed 1170730
bashed 1170731
main()
main()
kill 1170721
[1]+ Killed ./ctrl-c
Bash _ 1170729 EXITs
Bash q 1170730 EXITs
Bash c 1170731 EXITs
However, if the go binary is running in the foreground then only children who do deal with SIGINT will be kept running. This feels like almost the opposite of the above findings
$ ./ctrl-c
1186531
bashed 1186538
bashed 1186539
bashed 1186540
main()
main()
main()
main()
main()
main()
^C
Bash c 1186540 INTs quit
Bash q 1186539 INTs ignored
Bash c 1186540 EXITs
Bash _ 1186538 INTs ignored
go SIGN 2 interrupt
go SIGN 17 child exited
6q ELAPSED 2
Bash q 1186539 EXITs
6_ ELAPSED 2
Bash _ 1186538 EXITs
go SIGN 17 child exited
main()
main()
main() done.
Anyway, the takeaway for me is that ctrl+c is forwarded to children when Cmd.Start() is used. The behavior is the same if Cmd.Run() is used, but Cmd.Run() will wait before each sub-command exits. Running the Cmd in a go routine (go func(){}()) does not change anything. If the sub-commands are started "in parallel" as a go-routine or with Cmd.Start(), the the interrupt signal will reach all of them at the same time.
To keep the sub-commands running on an interactive terminal after an interrupt, I think the sub-commands have to handle the signal and ignore it.
The code I experimented with:
package main
import (
"fmt"
"log"
"os"
"os/exec"
"os/signal"
"syscall"
"time"
)
func signs(s ...os.Signal) chan os.Signal {
signals := make(chan os.Signal, 1)
signal.Notify(signals, s...)
signal.Notify(signals,
os.Interrupt, syscall.SIGINT, syscall.SIGQUIT, // keyboard
syscall.SIGKILL, syscall.SIGHUP, syscall.SIGTERM, // os termination
syscall.SIGUSR1, syscall.SIGUSR2, // user
syscall.SIGPIPE, syscall.SIGCHLD, syscall.SIGSEGV, // os other
)
return signals
}
func interpret(signals chan os.Signal) chan os.Signal {
go func() {
for ;; {
select {
case sign := <-signals:
elog("go SIGN %#v %s", sign, sign)
}
}
}()
return signals
}
func bash(script string) {
cmd := exec.Command("/bin/bash", "-c", script )
cmd.Stdout = os.Stderr
err := cmd.Start()
//err := cmd.Run()
if err != nil {
log.Fatal(err)
}
elog("bashed %d", cmd.Process.Pid)
}
func main() {
fmt.Println(os.Getpid())
signals := interpret(signs())
signals = signals
//go bash(`
bash(`
trap ' echo Bash _ $$ INTs ignored; ' SIGINT
trap ' echo Bash _ $$ QUITs ignored; ' SIGQUIT
trap ' echo Bash _ $$ EXITs' EXIT
sleep 6;
echo 6_ $( ps -o etimes -p $$ )
#for i in {1..60}; do echo -n _; sleep 0.1; done; echo
`)
// go bash(`
bash(`
trap ' echo Bash q $$ INTs ignored; ' SIGINT
trap ' echo Bash q $$ QUITs; exit ' SIGQUIT
trap ' echo Bash q $$ EXITs; ' EXIT
sleep 6;
echo 6q $( ps -o etimes -p $$ )
#for i in {1..60}; do echo -n q; sleep 0.1; done; echo
`)
//go bash(`
bash(`
trap ' echo Bash c $$ INTs quit; exit ' SIGINT
trap ' echo Bash c $$ QUITs ignored; ' SIGQUIT
trap ' echo Bash c $$ EXITs' EXIT
sleep 6;
echo 6c $( ps -o etimes -p $$ )
#for i in {1..60}; do echo -n c; sleep 0.1; done; echo
`)
go func() {
for ;; {
time.Sleep(time.Millisecond * 333)
elog("main()")
}
}()
time.Sleep(3 * time.Second)
elog("main() done.")
}
func echo(a ...interface{}) {
_, err := fmt.Println(a...)
if err != nil {
fmt.Println("ERR ", err.Error())
}
}
func elog(form string, arg ...interface{}) {
println(fmt.Sprintf(form, arg...))
}

golang exec background process and get its pid

Situation:
I want to run a command that puts itself into the background. If it makes it more possible, then I'll run the command in foreground and bring it into the background by myself.
Question:
When the process runs in background: how can I get it's pid using Go?
I tried the following:
cmd := exec.Command("ssh", "-i", keyFile, "-o", "ExitOnForwardFailure yes", "-fqnNTL", fmt.Sprintf("%d:127.0.0.1:%d", port, port), fmt.Sprintf("%s#%s", serverUser, serverIP))
cmd.Start()
pid := cmd.Process.Pid
cmd.Wait()
This returns instantly and leaves ssh running in the background. But it's pid is not the pid of the running ssh process. Moreover, it's the pid of the parent ssh process before it forked and backgrounded itself.
You don't need anything special, just don't tell ssh to background itself and don't Wait() for it. Example:
$ cat script.sh
#!/bin/sh
sleep 1
echo "I'm the script with pid $$"
for i in 1 2 3; do
sleep 1
echo "Still running $$"
done
$ cat proc.go
package main
import (
"log"
"os"
"os/exec"
)
func main() {
cmd := exec.Command("./script.sh")
cmd.Stdout = os.Stdout
err := cmd.Start()
if err != nil {
log.Fatal(err)
}
log.Printf("Just ran subprocess %d, exiting\n", cmd.Process.Pid)
}
$ go run proc.go
2016/09/15 17:01:03 Just ran subprocess 3794, exiting
$ I'm the script with pid 3794
Still running 3794
Still running 3794
Still running 3794
#Mostafa Hussein, can use goroutine waiting, manage process
function main()
cmd := exec.Command( "shell.sh" )
err := cmd.Start()
if err != nil {
return err
}
pid := cmd.Process.Pid
// use goroutine waiting, manage process
// this is important, otherwise the process becomes in S mode
go func() {
err = cmd.Wait()
fmt.Printf("Command finished with error: %v", err)
}()
return nil
}

How to output to a process substitution file in Go?

I must missing something basic because the following is not working as expected.
go run test_fd.go <(cat) under either bash or zsh results in the following:
Expected output
$ go run test_fd.go <(cat) # where n is some fd number
fd {{n}} filename /dev/fd/{{n}}
Hello!
Actual output (Zsh)
$ go run test_fd.go <(cat)
fd 11 filename /dev/fd/11
panic: Write() err write /dev/fd/11: bad file descriptor
goroutine 1 [running]:
panic(0xefe80, 0xc820010200)
/usr/local/Cellar/go/1.6/libexec/src/runtime/panic.go:464 +0x3e6
main.main()
/Users/bmf/Projects/piper/main.go:32 +0x515
exit status 2
Actual output (Bash)
$ go run main.go <(cat)
fd 63 filename /dev/fd/63
panic: Write() err write /dev/fd/63: bad file descriptor
goroutine 1 [running]:
panic(0xefe80, 0xc82006c240)
/usr/local/Cellar/go/1.6/libexec/src/runtime/panic.go:464 +0x3e6
main.main()
/Users/bmf/Projects/piper/main.go:32 +0x515
exit status 2
Source
// test_fd.go
package main
import (
"fmt"
"os"
"regexp"
"strconv"
)
var fdRegex = regexp.MustCompile(`\A/dev/fd/(\d+)\z`)
func main() {
for _, filename := range os.Args {
fdStrMatch := fdRegex.FindStringSubmatch(filename)
if len(fdStrMatch) != 2 {
continue
}
fd, _ := strconv.Atoi(fdStrMatch[1]) // fdStrMatch[1] is \d+
fmt.Fprintf(os.Stderr, "fd %d filename %s\n", fd, filename)
f := os.NewFile(uintptr(fd), filename)
/*
f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY, 0777) // have tried many combinations of modes
if err != nil {
panic(fmt.Sprintf("Create() err %v", err))
}
*/
_, err := f.Write([]byte("Hello!\n"))
if err != nil {
panic(fmt.Sprintf("Write() err %v", err))
}
}
}
Wrong form of process substitution:
$ go run test_fd.go >(cat)
<(...) is read-only from the process via some file, usually /dev/fd/*
>(...) is write-only from the process via some file usually /dev/fd/*

Exit Status 2 on running Grep command using Golang

I write simple script using Golang to grep a log file with some parameters. Here's my shell command
grep CRON var/log/sys | tail -5 | grep "cd /home/raka/repo && git status"
I want to run command above in Golang using os/exec package. Here's my code sniped.
var (
reader io.Reader
out []byte
err error
commandName string = "grep"
)
args := []string{"CRON", "/var/log/syslog", "| tail -6", "| grep \"git status\""}
cmd := exec.Command(commandName, args...)
r, err = cmd.StdoutPipe()
err = cmd.Start()
out, err = ioutil.ReadAll(r)
err = cmd.Wait()
return strings.Split(string(out), "\n")```
Currently, the sniped above doesn't work, because of exit status 2.
Any of you guys/ladies have solution for this problem? thank you so much.
Pipes (|) are implemented by a shell program (like bash). If you want to use them you should execute shell passing a command containing piped program invocations:
exec.Command("/bin/sh", "-c",
"grep CRON var/log/sys | tail -5 | grep \"cd /home/raka/repo && git status\"")

Call source from inside a Go program

For fun and to better learn Go, I'm trying to re-implement antigen in Go.
Problem is: source is a shell built-in function, so I can't call it with os/exec Command function, because it expects an executable in PATH.
How can I do this? And, is it possible to make a source from inside a go program affect the user shell?
You can write the command directly in the terminal device. But, to do that, first you need to know which device is using the user. A script that executes your program can be a solution.
#!/bin/bash
echo Running from foo script, pid = $$
go run foo.go `tty`
Then, the program has to write the commands to the terminal device.
package main
import (
"C"
"fmt"
"os"
"syscall"
"unsafe"
)
func main() {
// Get tty path
if len(os.Args) < 2 {
fmt.Printf("no tty path\n")
os.Exit(1)
}
ttyPath := os.Args[1]
// Open tty
tty, err := os.Open(ttyPath)
if err != nil {
fmt.Printf("error opening tty: %s\n", err.Error())
os.Exit(2)
}
defer tty.Close()
// Write a command
cmd := "echo Hello from go, pid = $$\n"
cmdstr := C.CString(cmd)
cmdaddr := uintptr(unsafe.Pointer(cmdstr))
for i := range []byte(cmd) {
_, _, err := syscall.Syscall(syscall.SYS_IOCTL, tty.Fd(), syscall.TIOCSTI, cmdaddr+uintptr(i))
if uintptr(err) != 0 {
fmt.Printf("syscall error: %s\n", err.Error())
os.Exit(3)
}
}
}
Here is an example output:
$ echo $$
70318
$ ./foo
Running from foo script, pid = 83035
echo Hello from go, pid = $$
$ echo Hello from go, pid = $$
Hello from go, pid = 70318
Note that I am executing the script with ./ not source, so the PID of the script differs. But later, the command executed by the go program has the same PID.

Resources