How to handle signals in bash during synchronous execution? - bash

I have a bash script process which at some point executes a long-running subprocess synchronously.
During the run of that subprocess, a signal is sent directly to the bash script process requesting the script to terminate.
Is there any way to intercept that signal, terminate the subprocess and then exit the bash process?
Apparently, bash's signal handling never interrupts synchronous calls?
I cannot control the fact that the termination signal is sent to the bash process. Although if the signal could propagate to the child process, that would also solve my issue.
thanks in advance,
Broes

See the man page of bash, chapter SIGNALS:
If bash is waiting for a command to complete and receives a signal for which a trap has been set, the trap will not be executed until the command completes. When bash is waiting for an asynchronous command via the wait builtin, the reception of a signal for which a trap has been set will cause the wait builtin to return immediately with an exit status greater than 128, immediately after which the trap is executed.
So, run your external program asynchronously and use wait. Kill it using $!.

Here's a bash utility function I wrote to handle this. It's proved useful and robust. I hope you find it useful.
# Run a command in a way that can be interrupted by a signal (eg SIGTERM)
#
# When bash receives a SIGTERM it normally simply exits. If it's executing a subprocess
# that subprocess isn't signaled. (Typically that's not a problem for interactive shells
# because the entire Process Group gets sent the signal.)
#
# When running a script it's sometimes useful for the script to propagate a SIGTERM
# to the command that was running. We can do that by using the trap builtin to catch
# the signal. But it's a little tricky, per the bash manual:
#
# If bash is waiting for a command to complete and receives a signal for
# which a trap has been set, the trap will not be executed until the
# command completes.
#
# so a script executing a long-running command with a signal trap set won't
# notice the signal until later. There's a way around that though...
#
# When bash is waiting for an asynchronous command via the wait builtin, the
# reception of a signal for which a trap has been set will cause the wait
# builtin to return immediately with an exit status greater than 128,
# immediately after which the trap is executed.
#
# Usage:
#
# interruptable [options] command [args]
#
# Options:
# --killall - put the child into a process group (via setsid)
# and send the SIGTERM to the process group
# --debug - print a message including pid of the child
#
# Usage examples:
#
# interruptable sleep 3600
#
# If not interrupted, the exit status of the specified command is returned.
# If interrupted, the specified command is sent a SIGTERM and the current
# shell exits with a status of 143.
interruptable() {
# handle options
local setsid=""
local debug=false
while true; do
case "${1:-}" in
--killall) setsid=setsid; shift ;;
--debug) debug=true; shift ;;
--*) echo "Invalid option: $1" 1>&2; exit 1;;
*) break;; # no more options
esac
done
# start the specified command
$setsid "$#" &
local child_pid=$!
# arrange to propagate a signal to the child process
trap '
exec 1>&2
set +e
trap "" SIGPIPE # ensure a possible sigpipe from the echo does not prevent the kill
echo "${BASH_SOURCE[0]} caught SIGTERM while executing $* (pid $child_pid), sending SIGTERM to it"
# (race) child may have exited in which case kill will report an error
# if setsid is used then prefix the pid with a "-" to indicate that the signal
# should be sent to the entire process group
kill ${setsid:+-}$child_pid
exit 143
' SIGTERM
# ensure that the trap doesn't persist after we return
trap 'trap - SIGTERM' RETURN
$debug && echo "interruptable wait (child $child_pid, self $$) for: $*"
# An error status from the child process will trigger an exception (via set -e)
# here unless the caller is checking the return status
wait $child_pid # last command, so status of waited for command is returned
}

Yes, the signal can be intercepted with the trap command. See the example below:
#!/bin/bash
function wrap {
local flag=0
trap "flag=1" SIGINT SIGTERM
xeyes &
subppid=$!
while :
do
if [ $flag -ne 0 ] ; then
kill $subppid
break
fi
sleep 1
done
}
flag=0
trap "flag=1" SIGINT SIGTERM
wrap &
wrappid=$!
while : # This is the same as "while true".
do
if [ $flag -ne 0 ] ; then
kill $wrappid
break
fi
sleep 1 # This script is not really doing anything.
done
echo 'end'
What trap basically does is that it executes the command between "". So here the main function is in the while loop below. In every iteration the script checks if the flag is set, if not, it sleeps for a second. Before that, we remembered the pid of the child process via $!. The trap issues the command when the SIGINT or SIGTERM is caught (for other signals see kill manual).
The wrapper function does the same, as the main function. Additionally it calls the actual subprocess function (in this case subprocess is xeyes). When the wrapper function receives SIGTERM signal from the main function (main function also caught one of the signals), the wrapper function can clean up things before actually killing the subprocess. After that it breaks from the while loop and exits the wrapper function. Then the main function also breaks and prints 'end'.
edit:
Hope I understand this correctly, you are forced to execute xeyes &. Then the steps would be as follows (in terminal):
xeyes &
subpid=$!
trap "kill $subpid && exit " SIGINT SIGTERM
.... other stuff
.... more stuff
^C #TERMINATE - this firstly kills xeyes and then exits the terminal

Related

How to kill a sleep process using a script

I want to write a script that will do the following things:
start a sleep 500 command in the foreground.
then send the sleep 500 command in the background so that I can get my prompt back
I have tried doing this inside the script:
sleep 500
kill `pidof sleep`
But when I am running the script, the sleep command does not stop and I do not get my prompt back.
I can't find the exact dup I was looking for so here is a short example. If I understand what you are wanting, you want to start a sleep process and then at any time you want to be able to quit the sleep process returning you to your prompt. You say "background" the sleep process, but I'm not sure whether you actually need to background it to let it expire on its own in the future, or if just need a way out of the sleep whenever you choose (that would make more sense)
While you can simply use Ctrl + C to interrupt the process, you won't be returned to your script. So if you need a way to return to your script, you need to trap SIGINT. When you set a trap 'some command' SIGINT your script will intercept and process the interrupt and control will return to the point following the sleep command, e.g.
#!/bin/bash
## set trap
trap 'printf "\n[Ctrl + C] SIGINT intercepted\n"' SIGINT
printf "sleeping 500...\n"
sleep 500
## rest of script after sleep
printf "control returned to script\ncontinuing to process commands\n"
Example Running Script Above
$ bash trap_int.sh
sleeping 500...
^C
[Ctrl + C] SIGINT intercepted
control returned to script
continuing to process commands
That may or may not satisfy what you need. However, you can build on that to control what happens after an interrupt a bit further. When you set the trap you can have it call a separate function within your script that can conditionally execute code and end the script on that separate branch by calling exit. That has the effect of transferring control back to your script to only execute commands from the path that starts with the function you specify. This provides away to avoid executing code that follows the sleep, while providing a separate execution path in your script to run whatever is needed after interrupt from sleep.
Conditionally Executing Commands After Interrupt
Rather than just sleep 500, in the next example we will repeatedly call two functions sleep_a which calls a nested sleep function sleep_b. Following SIGINT control will be passed to a function named handle_interrupt where additional commands can be executed calling exit to exit the script and return you to your prompt. An additional function called actual_exit is called to perform the exit.
#!/bin/bash
## function called to exit, returning you to prompt
function actual_exit
{
printf " Exiting after '%s'\n\n" "$1"
exit
}
## function called after SIGINT, calls actual_exit to disply signal caught
function handle_interrupt
{
printf "\n Signal SIGINT Caught! [ctrl + c]\n\n"
actual_exit 'SIGINT'
}
## set trap
trap handle_interrupt SIGINT # calls handle_interrupt on [ctrl + c]
declare -i idx=0 ## loop counter
function sleep_b
{
printf "\nentering sleep_b\n"
sleep 2
printf "executing post sleep_b index: %s\n" $((idx++))
}
function sleep_a
{
printf "\nentering sleep_a\n"
sleep 2
printf "executing post sleep_a index: %s\n" $((idx++))
sleep_b
}
## repeatedly call nested sleep until signal received
while :; do
sleep_a
done
## rest of script after sleep
printf "control returned to script\ncontinuing to process commands\n"
When the script is interrupted, the command following below the while loop are never reached as the script exits through the code path provided by the function called on SIGINT.
Providing Common Commands On exit
You can add yet another layer of execution by setting a trap on EXIT. If you have a collection of signal you set a trap for, you can have all end though a common function. For example if you monitored multiple signals each returning to a function ultimately calling exit, you can set a trap on EXIT calling a final function holding commands to be executed before the script exits.
For example, you could do:
trap final_function EXIT # calls final_function on exit() from script
where a call to exit anywhere within the script would invoke the trap on EXIT calling final_function before returning control back to you.
Let me know if something like one of the incremental levels of handling canceling sleep though an interrupt will do what you need. Technically it doesn't "background" sleep, but does allow you to cancel the sleep at any moment conditionally returning control within your script to wherever you choose.

Handling signals in bash script when it started as a background process

Okay, I have a script like this:
trap 'echo "CTRL-C signal was caught!" ' SIGINT
for ((i=0; i<15; i++))
do
sleep 3
done
When I start my script in a usual way, it immediately reacts to CTRL-C command and echo "CTRL-C signal was caught!", even if there is a sleep 3 command. But when I run my script as a background process, it waits until sleep 3 command is finished, and then echo "CTRL-C signal was caught!"
I do not understand this. I think trap should wait until previous command is finished, and then it should echo something, like when it started as a background process.
Bash manual states:
Background processes (...) are immune to keyboard-generated signals.
If bash is waiting for a command to complete and receives a signal for which a trap has been set, the trap will not be executed until the command completes.
Consequently:
If your script runs in the foreground: when you press "Ctrl-C", a SIGINT is sent to the currently running process (i.e. the sleep command). The exit status of sleep tells Bash that it was interrupted by the SIGINT signal and bash calls your trap.
If your script runs in the background, then the backgrounded sleep does not receive the signal and the SIGINT trap is only executed once sleep has ended.

Prevent SIGINT from interrupting current task while still passing information about SIGINT (and preserve the exit code)

I have a quite long shell script and I'm trying to add signal handling to it.
The main task of the script is to run various programs and then clean up their temporary files.
I want to trap SIGINT.
When the signal is caught, the script should wait for the current program to finish execution, then do the cleanup and exit.
Here is an MCVE:
#!/bin/sh
stop_this=0
trap 'stop_this=1' 2
while true ; do
result="$(sleep 2 ; echo success)" # run some program
echo "result: '$result'"
echo "Cleaning up..." # clean up temporary files
if [ $stop_this -ne 0 ] ; then
echo 'OK, time to stop this.'
break
fi
done
exit 0
The expected result:
Cleaning up...
result: 'success'
Cleaning up...
^Cresult: 'success'
Cleaning up...
OK, time to stop this.
The actual result:
Cleaning up...
result: 'success'
Cleaning up...
^Cresult: ''
Cleaning up...
OK, time to stop this.
The problem is that the currently running instruction (result="$(sleep 2 ; echo success)" in this case) is interrupted.
What can I do so it would behave more like I was set trap '' 2?
I'm looking for either a POSIX solution or one that is supported by most of shell interpreters (BusyBox, dash, Cygwin...)
I already saw answers for Prevent SIGINT from closing child process in bash script but this isn't really working for me. All of these solutions require to modify each line which shouldn't be interrupted. My real script is quite long and much more complicated than the example. I would have to modify hundreds of lines.
You need to prevent the SIGINT from going to the echo in the first place (or rewrite the cmd that you are running in the variable assignment to ignore SIGINT). Also, you need to allow the variable assignment to happen, and it appears that the shell is aborting the assignment when it receives the SIGINT. If you're only worried about user generated SIGINT from the tty, you need to disassociate that command from the tty (eg, get it out of the foreground process group) and prevent the SIGINT from aborting the assignment. You can (almost) accomplish both of those with:
#!/bin/sh
stop_this=0
while true ; do
trap 'stop_this=1' INT
{ sleep 1; echo success > tmpfile; } & # run some program
while ! wait; do : ; done
trap : INT
result=$(cat tmpfile& wait)
echo "result: '$result'"
echo "Cleaning up..." # clean up temporary files
if [ $stop_this -ne 0 ] ; then
echo 'OK, time to stop this.'
break
fi
done
exit 0
If you're worried about SIGINT from another source, you'll have to re-implement sleep (or whatever command I presume sleep is a proxy for) to handle SIGINT the way you want. The key here is to run the command in the background and wait for it to prevent the SIGINT from going to it and terminating it early. Note that we've opened at least 2 new cans of worms here. By waiting in a loop, we're effectively ignoring the any errors that the subcommand might raise (we're doing this to try and implement a SIGRESTART), so may potentially hang. Also, if the SIGINT arrives during the cat, we have attempted to prevent the cat from aborting by running it in the background, but now the variable assignment will be terminated and you'll get your original behavior. Signal handling is not clean in the shell! But this gets you closer to your desired goal.
Sighandling in shell scripts can get clumsy. It's pretty much impossible to
do it "right" without the support of C.
The problem with:
result="$(sleep 2 ; echo success)" # run some program
is that $() creates a subshell and in subshells, non-ignored (trap '' SIGNAL is how you ignore SIGNAL)
signals are reset to their default dispositions which for SIGINT is to terminate the process
($( ) gets its own process, thought it will receive the signal too because the terminal-generated SIGINT
is process-group targeted)
To prevent this, you could do something like:
result="$(
trap '' INT #ignore; could get killed right before the trap command
sleep 2; echo success)"
or
result="$( trap : INT; #no-op handler; same problem
sleep 2; while ! echo success; do :; done)"
but as noted, there will be a small race-condition window between the start of the
subshell and the registration of the signal handler during which
the subshell could get killed by the reset-to-default SIGINT signal.
Both answers from #PSkocik and #WilliamPursell have helped me to get on the right track.
I have a fully working solution. It ain't pretty because it needs to use an external file to indicate that the signal didn't occurred but beside that it should work reliably.
#!/bin/sh
touch ./continue
trap 'rm -f ./continue' 2
( # the whole main body of the script is in a separate background process
trap '' 2 # ignore SIGINT
while true ; do
result="$(sleep 2 ; echo success)" # run some program
echo "result: '$result'"
echo "Cleaning up..." # clean up temporary files
if [ ! -e ./continue ] ; then # exit the loop if file "./continue" is deleted
echo 'OK, time to stop this.'
break
fi
done
) & # end of the main body of the script
while ! wait ; do : ; done # wait for the background process to end (ignore signals)
wait $! # wait again to get the exit code
result=$? # exit code of the background process
rm -f ./continue # clean up if the background process ended without a signal
exit $result
EDIT: There are some problems with this code in Cygwin.
The main functionality regarding signals work.
However, it seems like the finished background process doesn't stay in the system as a zombie. This makes the wait $! to not work. The exit code of the script has incorrect value of 127.
Solution to that would be removing lines wait $!, result=$? and result=$? so the script always returns 0.
It should be also possible to keep the proper error code by using another layer of subshell and temporarily store the exit code in a file.
For disallowing interrupting the program:
trap "" ERR HUP INT QUIT TERM TSTP TTIN TTOU
But if a sub-command handles traps by itself, and that command must really complete, you need to prevent passing signals to it.
For people on Linux that don't mind installing extra commands, you can just use:
waitFor [command]
Alternatively you can adapt the latest source code of waitFor into your program as needed, or use the code from Gilles' answer. Although that has the disadvantage of not benefiting from updates upstream.
Just mind that other terminals and the service manager can still terminate "command". If you want the service manager to be unable to close "command", it shall be run as a service with the appropriate kill mode and kill signal set.
You may want to adapt the following:
#!/bin/sh
tmpfile=".tmpfile"
rm -f $tmpfile
trap : INT
# put the action that should not be interrupted in the innermost brackets
# | |
( set -m; (sleep 10; echo success > $tmpfile) & wait ) &
wait # wait will be interrupted by Ctrl+c
while [ ! -r $tmpfile ]; do
echo "waiting for $tmpfile"
sleep 1
done
result=`cat $tmpfile`
echo "result: '$result'"
This seems also to work with programs that install their own SIGINT handler like mpirun and mpiexec and so on.

Forwarding signals in bash script which is submitted on the cluster

I have a launch.sh script which I submit on the cluster with
bsub $settings < launch.sh
This launch.sh bash script looks simplified as the following:
function trap_with_arg() {
func="$1" ; shift
for sig ; do
echo "$ES Installing trap for signal $sig"
trap "$func $sig" "$sig"
done
}
function signalHandler() {
# do stuff depending in what stage the script is
}
# Setup the Trap
trap_with_arg signalHandler SIGINT SIGTERM SIGUSR1 SIGUSR2
./start.sh
mpirun process.sh
./end.sh
Where process.sh calls two binaries (as an example) as
./binaryA
./binaryB
My question is the following:
The cluster already sends SIGUSR1 (approx. 10min before SIGTERM) to the process (I think this is the bash shell running my launch.sh script).
At the moment I catch this signal in the launch.sh script and call some signal handler. The problem is, this signal handler only gets executed (at least what I know) after a running command is finished (e.g. that might be mpirun process.sh or ./start.sh )
How can I forward these signals to make the commands/binaries exit gracefully. Forwarding for example to process.sh (mpirun, as I experienced, already forwards somehow these received signals (how does it do that?)
What is the proper way of forwarding signals, (e.g. also to the binaries binaryA, binaryB ?
I have no really good clue how to do this? Making the commands execute in background, creating a child process?
Thanks for some enlightenment :-)
From bash manual at http://www.gnu.org/software/bash/manual/html_node/Signals.html:
If Bash is waiting for a command to complete and receives a signal for which a trap has been set, the trap will not be executed until the command completes. When Bash is waiting for an asynchronous command via the wait builtin, the reception of a signal for which a trap has been set will cause the wait builtin to return immediately with an exit status greater than 128, immediately after which the trap is executed.
Thus, the solution seems to place commands in background and use "wait":
something &
wait

How do I stop a signal from killing my Bash script?

I want an infinite loop to keep on running, and only temporarily be interrupted by a kill signal. I've tried SIGINT, SIGUSR1, SIGUSR2. All of them seem to halt the loop. I even tried SIGINFO, but that wasn't supported by Linux.
#!/bin/bash
echo $$ > /tmp/pid # Save the pid
function do_something {
echo "I am doing stuff" #let's do this now, and go back to doing the thing that is to be done over and over again.
#exit
}
while :
do
echo "This should be done over and over again, but always wait for someething else to be done in between"
trap do_something SIGINT
while `true`
do
sleep 1 #so we're waiting for that other thing.
done
done
My code runs the function once, after getting a INT signal from another script, but then never again. It halts.
EDIT: Although I accidentally put en exit at the end of the function, here on Stack Overflow, I didn't in the actual code I used. Either way, it made no difference. The solution is SIGTERM as described by Tiago.
I believe you're looking for SIGTERM:
Example:
#! /bin/bash
trap -- '' SIGINT SIGTERM
while true; do
date +%F_%T
sleep 1
done
Running this example cTRL+C won't kill it nor kill <pid> you can however kill it with kill -9 <pid>.
If you don't want CTRL+Z to interrupt use: trap -- '' SIGINT SIGTERM SIGTSTP
trap the signal, then either react to it appropriately, in the function associate with the trap, or ignore it by for example associate : as command to get executed when the signal occurs.
to trap signals, bash knows the trap command
Reset trap to former action by executing trap with signal name only.
Therefore you want to (i think that's what you say you want with "only temporarily be interrupted by a kill signal"):
trap the signal at the begin of your script: trap signal custom_action
just before you want the signal to allow interrupting your script, execute: trap signal
At the end of that phase, trap again by: signal custom_action
to specify signals, you can also use their respective signal numbers. A list of signal names is printed with the command:
trap -l
the default signal sent by kill is SIGTERM (15), unless you specify a different signal after the kill command
don't exit in your do_something function. Simply let the function return to the section in your code where it was interrupted when the signal occured.
The mentioned ":" command has another potential use in your script, if you feel thusly inclined:
while :
do
sleep 1
done
can be an alternative to "while true" - no backticks needed for that, btw.
You just want to ignore the exit status.
If you want your script to keep running and not exit, without worrying about handling traps.
(my_command) || true
The parentheses execute that command in a subshell. The true is for compatibility with set -e, if you use it. It simply overrides the status to always report a success.
See the source.
I found this question to be helpful:
How to run a command before a Bash script exits?

Resources