/bin/sh -version
GNU sh, version 1.14.7(1)
exitfn () {
# Resore signal handling for SIGINT
echo "exiting with trap" >> /tmp/logfile
rm -f /var/run/lockfile.pid # Growl at user,
exit # then exit script.
}
trap 'exitfn; exit' SIGINT SIGQUIT SIGTERM SIGKILL SIGHUP
The above is my function in shell script.
I want to call it in some special conditions...like
when:
"kill -9" fires on pid of this script
"ctrl + z" press while it is running on -x mode
server reboots while script is executing ..
In short, with any kind of interrupt in script, should do some action
eg. rm -f /var/run/lockfile.pid
but my above function is not working properly; it works only for terminal close or "ctrl + c"
Kindly don't suggest to upgrade "bash / sh" version.
SIGKILL cannot be trapped by the trap command, or by any process. It is a guarenteed kill signal, that by it's definition cannot be trapped. Thus upgrading you sh/bash will not work anyway.
You can't trap kill -9 that's the whole point of it, to destroy processes violently that don't respond to other signals (there's a workaround for this, see below).
The server reboot should first deliver a signal to your script which should be caught with what you have.
As to the CTRL-Z, that also gives you a signal, SIGSTOP from memory, so you may want to add that. Though that wouldn't normally be a reason to shut down your process since it may be then put into the background and restarted (with bg).
As to what do do for those situations where your process dies without a catchable signal (like the -9 case), the program should check for that on startup.
By that, I mean lockfile.pid should store the actual PID of the process that created it (by using echo $$ >/var/run/myprog_lockfile.pid for example) and, if you try to start your program, it should check for the existence of that process.
If the process doesn't exist, or it exists but isn't the right one (based on name usually), your new process should delete the pidfile and carry on as if it was never there. If the old process both exists and is the right one, your new process should log a message and exit.
Related
I have a launch.sh script which I submit on the cluster with
bsub $settings < launch.sh
This launch.sh bash script looks simplified as the following:
function trap_with_arg() {
func="$1" ; shift
for sig ; do
echo "$ES Installing trap for signal $sig"
trap "$func $sig" "$sig"
done
}
function signalHandler() {
# do stuff depending in what stage the script is
}
# Setup the Trap
trap_with_arg signalHandler SIGINT SIGTERM SIGUSR1 SIGUSR2
./start.sh
mpirun process.sh
./end.sh
Where process.sh calls two binaries (as an example) as
./binaryA
./binaryB
My question is the following:
The cluster already sends SIGUSR1 (approx. 10min before SIGTERM) to the process (I think this is the bash shell running my launch.sh script).
At the moment I catch this signal in the launch.sh script and call some signal handler. The problem is, this signal handler only gets executed (at least what I know) after a running command is finished (e.g. that might be mpirun process.sh or ./start.sh )
How can I forward these signals to make the commands/binaries exit gracefully. Forwarding for example to process.sh (mpirun, as I experienced, already forwards somehow these received signals (how does it do that?)
What is the proper way of forwarding signals, (e.g. also to the binaries binaryA, binaryB ?
I have no really good clue how to do this? Making the commands execute in background, creating a child process?
Thanks for some enlightenment :-)
From bash manual at http://www.gnu.org/software/bash/manual/html_node/Signals.html:
If Bash is waiting for a command to complete and receives a signal for which a trap has been set, the trap will not be executed until the command completes. When Bash is waiting for an asynchronous command via the wait builtin, the reception of a signal for which a trap has been set will cause the wait builtin to return immediately with an exit status greater than 128, immediately after which the trap is executed.
Thus, the solution seems to place commands in background and use "wait":
something &
wait
I have a bash script, its contents are:
function foo {
echo "Foo!"
}
function clean {
echo "exiting"
}
trap clean EXIT
trap foo SIGTERM
echo "Starting process with PID: $$"
while :
do
sleep 60
done
I execute this on a terminal with:
./my_script
And then do this on another terminal
kill -SIGTERM my_script_pid # obviously the PID is the one echoed from my_script
I would expect to see the message "Foo!" from the other terminal, but It's not working. SIGKILL works and the EXIT code is also executed.
Using Ctrl-C on the terminal my_script is running on triggers foo normally, but somehow I can't send the signal SIGTERM from another terminal to this one.
Replacing SIGTERM with any other signal doesn't change a thing (besides Ctrl-C not triggering anything, it was actually mapped to SIGUSR1 in the beginning).
It may be worth mentioning that just the signal being trapped is not working, and any other signal is having the default behaviour.
So, what am I missing? Any clues?
EDIT: I also just checked it wasn't a privilege issue (that would be weird as I'm able to send SIGKILL anyway), but it doesn't seem to be that.
Bash runs the trap only after sleep returns.
To understand why, think in C / Unix internals: While the signal is dispatched instantly to bash, the corresponding signal handler that bash has setup only does something like received_sigterm = true.
Only when sleep returns, and the wait system call which bash issued after starting the sleep process returns also, bash resumes its normal work and executes your trap (after noticing received_sigterm).
This is done this way for good reasons: Doing I/O (or generally calling into the kernel) directly from a signal handler generally results in undefined behaviour as far as I know - although I can't tell more about that.
Apart from this technical reason, there is another reason why bash doesn't run the trap instantly: This would actually undermine the fundamental semantics of the shell. Jobs (this includes pipelines) are executed strictly in a sequential manner unless you explicitly mess with background jobs.
The PID that you originally print is for the bash instance that executes your script, not for the sleep process that it is waiting on. During sleep, the signal is likely to be ignored.
If you want to see the effect that you are looking for, replace sleep with a shorter-lived process like ps.
function foo {
echo "Foo!"
}
function clean {
echo "exiting"
}
trap clean EXIT
trap foo SIGTERM
echo "Starting process with PID: $$"
while :
do
ps > /dev/null
done
My sample file
traptest.sh:
#!/bin/bash
trap 'echo trapped' TERM
while :
do
sleep 1000
done
$ traptest.sh &
[1] 4280
$ kill %1 <-- kill by job number works
Terminated
trapped
$ traptest.sh &
[1] 4280
$ kill 4280 <-- kill by process id doesn't work?
(sound of crickets, process isn't killed)
If I remove the trap statement completely, kill process-id works again?
Running some RHEL 2.6.18-194.11.4.el5 at work. I am really confused by this behaviour, is it right?
kill [pid]
send the TERM signal exclusively to the specified PID.
kill %1
send the TERM signal to the job #1's entire process group, in this case to the script pid + his children (sleep).
I've verified that with strace on sleep process and on script process
Anyway, someone got a similar problem here (but with SIGINT instead of SIGTERM): http://www.vidarholen.net/contents/blog/?p=34.
Quoting the most important sentence:
kill -INT %1 sends the signal to the job’s process group, not the backgrounded pid!
This is expected behavior. Default signal sent by kill is SIGTERM, which you are catching by your trap. Consider this:
#!/bin/bash
# traptest.sh
trap "echo Booh!" SIGINT SIGTERM
echo "pid is $$"
while : # This is the same as "while true".
do
a=1
done
(sleep really creates a new process and the behavior is clearer with my example I guess).
So if you run traptest.sh in one terminal and kill TRAPTEST_PROCESS_ID from another terminal, output in the terminal running traptest will be Booh! as expected (and the process will NOT be killed). If you try sending kill -s HUP TRAPTEST_PROCESS_ID, it will kill the traptest process.
This should clear up the %1 confusion.
Note: the code example is taken from tldp
Davide Berra explained the difference between kill %<jobspec> and kill <PID>, but not how that difference results in what you observed. After all, Unix signal handlers should be called pretty much instantaneously, so why does sending a SIGTERM to the script alone not trigger its trap handler?
The bash man page explains why, in the last paragraph of the SIGNALS section:
If bash is waiting for a command to complete and receives a signal for
which a trap has been set, the trap will not be executed until the
command completes.
So, the signal was delivered immediately, but the handler execution was deferred until sleep exited.
Hence, with kill %<jobspec>:
Both the script and sleep received SIGTERM
bash registered the signal, noticed that a trap was set for it, and queued the handler for future execution
sleep exited immediately
bash noted sleep's exit, and ran the trap handler
whereas with kill <script_PID>:
Only the script received SIGTERM
bash registered the signal, noticed that a trap was set for it, and queued the handler for future execution
sleep exited after 1000 seconds
bash noted sleep's exit, and ran the trap handler
Obviously, you didn't want long enough to see that last bit. :)
If you're interested in the gory details, download the bash source code and look in trap.c, specifically the trap_handler() and run_pending_traps() functions.
I can write shell scripts that trap SIGINT just fine, but I can't seem to trap SIGQUIT.
#!/bin/bash
function die {
echo "Dying on signal $1"
exit 0
}
trap 'die "SIGINT"' SIGINT
trap 'die "SIGQUIT"' SIGQUIT
while true; do
echo "sleeping..."
sleep 5
done
Executing this script and pressing CTRL-C has the desired effect, but pressing CTRL-\ (which, as I understand, should trigger SIGQUIT) does nothing except print ^\ in the terminal. Why?
I have two running theories. The first is that the semantics of SIGINT and SIGQUIT are different such that SIGQUIT only gets sent to the child process sleep, while SIGINT gets sent to both the child process and the parent bash process. If this is the case, where is it documented?
My second theory is that bash not only ignores (i.e., has a no-op handler for) SIGQUIT by default (as the man page suggests), but does not allow it to be trapped at all. This theory overlaps with the first theory, since it could be the case that SIGQUIT is going to both parent and child, but the parent (bash) just can't trap it. If this is the case, is there any way to trap SIGQUIT in a bash script?... perhaps some shopt I can set?
Edit: this is on Ubuntu 10.10 in gnome-terminal 2.32.0 running bash 4.1.5, and yes ^\ is configured to issue SIGQUIT (as reported by stty -a and confirmed by issuing ^\ SIGQUITs to other programs like ping).
UPDATE:
I just discovered that the problem must be due somehow to gnome-terminal. If I run this script from a virtual console (i.e., ctrl-alt-f1 to get out of X), it traps SIGQUIT perfectly fine when I press ^\. Same bash and everything, so the only difference must be the terminal emulator. So now my question becomes: how can I configure gnome-terminal to behave like the virtual console in this respect? I diff'd the outputs of stty -a in the virtual console and in gnome-terminal, and while there are differences, nothing seems immediately relevant (e.g., they both have quit = ^\;).
UPDATE 2:
Another experiment. Simply execute $ sleep 60 in gnome-terminal; press ^\ and the signal goes uncaught. Now execute $ sleep 60 in the virtual console; press ^\ and the signal is caught -- the process prints Quit and exits. But now run $ ping google.com in gnome-terminal and press ^\ -- the signal is caught and handled as expected. So there is something weird about gnome-terminal such that SIGQUIT can be caught by some programs, but not by others, even if those other others do catch it when invoked from the virtual console. Perhaps I should just upgrade my gnome-terminal.
I can only assume that this was some sort of bug in gnome-terminal 2.32.0; I have since upgraded to Ubuntu 11.04, with gnome-terminal 2.32.1 (and bash 4.2.8) and SIGQUIT is now trapped as expected.
I observe the same behavior on Fedora 19 with XFCE: according to ps s, bash in the xfce4-terminal has SIGQUIT ignored, running yes >/dev/null & and then ps s shows that even a subprocess has SIGQUIT ignored. When I (from the same terminal) run ssh localhost, the shell in the ssh session also has SIGQUIT ignored, but yes >/dev/null & has not.
Given the above comment which mentions gnome-terminal, I would guess the bug is in the common part of the two terminals: the vte library. Mine is vte-0.28.2-9.fc19.x86_64.
I am unable to trap a signal when running in a child / background process.
Here is my simple bash script:
#!/bin/bash
echo "in child"
trap "got_signal" SIGINT
function got_signal {
echo "trapped"
exit 0
}
while [ true ]; do
sleep 2
done
When running this and later do
kill -SIGINT (pid)
everything works as expected, it prints trapped and exits.
Now, if I start the same script from a parent script like this:
#!/bin/bash
echo "starting the child"
./child.sh &
Then the child does not trap the signal anymore.... ?
After changing to use SIGTERM instead of SIGINT, it seems to be working correctly... ?
The bash manpage on OSX (but it should be the same in other versions) has this to say about signal handling:
Non-builtin commands run by bash have signal handlers set to the values
inherited by the shell from its parent. When job control is not in
effect, asynchronous commands ignore SIGINT and SIGQUIT in addition to
these inherited handlers.
and further on, under the trap command:
Signals ignored upon entry to the shell cannot
be trapped or reset.
Since scripts don't use job control by default, this means the case you're talking about.
Per your note:
Signals ignored upon entry to the shell cannot be trapped or reset.
I have noticed that ZSH does not ignore the signals sent back and forth between parent and child process, but bash does. Here's the question I posted myself:
Trapping CHLD signal - ZSH works but ksh/bash/sh don't?