I am running a SpringBoot Application. I have added some cleanup action using #PreDestroy annotation. When I terminate the process using SIGTERM that is kill ${PID} of tomcat, it performs all the cleanup tasks by calling method marked with #PreDestroy. However, When I use SIGKILL i.e kill -9 ${PID} of tomcat , the clean up is not performed.
Is there any way to make the application perform all the pre destruction work when it is terminated using SIGKILL?
No, there is not. Regular kill waits for the process to exit while kill -9 does not and kills it immediately. What is important is that the signals SIGKILL and SIGSTOP cannot be caught, blocked, or ignored. man 7 signal shows a short summary of what each signal means:
Signal Value Action Comment
──────────────────────────────────────────────────────────────────────
SIGHUP 1 Term Hangup detected on controlling terminal
or death of controlling process
SIGINT 2 Term Interrupt from keyboard
SIGQUIT 3 Core Quit from keyboard
SIGILL 4 Core Illegal Instruction
SIGABRT 6 Core Abort signal from abort(3)
SIGFPE 8 Core Floating-point exception
SIGKILL 9 Term Kill signal
SIGSEGV 11 Core Invalid memory reference
SIGPIPE 13 Term Broken pipe: write to pipe with no
readers; see pipe(7)
SIGALRM 14 Term Timer signal from alarm(2)
SIGTERM 15 Term Termination signal
SIGUSR1 30,10,16 Term User-defined signal 1
SIGUSR2 31,12,17 Term User-defined signal 2
SIGCHLD 20,17,18 Ign Child stopped or terminated
SIGCONT 19,18,25 Cont Continue if stopped
SIGSTOP 17,19,23 Stop Stop process
SIGTSTP 18,20,24 Stop Stop typed at terminal
SIGTTIN 21,21,26 Stop Terminal input for background process
SIGTTOU 22,22,27 Stop Terminal output for background process
Related
I have a Bash script running a bunch of background jobs in parallel.
Under certain conditions, before a background job completes, it sends
a USR1 signal to the spawning Bash process (say, to inform that
some process that was run as a part of the job had terminated with
a nonzero exit code).
In a simplified form, the script is equivalent to the one shown below.
Here, for simplicity, each background job always sends a USR1 signal
before completion, unconditionally (via the signalparent() function).
signalparent() { kill -USR1 $$; }
handlesignal() { echo 'USR1 signal caught' >&2; }
trap handlesignal USR1
for i in {1..10}; do
{
sleep 1
echo "job $i finished" >&2
signalparent
} &
done
wait
When I run the above script (using Bash 3.2.57 on macOS 11.1, at least),
I observe some behavior that I cannot explain, which makes me think
that there is something in the interplay of Bash job management and
signal trapping that I overlook.
Specifically, I would like to acquire an explanation for the following
behaviors.
Almost always, when I run the script, I see fewer “signal caught”
lines in the output (from the handlesignal() function) than there
are jobs started in the for-loop—most of the time it is one to
four of those lines that are printed for ten jobs being started.
Why is it that, by the time the wait call completes, there
are still background jobs whose signaling kill commands had
not been yet executed?
At the same time, every so often, in some invocations of the script,
I observe the kill command (from the signalparent() function)
report an error regarding the originating process running the script
(i.e., the one with the $$ PID) no longer being present—see the
output below.
How come there are jobs whose signaling kill commands are still
running while the parent shell process had already terminated?
It was my understanding that it is impossible for the parent
process to terminate before all background jobs do, due to the
wait call.
job 2 finished
job 3 finished
job 5 finished
job 4 finished
job 1 finished
job 6 finished
USR1 signal caught
USR1 signal caught
job 10 finished
job 7 finished
job 8 finished
job 9 finished
bash: line 3: kill: (19207) - No such process
bash: line 3: kill: (19207) - No such process
bash: line 3: kill: (19207) - No such process
bash: line 3: kill: (19207) - No such process
Both of these behaviors signalize to me a presence of a race condition
of some kind, whose origins I do not quite understand. I would
appreciate if anyone could enlighten me on those, and perhaps even
suggest how the script could be changed to avoid such race conditions.
This is explained in the Bash Reference Manual as follows.
When bash is waiting for an asynchronous command via the wait builtin, the reception of a signal for which a trap has been set will cause the wait builtin to return immediately with an exit status greater than 128, immediately after which the trap is executed.
So, you need to repeat wait until it returns 0 to make sure all background jobs have terminated, e.g.:
until wait; do
:
done
It was my understanding that it is impossible for the parent process to terminate before all background jobs do, due to the wait call.
That is a misunderstanding; wait may return due to reception of a signal for which a trap has been set while there are running jobs at the background, and that may lead to normal completion of the program, with the side effect of leaving those jobs orphaned.
Regarding ‘Almost always, when I run the script, I see fewer “signal caught” lines in the output’—
According to signal(7):
Standard signals do not queue. If multiple instances of a standard signal are generated while that signal is blocked, then only one instance of the signal is marked as pending (and the signal will be delivered just once when it is unblocked).
One way to change your script so that the signals do not arrive at the same time is as follows:
signalparent() {
kill -USR1 $$
}
ncaught=0
handlesignal() {
(( ++ncaught ))
echo "USR1 signal caught (#=$ncaught)" >&2
}
trap handlesignal USR1
for i in {1..10}; do
{
sleep $i
signalparent
} &
done
nwaited=0
while (( nwaited < 10 )); do
wait && (( ++nwaited ))
done
Here is the output of the modified script with Bash 5.1 on macOS 10.15:
USR1 signal caught (#=1)
USR1 signal caught (#=2)
USR1 signal caught (#=3)
USR1 signal caught (#=4)
USR1 signal caught (#=5)
USR1 signal caught (#=6)
USR1 signal caught (#=7)
USR1 signal caught (#=8)
USR1 signal caught (#=9)
USR1 signal caught (#=10)
I am using a systemd service which calls a process when it's been "started" (e.g. systemctl start test.service). As per the design, the process stays in a loop forever, we are able to see process existence using the ps command. We have also seen that the process is getting killed (as intended) for systemctl stop command.
However, our requirement is that we want to do some safe shutdown operations from within the process before it gets killed. But I am not sure how to detect a systemd stop operation from within the process.
Does a systemctl stop test.service command send SIGKILL or SIGTERM signal to kill the process? How can I detect a systemctl stop operation from within a process?
By default, a SIGTERM is sent, followed by 90 seconds of waiting followed by a SIGKILL.
Killing processes with systemd is very customizable and well-documented.
I recommend reading all of man systemd.kill as well as reading about ExecStop= in man systemd.service.
To respond to those signals, refer to the signal handling documentation for the language you are using.
Does a systemctl stop test.service command send SIGKILL or SIGTERM signal to kill the
process? How can i detect a systemctl stop operation from within a process?
Systemd sends SIGTERM signal to process. In process you have to register signals, which are "caught".
In process, eg. SIGTERM signal can be registered like this:
void signal_callback()
{
printf("Process is going down\n");
}
signal(SIGTERM, signal_callback)
When SIGTERM is sent to the process, the signal_callback() function is executed.
I am running a program which invokes a shell script (for discussion sh1 with pid 100).
This script in turn invokes another script (for discussion sh2 with pid 101) and waits for it to finish. sh2(child script) takes about 50 seconds to finish.
The way I invoke the sh2 (/bin/sh2.sh )
During waiting for child to be done, I try to Terminate sh1 (using kill -15 100). I have a handler function in sh1 to handle this signal. However, I observe that my sh1(parent script) doesnot get terminated till the child is done with its work (for 50 seconds) and only after that this Signal is handled.
I modified my child script, to take 30 seconds to finish and I observe that after issuing the SIGTERM to sh1, it then takes around 30 seconds to terminate.
Is this the behavior while handling the SIGTERM ? that is to remain blocked by the child process ? and only then handle the signal. Doesn't the process get interrupted for signal handling?
SIGNAL Handling in parent script.
function clean_up()
{
//Do the cleanup
}
trap "clean_up $$; exit 0" TERM
If sh1 invokes sh2 and waits for it to finish, then it doesn't run the trap for the signal until after sh2 finishes. That is, if this is sh1:
#!/bin/sh
trap 'echo caught signal delayed' SIGTERM
sh2
then sh1 will catch the signal and do nothing until sh2 finishes, and then it will execute the trap. If you want the trap to fire as soon as the signal is sent, you can run sh2 asynchronously and wait for it explicitly:
#!/bin/sh
trap 'echo caught signal' SIGTERM
sh2&
wait
Unfortunately, this doesn't re-enter the wait. If you need to continue waiting, it's not really possible to do reliably, but you can get close with:
#!/bin/sh
trap 'echo caught signal' SIGTERM
sh2&
(exit 129) # prime the loop (eg, set $?) to simulate do/while
while test $? -gt 128; do wait; done
This isn't reliable because you can't tell the difference between catching a signal yourself and sh2 being terminated by a signal. If you need this to be reliable, you should re-write sh1 in a language which allows better control of signals.
I have an CLI app that is seg faulting during termination (After sending a Ctrl-C)
Pressing Ctrl-C in lldb naturally pauses execution.
Then I try:
(lldb)process signal SIGINT
(lldb)process continue
But that doesn't actually seem to do anything to terminate the app.
Also tried:
(lldb)process signal 2
The debugger uses ^C for interrupting the target, so it assumes that you don't actually want the ^C propagated to the target. You can change this behavior by using the "process handle" command:
(lldb) process handle SIGINT -p true
telling lldb to "pass" SIGINT to the process.
If you had stopped the process in lldb by issuing a ^C, then when you change the process handle as shown here and continue, that SIGINT will be forwarded to the process.
If you stopped for some other reason, after specifying SIGINT to be passed to the process, you can generate a SIGINT by getting the PID of the debugee using the process status and send a SIGINT directly to said process using platform shell:
(lldb) process status
(lldb) platform shell kill -INT {{PID from previous step}}
(lldb) continue
The easiest way I found was just to send the process a SIGINT directly. Take the pid of the debuggee process (which process status will show you) and run
kill -INT <pid>
from another terminal.
My sample file
traptest.sh:
#!/bin/bash
trap 'echo trapped' TERM
while :
do
sleep 1000
done
$ traptest.sh &
[1] 4280
$ kill %1 <-- kill by job number works
Terminated
trapped
$ traptest.sh &
[1] 4280
$ kill 4280 <-- kill by process id doesn't work?
(sound of crickets, process isn't killed)
If I remove the trap statement completely, kill process-id works again?
Running some RHEL 2.6.18-194.11.4.el5 at work. I am really confused by this behaviour, is it right?
kill [pid]
send the TERM signal exclusively to the specified PID.
kill %1
send the TERM signal to the job #1's entire process group, in this case to the script pid + his children (sleep).
I've verified that with strace on sleep process and on script process
Anyway, someone got a similar problem here (but with SIGINT instead of SIGTERM): http://www.vidarholen.net/contents/blog/?p=34.
Quoting the most important sentence:
kill -INT %1 sends the signal to the job’s process group, not the backgrounded pid!
This is expected behavior. Default signal sent by kill is SIGTERM, which you are catching by your trap. Consider this:
#!/bin/bash
# traptest.sh
trap "echo Booh!" SIGINT SIGTERM
echo "pid is $$"
while : # This is the same as "while true".
do
a=1
done
(sleep really creates a new process and the behavior is clearer with my example I guess).
So if you run traptest.sh in one terminal and kill TRAPTEST_PROCESS_ID from another terminal, output in the terminal running traptest will be Booh! as expected (and the process will NOT be killed). If you try sending kill -s HUP TRAPTEST_PROCESS_ID, it will kill the traptest process.
This should clear up the %1 confusion.
Note: the code example is taken from tldp
Davide Berra explained the difference between kill %<jobspec> and kill <PID>, but not how that difference results in what you observed. After all, Unix signal handlers should be called pretty much instantaneously, so why does sending a SIGTERM to the script alone not trigger its trap handler?
The bash man page explains why, in the last paragraph of the SIGNALS section:
If bash is waiting for a command to complete and receives a signal for
which a trap has been set, the trap will not be executed until the
command completes.
So, the signal was delivered immediately, but the handler execution was deferred until sleep exited.
Hence, with kill %<jobspec>:
Both the script and sleep received SIGTERM
bash registered the signal, noticed that a trap was set for it, and queued the handler for future execution
sleep exited immediately
bash noted sleep's exit, and ran the trap handler
whereas with kill <script_PID>:
Only the script received SIGTERM
bash registered the signal, noticed that a trap was set for it, and queued the handler for future execution
sleep exited after 1000 seconds
bash noted sleep's exit, and ran the trap handler
Obviously, you didn't want long enough to see that last bit. :)
If you're interested in the gory details, download the bash source code and look in trap.c, specifically the trap_handler() and run_pending_traps() functions.