If Ruby receives the TERM signal, it usually exits with the exit code 143, which according to this source indicates the process successfully responded to that signal. But if I make the script run without a shell, the exit code is 1.
With shell:
> cat Dockerfile
FROM ruby:alpine
CMD ruby -e "Process.kill('TERM', Process.pid)" # <- shell form
> docker build -t term_shell . > /dev/null
> docker run term_shell
Terminated
> echo $?
143
Without shell:
> cat Dockerfile
FROM ruby:alpine
CMD ["ruby", "-e", "Process.kill('TERM', Process.pid)"] # <- exec form
> docker build -t term_exec . > /dev/null
> docker run term_exec
> echo $?
1
But if I exit myself with 143, the exit code is as expected:
> cat Dockerfile
FROM ruby:alpine
CMD ["ruby", "-e", "exit(143)"] # <- exec form
> docker build -t exit_exec . > /dev/null
> docker run exit_exec
> echo $?
143
Why is that? Does the exit code when ruby receives TERM comes not from Ruby, but the shell?
The exit code of your second example is 1 because the call Process.kill('TERM', Process.pid) failed. ruby -e exited because of this failure, and the status code in that case is 1.
With CMD ruby -e "Process.kill('TERM', Process.pid)", docker executes the given command in a shell. In a running container, it means that the root process with pid 1 will be /bin/sh -c, and the ruby -e command will be executed in a child process with another pid (for example 6).
With CMD ["ruby", "-e", "Process.kill('TERM', Process.pid)"], docker executes directly ruby -e as the root process with pid 1.
The PID 1 on Linux behaves differently than the normal ones. From docker documentation:
Note: A process running as PID 1 inside a container is treated specially by Linux: it ignores any signal with the default action. So, the process will not terminate on SIGINT or SIGTERM unless it is coded to do so.
So in your case, the TERM signal won't be sent to you process.
You can find more information on PID 1 behavior on this article:
https://hackernoon.com/my-process-became-pid-1-and-now-signals-behave-strangely-b05c52cc551c
Related
The following script.sh is executed:
#!/bin/bash
set -eu
# code ...
su buser
mkdir /does/not/work
echo $?
echo This should not be printed
Output:
1
This should not be printed
How i execute the script:
docker exec -i fancy_container bash < script.sh
Question: Why does the script not terminate after the failing command even when set -e was defined and how can i get the script to exit on any failing command? I think the key point is the '<' operator, which i do not understand exactly how it executes the script.
Notes:
-e means: Abort script at first error, when a command exits with non-zero status (except in until or while loops, if-tests, list constructs)
Possible solution:
docker exec -i fancy_container bash -c "cat > tmp.sh; bash tmp.sh" < script.sh
How it works:
< script.sh - Pipe all rows of this file from the host, to the docker exec command.
cat > tmp.sh - Save the incoming piped content to a file inside the container.
bash tmp.sh - Execute the file as-whole inside the container, which means -e works again as expected!
But i still don't know why the initial approach isn't working.
I have the following two files in a directory:
Dockerfile
FROM debian
WORKDIR /app
COPY start.sh /app/
CMD ["/app/start.sh"]
start.sh (with permissions 755 using chmod +x start.sh)
#!/bin/bash
trap "echo SIGINT; exit" SIGINT
trap "echo SIGTERM; exit" SIGTERM
echo Starting script
sleep 100000
I then run the following commands:
$ docker build . -t tmp
$ docker run --name tmp tmp
I then expect that pressing Ctrl+C would send a SIGINT to the program, which would print SIGINT to the screen then exit, but that doesn't happen.
I also try running $ docker stop tmp, which I expect would send a SIGTERM to the program, but checking $ docker logs tmp after shows that SIGTERM was not caught.
Why are SIGINT and SIGTERM not being caught by the bash script?
Actually, your Dockerfile and start.sh entrypoint script work as is for me with Ctrl+C, provided you run the container with one of the following commands:
docker run --name tmp -it tmp
docker run --rm -it tmp
Documentation details
As specified in docker run --help:
the --interactive = -i CLI flag asks to keep STDIN open even if not attached
(typically useful for an interactive shell, or when also passing the --detach = -d CLI flag)
the --tty = -t CLI flag asks to allocate a pseudo-TTY
(which notably forwards signals to the shell entrypoint, especially useful for your use case)
Related remarks
For completeness, note that there are several related issues that can make docker stop take too much time and "fall back" to docker kill, which can arise when the shell entrypoint starts some other process(es):
First, when the last line of the shell entrypoint runs another, main program, don't forget to prepend this line with the exec builtin:
exec prog arg1 arg2 ...
But when the shell entrypoint is intended to run for a long time, trapping signals (at least INT / TERM, but not KILL) is very important;
{see also this SO question: Docker Run Script to catch interruption signal}
Otherwise, if the signals are not forwarded to the children processes, we run the risk of hitting the "PID 1 zombie reaping problem", for instance
{see also this SO question for details: Speed up docker-compose shutdown}
CTRL+C sends a signal to docker running on that console.
To send a signal to the script you could use
docker exec -it <containerId> /bin/sh -c "pkill -INT -f 'start\.sh'"
Or include echo "my PID: $$" on your script and send
docker exec -it <containerId> /bin/sh -c "kill -INT <script pid>"
Some shell implementations in docker might ignore the signal.
This script will correctly react to pkill -15. Please note that signals are specified without the SIG prefix.
#!/bin/sh
trap "touch SIGINT.tmp; ls -l; exit" INT TERM
trap "echo 'really exiting'; exit" EXIT
echo Starting script
while true; do sleep 1; done
The long sleep command was replaced by an infinite loop of short ones since sleep may ignore some signals.
The solution I found was to just use the --init flag.
docker run --init [MORE OPTIONS] IMAGE [COMMAND] [ARG...]
Per their docs...
For our application running inside a container it is preferable that it receives a SIGTERM when the container is being (gracefully) shutdown. At the same time, we want it's output to go to a log file.
In the startscript of our docker container, we had therefore been using bash's exec similar to this
exec command someParam >> stdout.log
That worked just fine, command replaced the shell that had been the container's root process and would receive the SIGTERM.
Since the application tends to log a lot, we decided to add log rotation by using Apache's rotatelogs tool, i.e.
exec command | rotatelogs -n 10 stdout.log 10M
Alas, it seems that by using the pipe, exec can no longer have command replace the shell. When looking at the processes in the running container with pstree -p, it now looks like this
mycontainer#/#pstree -p
start.sh(1)-+-command(118)
`-rotatelogs(119)
So bash remains the root process, and does not pass the SIGTERM on to command.
Before stumbling upon exec, I had found an approach that installs a signal handler into the bash script, which would then itself send a SIGTERM to the command process using kill. However, this became really convoluted, getting the PID was also not always straightforward, and I would like to preserve the convenience of exec when it comes to signal handling and get piping for log rotation.
Any idea how to accomplish this?
Perhaps you want
exec sh -c 'command | rotatelogs -n 10 stdout.log 10M'
I was able to get around this by using process substitution. For your specific case the following may work.
exec command > >(rotatelogs -n 10 stdout.log 10M)
To reproduce the scenario I built this simple Dockerfile
FROM perl
SHELL ["/bin/bash", "-c"]
# The following will gracefully terminate upon docker stop
CMD exec perl -e '$SIG{TERM} = sub { $|++; print "Caught a sigterm!\n"; sleep(5); die "is the end!" }; sleep(30);' 2>&1 > >(tee /my_log)
# The following won't gracefully terminate upon docker stop
#CMD exec perl -e '$SIG{TERM} = sub { $|++; print "Caught a sigterm!\n"; sleep(5); die "is the end!" }; sleep(30);' 2>&1 | tee /my_log
Build docker build -f Dockerfile.meu -t test .
Run docker run --name test --rm -ti test
Stop it docker stop test
Output:
Caught a sigterm!
is the end! at -e line 1.
I have a docker container that runs a script via the entrypoint directive. The container closes after the entrypoint script is finished. I need to get the exit code from the script in order to do some logging if the script fails. Right now I'm thinking of something like this
docker run container/myContainer:latest
if [ $? != 0 ];
then
do some stuff
fi
Is this proper way to achieve this? Specifically, will this be the exit code of docker run or of my entrypoint script?
Yes, the docker container run exit code is the exit code from your entrypoint/cmd:
$ docker container run busybox /bin/sh -c "exit 5"
$ echo $?
5
You may also inspect the state of an exited container:
$ docker container inspect --format '{{.State.ExitCode}}' \
$(docker container ls -lq)
5
Checking the value of $? is not needed if you just want to act upon the exit status of the previous command.
if docker run container/myContainer:latest; then
do_stuff
fi
The above example will run/execute do_stuff if the exit status of docker run is zero which is a success.
You can add an else and elif clause in that
Or if you want to negate the exit status of the command.
if ! docker run container/myContainer:latest; then
do_stuff
fi
The above example will run do_stuff if the exit status of docker run is anything but zero, e.g. 1 and going up, since the ! negates.
If the command has some output and if does not have a silent/quite flag/option you can redirect it to /dev/null
if docker run container/myContainer:latest >/dev/null; then
do_stuff
fi
Should not output anything to stdout
see help test | grep -- '^[[:blank:]]*!'
In some cases if some output is still showing then that might be stderr which you can silent with >/dev/null 2>&1 instead of just >/dev/null
I have to run a command using exec in a shell script, and I need to trap the exit code in case of error and run another command e.g
#!/bin/sh
set +e
exec command_that_will_fail
if [ $? -eq 1 ]; then
echo "command failed, running another command"
fi
I understand that exec replaces the current shell and carries on, my problem is that I need to run another command if the exec is not sucessfull.
Your code works if there's some immediate error when it tries to run the process:
$ echo 1
1
$ echo $?
0
$ exec asd123
-bash: exec: asd123: not found
$ echo $?
127
If the executable file was found, and is started, then it will not return, because it will overtake the whole script and never return to bash again.
For example this never returns:
$ exec grep asd /dev/null
(the exit code of grep is 1, but the parent shell is overtaken, so nobody can check)
If you want to get an exit code from the process in this case, you have to start it as a subprocess, i.e. not using exec (just command_that_will_fail). In this case the bash process will act as a supervisor that waits until the subprocess finishes and can inspect the exit code.