It all started from this article about setting up nginx and certbot in docker. In the end of the manual the author made the automatic certificate renewal for nginx with this command:
command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
I'm not the only one who didn't understand this part, so there was a question on SO: Why do sleep & wait in bash?
The answer was given that the original command was not perfect and here is the corrected version:
/bin/sh -c 'nginx -g \"daemon off;\" & trap exit TERM; while :; do sleep 6h & wait $${!}; nginx -s reload; done'
But in this command I see nginx -g \"daemon off;\" &
Why do we first put nginx on foreground and then stuff it in background? What are implications and why not just launch nginx in background at first?
Another question: as I understand, the while cycle stays in foreground for docker, unlike the original command. But if nginx if background, does it mean that if it dies, docker does not care? In foreground while is still working, no problem.
And the last question: why in this commands sometimes we see $${!} and sometimes ${!}. Example of ${!} from the same SO question:
docker run --name test --rm --entrypoint="/bin/sh" nginx -c 'nginx -g "daemon off;" & trap exit TERM; while :; do sleep 20 & wait ${!}; echo running; done'
I know it's a character escaping, but I don't figure out the rules for this case.
But in this command I see nginx -g \"daemon off;\" & Why do we first put nginx on foreground and then stuff it in background? What are implications and why not just launch nginx in background at first?
The reason was mainly to highlight the differences and there are no implications. The command is equivalent to:
"/bin/sh -c 'nginx; trap exit TERM; while :; do sleep 6h & wait $${!}; nginx -s reload; done'
Another question: as I understand, the while cycle stays in foreground for docker, unlike the original command. But if nginx if background, does it mean that if it dies, docker does not care? In foreground while is still working, no problem.
The command basically creates three processes: the shell process (/bin/sh), sleep 6H and the nginx server. A fourth process (nginx -s reload) is forked every 6 hours.
Docker always monitors the process with PID 1 which in this case is the shell (/bin/sh). If the shell dies the container exits. If the nginx server, which is a child of the shell process, dies docker, indeed doesn't care.
The "corrected" version doesn't address these issues. It has the same problems as the original one. The answer to the SO question only highlights that the sleep and wait is not needed unless you want to handle signals in a timely manner. It means that:
"/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx ..."
does exactly the same thing as:
"/bin/sh -c 'while :; do sleep 6h; nginx ..."
In conclusion, a proper implementation would have nginx as the main process (PID 1) and another process running in background waking up every 6h to signal the server to reload the configuration. Neither the original, nor the corrected command implement all this properly.
To fix the before mentioned problems the command should be like this:
'while :; do sleep 6h; nginx -s reload; done & exec nginx -g "daemon off;"'
The exec system call replaces the content of the shell process with the nginx server making nginx the main process in foreground.
All the signals are now propagated correctly to the server (see also Controlling nginx).
Note: This solution still has a flaw. The shell process (the while loop) is not monitored. If for any reason this process exits the only thing docker does
is to send an alert.
Hope this sheds some light.
Answer to my last question regarding ${!} and $${!}:
Apparently, if we write a command in docker-compose file with one dollar sign (${!}), it will be expanded by docker-compose into 'pid of last background command relative to shell that launched docker-compose'. So, the entrypoint in container will look like this:
/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait someUnknownLastPID; done;'"
With $${!} the dollar sign is escaped in docker-compose processing. The entrypoint in container will be something like
/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait ${!}; done;'"
Source: https://stackoverflow.com/a/40621373/11931043
Related
I want a custom bash script in the container that is called automatically before the container stops (docker stop or ctrl + c).
According to this docker doc and multiple StackOverflow threads, I need to catch the SIGTERM signal in the container and then run my custom script when the event appears. As I know SIGTERM can be only used from a root process with PID 1.
Relevand part of my Dockerfile:
...
COPY container-scripts/entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
I use [] to define the entrypoint and as I know this will run my script directly, without having a /bin/sh -c wrapper (PID 1), and when the script eventually exec another process, that process becomes the main process and will receive the docker stop signal.
entrypoint.sh:
...
# run the external bash script if it exists
BOOT_SCRIPT="/boot.sh"
if [ -f "$BOOT_SCRIPT" ]; then
printf ">> executing the '%s' script\n" "$BOOT_SCRIPT"
source "$BOOT_SCRIPT"
fi
# start something here
...
The boot.sh is used by child containers to execute something else that the child container wants. Everything is fine, my containers work like a charm.
ps axu in a child container:
PID USER TIME COMMAND
1 root 0:00 {entrypoint.sh} /bin/bash /entrypoint.sh
134 root 0:25 /usr/lib/jvm/java-17-openjdk/bin/java -server -D...
...
421 root 0:00 ps axu
Before stopping the container I need to run some commands automatically so I created a shutdown.sh bash script. This script works fine and does what I need. But I execute the shutdown script manually this way:
$ docker exec -it my-container /bin/bash
# /shutdown.sh
# exit
$ docker container stop my-container
I would like to automate the execution of the shutdown.sh script.
I tried to add the following to the entrypoint.sh but it does not work:
trap "echo 'hello SIGTERM'; source /shutdown.sh; exit" SIGTERM
What is wrong with my code?
Your help and comments guided me in the right direction.
I went through again the official documentations here, here, and here and finally I found what was the problem.
The issue was the following:
My entrypoint.sh script, which kept alive the container executed the following command at the end:
# start the ssh server
ssh-keygen -A
/usr/sbin/sshd -D -e "$#"
The -D option runs the ssh daemon in a NOT detach mode and sshd does not become a daemon. Actually, that was my intention, this is the way how I kept alive the container.
But this foreground process prevented to be executed properly the trap command. I changed the way how I started the sshd app and now it runs as a normal background process.
Then, I added the following command to keep alive my docker container (this is a recommended best practice):
tail -f /dev/null
But of course, the same issue appeared. Tail runs as a foreground process and the trap command does not do its job.
The only way how I can keep alive the container and let the entrypoint.sh runs as a foreign process in docker is the following:
while true; do
sleep 1
done
This way the trap command works fine and my bash function that handles the SIGINT, etc. signals runs properly when the time comes.
But honestly, I do not like this solution. This endless loop with a sleep looks ugly, but I have no idea at the moment how to manage it in a nice way :(
But this is another question that not belongs to this thread (but could be great if you can suggest my a better solution).
I'm building some CI pipelines, and part of it is a bash wrapper script around a docker container running ansible commands. The trouble I'm having is that on job abort the container keeps running, which is potentially dangerous.
What I have currently is:
#!/bin/bash
CONTAINER=ansible
function kill_container() {
echo "$0 caught $1" >&2
docker kill ${CONTAINER}
exit $?
}
trap 'kill_container SIGINT' SIGINT
trap 'kill_container SIGTERM' SIGTERM
function ansible_base() {
docker run -d --rm --name ${CONTAINER} someorg/ansible:latest $#
docker logs --follow ${CONTAINER}
}
ansible_base $#
and my local test is simply ./run.sh sleep 30.
For the purpose of reproducability, you can substitute alpine:latest as the image and it behaves the same.
Prior to adding -d to the run and the docker logs it did not respect SIGINT at all, but now it works as expected. Eg:
./ci/run.sh sleep 30
5f5d78cfea27cdc15f5fede2003352253ae3254f44489ab4689ebca8d0f91768
^C./ci/run.sh caught SIGINT
ansible
However, if I run a pkill run.sh from another terminal it still waits the full 30 seconds before handling the signal, raising an error that the container is already gone. Eg:
./ci/run.sh sleep 30
a642a1060dc9d340e92dc255d68a9d9cb26d62ec59c5ef8d4e3d4198f1692c3e
./ci/run.sh caught SIGTERM
Error response from daemon: Cannot kill container: ansible: Container a642a1060dc9d340e92dc255d68a9d9cb26d62ec59c5ef8d4e3d4198f1692c3e is not running
Ultimately, the observed behaviour in the CI system is the same. The process is issued a SIGTERM, and then after not responding for 30 seconds a SIGKILL. This terminates the wrapper script, but not the docker command.
As #brunson said, I needed an init process to handle signal propagation.
When I was originally writing this my thought was "it's just a command, it doesn't need an initd" which was somewhat true until the very instant I needed it to respect signals at all. Frankly it was a foolish thought in the first place.
Anyhow, to accomplish the fix I used tini.
Added to Dockerfile:
RUN apk add --no-cache tini
ENTRYPOINT ["/sbin/tini", "--"]
and run.sh is back down to a much more manageable:
#!/bin/bash
function ansible_base() {
docker run --rm someorg/ansible:latest "$#"
}
ansible_base "$#"
Not pure bash question, but requires a combo knowledge of bash and kubernetes cli to fully answer. I want to do some port forwarding thing with kubernetes alongside with other job (like telepresense), and my bash script doing like that:
# Killing all kubectl port forwards that might remain after previous launch.
kill $(pidof kubectl)
#run kube-proxy to tunnel port 2828 to the my pod on k8s
kubectl port-forward deployment/my 2828:2828 -n myns &
#wait for proxy to establish
sleep 10
this script has downsides
killing all kubectl, not only mine
does not kill kubectl at the end of script (could it just handle ctrl+c and "join" process gracefully instead of killing it)?
may sleep longer than needed (can i track if kubectl server is established and script can continue?)
kubectl error is not handled (should exit if error happened)
how could I solve mentioned drawbacks?
You should track the pid of the created kubectl process and possible store it in a "lock" file.
Something like: pid=$! and echo $! > lockfile
Then in the beginning of the script you could check that lockfile and and kill the process:
pid=$(<lockfile)
kill "$pid"
kubectl port-forward deployment/my 2828:2828 -n myns &
pid=$!
echo "$pid" > lockfile
This way you can also check to see if the process is already running or it have stopped:
pid=$(<lockfile)
if ps "$pid" 2>&1 >/dev/null
then
echo "Already running, no need to restart"
exit 0
fi
kubectl port-forward deployment/my 2828:2828 -n myns &
pid=$!
echo "$pid" > lockfile
This will however not work if kubectl forks and stops the parent process.
Let's consider following script:
#!/bin/bash
while true ; do: ; done
After running the script, the bash goes into loop, but can be interrupted (by pressing Ctrl-C or issuing kill -2 command) or terminated (by issuing kill command). All works perfectly well. But now let's consider another script:
#!/bin/bash
sleep 60
After running this script, bash process no longer reacts to SIGINT or SIGTERM signals. Of course it reacts to pressing Ctrl-C or to killing sleep process, but what I am interested in, is making bash process itself react to these signals. I need this because I am building Docker image with bash script as an entrypoint and Docker sends signals to PID 1 in the containter, which in my case will be bash process. I am struggling with making the container shut down gracefully. Bash process does not react to signals, so Docker kills it in order to shut down the containter.
Any help will be appreciated.
Consider this Docker file:
from centos:7
COPY entrypoint.sh /usr/bin/entrypoint.sh
RUN chmod 760 /usr/bin/entrypoint.sh
ENTRYPOINT ["/usr/bin/entrypoint.sh"]
with the corresponding entrypoint.sh script
#!/usr/bin/env bash
function finish {
# stop holding process here
echo "exciting gracefully . . ."
kill -TERM "$child" 2>/dev/null
exit 0
}
trap finish SIGHUP SIGINT SIGQUIT SIGTERM
# your process which holds the container, eg
sleep 60 &
child=$!
wait "$child
Build the image:
docker build --no-cache -t overflow .
Run the image:
docker run overflow:latest
if you CTRL+C within 60 seconds you'll see the output:
exciting gracefully . . .
Showing the signal has first killed your script and then the container.
A good resource on signals and containers can be found here
if your docker API 1.25+ you can run container
docker run --init -it
--init - Run an init inside the container that forwards signals and reaps processes
description from docker guide https://docs.docker.com/engine/reference/commandline/run/
I'm using the simplest docker ever:
FROM ubuntu
COPY script.sh /script.sh
CMD /script.sh
Where all the script does is:
#!/bin/bash
function sigterm() {
echo "Got SIGTERM"
exit
}
trap sigterm SIGTERM
i=1
while true; do
echo "$(date +%H:%M:%S) | $((i++)) | $HOSTNAME"
sleep 1
done
I'm running this container in Minikube, but I can't get it to catch any SIGTERM from kubernetes.
I tried deleting the pod/deployment or to scale it up and down. In no case it got SIGTERM before being deleted. It respects the terminationGracePeriodSeconds But doesn't seem to run the preStop command or send the SIGTERM before killing the pod.
Is that due to using minikube? or am I doing something else wrong?
(the deployment is not part of a service, it's just a deployment)
(SSH into the pod and manually kill-ing it works as expected)
Reading the Dockerfile documentation on CMD
The CMD instruction has three forms:
CMD ["executable","param1","param2"] (exec form, this is the preferred form)
CMD ["param1","param2"] (as default parameters to ENTRYPOINT)
CMD command param1 param2 (shell form)
If you use the shell form of the CMD, then the will execute in /bin/sh -c
So you are using the shell form, which means your command is /bin/sh -c script.sh. Then, when kubernetes sends a SIGTERM to the container, is not the script.sh process the one receiving the signal, but the /bin/sh process. That's why you don't see the "Got SIGTERM" message.
When creating a Dockerfile, make sure you use the exec form. Otherwise the application will be started as a subcommand of /bin/sh -c, which does not pass signals. The container’s PID1 will be the shell, your application will not receive any signals.
Try changing your Dockerfile to use the exec form
FROM ubuntu
COPY script.sh /script.sh
CMD ["/script.sh"]