Linux script run with run-this-one doesn't work with docker - bash

Question

I'm experiencing an issue in which I run a command in a cronjob and want to make sure that it's not already being executed. I achieve that running as run-one [command] (man-page).
If I want to cancel the already running command and force the new command to run, I run as run-this-one [command].
At least this is what I expected, but if the command runs a docker container, the other process seems to be terminated (but isn't), the terminal shows Terminated, but continues to show the command output that is running in the container (but the commands after the container ends running are not executed). In this case, the command that runs run-this-one is not executed (not expected).
Example:
/path/to/file.sh
#!/bin/bash
set -eou pipefail
echo "sleep started..." >&2
docker run --rm alpine /bin/sh -c 'echo "sleep started inside..." && sleep 5 && echo "sleep ended inside..."'
echo "sleep ended..." >&2
If I run in a terminal window sudo run-one /path/to/file.sh, and then run in another terminal (before the previous command ends running) the command sudo run-one /path/to/file.sh, this command is not executed, as expected, and that command ends succesfully.
Terminal1:
user#host:/path$ sudo run-one /path/to/file.sh
sleep started...
sleep started inside...
sleep ended inside...
sleep ended...
user#host:/path$
Terminal2:
user#host:/path$ sudo run-one /path/to/file.sh
user#host:/path$
But if I run in a terminal window sudo run-one /path/to/file.sh, and then run in another terminal (before the previous command ends running) the command sudo run-this-one /path/to/file.sh, this command is not executed, which is not expected, and that command shows in the terminal Terminated, with the terminal showing user#host:/path$, but the output in the container still shows (the command is still running in the container created in the 1st terminal).
Terminal1:
user#host:/path$ sudo run-one /path/to/file.sh
sleep started...
sleep started inside...
Terminated
user#host:/path$ sleep ended inside...
# terminal doesn't show new input from the keyboard, but I can run commands after
Terminal2:
user#host:/path$ sudo run-this-one /path/to/file.sh
user#host:/path$
It works if the file is changed to:
/path/to/file.sh
#!/bin/bash
set -eou pipefail
echo "sleep started..." >&2
sleep 5
echo "sleep ended..." >&2
The above script file with docker was just an example, in my case it's different, but the problem is the same, and occurs independently of running the container with or without -it.
Someone knows why this is happening? Is there a (not very complex and not very hackish) solution to this problem? I've executed the above commands in Ubuntu 20.04 inside a VirtualBox machine (with vagrant).
Update (2021-07-15)
Based on #ErikMD comment and #DannyB answer, I put a trap and a cleanup function to remove the container, as can be seen in the script below:
/path/to/test
#!/bin/bash
set -eou pipefail
trap 'echo "[error] ${BASH_SOURCE[0]}:$LINENO" >&2; exit 3;' ERR
RED='\033[0;31m'
NC='\033[0m' # No Color
function error {
msg="$(date '+%F %T') - ${BASH_SOURCE[0]}:${BASH_LINENO[0]}: ${*}"
>&2 echo -e "${RED}${msg}${NC}"
exit 2
}
file="${BASH_SOURCE[0]}"
command="${1:-}"
if [ -z "$command" ]; then
error "[error] no command entered"
fi
shift;
case "$command" in
"cmd1")
function cleanup {
echo "cleaning $command..."
sudo docker rm --force "test-container"
}
trap 'cleanup; exit 4;' ERR
args=( "$file" "cmd:unique" )
echo "$command: run-one ${args[*]}" >&2
run-one "${args[#]}"
;;
"cmd2")
function cleanup {
echo "cleaning $command..."
sudo docker rm --force "test-container"
}
trap 'cleanup; exit 4;' ERR
args=( "$file" "cmd:unique" )
echo "$command: run-this-one ${args[*]}" >&2
run-this-one "${args[#]}"
;;
"cmd:unique")
"$file" "cmd:container"
;;
"cmd:container")
echo "sleep started..." >&2
sudo docker run --rm --name "test-container" alpine \
/bin/sh -c 'echo "sleep started inside..." && sleep 5 && echo "sleep ended inside..."'
echo "sleep ended..." >&2
;;
*)
echo -e "${RED}[error] invalid command: $command${NC}"
exit 1
;;
esac
If I run /path/to/test cmd1 (run-one) and /path/to/test cmd2 (run-this-one) in another terminal, it works as expected (the cmd1 process is stopped and removes the container, and the cmd2 process runs successfully).
If I run /path/to/test cmd2 in 2 terminals, it also works as expected (the 1st cmd2 process is stopped and removes the container, and the 2nd cmd2 process runs successfully).
But not so good: in the 2 cases above, sometimes the 2nd process stops with an error before the 1st removes the container (this can occur intermittently, probably due to a race condition).
And it gets worse: if I run /path/to/test cmd1 in 2 terminals, both commands fail, although the 1st cmd1 should run successfully (it fails because the 2nd cmd1 removes the container in the cleanup).
I tried to put the cleanup in the cmd:unique command instead (removing from the other 2 places), so as to call only by the single process running, to avoid the problem above, but weirdly the cleanup is not called there, even if the trap is also defined there.

Answer 1

Just to simplify your question, I would use this command to reproduce the problem:
run-one docker run --rm -it alpine sleep 10
As can be seen - either with run-one and run-this-one - the behavior is definitely not the desired one.
Since the command creates a process managed by docker, I suspect that the run-one set of tools is not the right tool for the job, since docker containers should not be killed with pkill, but rather with docker kill.
One relatively easy solution, is to embrace the way docker wants you to kill containers, and create your short run-one scripts that handle docker properly.
run-one-docker.sh
#!/usr/bin/env bash
if [[ "$#" -lt 2 ]]; then
echo "Usage: ./run-one-docker.sh NAME COMMAND"
echo "Example: ./run-one-docker.sh temp alpine sleep 10"
exit 1
fi
name="$1"
command=("${#:2}")
container_is_running() {
[ "$( docker container inspect -f '{{.State.Running}}' "$1" 2> /dev/null)" == "true" ]
}
if container_is_running "$name"; then
echo "$name is already running, aborting"
exit 1
else
docker run --rm -it --name "$name" "${command[#]}"
fi
run-this-one-docker.sh
#!/usr/bin/env bash
if [[ "$#" -lt 2 ]]; then
echo "Usage: ./run-this-one-docker.sh NAME COMMAND"
echo "Example: ./run-this-one-docker.sh temp alpine sleep 10"
exit 1
fi
name="$1"
command=("${#:2}")
container_is_running() {
[ "$( docker container inspect -f '{{.State.Running}}' "$1" 2> /dev/null)" == "true" ]
}
if container_is_running "$name"; then
echo "killing old $name"
docker kill "$name" > /dev/null
fi
docker run --rm -it --name "$name" "${command[#]}"