golang webapp init.d script hangs - go

I have a go web app compiled to a single binary that I am trying to manage via init.d. Here is my init.d script:
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/bin/my-go-app
DAEMON_ARGS="--logFile /var/log/my-go-app/my-go-app.log"
NAME=my-go-app
DESC=my-go-app
RUNDIR=/var/run/my-go-app
PIDFILE=$RUNDIR/my-go-app.pid
test -x $DAEMON || exit 0
if [ -r /etc/default/$NAME ]
then
. /etc/default/$NAME
fi
. /lib/lsb/init-functions
set -e
case "$1" in
start)
echo -n "Starting $DESC: "
mkdir -p $RUNDIR
touch $PIDFILE
chown ubuntu:ubuntu $RUNDIR $PIDFILE
chmod 755 $RUNDIR
if [ -n "$ULIMIT" ]
then
ulimit -n $ULIMIT
fi
if start-stop-daemon --start --quiet --umask 007 --pidfile $PIDFILE --chuid ubuntu:ubuntu --exec $DAEMON -- $DAEMON_ARGS
then
echo "$NAME."
else
echo "failed"
fi
;;
stop)
echo -n "Stopping $DESC: "
if start-stop-daemon --stop --retry forever/TERM/1 --quiet --oknodo --pidfile $PIDFILE --exec $DAEMON
then
echo "$NAME."
else
echo "failed"
fi
rm -f $PIDFILE
sleep 1
;;
restart|force-reload)
${0} stop
${0} start
;;
status)
echo -n "$DESC is "
if start-stop-daemon --stop --quiet --signal 0 --name ${NAME} --pidfile ${PIDFILE}
then
echo "running"
else
echo "not running"
exit 1
fi
;;
*)
echo "Usage: /etc/init.d/$NAME {start|stop|restart|force-reload|status}" >&2
exit 1
;;
esac
exit 0
The problem is that when I run service my-go-app start, it just hangs, like this:
ubuntu#ip-10-0-0-40:~$ service my-go-app start
Starting my-go-app:
and never returns. In this state, if I open a separate terminal, I can see that the app is running by checking the log file but there is nothing in /var/run/my-go-app/my-go-app.pid (though the pid file does get created).
Has anyone encountered (and hopefully resolved) this before? How can I run my go app as an init.d daemon?
EDIT:
I was able to get this to work by adding the "-b -m" command line flags to start-stop-daemonwhen starting the service. That line now looks like this:
start-stop-daemon -b -m --start --quiet --umask 007 --pidfile $PIDFILE --chuid ubuntu:ubuntu --exec $DAEMON -- $DAEMON_ARGS
My concern with this approach is the warning in the start-stop-daemon manpage:
-b, --background
Typically used with programs that don't detach on their own. This option will force start-stop-daemon to fork before starting the process, and
force it into the background. WARNING: start-stop-daemon cannot check the exit status if the process fails to execute for any reason. This is
a last resort, and is only meant for programs that either make no sense forking on their own, or where it's not feasible to add the code for
them to do this themselves.
This seems like a bad idea to me, because it sounds like SysV won't know if the process dies. Am I understanding this correctly? Has anyone else tried this approach?

If you are running a system with Upstart you can use this script:
start on runlevel [2345]
stop on runlevel [016] or unmounting-filesystem
# Replace {soft} and {hard} with the soft and hard resource limits you desire
#limit nofile {soft} {hard}
umask 007
setuid ubuntu
setgid ubuntu
exec /usr/bin/my-go-app --logFile /var/log/my-go-app/my-go-app.log
You can also add the following code to your daemon at a point where your application has been started and initialized correctly:
if ("yes" == os.Getenv("MYAPP_RAISESTOP")) {
p, err := os.FindProcess(os.Getpid())
p.Signal(syscall.SIGSTOP)
}
and the following to lines to the above upstart job:
env MYAPP_RAISESTOP="yes"
expect stop
I am sorry if the if () { } is not real Go syntax; I am a C programmer haha (although the stuff inside the () and {} is real, I did a little research :).
Doing this last bit ensures that Upstart will wait until your application is set up correctly before firing off the started event. If no other jobs are waiting for your app, then you do not really need that.

You will need the --background flag if you want to use SysVinit and start-stop-daemon with Go programs.
I suggest using something like Supervisor or Circus instead.
Edit:
This is not strictly true, if your Go program daemonizes its self, the --background flag can be excluded.

Related

Linux script run with run-this-one doesn't work with docker

I'm experiencing an issue in which I run a command in a cronjob and want to make sure that it's not already being executed. I achieve that running as run-one [command] (man-page).
If I want to cancel the already running command and force the new command to run, I run as run-this-one [command].
At least this is what I expected, but if the command runs a docker container, the other process seems to be terminated (but isn't), the terminal shows Terminated, but continues to show the command output that is running in the container (but the commands after the container ends running are not executed). In this case, the command that runs run-this-one is not executed (not expected).
Example:
/path/to/file.sh
#!/bin/bash
set -eou pipefail
echo "sleep started..." >&2
docker run --rm alpine /bin/sh -c 'echo "sleep started inside..." && sleep 5 && echo "sleep ended inside..."'
echo "sleep ended..." >&2
If I run in a terminal window sudo run-one /path/to/file.sh, and then run in another terminal (before the previous command ends running) the command sudo run-one /path/to/file.sh, this command is not executed, as expected, and that command ends succesfully.
Terminal1:
user#host:/path$ sudo run-one /path/to/file.sh
sleep started...
sleep started inside...
sleep ended inside...
sleep ended...
user#host:/path$
Terminal2:
user#host:/path$ sudo run-one /path/to/file.sh
user#host:/path$
But if I run in a terminal window sudo run-one /path/to/file.sh, and then run in another terminal (before the previous command ends running) the command sudo run-this-one /path/to/file.sh, this command is not executed, which is not expected, and that command shows in the terminal Terminated, with the terminal showing user#host:/path$, but the output in the container still shows (the command is still running in the container created in the 1st terminal).
Terminal1:
user#host:/path$ sudo run-one /path/to/file.sh
sleep started...
sleep started inside...
Terminated
user#host:/path$ sleep ended inside...
# terminal doesn't show new input from the keyboard, but I can run commands after
Terminal2:
user#host:/path$ sudo run-this-one /path/to/file.sh
user#host:/path$
It works if the file is changed to:
/path/to/file.sh
#!/bin/bash
set -eou pipefail
echo "sleep started..." >&2
sleep 5
echo "sleep ended..." >&2
The above script file with docker was just an example, in my case it's different, but the problem is the same, and occurs independently of running the container with or without -it.
Someone knows why this is happening? Is there a (not very complex and not very hackish) solution to this problem? I've executed the above commands in Ubuntu 20.04 inside a VirtualBox machine (with vagrant).
Update (2021-07-15)
Based on #ErikMD comment and #DannyB answer, I put a trap and a cleanup function to remove the container, as can be seen in the script below:
/path/to/test
#!/bin/bash
set -eou pipefail
trap 'echo "[error] ${BASH_SOURCE[0]}:$LINENO" >&2; exit 3;' ERR
RED='\033[0;31m'
NC='\033[0m' # No Color
function error {
msg="$(date '+%F %T') - ${BASH_SOURCE[0]}:${BASH_LINENO[0]}: ${*}"
>&2 echo -e "${RED}${msg}${NC}"
exit 2
}
file="${BASH_SOURCE[0]}"
command="${1:-}"
if [ -z "$command" ]; then
error "[error] no command entered"
fi
shift;
case "$command" in
"cmd1")
function cleanup {
echo "cleaning $command..."
sudo docker rm --force "test-container"
}
trap 'cleanup; exit 4;' ERR
args=( "$file" "cmd:unique" )
echo "$command: run-one ${args[*]}" >&2
run-one "${args[#]}"
;;
"cmd2")
function cleanup {
echo "cleaning $command..."
sudo docker rm --force "test-container"
}
trap 'cleanup; exit 4;' ERR
args=( "$file" "cmd:unique" )
echo "$command: run-this-one ${args[*]}" >&2
run-this-one "${args[#]}"
;;
"cmd:unique")
"$file" "cmd:container"
;;
"cmd:container")
echo "sleep started..." >&2
sudo docker run --rm --name "test-container" alpine \
/bin/sh -c 'echo "sleep started inside..." && sleep 5 && echo "sleep ended inside..."'
echo "sleep ended..." >&2
;;
*)
echo -e "${RED}[error] invalid command: $command${NC}"
exit 1
;;
esac
If I run /path/to/test cmd1 (run-one) and /path/to/test cmd2 (run-this-one) in another terminal, it works as expected (the cmd1 process is stopped and removes the container, and the cmd2 process runs successfully).
If I run /path/to/test cmd2 in 2 terminals, it also works as expected (the 1st cmd2 process is stopped and removes the container, and the 2nd cmd2 process runs successfully).
But not so good: in the 2 cases above, sometimes the 2nd process stops with an error before the 1st removes the container (this can occur intermittently, probably due to a race condition).
And it gets worse: if I run /path/to/test cmd1 in 2 terminals, both commands fail, although the 1st cmd1 should run successfully (it fails because the 2nd cmd1 removes the container in the cleanup).
I tried to put the cleanup in the cmd:unique command instead (removing from the other 2 places), so as to call only by the single process running, to avoid the problem above, but weirdly the cleanup is not called there, even if the trap is also defined there.
Just to simplify your question, I would use this command to reproduce the problem:
run-one docker run --rm -it alpine sleep 10
As can be seen - either with run-one and run-this-one - the behavior is definitely not the desired one.
Since the command creates a process managed by docker, I suspect that the run-one set of tools is not the right tool for the job, since docker containers should not be killed with pkill, but rather with docker kill.
One relatively easy solution, is to embrace the way docker wants you to kill containers, and create your short run-one scripts that handle docker properly.
run-one-docker.sh
#!/usr/bin/env bash
if [[ "$#" -lt 2 ]]; then
echo "Usage: ./run-one-docker.sh NAME COMMAND"
echo "Example: ./run-one-docker.sh temp alpine sleep 10"
exit 1
fi
name="$1"
command=("${#:2}")
container_is_running() {
[ "$( docker container inspect -f '{{.State.Running}}' "$1" 2> /dev/null)" == "true" ]
}
if container_is_running "$name"; then
echo "$name is already running, aborting"
exit 1
else
docker run --rm -it --name "$name" "${command[#]}"
fi
run-this-one-docker.sh
#!/usr/bin/env bash
if [[ "$#" -lt 2 ]]; then
echo "Usage: ./run-this-one-docker.sh NAME COMMAND"
echo "Example: ./run-this-one-docker.sh temp alpine sleep 10"
exit 1
fi
name="$1"
command=("${#:2}")
container_is_running() {
[ "$( docker container inspect -f '{{.State.Running}}' "$1" 2> /dev/null)" == "true" ]
}
if container_is_running "$name"; then
echo "killing old $name"
docker kill "$name" > /dev/null
fi
docker run --rm -it --name "$name" "${command[#]}"

What is the proper way to terminate a script using start-stop-daemon?

I am using a start-stop-daemon to make a INIT script for my script. I am using --make-pidfile cause my script doesn't make its own pid. I can start my script using start and pid file generates with appropriate PID. But the stop function doesn't work. I am getting return code 0 with --oknodo and 1 without it. If I do
ps -ef | grep perl
and
cat /home/me/mydaemon/run
I always see the same PID. I can terminate the script using
kill -15 PID.
But not with the stop function of my init script.
What is the proper way to stop my process?
As per start-stop-daemon manual,
--stop Checks for the existence of a specified process. If such a process exists, start-stop-daemon sends it the signal specified by
--signal, and exits with error status 0. If such a process does not exist, start-stop-daemon exits with error status 1 (0 if --oknodo
is specified). If --retry is specified, then start-stop-daemon
will check that the process(es) have terminated.
I didn't find find any documentation for --signal itself. Like how to specify --signal if I want to send a SIGTERM.
#!/bin/sh
### BEGIN INIT INFO
# Provides: myd
# Required-Start: $local_fs $network $named $time $syslog
# Required-Stop: $local_fs $network $named $time $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Description: Diitalk daemon for sending push notifications
### END INIT INFO
. /lib/lsb/init-functions
PATH=/sbin:/bin:/usr/sbin:/usr/bin
DAEMON="/home/me/mydaemon/myd"
NAME="myd"
DESC="My Daemon"
HOMEDIR=/home/me/mydaemon/run
PIDFILE="$HOMEDIR/$NAME.pid"
USER=me
GROUP=me
SHM_MEMORY=64
PKG_MEMORY=8
DUMP_CORE=no
case "$1" in
start|debug)
log_daemon_msg "Starting $DESC: $NAME"
start-stop-daemon --start --quiet --background --make-pidfile --pidfile $PIDFILE \
--exec $DAEMON || log_failure_msg " already running"
log_end_msg 0
;;
stop)
log_daemon_msg "Stopping $DESC: $NAME"
start-stop-daemon --oknodo --stop --quiet --pidfile $PIDFILE \
--exec $DAEMON
echo $?
log_end_msg 0
;;
The issue was with the --exec that I used for matching the process name. As per the start-stop-daemon documentation :
-x, --exec executable
Check for processes that are instances of this executable
(according to /proc/pid/exe).
In my case as my script is Perl script, /proc/pid/exe is symlinked to /usr/bin/perl; therefore the exec couldnt match the process name. I removed the exec so that it matches only the PID. Now I can properly stop my process.

init.d script for daemon - cannot remove pidfile

Have problems removing a pidfile when stopping a daemon on ubuntu (14.04).
I start the daemon with:
log_daemon_msg "Starting $DAEMON_NAME"
start-stop-daemon --start --background --pidfile $PIDFILE --make-pid --user $DAEMON_USER --chuid $DAEMON_USER --startas $DAEMON
log_end_msg $?
Note the use of --make-pid which I need or no pidfile is created. To stop I have:
log_daemon_msg "Stopping $DAEMON_NAME"
start-stop-daemon --stop --pidfile $PIDFILE --retry 10
log_end_msg $?
rm $PIDFILE
I would like to use the flag --remove-pidfile rather than the rm (as I am trying to make this as generic (cross-distro)) as possible but it results in an error.
Checking the man page (http://manpages.ubuntu.com/manpages/karmic/man8/start-stop-daemon.8.html) there appears to be no --remove-flag for ubuntu although it is mentioned for other distros.
Does anyone know any flags that would do this or should I just stick with rm?
If the version of start-stop-daemon on ubutu doesn't have that flag then I think what you are discovering is that (in the interest of making this generic) you should avoid it and stick to doing it yourself. – Etan Reisner

rc.d start does not terminate?

So I wrote the Arch Linux rc.d script for mongod daemon (following an example), but when I do:
sudo rc.d start mongod
it just gets stuck on:
:: Starting /usr/bin/mongod [BUSY]
and never transitions to "DONE" phase. Any tips?
Here is my script:
#!/bin/bash
# import predefined functions
. /etc/rc.conf
. /etc/rc.d/functions
# Point to the binary
DAEMON=/usr/bin/mongod
# Get the ARGS from the conf
. /etc/conf.d/crond
# Function to get the process id
PID=$(get_pid $DAEMON)
case "$1" in
start)
stat_busy "Starting $DAEMON"
# Check the PID exists - and if it does (returns 0) - do no run
[ -z "$PID" ] && $DAEMON $ARGS &> /dev/null
if [ $? = 0 ]; then
add_daemon $DAEMON
stat_done
else
stat_fail
exit 1
fi
;;
stop)
stat_busy "Stopping $DAEMON"
kill -HUP $PID &>/dev/null
rm_daemon $DAEMON
stat_done
;;
restart)
$0 stop
sleep 1
$0 start
;;
*)
echo "usage: $0 {start|stop|restart}"
esac
I've looked at how apache does it, but I can't figure out what they are doing that's different. Here's a piece of their httpd script:
case "$1" in
start)
stat_busy "Starting Apache Web Server"
[ ! -d /var/run/httpd ] && install -d /var/run/httpd
if $APACHECTL start >/dev/null ; then
add_daemon $daemon_name
stat_done
else
stat_fail
exit 1
fi
;;
For one thing, you are passing an $ARGS variable that is never actually defined. You will probably want to either pass some configuration options, or the location of a mongodb.conf file using the -f or --config option, to inform the daemon of the location of your database, log file, IP bindings, etc.
The mongod defaults assume that you database location is /data/db/. If this does not exist, or the daemon does not have permissions to that location, then the init script will fail.
You should probably also run the daemon with a user account other than yourself or root (the default pacman package creates a user named mongodb), and give this user read/write access to the data path and log file.
[ -z "$PID" ] && /bin/su mongodb -c "/usr/bin/mongod --config /etc/mongodb.conf --fork" > /dev/null
I would suggest referring to the mongodb init script provided in the Arch Community package, and comparing that to what you have here. Or, install MongoDB using pacman, which sets all of this up for you.
If all else fails, add some 'echo' commands inside of your if and else blocks to track down exactly where the init script is hanging, check mongodb's logs, and report back to us.

start-stop-daemon works at command line but doesn't work in /etc/init.d script

I'm trying to get a starter script (for a ruby gem called ar_sendmail) working in /etc/init.d/ar_sendmail:
#! /bin/sh
echo "in /etc/init.d/ar_sendmail"
DIR=/home/max/work/e_learning_resource/trunk
PATH=/var/lib/gems/1.8/bin
DAEMON=/var/lib/gems/1.8/bin/ar_sendmail
DAEMON_OPTS="-e production -d --batch-size 100 --delay 150"
NAME=ar_sendmail
DESC=ar_sendmail
PID_FILE=/home/max/work/e_learning_resource/trunk/shared/log/ar_sendmail.pid
test -x $DAEMON || exit 0
set -e
case "$1" in
start)
echo -n "Starting $DESC: "
start-stop-daemon -d $DIR --start --quiet --pidfile $PID_FILE \
--exec $DAEMON -- $DAEMON_OPTS
echo "$NAME."
;;
stop)
echo -n "Stopping $DESC: "
kill -TERM `cat $PID_FILE`
rm $PID_FILE
echo "$NAME."
;;
restart)
echo -n "Restarting $DESC: "
kill -TERM `cat $PID_FILE`
rm $PID_FILE
sleep 1
start-stop-daemon -d $DIR --start --quiet --pidfile \
$PID_FILE --exec $DAEMON -- $DAEMON_OPTS
echo "$NAME."
;;
*)
N=/etc/init.d/$NAME
echo "Usage: $N {start|stop|restart|reload}" >&2
exit 1
;;
esac
exit 0
It's blowing up on the start-stop-daemon line, saying "start-stop-daemon: not found". But, when i plug the values into that line manually, and run it on the command line, it works.
My first thought was it was the shebang line but #! /bin/sh should be right shouldn't it? It's definitely the right folder and what i use in my other /etc/init.d scripts.
My second thought was that it's sudo related: i'd been testing start-stop-daemon in non-sudo and running /etc/init.d/ar_sendmail in sudo mode. But, i can run start-stop-daemon fine with sudo as well.
Kind of stumped, any ideas?
As #Dysaster points out, you're overwriting your PATH with this line:
PATH=/var/lib/gems/1.8/bin
Because you're giving the complete pathname for your daemon, I think you probably don't even need to add /var/lib/gems/1.8/bin to your path, unless ar_sendmail needs to execute programs in that directory without knowing their path. (That would sure be unfortunate, but easily fixed with: PATH=/var/lib/gems/1.8/bin:$PATH.)
Add a source /etc/profile to the start of the script, so you get your path setup.

Resources