I'm trying to get Upstart sending me e-mails when a process is respawned.
So, following upstart stanzas, here's my upstart script for ntpd service (just as an example):
/etc/init/ntpd.conf
### ntpd
script
mail -s "ntpd Service Respawned" my_email#gmail.com
control + D
end script
respawn
exec /etc/init.d/ntpd start
Then, I reload the process (initctl reload ntpd) in order to get upstart to reload ntpd.conf's config. Then kill -9 the process to force its respawn.
Here's /var/log/message.log:
init: ntpd main process (12446) killed by KILL signal
init: ntpd main process ended, respawning
And the e-mail is never sent. I've tried with post-start and exec but it doesn't work either.
Any advice?
echo "ntpd Service Respawned" | mail -s "ntpd Service Respawned" my_email#gmail.com
Try with this.
Just solved this one.
What I did was add the following in my Upstart script:
respawn
pre-start script
mail -s "ntpd Service Respawned" my_address#gmail.com
control + D
end script
exec /etc/init.d/ntpd start
That works like a charm.
I think Upstart does pay much attention to the statements order.
Thanks!!!
Related
I want a custom bash script in the container that is called automatically before the container stops (docker stop or ctrl + c).
According to this docker doc and multiple StackOverflow threads, I need to catch the SIGTERM signal in the container and then run my custom script when the event appears. As I know SIGTERM can be only used from a root process with PID 1.
Relevand part of my Dockerfile:
...
COPY container-scripts/entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
I use [] to define the entrypoint and as I know this will run my script directly, without having a /bin/sh -c wrapper (PID 1), and when the script eventually exec another process, that process becomes the main process and will receive the docker stop signal.
entrypoint.sh:
...
# run the external bash script if it exists
BOOT_SCRIPT="/boot.sh"
if [ -f "$BOOT_SCRIPT" ]; then
printf ">> executing the '%s' script\n" "$BOOT_SCRIPT"
source "$BOOT_SCRIPT"
fi
# start something here
...
The boot.sh is used by child containers to execute something else that the child container wants. Everything is fine, my containers work like a charm.
ps axu in a child container:
PID USER TIME COMMAND
1 root 0:00 {entrypoint.sh} /bin/bash /entrypoint.sh
134 root 0:25 /usr/lib/jvm/java-17-openjdk/bin/java -server -D...
...
421 root 0:00 ps axu
Before stopping the container I need to run some commands automatically so I created a shutdown.sh bash script. This script works fine and does what I need. But I execute the shutdown script manually this way:
$ docker exec -it my-container /bin/bash
# /shutdown.sh
# exit
$ docker container stop my-container
I would like to automate the execution of the shutdown.sh script.
I tried to add the following to the entrypoint.sh but it does not work:
trap "echo 'hello SIGTERM'; source /shutdown.sh; exit" SIGTERM
What is wrong with my code?
Your help and comments guided me in the right direction.
I went through again the official documentations here, here, and here and finally I found what was the problem.
The issue was the following:
My entrypoint.sh script, which kept alive the container executed the following command at the end:
# start the ssh server
ssh-keygen -A
/usr/sbin/sshd -D -e "$#"
The -D option runs the ssh daemon in a NOT detach mode and sshd does not become a daemon. Actually, that was my intention, this is the way how I kept alive the container.
But this foreground process prevented to be executed properly the trap command. I changed the way how I started the sshd app and now it runs as a normal background process.
Then, I added the following command to keep alive my docker container (this is a recommended best practice):
tail -f /dev/null
But of course, the same issue appeared. Tail runs as a foreground process and the trap command does not do its job.
The only way how I can keep alive the container and let the entrypoint.sh runs as a foreign process in docker is the following:
while true; do
sleep 1
done
This way the trap command works fine and my bash function that handles the SIGINT, etc. signals runs properly when the time comes.
But honestly, I do not like this solution. This endless loop with a sleep looks ugly, but I have no idea at the moment how to manage it in a nice way :(
But this is another question that not belongs to this thread (but could be great if you can suggest my a better solution).
I need to call systemctl start myservice near the end of a Bash script, but I really don't care about whether it will be successful or when it intends to return. I just need to start the action. It's others' task to monitor the status of that service. My script should return as quickly as possible, no matter whether that service has completed starting, I'm not depending on that.
My first thought was to use something like this:
# do other work
systemctl start myservice &
echo "done"
# end of script
But I've read that this is problematic with signals or in non-interactive environments, where my script is usually called. So I read on and found the nohup command, but that seems to write output files anywhere and might hang if you don't redirect stdin from /dev/null, they say.
So I still don't know how to do this correctly. I'm open for a generic way to start-and-forget any process from a Bash script, or for systemctl specifically as this will be my only use case for now.
I found a pretty easy solution to this:
systemctl start --no-block myservice
The --no-block option can be used for starting, stopping etc. and it won't wait for the actual process to finish. More details in the manpage of systemctl.
If you simply want to start systemctl and you don't want to wait for it, use exec to replace the current process with the systemctl call. For example, instead of backgrounding the process, simply use:
exec systemctl ....
You may want to include the --no-pager option to ensure that the process isn't piped to a pager which would block waiting for user input, e.g.
exec systemctl --no-pager ....
Of course your echo "done" will never be reached, but that wasn't pertinent to your script.
I want to start a long-running python process when I log-in to a server.
So I put something like this into ~/.bashrc
python long_running_process.py
This correctly starts the python process on login. The problem here is that I'm not able to suspend it later (with ctrl-z). I can only exit the process by presssing ctrl-c.
I'd like to be able to pause/suspend and resume the python process if possible. This is different from "backgrounding" the process which has been asked and answered here
EDIT: I found a hack that kindof works. I create a dummy shell-script called ~/run.sh and make it executable. Then I put
#!/bin/bash
set -m
python long_running_process.py
in ~/run.sh.
Lastly, I call ~/run.sh in ~/.bashrc instead of starting the python process directly. Giving set -m enables job-control in the script (which is disabled by default for shell-scripts it seems)
This is very hacky. If someone has a nicer/cleaner solution, I'd be very interested in hearing it
You can use POSIX signals, sent with the kill command. I'll run this little Python script that prints its process id and the time once per second, then pause it then resume it:
#!/usr/bin/env python3
import os
import time
while True:
print(f"Process id: {os.getpid()}, time: {int(time.time())}")
time.sleep(1)
It started with process id 63070, so I can pause it with:
kill -STOP 63070
and then resume it with:
kill -CONT 63070
You can stop the process with command kill -SIGSTOP pid where pid is the process id (you can find it with command ps -ef | grep process_name ) and then continue the process with signal SIGCONT.
I have a unix script that invokes another script on a remote unix server.
amongst other commands i am stopping a service. The stop command essentially translates to
ssh -t -t -q ${AEM_USER}#${SERVERIP} 'bash -l -c "service aem stop"'
The service is getting stopped but when i start back the service it just creates the .pid file and does not perform the start up. When i run the command for start i.e.
ssh -t -t -q ${AEM_USER}#${SERVERIP} 'bash -l -c "service aem start"'
it does not show any error. On going to the server and checking the status
service aemauthor status
Below message is displayed
aem dead but pid file exists
Also when starting the service by logging in to the server, it works as expected along with the message
Removing stale pidfile (pid: 8701)
Starting aem
We don't know the details of the service script of aem.
I guess the problem is related to the SIGHUP signal. When we log off from a shell or disconnect from ssh, the OS will send HUP signal to all processes that started in this terminated shell. If the process didn't handle the HUP signal, it would exit by default.
When we run a command via ssh remotely, the process started by this command will receive HUP signal after ssh session is terminated.
We can use the nohup command to ignore the HUP signal.
You can try
ssh -t -t -q ${AEM_USER}#${SERVERIP} 'bash -l -c "nohup service aem start"'
If it works, you can use nohup command to start aem in the service script.
As mentioned at the stale pidfile syndrome, there are different reasons for pidfiles getting stalled, like for instance some issues with the way your handles its removal when the process exits... but considering your only experiencing when running remotely, I would guess it might be related to what is being loaded or not by your profile... check the most voted solid answer at the post below for some insights:
Why Does SSH Remote Command Get Fewer Environment Variables
As described in the comments of the mentioned post, you can try sourcing /etc/profile or ~/.bash_profile before executing your script to test it, or even trying to execute env locally and remotelly to compare variables that are being sourced or not.
I want to set my screen as screensave status every 50minutes (3000 seconds).
cat /home/rest.sh
while true;do
sleep 3000
xscreensaver-command --lock 1>/dev/null
done
sh /home/rest.sh & can make it run.
Now i want to set it as a daemon.
sudo vim /etc/systemd/system/screensave.service
[Unit]
Description=screensave
[Service]
User=root
ExecStart=/bin/bash /home/rest.sh
StandardError=journal
[Install]
WantedBy=multi-user.target
To set it and enable as daemon.
systemctl enable screensave.service
I find that the service is not running as a daemon.
sudo journalctl -u screensave
Jan 24 12:16:50 user systemd[1]: Started screensave.
Jan 24 12:17:22 user bash[621]: xscreensaver-command: warning: $DISPLAY is not set: defaulting to ":0.0".
Jan 24 12:17:22 user bash[621]: No protocol specified
Jan 24 12:17:22 user bash[621]: xscreensaver-command: can't open display :0.0
How to run it as a daemon after $DISPLAY is set ?
This is a very common FAQ. A system daemon cannot easily connect to the X session of any individual user. On a multi-user system, how do you tell which user's session to connect to, anyway? On a single-user system, what should the daemon do if no session is running (as it often isn't at the time the daemon starts up)?
Trying to run a system daemon as any particular user won't work, and giving individual users access to a system daemon is a recipe for security problems. It can be done, but the solution is complex, and probably not something you want to attempt on your own. (Briefly, have the daemon listen to commands on a socket; create a user-space program which knows how to talk to the socket, and build some sort of authorization and authentication so the daemon knows whom it's talking to and can verify that this user is allowed to connect to this display.)
The drop-dead simple solution is to run this from your desktop environment's startup scripts instead. Most desktops have something like "session start-up items" or "autorun on login" hooks.
I'm not running linux and can't check now but the steps to daemonize a process are to close stdin stdout stderr change current working directory to / and to fork twice and setsid so that current process is a new session leader.
adding something like this at the beginning, before running, first thing to check is exec command creates a new session leader process with ps -Cbash -o sid,pgid,pid,ppid,comm,args
# checking if current process is a session leader to avoid infinite call
if [[ $(ps -p $$ -osid=) != $$ ]]; then
( cd / ; exec setsid /bin/bash /home/rest.sh & ) </dev/null 1>&0 2>&0 &
exit
fi