How to build a multi instance systemd service configuration? - systemd

I am trying to configure systemd to be able to execute multiple instances of the same service but it seems that I am doing something wrong and the documentation resources seem not to be quite so clear.
Created /lib/systemd/system/confluence#.service file with this content:
[Unit]
Description=Confluence %i
After=postgresql.service nginx.service
[Service]
Type=forking
ExecStart=/opt/atlassian/confluence-%i/bin/start-confluence.sh
ExecStartPre=/opt/atlassian/confluence-%i/bin/setenv.sh prestart
ExecStop=/opt/atlassian/confluence-%i/bin/stop-confluence.sh
TimeoutStopSec=5min
PIDFile=/opt/atlassian/confluence-%i/work/catalina.pid
[Install]
WantedBy=multi-user.target
So far, so good, the systemctl enable confluence.test reported success (and yes the /opt/atlassian/confluence-test/ "happens" to contain the what it needs.
Still, when I try to start the service using systemctl start confluence I get:
root#atlas:/lib/systemd/system# systemctl start confluence#test.service
Job for confluence#test.service failed. See "systemctl status confluence#test.service" and "journalctl -xe" for details.
root#atlas:/lib/systemd/system# systemctl status confluence#test.service
● confluence#test.service - Confluence test
Loaded: loaded (/lib/systemd/system/confluence#.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2015-10-09 13:25:28 BST; 7s ago
Process: 16352 ExecStartPre=/opt/atlassian/confluence-%i/bin/setenv.sh prestart (code=exited, status=203/EXEC)
Oct 09 13:25:28 atlas systemd[1]: Starting Confluence test...
Oct 09 13:25:28 atlas systemd[1]: confluence#test.service: control process exited, code=exited status=203
Oct 09 13:25:28 atlas systemd[1]: Failed to start Confluence test.
Oct 09 13:25:28 atlas systemd[1]: Unit confluence#test.service entered failed state.
Oct 09 13:25:28 atlas systemd[1]: confluence#test.service failed.
Somehow it seems that systemd does not expand the "%i" which is supposed to be the instance name.

I have been bouncing all around the Web looking for the pieces I need to create my own set of systemd unit, template and target files. This question was in the pile of search results. In this case, rather than finding help, I think I can give help.
I don't have Confluence, and won't install it to test, so I am mostly guessing for the first part.
I believe the problem is that the %i specifier is escaped and could be creating bad paths in the commands. If that is the whole problem, changing to the unescaped version, %I, would be the simplest solution. Your file /lib/systemd/system/confluence#.service would then become:
[Unit]
Description=Confluence %I
After=postgresql.service nginx.service
[Service]
Type=forking
ExecStart=/opt/atlassian/confluence-%I/bin/start-confluence.sh
ExecStartPre=/opt/atlassian/confluence-%I/bin/setenv.sh prestart
ExecStop=/opt/atlassian/confluence-%I/bin/stop-confluence.sh
TimeoutStopSec=5min
PIDFile=/opt/atlassian/confluence-%I/work/catalina.pid
[Install]
WantedBy=multi-user.target
If, on the other hand, the problem has a different root, such as specifiers not expanding in the commands, you can still accomplish the same results using a wrapper script. systemd is supposed to eliminate the need for them, but using older systems in newer processes often results in the theory not matching practice.
Create a shell script, perhaps in the directory that is the base for everything else anyway.
Create /opt/atlassian/conflencectl with this:
#!/usr/bin/env bash
# Check that there is a command to execute
if ( ! test -n "$1" ) {
exit 1
}
# Check that there is an instance name to use
if ( ! test -n "$2" ) {
exit 1
}
# Make the directory name stub for this instance
InstanceDirName="confluence-$2"
# Execute the proper command, based on the command from the unit file
case $1 in
Start)
/opt/atlassian/$InstanceDirName/bin/start-confluence.sh;
;;
StartPre )
/opt/atlassian/$InstanceDirName/bin/setenv.sh prestart;
;;
Stop )
/opt/atlassian/$InstanceDirName/bin/stop-confluence.sh;
;;
*)
exit 1;
esac
Depending on what is, or is not, needed for the operations, you could add additional safety, sanity, or dependency checks, such as testing for the existence of the directory or needed config files.
Your systemd unit file then becomes:
[Unit]
Description=Confluence %I
After=postgresql.service nginx.service
[Service]
Type=forking
ExecStart=/opt/atlassian/conflencectl "Start" "%i"
ExecStartPre=/opt/atlassian/conflencectl "StartPre" "%i"
ExecStop=/opt/atlassian/conflencectl "Stop" "%i"
TimeoutStopSec=5min
PIDFile=/opt/atlassian/confluence-%I/work/catalina.pid
[Install]
WantedBy=multi-user.target
Even with this version I'm not too sure what will happen with the PIDFile declaration, nor how to compensate, if needed, using the wrapper script. The documentation I've scanned seems to imply that systemd is pretty good at knowing the PID anyway, and it may not be needed. Only testing will prove the final effects, however.

You can't have a separate start/stop script for each instance.
I would suggest ExecStart=/opt/atlassian/confluence/bin/start-confluence.sh
This will start a new instance every-time it is called.
Same goes for ExecStartPre and ExecStop

Related

Systemd run script after wakeup: "Can't open display"

I have a script which does things with screen brightness, works fine that's cool and now I want to make it run after wake up from suspend.
So I tried using systemd, I have a file under /etc/systemd/system/myscript.service which is as follows:
[Unit]
Description=Run myscript after wakeup
After=suspend.target hibernate.target hybrid-sleep.target suspend-then-hibernate.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/myscript
User=me
#Environment=DISPLAY=:0
[Install]
WantedBy=suspend.target hibernate.target hybrid-sleep.target suspend-then-hibernate.target
Note: User is set because the myscript needs HOME variable.
After I run sudo systemctl enable myscript and try suspend/wakeup, myscript is not run and journalctl -u myscript.service outputs the following message:
Jan 25 13:42:53 mymachine myscript[24489]: Can't open display
Jan 25 13:42:53 mymachine systemd[1]: myscript.service: Succeeded.
Jan 25 13:42:53 mymachine systemd[1]: Finished Run myscript after wakeup.
If I uncomment the line #Environment=DISPLAY=:0 in myscript.service the error is "Can't open display :0"
Any help would be great :^)
This worked on my Arch system. I tested a script in that location with xbacklight going up and down by 75% a few times after a resume from hibernate or suspend (systemctl hibernate / suspend).
I can only think that you do not have the DISPLAY=:0 in your environment (verify with env) for the user you are running the script as.
I was having a similar problem. Fixed it by adding the following to my systemd service:
Environment="DISPLAY=<DISP>"
Environment="XAUTHORITY=/path/to/xauthority"
Replace <DISP> with the value of your $DISPLAY variable, this is usually :0.

systemctl service systemd-notify not working with non-root user

I have a simple example of a service unit and bash script on Red Hat Enterprise Linux 7 using Type=notify that I am trying to get working.
When the service unit is configured to start the script as root, things work as expected. When adding User=testuser it fails. While the script initially starts (as seen on process list) the systemctl service never receives the notify message indicating ready so it hangs and eventually times out.
[Unit]
Description=My Test
[Service]
Type=notify
User=testuser
ExecStart=/home/iatf/test.sh
[Install]
WantedBy=multi-user.target
Test.sh (owned by testuser with execute permission)
#!/bin/bash
systemd-notify --status="Starting..."
sleep 5
systemd-notify --ready --status="Started"
while [ 1 ] ; do
systemd-notify --status="Processing..."
sleep 3
systemd-notify --status="Waiting..."
sleep 3
done
When run as root systemctl status test displays the correct status and status messages as sent from my test.sh bash script. When User=testuser the service hangs and then timesout and journalctl -xe reports:
Jul 15 13:37:25 tstcs03.ingdev systemd[1]: Cannot find unit for notify message of PID 7193.
Jul 15 13:37:28 tstcs03.ingdev systemd[1]: Cannot find unit for notify message of PID 7290.
Jul 15 13:37:31 tstcs03.ingdev systemd[1]: Cannot find unit for notify message of PID 7388.
Jul 15 13:37:34 tstcs03.ingdev systemd[1]: Cannot find unit for notify message of PID 7480.
I am not sure what those PIDs are as they do not appear on ps -ef list
This appears to be known limitation in the notify service type
From a pull request to the systemd man pages
Due to current limitations of the Linux kernel and the systemd, this
command requires CAP_SYS_ADMIN privileges to work
reliably. I.e. it's useful only in shell scripts running as a root
user.
I've attempted some hacky workarounds with sudo and friends but they won't work as systemd - generally failing with
No status data could be sent: $NOTIFY_SOCKET was not set
This refers to the socket that systemd-notify is trying to send data to - its defined in the service environment but I could not get it reliably exposed to a sudo environment
You could also try using a Python workaround described here
python -c "import systemd.daemon, time; systemd.daemon.notify('READY=1'); time.sleep(5)"
Its basically just a sleep which is not reliable and the whole point of using notify is reliable services.
In my case - I just refactored to use root as the user - with the actual service as a child under the main service with the desired user
sudo -u USERACCOUNT_LOGGED notify-send "hello"

ExecStartPost script an infinite loop script - Is it possible to do?

I am tweaking snmpd systemd service and I want to execute a script in ExecStartPost. This will be an infinite loop script.
This script is an implementation of agentx ..
I have tried running in a normal manner with and without & , but after sometime the systemd service is timing out..
it is timing out expecting a exit status from the script is what is believe.
Is there any way to run the script in background without the systemd snmp service timing out ?
[Unit]
Description=Simple Network Management Protocol (SNMP) Daemon.
After=syslog.target network.target
[Service]
Type=notify
Environment=OPTIONS="-LS0-6d"
EnvironmentFile=-/etc/sysconfig/snmpd
ExecStart=/usr/sbin/snmpd $OPTIONS -f
ExecStartPost=/usr/bin/python /usr/local/bin/pyagent.py
ExecReload=/bin/kill -HUP $MAINPID
[Install]
WantedBy=multi-user.target
From reading the documentation, I believe you want to keep your & but add - prefix to skip result checking:
ExecStartPost=-/usr/bin/python /usr/local/bin/pyagent.py &

Embedded linux application start script works better from command line

I'm running embedded linux on an Altera FPGA. It uses SystemD to run startup, and I have a script in the "multi-user.target.wants" section that runs my application.
When it runs from startup my code runs slower than when I run the identical script from an ssh shell.
I have checked that paths are the same, that permissions are correct on the scripts, that full paths are used in the scripts. Using 'top' I can see that priorities are set the same for the various threads started, yet somehow performance is completely different between the two ways of starting.
The script in full is:
#!/bin/sh
sleep 5s
mount /dev/mmcblk0p5 /home/root/linux
cd /home/root/linux/mem_driver
./memdev_load
cd /home/root/linux/gpio_driver
insmod ./gpiodev.ko
mknod /dev/gpiodev c 249 0
sleep 5s
cd /home/root/src/control
mysqld_safe &
up=0
while [ $up -ne 2 ]
do
up=$(pgrep mysql | wc -l);
echo $up
done
sleep 3s
cd /home/root/studio_web/myapp
npm start &
sleep 1s
cd /home/root/src/control
#sleep 1s
./control > /home/root/linux/output.log
various sleep commands have been inserted to try and make sure things start up in the right order.
Any help in diagnosing why this behaves differently would be greatly appreciated.
Is that the only shell script you are using? or do you have a systemd service file that executes that single shell script?
Using sleep is ineffective here. You should separate them into separate shell scripts and then use systemd to ensure that the shell scripts are run in order.
For example, we want to mount the directory first, because if this fails then nothing following will be successful. So we create a systemd mount service:
# home-root-linux.mount
[Unit]
Description=Mount /home/root/linux
Before=gpiodev.service
[Mount]
What=/dev/mmcblk0p5
Where=/home/root/linux
Options=defaults
[Install]
WantedBy=multi-user.target
Then we can create another systemd service which depends on the mount above before executing the three parts of the shell script which were previously separated by sleep to ensure that they were run in order.
# gpiodev.service
[Unit]
Description=Handle gpiodev kernel module
After=home-root-linux.mount
Before=mysqlsafe.service
[Service]
Type=oneshot
ExecStartPre=/home/root/linux/mem_driver/memdev_load
ExecStart=/sbin/insmod gpiodev.ko; /bin/mknod /dev/gpiodev c 249 0
WorkingDirectory=/home/root/linux/gpio_driver
RemainAfterExit=yes
StandardOutput=journal
[Install]
WantedBy=multi-user.target
Second part of the systemd service (following the sleep). We have a separate shellscript which is placed in /sbin/ in this example as it contains a while loop so it would be best to separate this:
# mysqlsafe.service
[Unit]
Description=MySQL safe
After=gpiodev.service
Before=npmoutput.service
[Service]
Type=oneshot
ExecStart=/sbin/mysqlsafe.sh
WorkingDirectory=/home/root/src/control
RemainAfterExit=yes
StandardOutput=journal
[Install]
WantedBy=multi-user.target
Second part of the shell script which is executed in the systemd service above (separated to a separate file due to the complexity):
# /sbin/mysqlsafe.sh
#!/bin/sh
mysqld_safe &
up=0
while [ $up -ne 2 ]
do
up=$(pgrep mysql | wc -l);
echo $up
done
Third part of the systemd service (the third section of the original shell script which was separated by sleep):
# mpmoutput.service
[Unit]
Description=npm and output to log
After=mysqlsafe.service
[Service]
Type=oneshot
ExecStartPre=/usr/bin/npm &
ExecStart=/home/root/src/control > /home/root/linux/output.log
WorkingDirectory=/home/root/studio_web/myapp
RemainAfterExit=yes
StandardOutput=journal
[Install]
WantedBy=multi-user.target
The idea behind this approach is that systemd recognises the importance of each service and the reliance upon the following service i.e. if one service fails the following services in queue will not execute. You can then check this using systemctl and see logging in journalctl.
Just a quick copy, paste and edit. Could contain errors as it was not tested or checked.
More reading can be found here regarding systemd service files: https://www.freedesktop.org/software/systemd/man/systemd.service.html

Fixing a systemd service 203/EXEC failure (no such file or directory)

I'm trying to set up a simple systemd timer to run a bash script every day at midnight.
systemctl --user status backup.service fails and logs the following:
backup.service: Failed at step EXEC spawning /home/user/.scripts/backup.sh: No such file or directory.
backup.service: Main process exited, code=exited, status=203/EXEC
Failed to start backup.
backup.service: Unit entered failed state.
backup.service: Failed with result 'exit-code'.
I'm lost, since the files and directories exist. The script is executable and, just to check, I've even set permissions to 777.
Some background:
The backup.timer and backup.service unit files are located in /home/user/.config/systemd/user.
backup.timer is loaded and active, and currently waiting for midnight.
Here's what it looks like:
[Unit]
Description=Runs backup at 0000
[Timer]
OnCalendar=daily
Unit=backup.service
[Install]
WantedBy=multi-user.target
Here's backup.service:
[Unit]
Description=backup
[Service]
Type=oneshot
ExecStart=/home/user/.scripts/backup.sh
[Install]
WantedBy=multi-user.target
And lastly, this is a paraphrase of backup.sh:
#!/usr/env/bin bash
rsync -a --delete --quiet /home/user/directory/ /mnt/drive/directory-backup/
The script runs fine if I execute it myself.
Not sure if it matters, but I use fish as my shell (started from .bashrc).
I'm happy to post the full script if that's helpful.
I think I found the answer:
In the .service file, I needed to add /bin/bash before the path to the script.
For example, for backup.service:
ExecStart=/bin/bash /home/user/.scripts/backup.sh
As opposed to:
ExecStart=/home/user/.scripts/backup.sh
I'm not sure why. Perhaps fish. On the other hand, I have another script running for my email, and the service file seems to run fine without /bin/bash. It does use default.target instead multi-user.target, though.
Most of the tutorials I came across don't prepend /bin/bash, but I then saw this SO answer which had it, and figured it was worth a try.
The service file executes the script, and the timer is listed in systemctl --user list-timers, so hopefully this will work.
Update: I can confirm that everything is working now.
To simplify, make sure to add a hash bang to the top of your ExecStart script, i.e.
#!/bin/bash
python -u alwayson.py
When this happened to me it was because my script had DOS line endings, which always messes up the shebang line at the top of the script. I changed it to Unix line endings and it worked.
I ran across a Main process exited, code=exited, status=203/EXEC today as well and my bug was that I forgot to add the executable bit to the file.
If that is a copy/paste from your script, you've permuted this line:
#!/usr/env/bin bash
There's no #!/usr/env/bin, you meant #!/usr/bin/env.
try running:
systemctl daemon-reload
and then again run
service <yourservice> status
I faced a similar issue, changed the permission and added executable permission
use chmod +x /etc/systemd/system/<service-filename>
This worked for me
I actually used the answer from How do I run a node.js app as a background service? combined with what dwrz said above. In my case, I was creating a Discord bot that needed to be able to run when I was not around.
With this service in place, I initially got the same error that the initial poster did, which brought me here. I was missing the #!/usr/bin/env node at the top of my executed node.js script.
Since then, no problems, although I intend to see what else can be extended to the service itself.

Resources