I have a strange situation with a web service hosted on a debian instance, that sometimes stops, and does not restart automatically. However, when SSH-ing into the machine, the service seems to restart automatically.
I originally wanted the service to always be up and restart, could you help me figure out what's wrong ? I may have misunderstood how systemctl --user services are meant to run.
The service in question is a Rails application running with passenger standalone, but I believe the problem might just be a misconfiguration in the systemd file.
My systemd file
# .config/systemd/user/my_service.service
[Unit]
Description=passenger with rails server for my_service (production)
After=syslog.target network.target
[Service]
Type=forking
PrivateTmp=yes
WorkingDirectory=/websites/xxx/current
PIDFile=/websites/xxx/shared/tmp/pids/passenger.8080.pid
ExecStart=/home/outscale/.asdf/shims/bundle exec passenger start /websites/xxx/current
ExecStop=/home/outscale/.asdf/shims/bundle exec passenger stop /websites/xxx/current
MemoryAccounting=true
MemoryLimit=3584M
Restart=always
RestartSec=1
TimeoutStopSec=30
KillMode=mixed
StandardInput=null
SyslogIdentifier=%p
# Environment
Environment="RAILS_ENV=production"
Environment="NODE_ENV=production"
[Install]
WantedBy=default.target
I have copied this installed the service using
systemctl --user daemon-reload
systemctl --user enable my_service
Was I meant to use something else, like systemctl --global enable unit ? I want my service to run with the "outscale" user installing the service (otherwise my version manager asdf does not work as expected)
I found the solution to my problem there. I had misunderstood the behavior of the --user flag (VS using the User= property in the service file)
I was running under debian 11 and as stated in the mentioned answer, my service would not necessarily shut down after logging out of ssh, but only at some point (not clear if it happened when my service crashed for the first time or some sort of garbage collection)
And the service would boot up again magically when SSHing in the instance as a reaction to a user login and starting all the services.
So the fix was to reimplement the services using User= and without the --user flag to make it a globally available service.
Related
I have an opensplice publisher on Ubuntu 20.04 that is started via systemd.
If the publisher starts via systemd then the data is not pubished, but also no errors are reported or present in the opensplice log files.
The publisher works if I run it from a command line or if I stop and restart the service.
The QoS are the same for the publisher and subscriber.
The publisher and subscriber applications are running on different machines.
There are no other participants on the network. All the machines are rebooted and the order of reboot does not change the observed behaviour.
The systemd service is:
[Unit]
Description=Publisher Process
Documentation=
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
WorkingDirectory=/opt/publisher/bin
ExecStart=/opt/publisher/bin/publisher.sh
Restart=always
RestartSec=2
[Install]
WantedBy=multi-user.target
The publisher.sh is:
#!/bin/bash
cd /opt/publisher/bin
source bashrc_local
# We just keep running the application (in case of a crash)
while true; do
./publisher
sleep 15
done
I have a work around that feels a little bit naff.
#!/bin/bash
cd /opt/publisher/bin
source bashrc_local
timeout 30 ./remote_processor
killall remote_processor
# We just keep running the application (in case of a crash)
while true; do
./publisher
sleep 15
done
Any ideas on how I can remove my work around?
Edit 16 Sept 22
The issue appears to be systemd start order and dependencies as I have run into the same issue with a program publishing data via UDP which is not using DDS.
Changing the dependencies so the services are started just before the user login does not help.
check your environment variables as systemd will not run with the same environment as your bash console
in particular have you set the OSPL_URI variable to point at the config?
if using the commercial version, OSPL_HOME and ADLINK_LICENSE will also need to be set
Does the PATH variable include your OSPL shared libraries?
These are all setup by running the $OSPL_HOME\release.com script in your bash session
I tend to manually add the required ones to the service file
e.g.
Environment=OSPL_URI=file:///opt/ospl.xml
I'm trying to launch Neo4J graph database on AWS using their AIM image (enteprise 3.3.9)
However, the server fails to launch the instance automatically how it's supposed to.
When I try to relaunch it using
systemctl restart neo4j
It also fails.
When I do
systemctl cat neo4j
I find the /etc/neo4j/pre-neo4j.sh file, which is apparently launched on the instance's startup, which, in turn launches Neo4J (when it's supposed to work):
[Unit]
Description=Neo4j Graph Database
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/etc/neo4j/pre-neo4j.sh
Restart=on-failure
User=neo4j
Group=neo4j
Environment="NEO4J_CONF=/etc/neo4j" "NEO4J_HOME=/var/lib/neo4j"
LimitNOFILE=60000
TimeoutSec=120
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
So then I launch it manually via the bash script using the sudo prefix and then it starts up fine.
sudo /etc/neo4j/pre-neo4j.sh
The documentation on deploying Neo4J on an AWS server doesn't mention anything about permissions if you use their image. So what can be the problem?
I don't want to have manually launch the DB using the sudo — is it possible to resolve this problem by modifying the bash script itself?
..
The file /etc/neo4j/pre-neo4j.sh sets some environmental parameters and then launches neo4j via:
/usr/share/neo4j/bin/neo4j console
Based on the comments.
The solution was to use
journalctl -u neo4j
to inspect the logs associated with the failed start of neo4j. This enabled to identify the root cause, and subsequently, to fix the issue.
Is there some way to trigger an event (e.g. running a script to push some logs to S3) when an EC2 instance is stopped/terminated?
I have looked into triggering the script using a service in /usr/lib/systemd/system but I haven't had any luck with that yet. I have heard that networking capabilities on the instance can be shutdown before a service is triggered and if true, that could be why the script is not executing correctly.
So the answer is not really AWS specific, but it is working for me now (tested on EC2 instance stopping and terminating).
I've created a system.d service file:
/usr/lib/systemd/system/my_shutdown.service
[Unit]
Description=my_shutdown Service
Before=shutdown.target reboot.target halt.target
Requires=network-online.target network.target
[Service]
KillMode=none
ExecStart=/bin/true
ExecStop=/path/to/my_script.sh
RemainAfterExit=yes
Type=oneshot
[Install]
WantedBy=multi-user.target
Added this service to multi-user.target:
systemctl enable my_shutdown.service
Alternatively you can manually create the symlink:
ln -s /usr/lib/systemd/system/my_shutdown.service /etc/systemd/system/multi-user.target.wants/my_shutdown.service
Started the service and tested by stopping/terminating the instance.
systemctl start my_shutdown.service
My understanding:
Description: a description of our service.
Before: we want our service to stop before these targets are started.
Requires: our service requires that network capabilities are available. These targets must not be stopped before our service starts/stops.
KillMode: none; do not kill our process.
ExecStart: /bin/true; a command that does nothing but returns a success. Run when are service is started.
ExecStop: the script to run. Run when are service is being stopped.
RemainAfterExit: consider our service active even when all its processes exited.
Type: oneshot; it is expected that the process has to exit before systemd starts follow-up units.
WantedBy: the target we want to add our service to.
References:
https://www.freedesktop.org/software/systemd/man/systemd.service.html
https://www.freedesktop.org/software/systemd/man/systemd.kill.html#
https://www.freedesktop.org/software/systemd/man/systemd.special.html
https://www.freedesktop.org/software/systemd/man/systemd.target.html
You can trigger events, such as pushing logs to S3 on specific events, with CloudWatch... Learn more here: https://aws.amazon.com/cloudwatch/
I am trying to migrate from PHP to Go and planning to drop nginx alltogether. But I don't know how to run the go http webserver as a deamon in the background and I also don't know how to automatically start the webserver if I make a reboot, or how to kill the process.
With nginx all I do is
$ systemctl start nginx.service
$ systemctl restart nginx.service
$ systemctl stop nginx.service
$ systemctl enable nginx.service
$ systemctl disable nginx.service
This is very convenient, but it seems like I can't do this with Go http server. I have to compile and run it as any other Go program. What solutions do exist for these concerns?
This is less of a Go question and more of a Systems Administration question. There are ways to add a command to systemd (like in this blog post).
Personally, I prefer to keep my applications separate from my services, so I tend to use supervisord for my programs that tend to be started, stopped, or restarted frequently. The documentation for supervisord is pretty straightforward, but essentially you can create a config file to describe the services you want to run, the command used to run it (such as /path/to/go/binary -flag) how you want to handle starting, stopping, failure recovery, logging, etc....
I am trying to setup tomcat 7 on Digital Ocean CoreOS machine but facing some problem, not sure how to solve them. I am following below tutorial provided by the Digital Ocean to setup Apache.
https://www.digitalocean.com/community/tutorials/how-to-create-and-run-a-service-on-a-coreos-cluster
I created docker container and run it using following command.
docker run -i -t ubuntu:14.04 /bin/bash
I was successfully able to install tomcat 7 by using below commands. (I followed this tutorial to setup tomcat 7 within the docker container: https://www.digitalocean.com/community/tutorials/how-to-install-apache-tomcat-7-on-ubuntu-14-04-via-apt-get)
sudo apt-get update
sudo apt-get install tomcat7
Then I can created service unit file named as tomcat#.service
[Unit]
Description=Tomcat 7 web server service
After=etcd.service After=docker.service
Requires=tomcat-discovery#%i.service
[Service]
TimeoutStartSec=0 KillMode=none
EnvironmentFile=/etc/environment
ExecStartPre=-/usr/bin/docker kill tomcat%i
ExecStartPre=-/usr/bin/docker rm tomcat%i
ExecStartPre=/usr/bin/docker pull attacomsian/tomcat
ExecStart=/usr/bin/docker run –name tomcat%i -p ${COREOS_PUBLIC_IPV4}:%i:8080 attacomsian/tomcat `service tomcat7 start` -D FOREGROUND
ExecStop=/usr/bin/docker stop tomcat%i
[X-Fleet]
X-Conflicts=tomcat#*.service
Then I created tomcat-discovery#.service to register service states with Etcd as below
[Unit]
Description=Announce Tomcat#%i service
BindsTo=tomcat#%i.service
[Service]
EnvironmentFile=/etc/environment
ExecStart=/bin/sh -c “while true; do etcdctl set /announce/services/tomcat%i ${COREOS_PUBLIC_IPV4}:%i –ttl 60; sleep 45; done”
ExecStop=/usr/bin/etcdctl rm /announce/services/tomcat%i
[X-Fleet]
X-ConditionMachineOf=tomcat#%i.service
I submitted and loaded files to Fleet as below
fleetctl submit tomcat#.service tomcat-discovery#.service
fleetctl load tomcat#8080.service
fleetctl load tomcat-discovery#8080.service
Everything worked fine so far. I did not see any error. But when I tried to run the service as below
fleetctl start tomcat#8080.service
But it did not started. I can see it is appearing as dead.
I am new to CoreOS and still learning. I am managing servers at Digital Ocean and I quite about it quite well. I googled about this issue but did not found any help. I personally think following line is actually causing the trouble.
ExecStart=/usr/bin/docker run –name tomcat%i -p ${COREOS_PUBLIC_IPV4}:%i:8080 attacomsian/tomcat `service tomcat7 start` -D FOREGROUND
I would really appreciate any kind help to get this up.
Many Thanks
Attacomsian
I was going so suggest you take a look at what others have done and then discovered you have posted a similar question on the Docker Hub registry.
Did you take a look at the Docker file used by the tutum/tomcat image?
https://github.com/tutumcloud/tutum-docker-tomcat/blob/master/7.0/Dockerfile
https://github.com/tutumcloud/tutum-docker-tomcat/blob/master/7.0/run.sh
It runs a script called "run.sh" that runs tomcat in the foreground.
The thing that is tricky to understand is that Docker is not a virtual machine and therefore does not have any services running. You must run the docker processes explicitly or setup a process manager like runit or supervisord.
Hope this helps.