why systemd cannot start golang web app, no answer - go

My golang web app cannot start when I use systemd, but it works fine when manually start. What are problems with my systemd configuration?
goweb.service
$ cat goweb.service
[Unit]
Description=Backend service
After=network.target
[Service]
User=myapp
Group=myapp
Restart=on-failure
ExecStart=/u01/backend
[Install]
WantedBy=multi-user.target
backend is a binary file compiled with command: env GOOS=linux GOARCH=amd64 go build -v bitbucket.org/myapp/backend
systemd service status
$ sudo service goweb status
Redirecting to /bin/systemctl status goweb.service
● goweb.service - Backend service
Loaded: loaded (/usr/lib/systemd/system/goweb.service; disabled; vendor preset: disabled)
Active: inactive (dead)
May 18 10:55:56 instance-1 systemd[1]: Started Backend service.
May 18 10:55:56 instance-1 systemd[1]: Starting Backend service...
P/S: It looks like my web app started, but then stoped immediately.
I try config Type=forking, then service status show as below. Could someone explains why the log Started Backend service. and Starting Backend service... order is reversed.
$ sudo service goweb status
Redirecting to /bin/systemctl status goweb.service
● goweb.service - Backend service
Loaded: loaded (/usr/lib/systemd/system/goweb.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Wed 2016-05-18 11:06:02 UTC; 2s ago
Process: 25847 ExecStart=/u01/backend (code=exited, status=0/SUCCESS)
May 18 11:06:02 instance-1 systemd[1]: Starting Backend service...
May 18 11:06:02 instance-1 systemd[1]: Started Backend service.
Result when I run web app manually (from terminal):
$ /u01/backend
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] POST /upload --> main.uploadFileHandler (3 handlers)
[GIN-debug] Environment variable PORT="9005"
[GIN-debug] Listening and serving HTTP on :9005
Update:
I try supervisord and it works fine (same binary file)
With systemd, I changed service config to Restart=always, RestartSec=15. And systemd keeps restarting my web app.
Does anyone know why supervisord work fine but systemd not? I think that systemd would work fine with a basic task like that!

I know this is old, but saw that there weren't any sufficient answers, so thought I would post. My app was also being run manually, but when using systemd it wouldn't work. I realized it had to do with the filepaths that the Go program would state, such as:
tpl = template.Must(template.ParseGlob("templates/*"))
When running the app manually, the templates folder was there relative to my main.go file, but when systemd would run it, the filepath was different (not sure why or how to solve this yet), but for now I just hardcoded an absolute filepath, such as:
tpl = template.Must(template.ParseGlob("./home/ubuntu/templates/*"))
Now systemd works.
Hope this helps. If anyone can elaborate on how to come up with a better solution, would totally help me as well!

I'll take a wild stab and guess your app listens on port 80 and/or 443, so your best bet is to use setcap on it to give it permission.
example: sudo setcap 'cap_net_bind_service=+ep' /u01/backend, this will need to be done every time you compile your app.

From #RijulSudhir comment above, insert:
WorkingDirectory=/home/ubuntu/
below:
[service]
of systemd service file.

Replace your app temporarily with a minimal version that just dumps the environment out. Are the environment for running manually and under systemd sufficiently similar?

In this specific instance, I guess it's because you don't specify
ExecStart=/path/to/app
As a result, there's nothing to start.

Related

Starting an opensplice publisher via systemd does not publish data

I have an opensplice publisher on Ubuntu 20.04 that is started via systemd.
If the publisher starts via systemd then the data is not pubished, but also no errors are reported or present in the opensplice log files.
The publisher works if I run it from a command line or if I stop and restart the service.
The QoS are the same for the publisher and subscriber.
The publisher and subscriber applications are running on different machines.
There are no other participants on the network. All the machines are rebooted and the order of reboot does not change the observed behaviour.
The systemd service is:
[Unit]
Description=Publisher Process
Documentation=
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
WorkingDirectory=/opt/publisher/bin
ExecStart=/opt/publisher/bin/publisher.sh
Restart=always
RestartSec=2
[Install]
WantedBy=multi-user.target
The publisher.sh is:
#!/bin/bash
cd /opt/publisher/bin
source bashrc_local
# We just keep running the application (in case of a crash)
while true; do
./publisher
sleep 15
done
I have a work around that feels a little bit naff.
#!/bin/bash
cd /opt/publisher/bin
source bashrc_local
timeout 30 ./remote_processor
killall remote_processor
# We just keep running the application (in case of a crash)
while true; do
./publisher
sleep 15
done
Any ideas on how I can remove my work around?
Edit 16 Sept 22
The issue appears to be systemd start order and dependencies as I have run into the same issue with a program publishing data via UDP which is not using DDS.
Changing the dependencies so the services are started just before the user login does not help.
check your environment variables as systemd will not run with the same environment as your bash console
in particular have you set the OSPL_URI variable to point at the config?
if using the commercial version, OSPL_HOME and ADLINK_LICENSE will also need to be set
Does the PATH variable include your OSPL shared libraries?
These are all setup by running the $OSPL_HOME\release.com script in your bash session
I tend to manually add the required ones to the service file
e.g.
Environment=OSPL_URI=file:///opt/ospl.xml

User systemd service restarting only when SSH-ing into the machine

I have a strange situation with a web service hosted on a debian instance, that sometimes stops, and does not restart automatically. However, when SSH-ing into the machine, the service seems to restart automatically.
I originally wanted the service to always be up and restart, could you help me figure out what's wrong ? I may have misunderstood how systemctl --user services are meant to run.
The service in question is a Rails application running with passenger standalone, but I believe the problem might just be a misconfiguration in the systemd file.
My systemd file
# .config/systemd/user/my_service.service
[Unit]
Description=passenger with rails server for my_service (production)
After=syslog.target network.target
[Service]
Type=forking
PrivateTmp=yes
WorkingDirectory=/websites/xxx/current
PIDFile=/websites/xxx/shared/tmp/pids/passenger.8080.pid
ExecStart=/home/outscale/.asdf/shims/bundle exec passenger start /websites/xxx/current
ExecStop=/home/outscale/.asdf/shims/bundle exec passenger stop /websites/xxx/current
MemoryAccounting=true
MemoryLimit=3584M
Restart=always
RestartSec=1
TimeoutStopSec=30
KillMode=mixed
StandardInput=null
SyslogIdentifier=%p
# Environment
Environment="RAILS_ENV=production"
Environment="NODE_ENV=production"
[Install]
WantedBy=default.target
I have copied this installed the service using
systemctl --user daemon-reload
systemctl --user enable my_service
Was I meant to use something else, like systemctl --global enable unit ? I want my service to run with the "outscale" user installing the service (otherwise my version manager asdf does not work as expected)
I found the solution to my problem there. I had misunderstood the behavior of the --user flag (VS using the User= property in the service file)
I was running under debian 11 and as stated in the mentioned answer, my service would not necessarily shut down after logging out of ssh, but only at some point (not clear if it happened when my service crashed for the first time or some sort of garbage collection)
And the service would boot up again magically when SSHing in the instance as a reaction to a user login and starting all the services.
So the fix was to reimplement the services using User= and without the --user flag to make it a globally available service.

How to properly override generated systemd unit file to start after a ZFS mount has mounted

I'm using Ubuntu 18.04.4 LTS which uses systemd, but the squid package packaged with this version of Ubuntu is configured to start via init.d. It starts and runs via systemctl start squid.service if I start it manually after the system has booted.
However, I'm using a ZFS mount point ("/media") to store the cache data, and during the boot process squid is starting before this mount point is active. Consequently I'm getting the error "Failed to verify one of the swap directories". Full output of systemctl status squid is here
I'd like to tell systemd to wait until after media.mount has completed in the most minimally invasive way possible (e.g. without modifying the /etc/init.d/squid file that is maintained by the package). To that end I created the /etc/systemd/system/squid.service.d/override.conf file like so:
% cat /etc/systemd/system/squid.service.d/override.conf
[Unit]
Wants=network.target network-online.target nss-lookup.target media.mount
After=network.target network-online.target nss-lookup.target media.mount
[Install]
WantedBy=multi-user.target
But squid is still starting too early.
Is what I want to do possible? Or do I have to bite the bullet and define a native /etc/systemd/system/squid.service file and remove the /etc/init.d/squid init script?

Trigger event on AWS EC2 instance stop/terminate

Is there some way to trigger an event (e.g. running a script to push some logs to S3) when an EC2 instance is stopped/terminated?
I have looked into triggering the script using a service in /usr/lib/systemd/system but I haven't had any luck with that yet. I have heard that networking capabilities on the instance can be shutdown before a service is triggered and if true, that could be why the script is not executing correctly.
So the answer is not really AWS specific, but it is working for me now (tested on EC2 instance stopping and terminating).
I've created a system.d service file:
/usr/lib/systemd/system/my_shutdown.service
[Unit]
Description=my_shutdown Service
Before=shutdown.target reboot.target halt.target
Requires=network-online.target network.target
[Service]
KillMode=none
ExecStart=/bin/true
ExecStop=/path/to/my_script.sh
RemainAfterExit=yes
Type=oneshot
[Install]
WantedBy=multi-user.target
Added this service to multi-user.target:
systemctl enable my_shutdown.service
Alternatively you can manually create the symlink:
ln -s /usr/lib/systemd/system/my_shutdown.service /etc/systemd/system/multi-user.target.wants/my_shutdown.service
Started the service and tested by stopping/terminating the instance.
systemctl start my_shutdown.service
My understanding:
Description: a description of our service.
Before: we want our service to stop before these targets are started.
Requires: our service requires that network capabilities are available. These targets must not be stopped before our service starts/stops.
KillMode: none; do not kill our process.
ExecStart: /bin/true; a command that does nothing but returns a success. Run when are service is started.
ExecStop: the script to run. Run when are service is being stopped.
RemainAfterExit: consider our service active even when all its processes exited.
Type: oneshot; it is expected that the process has to exit before systemd starts follow-up units.
WantedBy: the target we want to add our service to.
References:
https://www.freedesktop.org/software/systemd/man/systemd.service.html
https://www.freedesktop.org/software/systemd/man/systemd.kill.html#
https://www.freedesktop.org/software/systemd/man/systemd.special.html
https://www.freedesktop.org/software/systemd/man/systemd.target.html
You can trigger events, such as pushing logs to S3 on specific events, with CloudWatch... Learn more here: https://aws.amazon.com/cloudwatch/

CoreOS EnvironmentFile directive failing

I am attempting to start gliderlabs/registrator and have it connect to consul on the COREOS_PRIVATE_IPV4 ip address.
[Unit]
Description=registrator
After=consul-server#%i.service
Requires=consul-server#%i.service
[Service]
EnvironmentFile=/etc/environment
ExecStartPre=-/usr/bin/docker kill registrator
ExecStartPre=-/usr/bin/docker rm registrator
ExecStartPre=/usr/bin/docker pull gliderlabs/registrator
ExecStart=/usr/bin/docker run --volume=/var/run/docker.sock:/var/run/docker.sock --net=host --hostname ${HOSTNAME} --name=registrator gliderlabs/registrator:latest consul://${COREOS_PRIVATE_IPV4}:8500
ExecStop=/usr/bin/docker stop registrator
[X-Fleet]
#Global=true
I am running into an error on starting the service that complains about the EnvironmentFile directive.
Dec 13 16:23:41 core-01 systemd[1]: [/run/fleet/units/registrator.service:5] Unknown lvalue 'EnvironmentFile' in section 'Unit'
I am currently running coreos 835.9.0. Does anyone have any thoughts on why this might be failing?
The unit provided does not match up with what the error is returning. The error is essentially saying that the EnvironmentFile=option is in the [Unit] section of your systemd-unit, and that the option isn't valid within that section.
Either what you actually have in the unit is different from what you put here, or it's possible fleet messed it up when it parsed and rendered out the unit.
If you look at the file in /run/fleet/units/registrator.service you should be able to verify where the EnvironmentFile option is. Make sure it's in the [Service] section and not the [Unit] section.
You can also run fleetctl cat registrator.service and fleet will output the unit file definition. It's possible you submitted the unit, made changes and then didn't destroy the unit before re-submitting it.

Resources