start request repeated too quickly - systemd

I'm writing a bash-script but I often face this issue.
When I try to start or stop a service I often get:
start request repeated too quickly
How can I solve this problem?
It's for example when I try to restart docker or openshift-origin master.
sudo service origin-master restart
● origin-master.service - Origin Master Service
Loaded: loaded (/usr/lib/systemd/system/origin-master.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Wed 2016-02-17 08:22:11 UTC; 44s ago
Docs: https://github.com/openshift/origin
Process: 2296 ExecStart=/usr/bin/openshift start master --config=${CONFIG_FILE} $OPTIONS (code=exited, status=255)
Main PID: 2296 (code=exited, status=255)
Feb 17 08:22:10 ip-172-xx-xx-xx.eu-central-1.compute.internal systemd[1]: origin-master.service: main process exited, code=exited, status=255/n/a
Feb 17 08:22:10 ip-172-xx-xx-xx.eu-central-1.compute.internal systemd[1]: Failed to start Origin Master Service.
Feb 17 08:22:10 ip-172-xx-xx-xx.eu-central-1.compute.internal systemd[1]: Unit origin-master.service entered failed state.
Feb 17 08:22:10 ip-172-xx-xx-xx.eu-central-1.compute.internal systemd[1]: origin-master.service failed.
Feb 17 08:22:11 ip-172-xx-xx-xx.eu-central-1.compute.internal systemd[1]: origin-master.service holdoff time over, scheduling restart.
Feb 17 08:22:11 ip-172-xx-xx-xx.eu-central-1.compute.internal systemd[1]: start request repeated too quickly for origin-master.service
Feb 17 08:22:11 ip-172-xx-xx-xx.eu-central-1.compute.internal systemd[1]: Failed to start Origin Master Service.
Feb 17 08:22:11 ip-172-xx-xx-xx.eu-central-1.compute.internal systemd[1]: Unit origin-master.service entered failed state.
Feb 17 08:22:11 ip-172-xx-xx-xx.eu-central-1.compute.internal systemd[1]: origin-master.service failed.
My script is just doing:
if [ $1 = "-u" ]
then
sudo service origin-master restart
fi
A manual restart is possible before I've executed the script. But after it it remains giving the error

This is a "feature" of systemctl. There is a parameter in the file that limits the restart frequency in seconds. Lower this while testing.
Edit the file
/etc/systemd/system/multi-user.target.wants/<your service here>
my example:
Restart=on-failure
StartLimitBurst=2
# Restart, but not more than once every 10 minutes
#StartLimitInterval=600
# Restart, but not more than once every 30s (for testing purposes)
StartLimitInterval=30

I suggest you familiarize yourself with systemd. That's what you're using under the hood when you run service. As #chepner says, the service is failing (as you can see from the second line of the log), and it's being restarted too quickly, triggering the error.
Try running journalctl -u origin-master.service to figure out why the error is happening.
Also, systemd cat origin-master.service will show you the Service Unit file that describes your service - there might be errors.

I had this problem on Ubuntu 20.4. And by adding execute permission to the ExecStart file the problem was solved.
sudo chmod +x /path/to/execfile

I have faced same issue and solved this problem like that:
if /var/log/mysql folder not exists:
sudo mkdir /var/log/mysql
and then give permission this folder:
sudo chown -R mysql:mysql /var/log/mysql
sudo systemctl stop mysql
sudo systemctl start mysql

in my case, there was a typing mistake in this file -> /etc/systemd/system/multi-user.target.wants/<your service here> so after tweaking necessary parameters, if you are still facing the same error, don't forget to check the file

In my case , is my /etc/docker/daemon.json file format error, when i make this true, run systemctl start docker the server start success.

Please try running the command :
td-agent --dry-run
This will give you the root cause.

Related

Avoid waiting for user when checking the Apache Tomcat status

As part of a bash script I check the recently installed Apache Tomcat status with
sudo systemctl status tomcat
The output is as follows
● tomcat.service
Loaded: loaded (/etc/systemd/system/tomcat.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-01-30 16:25:48 UTC; 3min 9s ago
Process: 175439 ExecStart=/opt/tomcat/bin/startup.sh (code=exited, status=0/SUCCESS)
Main PID: 175447 (java)
Tasks: 30 (limit: 4546)
Memory: 253.0M
CPU: 9.485s
CGroup: /system.slice/tomcat.service
└─175447 /usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.uti>
Jan 30 16:25:48 vps-06354c04 systemd[1]: Starting tomcat.service...
Jan 30 16:25:48 vps-06354c04 startup.sh[175439]: Tomcat started.
Jan 30 16:25:48 vps-06354c04 systemd[1]: Started tomcat.service.
Jan 30 16:25:48 vps-06354c04 systemd[1]: /etc/systemd/system/tomcat.service:1: Assignment outside of section. Ignoring.
Jan 30 16:25:48 vps-06354c04 systemd[1]: /etc/systemd/system/tomcat.service:2: Assignment outside of section. Ignoring.
This is the info I expect to see, but after printing it, systemctl keeps waiting for the user to type a key, breaking the automation I expect to deliver.
How can I avoid this behaviour?
I'm pretty sure the --no-pager option would keep that from happening. I just confirmed that on my own system on a different service. Otherwise, it goes interactive.
I don't recall ever seeing systemctl status asking for input, so perhaps it's the sudo used in this command doing that, in which case you could ask your system administrator to enable passwordless sudo on the account that runs this command.
A general solution for automating user input in shell scripts is to use expect, but for a simple case where you only need to send a single value one time, you can often get by with using echo and piping the value to the command (e.g., echo 'foo' | sudo systemctl status tomcat), although you should never do this to pass sensitive information such as passwords because that will potentially be accessible to other users on that system.

Can't launch ClickHouse. Exit code 203

I tried a lot but CH did not work. Please help me/
I installen ClickHouse on CentOs7 according to the https://clickhouse.tech/docs/ru/getting-started/install/
I entered
sudo systemctl status clickhouse-server
and did not see any result
I entered sudo clickhouse-client -h 127.0.0.1 --user default
sudo: unable to execute /bin/clickhouse-client: Operation not permitted
I entered sudo systemctl status clickhouse-server
clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)
Loaded: loaded (/usr/lib/systemd/system/clickhouse-server.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2020-12-24 18:56:34 UTC; 5s ago
Process: 7871 ExecStart=/usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid (code=exited, status=203/EXEC)
Main PID: 7871 (code=exited, status=203/EXEC)
CGroup: /machine.slice/libpod-5371acee9c1a12529197f3a6dadcbd14393f20d7e91c2e347e9147508002aee7.scope/system.slice/clickhouse-server.service
Dec 24 18:56:34 ok-parser systemd[1]: Unit clickhouse-server.service entered failed state.
Log file /var/log/clickhouse-server/clickhouse-server.log is empty
I have faced a very similar or exactly the same issue recently.
It seems to be a ClickHouse issue: ClickHouse server fails to start after upgrading from 22.3.6.5-lts to 22.4.5.9-stable · Issue #37099 · ClickHouse/ClickHouse.
Here is the solution comment:
den-crane commented May 12, 2022, 9:48 PM GMT+3
#svbrunov you need to do
rm /etc/systemd/system/clickhouse-server.service
systemctl daemon-reload
service clickhouse-server restart

Ubuntu - start a shell with multiple commands as a service

Script location: /usr/bin/Startup_tc_and_agents.sh
Script content (all 3 command are on the first line in the script):
sh /home/tb/Desktop/Installed/TeamCity/bin/runAll.sh start; sh /home/tb/Desktop/Installed/TeamCity/buildAgent_2/bin/agent.sh start; sh /home/tb/Desktop/Installed/TeamCity/buildAgent_3/bin/agent.sh start;
if I run this script manually in the terminal - everthing works as expected. But if I execute it from a .service it fails?
Service location: /etc/systemd/system/TeamCity_and_agents_startup.service
Service content:
[Unit]
Description=Service that startup TeamCity and its 3 agents
After=network.target
[Service]
Type=simple
User=root
ExecStart=/usr/bin/Startup_tc_and_agents.sh
[Install]
WantedBy=multi-user.target
I have also granted permissions like:
sudo chmod -R a+rwx /usr/bin/Startup_tc_and_agents.sh
sudo chmod -R a+rwx /etc/systemd/system/TeamCity_and_agents_startup.service
I have also run these commands:
systemctl daemon-reload
systemctl enable TeamCity_and_agents_startup.service
I have also tried to add #!/bin/sh as the first line and just above the line 'ExecStart' - but it doesn't change anything.
When I run the service with: systemctl start TeamCity_and_agents_startup.service I get the error below:
TeamCity_and_agents_startup.service - Service that startup TeamCity and its 3 agents
Loaded: loaded (/etc/systemd/system/TeamCity_and_agents_startup.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2020-02-10 16:17:24 +04; 13s ago
Process: 56412 ExecStart=/usr/bin/Startup_tc_and_agents.sh (code=exited, status=203/EXEC)
Main PID: 56412 (code=exited, status=203/EXEC)
Feb 10 16:17:24 Monster systemd[1]: Started Service that startup TeamCity and its 3 agents.
Feb 10 16:17:24 Monster systemd[56412]: TeamCity_and_agents_startup.service: Failed to execute command: Exec format error
Feb 10 16:17:24 Monster systemd[56412]: TeamCity_and_agents_startup.service: Failed at step EXEC spawning /usr/bin/Startup_tc_and_agents.sh: Exec format error
Feb 10 16:17:24 Monster systemd[1]: TeamCity_and_agents_startup.service: Main process exited, code=exited, status=203/EXEC
Feb 10 16:17:24 Monster systemd[1]: TeamCity_and_agents_startup.service: Failed with result 'exit-code'.
I changed this line:
ExecStart=/usr/bin/Startup_tc_and_agents.sh
to this:
ExecStart=/bin/sh /usr/bin/Startup_tc_and_agents.sh
PS: #!/bin/sh was not necessary to include in the script in my case.

custom systemd service can't start on Ubuntu 18.04

and thanks in advance for any assistance
I run original QT wallets (command-line based) for various cryptocurrencies. Earlier this year, I set them up as a custom systemd service, and that has been invaluable. It starts them up and shuts them down with the system just like all the normal services. I recently discovered an issue with one in particular, blackcoin.
This service worked fine in the past (I don't know how long it was down for before I found it)
If I run the command after execstart= command manually, everything works just fine. If I try to start the service (via systemctl start blackcoin), it fails with the following service status:
blackcoin.service - blackcoin wallet daemon
Loaded: loaded (/etc/systemd/system/blackcoin.service; enabled; vendor preset: enabled)
Active: failed (Result: core-dump) since Tue 2018-11-20 10:44:01 MST; 2h 51min ago
Process: 12272 ExecStart=/usr/bin/blackcoind -datadir=/coindaemon-rundirectory/blackcoin/ -conf=/coindaemon-rundirectory/blackcoin/blackcoin.conf -daemon (code=exited, status=0/SUCCESS)
Main PID: 12283 (code=dumped, signal=ABRT)
Nov 20 10:44:01 knox systemd[1]: blackcoin.service: Service hold-off time over, scheduling restart.
Nov 20 10:44:01 knox systemd[1]: blackcoin.service: Scheduled restart job, restart counter is at 5.
Nov 20 10:44:01 knox systemd[1]: Stopped blackcoin wallet daemon.
Nov 20 10:44:01 knox systemd[1]: blackcoin.service: Start request repeated too quickly.
Nov 20 10:44:01 knox systemd[1]: blackcoin.service: Failed with result 'core-dump'.
Nov 20 10:44:01 knox systemd[1]: Failed to start blackcoin wallet daemon.
Here is the body of the systemd service:
##################################################################
## Blackcoin Systemd service ##
##################################################################
[Unit]
Description=blackcoin wallet daemon
After=network.target
[Service]
Type=forking
User=somedude
RuntimeDirectory=blackcoind
PIDFile=/run/blackcoind/blackcoind.pid
Restart=on-failure
ExecStart=/usr/bin/blackcoind \
-datadir=/home/somedude/blackcoin/ \
-conf=/home/somedude/blackcoin/blackcoin.conf \
-daemon
ExecStop=/usr/bin/blackcoind \
-datadir=/home/somedude/blackcoin/ \
-conf=/home/somedude/blackcoin/blackcoin.conf \
stop
# Recommended hardening
# Provide a private /tmp and /var/tmp.
PrivateTmp=true
# Mount /usr, /boot/ and /etc read-only for the process.
ProtectSystem=full
# Disallow the process and all of its children to gain
# new privileges through execve().
NoNewPrivileges=true
# Use a new /dev namespace only populated with API pseudo devices
# such as /dev/null, /dev/zero and /dev/random.
PrivateDevices=true
# Deny the creation of writable and executable memory mappings.
MemoryDenyWriteExecute=true
[Install]
WantedBy=multi-user.target
And this is what blackcoin.conf contains:
rpcuser=somedude
rpcpassword=12345 (please don't rob my coins!)
# Wallets
wallet=wallet-blackcoin.dat
pid=/run/blackcoind/blackcoind.pid
rpcport=56111
port=56112
I'm going to keep testing and will post anything new that I find. Thanks for looking!

Cannot start prometheus by using systemd

OS level: CentOS Linux release 7.4.1708
Prometheus level: 2.4.2
prometheus.service:
[Unit]
Description=Prometheus
[Service]
User=prometheus
ExecStart=/usr/local/prometheus/prometheus
[Install]
WantedBy=default.target
When I use systemctl start prometheus to start the prometheus service, it always exit the main process by itself. And the systemctl's log shows like this:
● prometheus.service - Prometheus
Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2018-09-25 10:43:56 CST; 6s ago
Process: 5174 ExecStart=/usr/local/prometheus/prometheus (code=exited, status=1/FAILURE)
Main PID: 5174 (code=exited, status=1/FAILURE)
Sep 25 10:43:56 devtestserver systemd[1]: Started Prometheus.
Sep 25 10:43:56 devtestserver systemd[1]: Starting Prometheus...
Sep 25 10:43:56 devtestserver prometheus[5174]: level=info ts=2018-09-25T02:43:56.736457704Z caller=main.go:238 msg="Starting Prometheus" version="(version=2.4.2, branch=HE...13b1190a0)"
Sep 25 10:43:56 devtestserver systemd[1]: prometheus.service: main process exited, code=exited, status=1/FAILURE
Sep 25 10:43:56 devtestserver systemd[1]: Unit prometheus.service entered failed state.
Sep 25 10:43:56 devtestserver systemd[1]: prometheus.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
I have no ideas with this problem. I use the same config for the node_exporter, but node_exporter can start as normal. Please help. Thanks a lot.
You have not added configuration file i.e. prometheus.yml
Considering Service part of your prometheus.service file,
ExecStart=/usr/local/prometheus/prometheus \
--config.file /prometheus-2.26.0.linux-amd64/prometheus.yml
here, my .yml file is in /prometheus-2.26.0.linux-amd64/ location.
your might be different. Befre running check your both paths i.e. your executable file is on the path given in "ExecStart" and yml file is in --config.file
then reload your system by
systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus
then check the status using,
systemctl status prometheus
It should be active(running).
This should solve your problem. Let me know if it helped : )
There is an extra "i" at the end of WantedBy=default.target.
To get more details about services failing to start, try sudo journalctl -ex
My guess is it's either the extra "i" or Prometheus might not be able to parse your scrape rules or alerts files. It comes with "promtool" to check your configuration files and is installed in the same directory as prometheus. Your first step should be to try "promtool check config /path/to/prometheus.yml"
I encountered the same issue with Ubuntu 16.04. Turned out to be a permissions issue.
You should check that you user owns the directories in which you installed the binaries and the files inside these directories.
Where is the config file located? systemd is executed by / by default. prometheus reads the setting of ./prometheus.yml by default. Perhaps you need to add the following config option to the unit file of systemd.
[Unit]
Description=Prometheus
[Service]
User=prometheus
ExecStart=/usr/local/prometheus/prometheus --config.file /path/to/your/config
[Install]
WantedBy=default.target
This problem is caused because the data storage directory does not have permission. The default Prometheus data directory is /data.
chown -R prometheus:prometheus /data
copy and paste this in your command line:
sudo tee /etc/systemd/system/prometheus.service<<EOF
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP \$MAINPID
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090 \
--web.external-url=
SyslogIdentifier=prometheus
Restart=always
[Install]
WantedBy=multi-user.target
EOF
I encountered similar issue in redhat/Centos.I solved it by temporarily running "sudo setenforce 0". You can also edit the /etc/selinux/config file and set the SELINUX to disabled

Resources