Troubles with Aerospike 3.8.x monitored with monit on CentOS 7 - shell

Since Aerospike went 3.8.0+, it changed the way the process ASD is launched. There is no more /etc/init.d/aerospike instead systemd is now used.
I have a problem as I use monit as a watchdog for aerospike and it is quite difficult to start / stop the asd process through systemd called by monit. Monit uses a limited path so maybe the problem comes from missing environment vars.
Here is what I use in monit:
check process aerospike with pidfile /var/run/aerospike/asd.pid
start = "/usr/bin/bash -c '/opt/aerospike/aerospike-service.sh start'"
stop = "/usr/bin/bash -c '/opt/aerospike/aerospike-service.sh stop'"
I'm calling explicitely a shell script so that I can add environment vars on startup.
Here is my basic shell script:
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin"
case "$1" in
start)
echo "Starting aerospike Server ..."
systemctl start aerospike.service
;;
stop)
echo "Stopping aerospike Server ..."
systemctl stop aerospike.service
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
I blindly copy-pasted my path at the start of the script. The script works if it is launched through CLI, but not if launched through monit.
I use CentOS 7.
I have not found a raw command to launch the asd process (and stop it) as it could be a way to not use systemd and sidestepping this problem.
Is there a way to configure monit to launch nicely systemd services, or are there raw commands for aerospike that can be used directly through monit?

Related

Keep Track of laravel websocket with monit centos

Im trying to monitor laravel-websocket with monit instead of supervisord because of more options it provides
So In my /home/rabter/laravelwebsocket.sh :
#!/bin/bash
case $1 in
start)
echo $$ > /var/run/laravelwebsocket.pid;
exec 2>&1 php /home/rabter/core/artisan websockets:serve 1>/tmp/laravelwebsocket.out
;;
stop)
kill `cat /var/run/laravelwebsocket.pid` ;;
*)
echo "usage: laravelwebsocket.sh {start|stop}" ;;
esac
exit 0
And in etc/monit.d I made a file named cwp.laravelwebsocket with code
check process laravelwebsocket with pidfile /var/run/laravelwebsocket.pid
start program "/bin/bash -c /home/rabter/laravelwebsocket.sh start"
stop program "/bin/bash -c /home/rabter/laravelwebsocket.sh stop"
if failed port 6001 then restart
if 4 restarts within 8 cycles then timeout
unfortunately with I run monit everything starts to get monitord but laravel websocket, and it does not start once and in monit table infront I see
Process - laravelwebsocket Execution failed | Does not exist
How can I make monit monitor and start laravel-websocket on startup and on fails or errors or crashes?
I have looked into Monitor a Laravel Queue Worker with Monit
but no luck!
Your bash script inserts its own pid into your pid file. Additionally, the php process should be send to background if using monit, because monit is a monitoring tool, rather then a supervisor.
#!/usr/bin/env bash
case $1 in
start)
php /home/rabter/core/artisan websockets:serve & 2>&1 >/tmp/laravelwebsocket.out
echo $! > /var/run/laravelwebsocket.pid;
;;
stop)
kill $(cat /var/run/laravelwebsocket.pid) ;;
*)
echo "usage: $(basename $0) {start|stop}" ;;
esac
exit 0
Then make that file executable with chmod +x FILEPATH.
This should now work:
check process laravelwebsocket with pidfile /var/run/laravelwebsocket.pid
start program "/home/rabter/laravelwebsocket.sh start"
stop program "/home/rabter/laravelwebsocket.sh stop"
if failed port 6001 then restart
if 4 restarts within 8 cycles then timeout
Do you use monit as init-system for a container? If so, please let me know. Then a few more details apply.

How to create solr service for starting solr on reboot

I am trying to create a solr service script that I can use to start solr automatically on reboot. Here is a script I saw recommended:
#!/bin/sh
# Starts, stops, and restarts Apache Solr.
#
# chkconfig: 35 92 08
# description: Starts and stops Apache Solr
SOLR_DIR="/var/www/html/fas/solr/solr-latest"
JAVA_OPTIONS="-Xmx1024m -DSTOP.PORT=8983 -DSTOP.KEY=mustard -jar /var/www/html/fas/solr/solr-latest/server/start.jar"
LOG_FILE="/var/log/solr.log"
JAVA="/bin/java"
case $1 in
start)
echo "Starting Solr"
cd $SOLR_DIR
$JAVA $JAVA_OPTIONS 2> $LOG_FILE &
;;
stop)
echo "Stopping Solr"
cd $SOLR_DIR
$JAVA $JAVA_OPTIONS --stop
;;
restart)
$0 stop
sleep 1
$0 start
;;
*)
echo "Usage: $0 {start|stop|restart}" >&2
exit 1
;;
esac
I think I have set the appropriate values for the variables in the script. But when I try to run the script, I get "Connection refused."
$ service solr stop
Stopping Solr
java.net.ConnectException: Connection refused (Connection refused)
I get the same result whether I run the script as root or not.
I can stop and start solr this way, though:
/path/to/my/solr/bin/solr start
So I also tried creating this script at /etc/init.d/solr-start
#!/bin/sh
# Starts Apache Solr.
#
# chkconfig: 35 92 08
# description: Starts Apache Solr
/var/www/html/fas/solr/solr-latest/bin/solr start
This script works at the command line, but it doesn't work on reboot. To try to make it run on reboot, I did...
sudo systemctl enable solr-start
But solr is not started on reboot.
My versions:
RHEL 7, Solr 6.6.6
Unfortunately you're providing nearly no details about your specific Solr installation. The following systemd unit example might provide a starting point though.
Create the file /etc/systemd/system/solr.service with the following content (and perform adaptions to make it suitable to your Solr installation):
[Unit]
Description=Apache SOLR
After=syslog.target network.target remote-fs.target nss-lookup.target systemd-journald-dev-log.socket
Before=multi-user.target
Conflicts=shutdown.target
[Service]
User=solr
# Assumes SOLR_PID_DIR; change port if it differs
PIDFile=/var/lib/solr/solr-8983.pid
# Assumes proper configuration in /etc/default/solr.in.sh
Environment=SOLR_INCLUDE=/etc/default/solr.in.sh
ExecStart=/path/to/my/solr/bin/solr start
ExecStop=/path/to/my/solr/bin/solr stop
Restart=on-failure
[Install]
WantedBy=multi-user.target
Perform the following commands either as root or prefix them with sudo:
systemctl daemon-reload
systemctl enable solr.service
systemctl start solr.service
systemctl status solr.service
If you need different (or more) options in the systemd unit, this GitHub Gist was suggested to upstream as possible starting point for upstream inclusion.

Embedded linux application start script works better from command line

I'm running embedded linux on an Altera FPGA. It uses SystemD to run startup, and I have a script in the "multi-user.target.wants" section that runs my application.
When it runs from startup my code runs slower than when I run the identical script from an ssh shell.
I have checked that paths are the same, that permissions are correct on the scripts, that full paths are used in the scripts. Using 'top' I can see that priorities are set the same for the various threads started, yet somehow performance is completely different between the two ways of starting.
The script in full is:
#!/bin/sh
sleep 5s
mount /dev/mmcblk0p5 /home/root/linux
cd /home/root/linux/mem_driver
./memdev_load
cd /home/root/linux/gpio_driver
insmod ./gpiodev.ko
mknod /dev/gpiodev c 249 0
sleep 5s
cd /home/root/src/control
mysqld_safe &
up=0
while [ $up -ne 2 ]
do
up=$(pgrep mysql | wc -l);
echo $up
done
sleep 3s
cd /home/root/studio_web/myapp
npm start &
sleep 1s
cd /home/root/src/control
#sleep 1s
./control > /home/root/linux/output.log
various sleep commands have been inserted to try and make sure things start up in the right order.
Any help in diagnosing why this behaves differently would be greatly appreciated.
Is that the only shell script you are using? or do you have a systemd service file that executes that single shell script?
Using sleep is ineffective here. You should separate them into separate shell scripts and then use systemd to ensure that the shell scripts are run in order.
For example, we want to mount the directory first, because if this fails then nothing following will be successful. So we create a systemd mount service:
# home-root-linux.mount
[Unit]
Description=Mount /home/root/linux
Before=gpiodev.service
[Mount]
What=/dev/mmcblk0p5
Where=/home/root/linux
Options=defaults
[Install]
WantedBy=multi-user.target
Then we can create another systemd service which depends on the mount above before executing the three parts of the shell script which were previously separated by sleep to ensure that they were run in order.
# gpiodev.service
[Unit]
Description=Handle gpiodev kernel module
After=home-root-linux.mount
Before=mysqlsafe.service
[Service]
Type=oneshot
ExecStartPre=/home/root/linux/mem_driver/memdev_load
ExecStart=/sbin/insmod gpiodev.ko; /bin/mknod /dev/gpiodev c 249 0
WorkingDirectory=/home/root/linux/gpio_driver
RemainAfterExit=yes
StandardOutput=journal
[Install]
WantedBy=multi-user.target
Second part of the systemd service (following the sleep). We have a separate shellscript which is placed in /sbin/ in this example as it contains a while loop so it would be best to separate this:
# mysqlsafe.service
[Unit]
Description=MySQL safe
After=gpiodev.service
Before=npmoutput.service
[Service]
Type=oneshot
ExecStart=/sbin/mysqlsafe.sh
WorkingDirectory=/home/root/src/control
RemainAfterExit=yes
StandardOutput=journal
[Install]
WantedBy=multi-user.target
Second part of the shell script which is executed in the systemd service above (separated to a separate file due to the complexity):
# /sbin/mysqlsafe.sh
#!/bin/sh
mysqld_safe &
up=0
while [ $up -ne 2 ]
do
up=$(pgrep mysql | wc -l);
echo $up
done
Third part of the systemd service (the third section of the original shell script which was separated by sleep):
# mpmoutput.service
[Unit]
Description=npm and output to log
After=mysqlsafe.service
[Service]
Type=oneshot
ExecStartPre=/usr/bin/npm &
ExecStart=/home/root/src/control > /home/root/linux/output.log
WorkingDirectory=/home/root/studio_web/myapp
RemainAfterExit=yes
StandardOutput=journal
[Install]
WantedBy=multi-user.target
The idea behind this approach is that systemd recognises the importance of each service and the reliance upon the following service i.e. if one service fails the following services in queue will not execute. You can then check this using systemctl and see logging in journalctl.
Just a quick copy, paste and edit. Could contain errors as it was not tested or checked.
More reading can be found here regarding systemd service files: https://www.freedesktop.org/software/systemd/man/systemd.service.html

How do I write a watchdog daemon in bash?

I want a way to write a daemon in a shell script, which runs another application in a loop, restarting it if it dies.
When run using ./myscript.sh from an SSH session, it shall launch a new instance of the daemon, except if the daemon is already running.
When the SSH session ends, the daemon shall persist.
There shall be a parameter (./myscript -stop) that kills any existing daemon.
(Notes on edit - The original question specified that nohup and similar tools may not be used. This artificial requirement was an "XY question", and the accepted answer in fact uses all the tools the OP claimed were not possible to use.)
Based on clarifications in comments, what you actually want is a daemon process that keeps a child running, relaunching it whenever it exits. You want a way to type "./myscript.sh" in an ssh session and have the daemon started.
#!/usr/bin/env bash
PIDFILE=~/.mydaemon.pid
if [ x"$1" = x-daemon ]; then
if test -f "$PIDFILE"; then exit; fi
echo $$ > "$PIDFILE"
trap "rm '$PIDFILE'" EXIT SIGTERM
while true; do
#launch your app here
/usr/bin/server-or-whatever &
wait # needed for trap to work
done
elif [ x"$1" = x-stop ]; then
kill `cat "$PIDFILE"`
else
nohup "$0" -daemon
fi
Run the script: it will launch the daemon process for you with nohup. The daemon process is a loop that watches for the child to exit, and relaunches it when it does.
To control the daemon, there's a -stop argument the script can take that will kill the daemon. Look at examples in your system's init scripts for more complete examples with better error checking.
The pid of the most recently "backgrounded" process is stored in $!
$ cat &
[1] 7057
$ echo $!
7057
I am unaware of a fork command in bash. Are you sure bash is the right tool for this job?

run nodejs forever

im runnig an http server using nodejs. i want also the server to run forever, even when my machine restarts,i want node to run again upon restart. so i created a script to put in the
/etc/init.d/
here is the script
#! /bin/sh -e
set -e
PATH=/usr/local/bin/node:/bin:/usr/bin:/sbin:/usr/sbin
DAEMON=/opt/node-v0.4.7/examples/app.js
case "$1" in
start) forever start $DAEMON ;;
stop) forever stop $DAEMON ;;
force-reload|restart)
forever restart $DAEMON ;;
*) echo "Usage: /etc/init.d/node {start|stop|restart|force-reload}"
exit 1 ;;
esac
exit 0
however when i run
/etc/init.d/node
i keep getting the same error saying
/etc/init.d/node: 13: Syntax error: word unexpected (expecting ")")
can you guys see the error ? i'm sure it's probably some easy syntax error but it's kinda late and i'm really tired.
thanks for the help
I changed the shebang from:
#! /bin/sh -e
To:
#! /bin/bash
And now that script works for me.
You can create upstart service in Ubuntu(if you use it).
Create myapp.conf in /etc/init and write something like:
start on startup
respawn
exec node /path/to/your/script.js
Then your app will start after reboot and you can manage it via start and stop commands.
And look at forever node.js module

Resources