I have a requirement wherein I have to constantly fetch messages from AWS-SQS(simple queue service) and update the related records of a model. The message contains data that needs to be displayed to the related users as notifications as soon as they are fetched. This has been successfully managed by using Action Cable. I have created a rake task that fetches the messages from the queue and does the required processing. This task is supposed to run in an infinite loop. I have 2 questions regarding it?
namespace :sqs_consumer do
desc 'Get data from the AWS-SQS and process it.'
task start: :environment do
# initialize the sqs client
loop do
#read the queue for messages and process them in batches(if any)
end
end
end
1) Is it right to create a rake task for the above requirement? A rake task that runs infinitely is the right way? If not, what is the right approach. I cant run the task periodically since i need data in real time.
2) I want to monitor the task. I am using monit for the same. Unfortunately, Monit conf for the same doesn't seem to work. What am I doing wrong or missing?
check process aws_sqs_consumer with pidfile /var/www/myproject/shared/pids/sqs_consumer.pid
start program = "/bin/sh -c 'cd /var/www/myproject/current; nohup bundle exec rake sqs_consumer:start RAILS_ENV=staging -i 0 -P /var/www/myproject/shared/pids/sqs_consumer.pid >> log/sqs_consumer.log 2>&1 &'" as uid ubuntu and gid ubuntu
stop program = "/bin/sh -c 'kill $(cat /var/www/myproject/shared/pids/sqs_consumer.pid)'" as uid ubuntu and gid ubuntu
This monit configuration worked for me
check process aws_sqs_consumer with pidfile /var/www/myproject/shared/tmp/pids/sqs_consumer.pid
start program = "/bin/sh -c 'cd /var/www/myproject/current && BACKGROUND=y PIDFILE=/var/www/myproject/shared/tmp/pids/sqs_consumer.pid LOG_LEVEL=info bundle exec rake sqs_consumer:start RAILS_ENV=staging'"
stop program = "/bin/sh -c 'kill $(cat /var/www/myproject/shared/tmp/pids/sqs_consumer.pid)'" as uid ubuntu and gid ubuntu with timeout 90 seconds
Related
I have a unix script that invokes another script on a remote unix server.
amongst other commands i am stopping a service. The stop command essentially translates to
ssh -t -t -q ${AEM_USER}#${SERVERIP} 'bash -l -c "service aem stop"'
The service is getting stopped but when i start back the service it just creates the .pid file and does not perform the start up. When i run the command for start i.e.
ssh -t -t -q ${AEM_USER}#${SERVERIP} 'bash -l -c "service aem start"'
it does not show any error. On going to the server and checking the status
service aemauthor status
Below message is displayed
aem dead but pid file exists
Also when starting the service by logging in to the server, it works as expected along with the message
Removing stale pidfile (pid: 8701)
Starting aem
We don't know the details of the service script of aem.
I guess the problem is related to the SIGHUP signal. When we log off from a shell or disconnect from ssh, the OS will send HUP signal to all processes that started in this terminated shell. If the process didn't handle the HUP signal, it would exit by default.
When we run a command via ssh remotely, the process started by this command will receive HUP signal after ssh session is terminated.
We can use the nohup command to ignore the HUP signal.
You can try
ssh -t -t -q ${AEM_USER}#${SERVERIP} 'bash -l -c "nohup service aem start"'
If it works, you can use nohup command to start aem in the service script.
As mentioned at the stale pidfile syndrome, there are different reasons for pidfiles getting stalled, like for instance some issues with the way your handles its removal when the process exits... but considering your only experiencing when running remotely, I would guess it might be related to what is being loaded or not by your profile... check the most voted solid answer at the post below for some insights:
Why Does SSH Remote Command Get Fewer Environment Variables
As described in the comments of the mentioned post, you can try sourcing /etc/profile or ~/.bash_profile before executing your script to test it, or even trying to execute env locally and remotelly to compare variables that are being sourced or not.
I want to set my screen as screensave status every 50minutes (3000 seconds).
cat /home/rest.sh
while true;do
sleep 3000
xscreensaver-command --lock 1>/dev/null
done
sh /home/rest.sh & can make it run.
Now i want to set it as a daemon.
sudo vim /etc/systemd/system/screensave.service
[Unit]
Description=screensave
[Service]
User=root
ExecStart=/bin/bash /home/rest.sh
StandardError=journal
[Install]
WantedBy=multi-user.target
To set it and enable as daemon.
systemctl enable screensave.service
I find that the service is not running as a daemon.
sudo journalctl -u screensave
Jan 24 12:16:50 user systemd[1]: Started screensave.
Jan 24 12:17:22 user bash[621]: xscreensaver-command: warning: $DISPLAY is not set: defaulting to ":0.0".
Jan 24 12:17:22 user bash[621]: No protocol specified
Jan 24 12:17:22 user bash[621]: xscreensaver-command: can't open display :0.0
How to run it as a daemon after $DISPLAY is set ?
This is a very common FAQ. A system daemon cannot easily connect to the X session of any individual user. On a multi-user system, how do you tell which user's session to connect to, anyway? On a single-user system, what should the daemon do if no session is running (as it often isn't at the time the daemon starts up)?
Trying to run a system daemon as any particular user won't work, and giving individual users access to a system daemon is a recipe for security problems. It can be done, but the solution is complex, and probably not something you want to attempt on your own. (Briefly, have the daemon listen to commands on a socket; create a user-space program which knows how to talk to the socket, and build some sort of authorization and authentication so the daemon knows whom it's talking to and can verify that this user is allowed to connect to this display.)
The drop-dead simple solution is to run this from your desktop environment's startup scripts instead. Most desktops have something like "session start-up items" or "autorun on login" hooks.
I'm not running linux and can't check now but the steps to daemonize a process are to close stdin stdout stderr change current working directory to / and to fork twice and setsid so that current process is a new session leader.
adding something like this at the beginning, before running, first thing to check is exec command creates a new session leader process with ps -Cbash -o sid,pgid,pid,ppid,comm,args
# checking if current process is a session leader to avoid infinite call
if [[ $(ps -p $$ -osid=) != $$ ]]; then
( cd / ; exec setsid /bin/bash /home/rest.sh & ) </dev/null 1>&0 2>&0 &
exit
fi
I'm trying to get Upstart sending me e-mails when a process is respawned.
So, following upstart stanzas, here's my upstart script for ntpd service (just as an example):
/etc/init/ntpd.conf
### ntpd
script
mail -s "ntpd Service Respawned" my_email#gmail.com
control + D
end script
respawn
exec /etc/init.d/ntpd start
Then, I reload the process (initctl reload ntpd) in order to get upstart to reload ntpd.conf's config. Then kill -9 the process to force its respawn.
Here's /var/log/message.log:
init: ntpd main process (12446) killed by KILL signal
init: ntpd main process ended, respawning
And the e-mail is never sent. I've tried with post-start and exec but it doesn't work either.
Any advice?
echo "ntpd Service Respawned" | mail -s "ntpd Service Respawned" my_email#gmail.com
Try with this.
Just solved this one.
What I did was add the following in my Upstart script:
respawn
pre-start script
mail -s "ntpd Service Respawned" my_address#gmail.com
control + D
end script
exec /etc/init.d/ntpd start
That works like a charm.
I think Upstart does pay much attention to the statements order.
Thanks!!!
I have these rake tasks that will occasionally fail. I want to use monit to monitor them and to restart them if necessary.
I have read the other ruby/monit threads on StackOverflow. My case is different in that these programs require my Rails environment in order to work. That's why I have them as rake tasks now.
Here is one of the tasks I need to monitor, in it's entirety:
task(process_updates: :environment) do
`echo "#{Process.pid}" > #{Rails.root}/log/process_alerts.pid`
`echo "#{Process.ppid}" > #{Rails.root}/log/process_alerts.ppid`
SynchronizationService::process_alerts
end
My question is, do I leave this as a rake task, since SynchronizationService::process_alerts requires the Rails environment to work? Or is there some other wrapper I should invoke and then just run some *.rb file?
Monit can check for running pid, since you're creating pid when you run task you can create a monit config which should look something like this:
check process alerts with pidfile RAILSROOT/log/process_alerts.pid
start program = "cd PATH_TO_APP; rake YOURTASK" with timeout 120 seconds
alert your#mail.com on { nonexist, timeout }
Of course RAILSROOT, PATH_TO_APP, YOURTASK should correspond to your paths/rake task.
Monit then will check for running process in system using the pidfile value and will start the process using start program command if it can't find running process.
I want to control many different Sinatra apps from a central (Sinatra) app.
The problem I have is no matter which way I exec/spawn/fork the call to start it, I cannot get the pid of the Sinatra server so that I can kill (:int) it?
This is due to my shell exec string, which contains a few other commands first, so I get the pid of the first.
My command string is like
command = source ~/.profile; rbenv #{ver}; some_env=1234 ruby app.rb
So I get the pid of the sh process of the sourcing command.
The question is, how can I get the pid of the ruby command launched above?
I am currently using spawn, but have tried most others as well, but I don't think that is the problem!?
pid = Process.spawn(command)
pid # => 1234
The ruby app itself starts
$ ps aux
1234 sh -c . ~/.profile; shell_script
4567 shell_script
I want to know 4567!?
There's no easy way of getting your '4567', but you should be able to make your process have the same pid as Process.spawn returns.
Try ending your command with an exec rather than a straight call to ruby, i.e.:
source ~/.profile; rbenv #{ver}; export some_env=1234; exec ruby app.rb
you can check whether the process "shell_script" is a child of "sh -c . ~/.profile; shell_script".you can check this through "ps -axgf" command.
if it is a parent then u can use the group id of pid 1234 (get it form the output of ps -axgf) to kill the child with pid 4567 using this command .
kill -9 -1234(assumming 1234 is the group id)