I have a plain ruby application (it's not a Web app, so not using a pre-existing platform like rails, sinatra...) processing data continuously.
I plan to deploy it with Capistrano and simply start it with the ruby command. The problem is that I get data by batches, and it can take few minutes to process them.
When I deploy a new version I would like to introduce a soft restart, meaning that the app will be first notify about the new deploy: so it can finish the batch, and then say 'I m ready for an update' (the deployment script will wait for that message).
Is there any Gem for that? Maybe Capistrano includes that option?
Allow the application to trap POSIX signals. Take a look at the Signal class.
If you send a kill <signal type> to the process, any signal handlers will be invoked, regardless of what the process is currently doing. You can, for example, set some sort of flag that is checked at a sensible point in your logic (at the end of a run loop, for example), terminating the process if that flag is set. There are many signals you can respond to, but SIGHUP or one of the SIGUSR signals probably make sense for what you're doing... you can respond to whatever signal you like in whatever way you like, but it makes sense to allow the default behaviour for most of the typically handled ones (like SIGTERM and SIGKILL). For really complex stuff, you can actually accept a coded series of signals to trigger particular events too.
Signal.trap("HUP") do
puts "Huh?"
end
loop do
puts "Looping..."
sleep 2
end
Output
[chris#chipbook:~%] ruby sig_demo.rb
Looping...
Looping...
Looping...
Looping...
Looping...
Huh?
Looping...
Looping...
Looping...
Huh?
Looping...
Looping...
Looping...
Because in another terminal window I'd done:
[chris#chipbook:/usr/local%] ps aux | grep ruby
chris 69487 0.0 0.0 2425480 188 s005 R+ 11:26pm 0:00.00 grep ruby
chris 69462 0.0 0.1 2440224 4060 s004 S+ 11:26pm 0:00.03 ruby sig_demo.rb
[chris#chipbook:/usr/local%] kill -HUP 69462
[chris#chipbook:/usr/local%] kill -HUP 69462
Related
I need to call commands sporadically in my ruby app, most commands will end quite quickly, but some will run for longer (like playing a song) and I need to be able to kill that process if wanted.
The commands are called in a sub thread as not to lock the rest of the app.
I also need to store info on running commands (so I know what to kill if needed), but I also need to send out info when the command ended.
The problems:
wait or waitpid doesn't work (they never return)
I tried to use trap, which detects when the command ended, but it only works for the latest started command.
Ruby leaves the commands as zombie process (until killing the app) and trying to use the kill(both Process.kill and calling the command directly) doesn't remove them. (Perhaps the reason to why wait doesn't work??)
# Play sound (player can be either 'mplayer' or 'aplay' )
pid = spawn player, sound_dir + file
Eddie::common[:commands][:sound_pids][pid] = file
Signal.trap("CLD") {
# This runs, but doesn't clean up zombies
Process.kill "KILL", pid
p = system 'kill ' + pid.to_s
Eddie::common[:commands][:sound_pids].delete pid
}
If I run this twice (once while the first command is running) it will look like this after both commands ended:
Eddie::common[:commands][:sound_pids] => {3018=>"firstfile.wav"}
and this is the result of ps
chris 3018 0.0 0.0 0 0 pts/5 Z+ 23:50 0:00 [aplay] <defunct>
chris 3486 0.2 0.0 0 0 pts/5 Z+ 23:51 0:00 [aplay] <defunct>
Added note: Using system to call the command doesn't leave a zombie process, but in turn it's not killable..
I need to execute a process that runs several admin system commands. I want to keep the sudo timestamp current while it runs in case the process run too long.
i have the following code, but it does not seem to work.
sudo_keep_alive = Thread.start do
def sudo
sleep 5.minutes
`sudo -v`
sudo
end
sudo
end
at_exit do
sudo_keep_alive.kill
end
Is there a convention for this?
UPDATE
The reason i cannot run the script has root, is there are other system commands the script runs that cannot run as root. Each command needs to be responsible for running it's own admin commands. The script can potentially take a considerable amount of time to run, so i simply wanted to keep the sudo timestamp fresh in the event a command needs it.
To answer your other question, there is a better way to run an asynchronous loop.
By using head-tail recursion (def sudo; do_something; sudo; end) you risk running into a SystemStackError at around 10000 calls (see How does your favorite language handle deep recursion? ).
Instead, just use a regular old ruby loop.
Thread::new do
loop do
sleep 300 # 5.minutes is not base ruby, it comes from ActiveSupport
call_my_function
end
end
As mentioned by David Unric, there is no need to kill the thread using at_exit, as your main process will automatically kill any active threads when it finishes.
Scrap all this and execute your ruby script as root instead.
$ sudo ruby myscript.rb
bundle exec thin start -p 3111 gives the following output:
Using rack adapter
Thin web server (v1.2.11 codename Bat-Shit Crazy)
Maximum connections set to 1024
Listening on 0.0.0.0:3111, CTRL+C to stop
^C
Ctrl-C doesn't do anything (SIGINT). Neither does kill (SIGTERM).
I've found a few references to this behavior, but no solutions. The problem seems to be either in eventmachine (bundled with latest thin), in ruby 1.9.2-r290, or in the linux kernel (Ubuntu 10.4 LTS, 2.6.38.3-linode32).
It happens with my project, but not with a brand new rails project.
References:
http://groups.google.com/group/thin-ruby/browse_thread/thread/4b7c28e8964b5001?fwc=2
My guess is that either something's tying up the EventMachine reactor loop preventing it from exiting, or something's trapping SIGINT.
As a simple example of the former, put this into config.ru and run with thin -p 4567 start:
require 'thin'
require 'sinatra'
require 'eventmachine'
get '/' do
"hello world"
end
run Sinatra::Application
EventMachine.schedule do
trap("INT") do
puts "Caught SIGINT"
EventMachine.stop # this is useless
# exit # this stops the EventMachine
end
i = 0
while i < 10
puts "EM Running"
i += 1
sleep 1
end
end
Without trapping the SIGINT, you get the same behavior as when trapping it and calling EM.stop. EM.stop (at least in the pure ruby version, which you can run with EVENTMACHINE_LIBRARY="pure_ruby" thin start) sets a flag that a stop has been requested, which is picked up inside the reactor loop. If the reactor loop is stuck on a step (as in the above case), then it won't exit.
So a couple options:
use the workaround above of trapping SIGINT and forcing an exit. This could leave connections in an unclean state, but they don't call it quick & dirty for nothing ;)
you could put the blocking code inside a Thread or a Fiber, which will allow the reactor to keep running.
look for long-running tasks or loops inside your code, and convert these to be EventMachine aware. em-http-request is a great library for external http requests, and em-synchrony has several other protocols (for database connections, tcp connection pools, etc.). In the above example, this is straightforward: EventMachine.add_periodic_timer(1) { puts "EM Running" }
In your actual code, this might be harder to track down, but look for any places where you spawn threads and join them, or large loops. A profiling tool can help show what code is running when you try to exit, and lastly you can try disabling various parts of the system and libraries to figure out where the culprit is.
Given a very simple ruby script:
child = fork do
system 'sleep 10000'
end
5.times do
sleep 1
puts "send kill to #{child}"
Process.kill("QUIT", child)
end
QUIT signal is just lost. Where does it go? Something with default handler which just ignores it?
How to send signal to all processes created by that fork? Is it possible to do that without searching for all child processes?
The problem is that the system call creates yet another child process running the given command in a subshell, so there are actually three processes running in your example. Additionally, the Ruby Kernel#system command is implemented via the standard C function system(3), which calls fork and exec to create the new process and (on most systems) ignores SIGINT and SIGQUIT, and blocks SIGCHLD.
If you simply call sleep(10000) instead of system("sleep 10000") then things should work as you expect. You can also trap SIGQUIT in the child to handle it gracefully:
child = fork do
Signal.trap("QUIT") { puts "CHILD: ok, quitting time!"; exit }
sleep(10000)
end
If you really need to use a "system" call from the child process then you might be better off using an explicit fork/exec pair (instead of the implicit ones in the system call), so that you can perform your own signal handling in the third forked child.
I think that you are sending signal to fork process corectly. I think that the problem is with the system command. System command creates new fork and waits until it ends and I think that this waiting is blocking your quit signal. If you run your example as test.rb you'll see three processes:
test.rb
test.rb
sleep 10000
If you send signal "TERM" or "KILL" instead of "QUIT" the second test.rb will die but sleep 10000 will continue!
i'm trying to set up a server for integration tests (specs actually) via ruby and can't figure out how to control the process.
so, what i'm trying to do is:
run a rake task for my gem that executes the integration specs
the task needs to first start a server (i use webrick) and then run the specs
after executing the specs it should kill the webrick so i'm not left with some unused background process
webrick is not a requirement, but it's included in the ruby standard library so being able to use it would be great.
hope anyone is able to help!
ps. i'm running on linux, so having this work for windows is not my main priority (right now).
The standard way is to use the system functions fork (to duplicate the current process), exec (to replace the current process by an executable file), and kill (to send a signal to a process to terminate it).
For example :
pid = fork do
# this code is run in the child process
# you can do anything here, like changing current directory or reopening STDOUT
exec "/path/to/executable"
end
# this code is run in the parent process
# do your stuffs
# kill it (other signals than TERM may be used, depending on the program you want
# to kill. The signal KILL will always work but the process won't be allowed
# to cleanup anything)
Process.kill "TERM", pid
# you have to wait for its termination, otherwise it will become a zombie process
# (or you can use Process.detach)
Process.wait pid
This should work on any Unix like system. Windows creates process in a different way.
I just had to do something similar and this is what I came up with. #Michael Witrant's answer got me started, but I changed some things like using Process.spawn instead of fork (newer and better).
# start spawns a process and returns the pid of the process
def start(exe)
puts "Starting #{exe}"
pid = spawn(exe)
# need to detach to avoid daemon processes: http://www.ruby-doc.org/core-2.1.3/Process.html#method-c-detach
Process.detach(pid)
return pid
end
# This will kill off all the programs we started
def killall(pids)
pids.each do |pid|
puts "Killing #{pid}"
# kill it (other signals than TERM may be used, depending on the program you want
# to kill. The signal KILL will always work but the process won't be allowed
# to cleanup anything)
begin
Process.kill "TERM", pid
# you have to wait for its termination, otherwise it will become a zombie process
# (or you can use Process.detach)
Process.wait pid
rescue => ex
puts "ERROR: Couldn't kill #{pid}. #{ex.class}=#{ex.message}"
end
end
end
# Now we can start processes and keep the pids for killing them later
pids = []
pids << start('./someprogram')
# Do whatever you want here, run your tests, etc.
# When you're done, be sure to kill of the processes you spawned
killall(pids)
That's about all she wrote, give it a try and let me know how it works.
I have tried fork, but it has kind of problems when ActiveRecord is involved in both the processes. I would suggest Spawn plugin (http://github.com/tra/spawn). It does fork only but takes care of ActiveRecord.