I'm a little confused about my deploy strategy here, when deploying under what circumstances would I want to send a reload signal to unicorn? For example in my case it would be like:
sudo kill -s USR2 `cat /home/deploy/apps/my_app/current/tmp/pids/unicorn.pid`
I've been deploying my apps by killing that pid, then starting unicorn again via something like:
bundle exec unicorn -c config/unicorn/production.rb -E production -D
I'm just wondering why I'd want to use reload? Can I gain any performance for my deployment by doing so?
When you kill unicorn you cause downtime, until unicorn can start back up. When you use the USR2 signal, unicorn starts new workers first, then once they are running, it kills the old workers. It's basically all about removing the need to "turn off" unicorn.
Note, the assumption is that you have the documented before_fork hook in your unicorn configuration, in order to handle the killing of the old workers, should an ".oldbin" file be found, containing the PID of the old unicorn process:
before_fork do |server, worker|
# a .oldbin file exists if unicorn was gracefully restarted with a USR2 signal
# we should terminate the old process now that we're up and running
old_pid = "#{pids_dir}/unicorn.pid.oldbin"
if File.exists?(old_pid)
begin
Process.kill("QUIT", File.read(old_pid).to_i)
rescue Errno::ENOENT, Errno::ESRCH
# someone else did our job for us
end
end
end
Related
I have a server which is running Unicorn and nginx. I've just done a git pull to get the most recent updates to my own code, and now I need to restart it.
In setting it up, I followed this guide, and did these steps:
Start Unicorn
cd /var/www/my_server
unicorn -c unicorn.rb -D
Restart nginx
service nginx restart
I now need to know how to restart it. Ideally, it should be a quick process, so that my server doesn't have any/much downtime when doing this in the future.
EDIT: I tried a few other things as suggested elsewhere, such as killall ruby, and rebooting my server. Now I'm at a point where I've done the above, it doesn't give me any errors, but when I try and load a page, it doesn't respond, and likely eventually times out (though I didn't leave it that long). If I stop nginx, it says "connection refused", so it's obvious that nginx is working, but for some reason it's not able to connect to Unicorn.
EDIT: On a whim, I typed in just unicorn and it seems to be having an issue with my project - missing gems. Makes sense. So that first edit is no longer an issue, I'm still interested in the most elegant way of restarting it.
You can try send a HUP signal to the master process, doing
kill -HUP <processID>
HUP - reloads config file and gracefully restart all workers.
If the "preload_app" directive is false (the default), then workers
will also pick up any application code changes when restarted. If
"preload_app" is true, then application code changes will have no
effect; USR2 + QUIT (see below) must be used to load newer code in
this case. When reloading the application, +Gem.refresh+ will
be called so updated code for your application can pick up newly
installed RubyGems. It is not recommended that you uninstall
libraries your application depends on while Unicorn is running,
as respawned workers may enter a spawn loop when they fail to
load an uninstalled dependency.
If you want to read more about unicorn's signals you can read more here
I would recommend to use the init script they provide
One example init script is distributed with unicorn:
http://unicorn.bogomips.org/examples/init.sh
because there you have the upgrade method which takes care of your
use case, restarting the app with zero downtime.
...
upgrade)
if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
then
n=$TIMEOUT
while test -s $old_pid && test $n -ge 0
do
printf '.' && sleep 1 && n=$(( $n - 1 ))
done
echo
if test $n -lt 0 && test -s $old_pid
then
echo >&2 "$old_pid still exists after $TIMEOUT seconds"
exit 1
fi
exit 0
fi
...
On Ubuntu 14.04 try
sudo service unicorn restart
If you are using capistrano
add this in deploy.rb
desc "Zero-downtime restart of Unicorn"
task :restart, :except => { :no_release => true } do
run "kill -s USR2 unicorn_pid"
end
I pretty much need what my Question statement says, currently I have a Capistrano task that looks like this:
desc "stops private pub"
task :stop_private_pub do
run "kill -9 $(lsof -i:9292 -t)"
end
before 'deploy', 'servers:stop_private_pub'
...And it works well when in fact the process in port 9292 is running, The problem is that when the process Isn't running this task will FAIL! and it Halts the whole Deployment Process!
I'm not a UNIX Shell expert, nor am I a Capistrano master, so.. I really need help improving this Capistrano Task, Is there a way to kill -9 only if the process is running?
How can I do this?
Thanks in advance.
You could use Capistrano's capture command (in V3 at least, probably a V2 equivalent) to grab the output from your lsof command, and then only if you get a PID run the kill command.
pid = capture 'lsof', '-i:9292', '-t'
if pid # ensure it's valid here
run "kill -9 #{pid}" # make darn sure pid is an integer if you embed it
end
You could also do:
run "kill -9 $(lsof -i:9292 -t); true"
or add on error continue:
task :stop_web, :roles => :app, :on_error => :continue do
run "dosomething.sh; true"
end
I'm trying to start or restart Unicorn when I do cap production deploy with Capistrano 3.0.1. I have some examples that I got working with Capistrano 2.x using something like:
namespace :unicorn do
desc "Start unicorn for this application"
task :start do
run "cd #{current_path} && bundle exec unicorn -c /etc/unicorn/myapp.conf.rb -D"
end
end
But when I try and use run in the deploy.rb for Capistrano 3.x I get an undefined method error.
Here are a couple of the things I tried:
# within the :deploy I created a task that I called after :finished
namespace :deploy do
...
task :unicorn do
run "cd #{current_path} && bundle exec unicorn -c /etc/unicorn/myapp.conf.rb -D"
end
after :finished, 'deploy:unicorn'
end
I have also tried putting the run within the :restart task
namespace :deploy do
desc 'Restart application'
task :restart do
on roles(:app), in: :sequence, wait: 5 do
# Your restart mechanism here, for example:
# execute :touch, release_path.join('tmp/restart.txt')
execute :run, "cd #{current_path} && bundle exec unicorn -c /etc/unicorn/deployrails.conf.rb -D"
end
end
If I use just run "cd ... " then I'll get awrong number of arguments (1 for 0)` in the local shell.
I can start the unicorn process with unicorn -c /etc/unicorn/deployrails.conf.rb -D from my ssh'd VM shell.
I can kill the master Unicorn process from the VM shell using kill USR2, but even though the process is killed I get an error. I can then start the process again using unicorn -c ...
$ kill USR2 58798
bash: kill: USR2: arguments must be process or job IDs
I'm very new to Ruby, Rails and Deployment in general. I have a VirtualBox setup with Ubuntu, Nginx, RVM and Unicorn, I'm pretty excited so far, but this one is really messing with me, any advice or insight is appreciated.
I'm using following code:
namespace :unicorn do
desc 'Stop Unicorn'
task :stop do
on roles(:app) do
if test("[ -f #{fetch(:unicorn_pid)} ]")
execute :kill, capture(:cat, fetch(:unicorn_pid))
end
end
end
desc 'Start Unicorn'
task :start do
on roles(:app) do
within current_path do
with rails_env: fetch(:rails_env) do
execute :bundle, "exec unicorn -c #{fetch(:unicorn_config)} -D"
end
end
end
end
desc 'Reload Unicorn without killing master process'
task :reload do
on roles(:app) do
if test("[ -f #{fetch(:unicorn_pid)} ]")
execute :kill, '-s USR2', capture(:cat, fetch(:unicorn_pid))
else
error 'Unicorn process not running'
end
end
end
desc 'Restart Unicorn'
task :restart
before :restart, :stop
before :restart, :start
end
Can't say anything specific about capistrano 3(i use 2), but i think this may help: How to run shell commands on server in Capistrano v3?.
Also i can share some unicorn-related experience, hope this helps.
I assume you want 24/7 graceful restart approach.
Let's consult unicorn documentation for this matter. For graceful restart(without downtime) you can use two strategies:
kill -HUP unicorn_master_pid It requires your app to have 'preload_app' directive disabled, increasing starting time of every one of unicorn workers. If you can live with that - go on, it's your call.
kill -USR2 unicorn_master_pid
kill -QUIT unicorn_master_pid
More sophisticated approach, when you're already dealing with performance concerns. Basically it will reexecute unicorn master process, then you should kill it's predecessor. Theoretically you can deal with usr2-sleep-quit approach. Another(and the right one, i may say) way is to use unicorn before_fork hook, it will be executed, when new master process will be spawned and will try to for new children for itself.
You can put something like this in config/unicorn.rb:
# Where to drop a pidfile
pid project_home + '/tmp/pids/unicorn.pid'
before_fork do |server, worker|
server.logger.info("worker=#{worker.nr} spawning in #{Dir.pwd}")
# graceful shutdown.
old_pid_file = project_home + '/tmp/pids/unicorn.pid.oldbin'
if File.exists?(old_pid_file) && server.pid != old_pid_file
begin
old_pid = File.read(old_pid_file).to_i
server.logger.info("sending QUIT to #{old_pid}")
# we're killing old unicorn master right there
Process.kill("QUIT", old_pid)
rescue Errno::ENOENT, Errno::ESRCH
# someone else did our job for us
end
end
end
It's more or less safe to kill old unicorn when the new one is ready to fork workers. You won't get any downtime that way and old unicorn will wait for it's workers to finish.
And one more thing - you may want to put it under runit or init supervision. That way your capistrano tasks will be as simple as sv reload unicorn, restart unicorn or /etc/init.d/unicorn restart. This is good thing.
I'm just going to throw this in the ring: capistrano 3 unicorn gem
However, my issue with the gem (and any approach NOT using an init.d script), is that you may now have two methods of managing your unicorn process. One with this cap task and one with init.d scripts. Things like Monit / God will get confused and you may spend hours debugging why you have two unicorn processes trying to start, and then you may start to hate life.
Currently I'm using the following with capistrano 3 and unicorn:
namespace :unicorn do
desc 'Restart application'
task :restart do
on roles(:app) do
puts "restarting unicorn..."
execute "sudo /etc/init.d/unicorn_#{fetch(:application)} restart"
sleep 5
puts "whats running now, eh unicorn?"
execute "ps aux | grep unicorn"
end
end
end
The above is combined with the preload_app: true and the before_fork and after_fork statements mentioned by #dredozubov
Note I've named my init.d/unicorn script unicorn_application_name.
The new worker that is started should kill off the old one. You can see with ps aux | grep unicorn that the old master hangs around for a few seconds before it disappears.
To view all caps:
cap -T
and it shows:
***
cap unicorn:add_worker # Add a worker (TTIN)
cap unicorn:duplicate # Duplicate Unicorn; alias of unicorn:re...
cap unicorn:legacy_restart # Legacy Restart (USR2 + QUIT); use this...
cap unicorn:reload # Reload Unicorn (HUP); use this when pr...
cap unicorn:remove_worker # Remove a worker (TTOU)
cap unicorn:restart # Restart Unicorn (USR2); use this when ...
cap unicorn:start # Start Unicorn
cap unicorn:stop # Stop Unicorn (QUIT)
***
So, to start unicorn in production:
cap production unicorn:start
and restart:
cap production unicorn:restart
PS do not forget to correct use gem capistrano3-unicorn
https://github.com/tablexi/capistrano3-unicorn
You can try to use native capistrano way as written here:
If preload_app:true and you need capistrano to cleanup your oldbin pid use:
after 'deploy:publishing', 'deploy:restart'
namespace :deploy do
task :restart do
invoke 'unicorn:legacy_restart'
end
end
Perhaps you've seen this...
2012-03-07T15:36:25+00:00 heroku[web.1]: Stopping process with SIGTERM
2012-03-07T15:36:36+00:00 heroku[web.1]: Stopping process with SIGKILL
2012-03-07T15:36:36+00:00 heroku[web.1]: Error R12 (Exit timeout) -> Process failed to exit within 10 seconds of SIGTERM
2012-03-07T15:36:38+00:00 heroku[web.1]: Process exited with status 137
This is a well known problem when running unicorn on heroku...
heroku uses SIGTERM for graceful shutdown
unicorn uses SIGTERM for quick shutdown
Can I tell heroku to send SIGQUIT? Or can I tell unicorn to treat SIGTERM as graceful shutdown?
Heroku now provides instruction for this here:
https://blog.heroku.com/archives/2013/2/27/unicorn_rails
Their suggested unicorn.rb file is:
# config/unicorn.rb
worker_processes 3
timeout 30
preload_app true
before_fork do |server, worker|
Signal.trap 'TERM' do
puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
Process.kill 'QUIT', Process.pid
end
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
end
after_fork do |server, worker|
Signal.trap 'TERM' do
puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to sent QUIT'
end
defined?(ActiveRecord::Base) and
ActiveRecord::Base.establish_connection
end
This is a hack, but I've successfully created a unicorn config file that traps the TERM signal, preventing unicorn from receiving it and performing its quick shutdown. My signal handler then sends QUIT signal back to itself to trigger the unicorn graceful shutdown.
Tested with Ruby 1.9.2, Unicorn 4.0.1 and 4.2.1, Mac OS X.
listen 9292
worker_processes 1
# This is a hack. The code is run with 'before_fork' so it runs
# *after* Unicorn installs its own TERM signal handler (which makes
# this highly dependent on the Unicorn implementation details).
#
# We install our own signal handler for TERM and simply re-send a QUIT
# signal to our self.
before_fork do |_server, _worker|
Signal.trap 'TERM' do
puts 'intercepting TERM and sending myself QUIT instead'
Process.kill 'QUIT', Process.pid
end
end
One concern is that (I believe) this signal handler is inherited by worker processes. But, the worker process installs its own TERM handler, which should overwrite this one, so I would not expect any issue. (See Unicorn::HttpServer#init_worker_process # lib/unicorn/http_server.rb:551.
Edit: one more detail, this block that installs the signal handler will run once per worker process (because before_fork), but this merely redundant and won't affect anything.
i'm trying to set up a server for integration tests (specs actually) via ruby and can't figure out how to control the process.
so, what i'm trying to do is:
run a rake task for my gem that executes the integration specs
the task needs to first start a server (i use webrick) and then run the specs
after executing the specs it should kill the webrick so i'm not left with some unused background process
webrick is not a requirement, but it's included in the ruby standard library so being able to use it would be great.
hope anyone is able to help!
ps. i'm running on linux, so having this work for windows is not my main priority (right now).
The standard way is to use the system functions fork (to duplicate the current process), exec (to replace the current process by an executable file), and kill (to send a signal to a process to terminate it).
For example :
pid = fork do
# this code is run in the child process
# you can do anything here, like changing current directory or reopening STDOUT
exec "/path/to/executable"
end
# this code is run in the parent process
# do your stuffs
# kill it (other signals than TERM may be used, depending on the program you want
# to kill. The signal KILL will always work but the process won't be allowed
# to cleanup anything)
Process.kill "TERM", pid
# you have to wait for its termination, otherwise it will become a zombie process
# (or you can use Process.detach)
Process.wait pid
This should work on any Unix like system. Windows creates process in a different way.
I just had to do something similar and this is what I came up with. #Michael Witrant's answer got me started, but I changed some things like using Process.spawn instead of fork (newer and better).
# start spawns a process and returns the pid of the process
def start(exe)
puts "Starting #{exe}"
pid = spawn(exe)
# need to detach to avoid daemon processes: http://www.ruby-doc.org/core-2.1.3/Process.html#method-c-detach
Process.detach(pid)
return pid
end
# This will kill off all the programs we started
def killall(pids)
pids.each do |pid|
puts "Killing #{pid}"
# kill it (other signals than TERM may be used, depending on the program you want
# to kill. The signal KILL will always work but the process won't be allowed
# to cleanup anything)
begin
Process.kill "TERM", pid
# you have to wait for its termination, otherwise it will become a zombie process
# (or you can use Process.detach)
Process.wait pid
rescue => ex
puts "ERROR: Couldn't kill #{pid}. #{ex.class}=#{ex.message}"
end
end
end
# Now we can start processes and keep the pids for killing them later
pids = []
pids << start('./someprogram')
# Do whatever you want here, run your tests, etc.
# When you're done, be sure to kill of the processes you spawned
killall(pids)
That's about all she wrote, give it a try and let me know how it works.
I have tried fork, but it has kind of problems when ActiveRecord is involved in both the processes. I would suggest Spawn plugin (http://github.com/tra/spawn). It does fork only but takes care of ActiveRecord.