How can I tell unicorn to understand Heroku's signals? - ruby

Question

Perhaps you've seen this...
2012-03-07T15:36:25+00:00 heroku[web.1]: Stopping process with SIGTERM
2012-03-07T15:36:36+00:00 heroku[web.1]: Stopping process with SIGKILL
2012-03-07T15:36:36+00:00 heroku[web.1]: Error R12 (Exit timeout) -> Process failed to exit within 10 seconds of SIGTERM
2012-03-07T15:36:38+00:00 heroku[web.1]: Process exited with status 137
This is a well known problem when running unicorn on heroku...
heroku uses SIGTERM for graceful shutdown
unicorn uses SIGTERM for quick shutdown
Can I tell heroku to send SIGQUIT? Or can I tell unicorn to treat SIGTERM as graceful shutdown?

Answer 1

Heroku now provides instruction for this here:
https://blog.heroku.com/archives/2013/2/27/unicorn_rails
Their suggested unicorn.rb file is:
# config/unicorn.rb
worker_processes 3
timeout 30
preload_app true
before_fork do |server, worker|
Signal.trap 'TERM' do
puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
Process.kill 'QUIT', Process.pid
end
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
end
after_fork do |server, worker|
Signal.trap 'TERM' do
puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to sent QUIT'
end
defined?(ActiveRecord::Base) and
ActiveRecord::Base.establish_connection
end

Answer 2

This is a hack, but I've successfully created a unicorn config file that traps the TERM signal, preventing unicorn from receiving it and performing its quick shutdown. My signal handler then sends QUIT signal back to itself to trigger the unicorn graceful shutdown.
Tested with Ruby 1.9.2, Unicorn 4.0.1 and 4.2.1, Mac OS X.
listen 9292
worker_processes 1
# This is a hack. The code is run with 'before_fork' so it runs
# *after* Unicorn installs its own TERM signal handler (which makes
# this highly dependent on the Unicorn implementation details).
#
# We install our own signal handler for TERM and simply re-send a QUIT
# signal to our self.
before_fork do |_server, _worker|
Signal.trap 'TERM' do
puts 'intercepting TERM and sending myself QUIT instead'
Process.kill 'QUIT', Process.pid
end
end
One concern is that (I believe) this signal handler is inherited by worker processes. But, the worker process installs its own TERM handler, which should overwrite this one, so I would not expect any issue. (See Unicorn::HttpServer#init_worker_process # lib/unicorn/http_server.rb:551.
Edit: one more detail, this block that installs the signal handler will run once per worker process (because before_fork), but this merely redundant and won't affect anything.