I daemonized a Ruby scheduler script (using Rufus) with Rufus-Scheduler DaemonKit and I'm trying to trap the TERM or INT signals to have the application try to save state before quitting.
DaemonKit has its own trap_state (private) method and it catches the signal before the daemon script so even though I have this block, it doesn't do much.
DaemonKit::Application.running! do |config|
surprise = Surprise.new(interval, frequency, false)
surprise.start
config.trap( 'SIGINT' ) do #tried INT and TERM as well
puts 'Exiting'
surprise.stop
File.delete($lock)
end
end
As a side effect (maybe a mistake in my implementation ?) after sigterm the .rufus lockfile is still there
The behavior on ctrl-c right now is this
[daemon-kit]: DaemonKit (0.3.1) booted, now running surprise
log writing failed. can't be called from trap context
[daemon-kit]: Running signal traps for INT
log writing failed. can't be called from trap context
[daemon-kit]: Running shutdown hooks
log writing failed. can't be called from trap context
[daemon-kit]: Shutting down surprise
The start method is a pretty simple schedule
def start
#scheduler = Rufus::Scheduler.new(:lockfile => $lock)
#scheduler.every '1d', :first_at => #first, :overlap => false do |job|
... # some work
end
#scheduler.join
end
def stop
# save state
#scheduler.shutdown
end
Looking at your own answer, and the following code you pasted:
def start
#scheduler = Rufus::Scheduler.new(:lockfile => $lock)
# ...
#scheduler.join # <- NOT NEEDED
end
DaemonKit's DaemonKit::Application.running! block actually never finishes running, so you could safely skip calling #join on any thread.
We should work on making this use-case more clear, as I would love see it used more widely for this kinda work.
So it's very simple, I need to configure the trap proc (or block in my case) BEFORE I run the scheduler in the start method. Not feeling very clever right about now, but the following code works as expected. For reference, the set_trap is private in DK but the public trap method overrides the defaults that come with the DK startup.
DaemonKit::Application.running! do |config|
surprise = Surprise.new(interval, frequency, false)
config.trap("TERM") { surprise.stop }
config.trap( "INT" ) { surprise.stop }
surprise.start
end
Interestingly I saw this line on startup that I hadn't noticed before
[daemon-kit]: Trapping SIGINT signals not supported on this platform
INT and TERM both work though
Related
I'm trying to understand Ruby's traps for standard signals.
In specific, I'm trying to have multiple signal handlers ("traps") for the same signal. It seems impossible. Here's a super simplified code to demonstrate the problem:
file traps.rb:
should_stop = false
Signal.trap 'INT' do
# won't be executed :(
puts 'int --> A'
should_stop = true
end
Signal.trap 'INT' do
# will be executed
puts 'int --> B'
should_stop = true
end
times = 0
until should_stop
puts 'waiting to stop'
sleep 1
times += 1
break if times >= 5
end
puts 'done'
Run the code:
ruby traps.rb
Output without pressing CTRL+C:
waiting to stop
waiting to stop
waiting to stop
waiting to stop
waiting to stop
done
Output with pressing CTRL+C after 2 seconds:
waiting to stop
waiting to stop
^Cint --> B
done
It seems that only the last signal trap to be declared is the one which would be executed.
Is this behavior by design?
If not, how can we have multiple handlers executed to the same signal?
The main reason for asking this is that third party libraries might add their trap to a signal.
If we have two different third party libraries that add their trap to the same signal, only one of them would actually be executed. That's where the fun begins :(
It seems that only the last signal trap to be declared is the one which would be executed.
Is this behavior by design?
It is not very explicit in the documentation of Signal::trap, but it is by design:
The command or block specifies code to be run when the signal is raised.
Note the use of the singular, and the absence of any mention of something like "The command or block is added to the list of trap handlers to be run when the signal is raised."
It becomes clearer if you look at the POSIX trap shell builtin after which Signal::trap is modeled:
The action of trap shall override a previous action (either default action or one explicitly set).
The POSIX sigaction function which is the C equivalent to trap says more or less the same thing. Note, however, that the sigaction function also gives a way of retrieving the function pointer to the old action, so theoretically, you could set the action to a function pointer to a new action which uses the function pointer to the old action to call the old action as part of itself, thus in some way chaining the actions.
Note, however, that this would require the old and new action cooperate in some way. Also note that this mode of operation is not modeled by POSIX trap.
If not, how can we have multiple handlers executed to the same signal?
From the documentation:
trap returns the previous handler for the given signal.
So, Signal::trap implements the behavior from sigaction that gives you access to the "old" handler. You could save that old handler somewhere and chain the calls by explicitly calling it from your new handler.
Like with sigaction, this requires some form of cooperation between the handlers.
Combining the answer from Jörg W Mittag and the documentation, here's a simplified solution:
# file traps.rb
should_stop = false
Signal.trap('INT') do
puts 'int --> A'
should_stop = true
end
$prev_trap = Signal.trap('INT') do
puts 'int --> B'
should_stop = true
$prev_trap&.call
end
times = 0
until should_stop
puts 'waiting to stop'
sleep 1
times += 1
break if times >= 5
end
puts 'done'
After running ruby traps.rb, and pressing CTRL+C after 3 seconds, the output looks like this:
waiting to stop
waiting to stop
waiting to stop
^Cint --> B
int --> A
done
I am developing a long-running program in Ruby. I am writing some integration tests for this. These tests need to kill or stop the program after starting it; otherwise the tests hang.
For example, with a file bin/runner
#!/usr/bin/env ruby
while true do
puts "Hello World"
sleep 10
end
The (integration) test would be:
class RunReflectorTest < TestCase
test "it prints a welcome message over and over" do
out, err = capture_subprocess_io do
system "bin/runner"
end
assert_empty err
assert_includes out, "Hello World"
end
end
Only, obviously, this will not work; the test starts and never stops, because the system call never ends.
How should I tackle this? Is the problem in system itself, and would Kernel#spawn provide a solution? If so, how? Somehow the following keeps the out empty:
class RunReflectorTest < TestCase
test "it prints a welcome message over and over" do
out, err = capture_subprocess_io do
pid = spawn "bin/runner"
sleep 2
Process.kill pid
end
assert_empty err
assert_includes out, "Hello World"
end
end
. This direction also seems like it will cause a lot of timing-issues (and slow tests). Ideally, a reader would follow the stream of STDOUT and let the test pass as soon as the string is encountered and then immediately kill the subprocess. I cannot find how to do this with Process.
Test Behavior, Not Language Features
First, what you're doing is a TDD anti-pattern. Tests should focus on behaviors of methods or objects, not on language features like loops. If you must test a loop, construct a test that checks for a useful behavior like "entering an invalid response results in a re-prompt." There's almost no utility in checking that a loop loops forever.
However, you might decide to test a long-running process by checking to see:
If it's still running after t time.
If it's performed at least i iterations.
If a loop exits properly given certain input or upon reaching a boundary condition.
Use Timeouts or Signals to End Testing
Second, if you decide to do it anyway, you can just escape the block with Timeout::timeout. For example:
require 'timeout'
# Terminates block
Timeout::timeout(3) { `sleep 300` }
This is quick and easy. However, note that using timeout doesn't actually signal the process. If you run this a few times, you'll notice that sleep is still running multiple times as a system process.
It's better is to signal the process when you want to exit with Process::kill, ensuring that you clean up after yourself. For example:
pid = spawn 'sleep 300'
Process::kill 'TERM', pid
sleep 3
Process::wait pid
Aside from resource issues, this is a better approach when you're spawning something stateful and don't want to pollute the independence of your tests. You should almost always kill long-running (or infinite) processes in your test teardown whenever you can.
Ideally, a reader would follow the stream of STDOUT and let the test pass as soon as the string is encountered and then immediately kill the subprocess. I cannot find how to do this with Process.
You can redirect stdout of spawned process to any file descriptor by specifying out option
pid = spawn(command, :out=>"/dev/null") # write mode
Documentation
Example of redirection
With the answer from CodeGnome on how to use Timeout::timeout and the answer from andyconhin on how to redirect Process::spawn IO, I came up with two Minitest helpers that can be used as follows:
it "runs a deamon" do
wait_for(timeout: 2) do
wait_for_spawned_io(regexp: /Hello World/, command: ["bin/runner"])
end
end
The helpers are:
def wait_for(timeout: 1, &block)
Timeout::timeout(timeout) do
yield block
end
rescue Timeout::Error
flunk "Test did not pass within #{timeout} seconds"
end
def wait_for_spawned_io(regexp: //, command: [])
buffer = ""
begin
read_pipe, write_pipe = IO.pipe
pid = Process.spawn(command.shelljoin, out: write_pipe, err: write_pipe)
loop do
buffer << read_pipe.readpartial(1000)
break if regexp =~ buffer
end
ensure
read_pipe.close
write_pipe.close
Process.kill("INT", pid)
end
buffer
end
These can be used in a test which allows me to start a subprocess, capture the STDOUT and as soon as it matches the test Regular Expression, it passes, else it will wait 'till timeout and flunk (fail the test).
The loop will capture output and pass the test once it sees matching output. It uses a IO.pipe because that is most transparant for subprocesses (and their children) to write to.
I doubt this will work on Windows. And it needs some cleaning up of the wait_for_spawned_io which is doing slightly too much IMO. Antoher problem is that the Process.kill('INT') might not reach the children which are orphaned but still running after this test has ran. I need to find a way to ensure the entire subtree of processes is killed.
I have code that is running on a server, before the server is hard shut down, a signal SIGTERM is sent to let my code know it needs to clean up. I want to run code when this happens and send the signal back to the same program so any other code that needs to clean up can do so. I do not want to trap the signal or change signal behavior, I only need to run something before the rest of my program interprets the SIGTERM.
Currently I can do something like
Signal.trap('TERM') do
puts "Graceful shutdown"
exit
end
but it doesn't work if multiple pieces of code in the same app try to do the same thing. For example:
Signal.trap('TERM') do
puts "Graceful shutdown"
exit
end
Signal.trap('TERM') do
puts "Another graceful shutdown"
exit
end
You will only ever see "Another graceful shutdown" and the first code block will not run.
Ideally I would be able to invoke current behavior with something like:
Signal.trap('TERM') do
puts "another graceful shutdown"
super
end
But this doesn't work for obvious reasons. So the question is this: how can I run code when i get a SIGTERM without trapping it and preventing other code from doing the same?
Signal.trap returns the previous handler so you can do something like
def prepend_handler(signal, &handler)
previous = Signal.trap(signal) do
previous = -> { raise SignalException, signal} unless previous.respond_to?(:call)
handler.call(previous)
end
end
prepend_handler("TERM") do |old|
do_something
old.call
end
The respond_to? business is because a handler is either a callable or a string (the string values are documented here). Unless you use those string handlers yourself you are most likely to run into 'DEFAULT', i.e. the default ruby behaviour
I'm trying to create an app in ruby which can be started from command line and it does two things: runs a continous job (loop with sleep which runs some action [remote feed parsing]) with one thread and sinatra in a second thread. My code (simplified) looks like that:
require 'sinatra'
class MyApp < Sinatra::Base
get '/' do
"Hello!"
end
end
threads = []
threads << Thread.new do
loop do
# do something heavy
sleep 10
end
end
threads << Thread.new do
MyApp.run!
end
threads.each { |t| t.join }
The above code actually does it's job very well - the sinatra app is started an available under 4567 port and the do something heavy task is beeing fired each 10 seconds. However, i'm not able to kill that script.
I'm running it with ruby app.rb but killing it with ctrl + c is not working. It kills just the sinatra thread but the second one is still running and, to stop the script, i need to close the terminal window.
I was trying to kill all the threads on SIGNINT but it's also not working as expected
trap "SIGINT" do
puts "Exiting"
threads.each { |t| Thread.kill t }
exit 130
end
Can you help me with this? Thanks in advance.
To trap ctrl-c, change "SIGINT" to "INT".
trap("INT") {
puts "trapping"
threads.each{|t|
puts "killing"
Thread.kill t
}
}
To configure Sinatra to skip catching traps:
class MyApp < Sinatra::Base
configure do
set :traps, false
end
...
Reference: Ruby Signal module
To list the available Ruby signals: Signal.list.keys
Reference: Sinatra Intro
(When I run your code and trap INT, I do get a Sinatra socket warning "Already in use". I presume that's fine for your purposes, or you can solve that by doing a Sinatra graceful shutdown. See Sinatra - terminate server from request)
Late to the party, but Trap has one big disadvantage - it gets overriden by the webserver. For example, Puma sets several traps which basically makes your one never to be called.
The best workaround is to use at_exit which can be defined multiple times and Ruby makes sure all blocks are called. I haven't tested this if it would work for your case tho.
Heroku may send a SIGTERM to your application for various reasons, so I have created a handler to take care of some cleanup in case this happens. Some googling hasn't yielded any answers or examples on how to test this in RSpec. Here's the basic code:
Signal.trap('TERM') do
cleanup
end
def cleanup
puts "doing some cleanup stuff"
...
exit
end
What's the best way to test that this cleanup method is called when the program receives a SIGTERM?
Send the signal to RSpec with Process.kill 'TERM', 0 and test that the handler is called. It's true that if the signal isn't trapped the test will crash rather than nicely reporting a failure, but at least you'll know there's a problem in your code.
For example:
class SignalHandler
def self.trap_signals
Signal.trap('TERM') { term_handler }
end
def self.term_handler
# ...
end
end
describe SignalHandler do
describe '#trap_signals' do
it "traps TERM" do
# The MRI default TERM handler does not cause RSpec to exit with an error.
# Use the system default TERM handler instead, which does kill RSpec.
# If you test a different signal you might not need to do this,
# or you might need to install a different signal's handler.
old_signal_handler = Signal.trap 'TERM', 'SYSTEM_DEFAULT'
SignalHandler.trap_signals
expect(SignalHandler).to receive(:term_handler).with no_args
Process.kill 'TERM', 0 # Send the signal to ourself
# Put the Ruby default signal handler back in case it matters to other tests
Signal.trap 'TERM', old_signal_handler
end
end
end
I merely tested that the handler was called, but you could equally well test a side effect of the handler.