How do I get my programm to display the backtrace / caller when killed ?
I have an issue with an infinite loop in a gem that isn't mine and need to know where the issue is to report it
def hello
puts 'hello'
end
def test
while true
sleep 2
hello
end
end
test
In this exemple, when a kill signal is sent to the programm, I wish to know what the programm what doing ( display the caller )
Currently all I get displayed is "Killed" on the output
I guess you need to catch a signal,
Did you try
https://gist.github.com/sauloperez/6592971
It is not possible to catch sigkill signals as they are sent to the kernel and not the process : https://major.io/2010/03/18/sigterm-vs-sigkill/
you can, however, use a "parent" process that watches the "children" and will react accordingly SIGKILL signal Handler
This means that I cannot display the backtrace, I will probably have to use a log file or something like that.
I suppose anti-virus have a different way of working to avoid the issue of being killed
Taken from https://robots.thoughtbot.com/using-gdb-to-inspect-a-running-ruby-process
As a last resort you can use gdb to connect to your running ruby process
gdb </path/to/ruby> <PID>
# inside ~/.gdbinit
define redirect_stdout
call rb_eval_string("$_old_stdout, $stdout = $stdout,
File.open('/tmp/ruby-debug.' + Process.pid.to_s, 'a'); $stdout.sync = true")
end
define ruby_eval
call(rb_p(rb_eval_string_protect($arg0,(int*)0)))
end
Related
I am developing a long-running program in Ruby. I am writing some integration tests for this. These tests need to kill or stop the program after starting it; otherwise the tests hang.
For example, with a file bin/runner
#!/usr/bin/env ruby
while true do
puts "Hello World"
sleep 10
end
The (integration) test would be:
class RunReflectorTest < TestCase
test "it prints a welcome message over and over" do
out, err = capture_subprocess_io do
system "bin/runner"
end
assert_empty err
assert_includes out, "Hello World"
end
end
Only, obviously, this will not work; the test starts and never stops, because the system call never ends.
How should I tackle this? Is the problem in system itself, and would Kernel#spawn provide a solution? If so, how? Somehow the following keeps the out empty:
class RunReflectorTest < TestCase
test "it prints a welcome message over and over" do
out, err = capture_subprocess_io do
pid = spawn "bin/runner"
sleep 2
Process.kill pid
end
assert_empty err
assert_includes out, "Hello World"
end
end
. This direction also seems like it will cause a lot of timing-issues (and slow tests). Ideally, a reader would follow the stream of STDOUT and let the test pass as soon as the string is encountered and then immediately kill the subprocess. I cannot find how to do this with Process.
Test Behavior, Not Language Features
First, what you're doing is a TDD anti-pattern. Tests should focus on behaviors of methods or objects, not on language features like loops. If you must test a loop, construct a test that checks for a useful behavior like "entering an invalid response results in a re-prompt." There's almost no utility in checking that a loop loops forever.
However, you might decide to test a long-running process by checking to see:
If it's still running after t time.
If it's performed at least i iterations.
If a loop exits properly given certain input or upon reaching a boundary condition.
Use Timeouts or Signals to End Testing
Second, if you decide to do it anyway, you can just escape the block with Timeout::timeout. For example:
require 'timeout'
# Terminates block
Timeout::timeout(3) { `sleep 300` }
This is quick and easy. However, note that using timeout doesn't actually signal the process. If you run this a few times, you'll notice that sleep is still running multiple times as a system process.
It's better is to signal the process when you want to exit with Process::kill, ensuring that you clean up after yourself. For example:
pid = spawn 'sleep 300'
Process::kill 'TERM', pid
sleep 3
Process::wait pid
Aside from resource issues, this is a better approach when you're spawning something stateful and don't want to pollute the independence of your tests. You should almost always kill long-running (or infinite) processes in your test teardown whenever you can.
Ideally, a reader would follow the stream of STDOUT and let the test pass as soon as the string is encountered and then immediately kill the subprocess. I cannot find how to do this with Process.
You can redirect stdout of spawned process to any file descriptor by specifying out option
pid = spawn(command, :out=>"/dev/null") # write mode
Documentation
Example of redirection
With the answer from CodeGnome on how to use Timeout::timeout and the answer from andyconhin on how to redirect Process::spawn IO, I came up with two Minitest helpers that can be used as follows:
it "runs a deamon" do
wait_for(timeout: 2) do
wait_for_spawned_io(regexp: /Hello World/, command: ["bin/runner"])
end
end
The helpers are:
def wait_for(timeout: 1, &block)
Timeout::timeout(timeout) do
yield block
end
rescue Timeout::Error
flunk "Test did not pass within #{timeout} seconds"
end
def wait_for_spawned_io(regexp: //, command: [])
buffer = ""
begin
read_pipe, write_pipe = IO.pipe
pid = Process.spawn(command.shelljoin, out: write_pipe, err: write_pipe)
loop do
buffer << read_pipe.readpartial(1000)
break if regexp =~ buffer
end
ensure
read_pipe.close
write_pipe.close
Process.kill("INT", pid)
end
buffer
end
These can be used in a test which allows me to start a subprocess, capture the STDOUT and as soon as it matches the test Regular Expression, it passes, else it will wait 'till timeout and flunk (fail the test).
The loop will capture output and pass the test once it sees matching output. It uses a IO.pipe because that is most transparant for subprocesses (and their children) to write to.
I doubt this will work on Windows. And it needs some cleaning up of the wait_for_spawned_io which is doing slightly too much IMO. Antoher problem is that the Process.kill('INT') might not reach the children which are orphaned but still running after this test has ran. I need to find a way to ensure the entire subtree of processes is killed.
I have code that is running on a server, before the server is hard shut down, a signal SIGTERM is sent to let my code know it needs to clean up. I want to run code when this happens and send the signal back to the same program so any other code that needs to clean up can do so. I do not want to trap the signal or change signal behavior, I only need to run something before the rest of my program interprets the SIGTERM.
Currently I can do something like
Signal.trap('TERM') do
puts "Graceful shutdown"
exit
end
but it doesn't work if multiple pieces of code in the same app try to do the same thing. For example:
Signal.trap('TERM') do
puts "Graceful shutdown"
exit
end
Signal.trap('TERM') do
puts "Another graceful shutdown"
exit
end
You will only ever see "Another graceful shutdown" and the first code block will not run.
Ideally I would be able to invoke current behavior with something like:
Signal.trap('TERM') do
puts "another graceful shutdown"
super
end
But this doesn't work for obvious reasons. So the question is this: how can I run code when i get a SIGTERM without trapping it and preventing other code from doing the same?
Signal.trap returns the previous handler so you can do something like
def prepend_handler(signal, &handler)
previous = Signal.trap(signal) do
previous = -> { raise SignalException, signal} unless previous.respond_to?(:call)
handler.call(previous)
end
end
prepend_handler("TERM") do |old|
do_something
old.call
end
The respond_to? business is because a handler is either a callable or a string (the string values are documented here). Unless you use those string handlers yourself you are most likely to run into 'DEFAULT', i.e. the default ruby behaviour
I daemonized a Ruby scheduler script (using Rufus) with Rufus-Scheduler DaemonKit and I'm trying to trap the TERM or INT signals to have the application try to save state before quitting.
DaemonKit has its own trap_state (private) method and it catches the signal before the daemon script so even though I have this block, it doesn't do much.
DaemonKit::Application.running! do |config|
surprise = Surprise.new(interval, frequency, false)
surprise.start
config.trap( 'SIGINT' ) do #tried INT and TERM as well
puts 'Exiting'
surprise.stop
File.delete($lock)
end
end
As a side effect (maybe a mistake in my implementation ?) after sigterm the .rufus lockfile is still there
The behavior on ctrl-c right now is this
[daemon-kit]: DaemonKit (0.3.1) booted, now running surprise
log writing failed. can't be called from trap context
[daemon-kit]: Running signal traps for INT
log writing failed. can't be called from trap context
[daemon-kit]: Running shutdown hooks
log writing failed. can't be called from trap context
[daemon-kit]: Shutting down surprise
The start method is a pretty simple schedule
def start
#scheduler = Rufus::Scheduler.new(:lockfile => $lock)
#scheduler.every '1d', :first_at => #first, :overlap => false do |job|
... # some work
end
#scheduler.join
end
def stop
# save state
#scheduler.shutdown
end
Looking at your own answer, and the following code you pasted:
def start
#scheduler = Rufus::Scheduler.new(:lockfile => $lock)
# ...
#scheduler.join # <- NOT NEEDED
end
DaemonKit's DaemonKit::Application.running! block actually never finishes running, so you could safely skip calling #join on any thread.
We should work on making this use-case more clear, as I would love see it used more widely for this kinda work.
So it's very simple, I need to configure the trap proc (or block in my case) BEFORE I run the scheduler in the start method. Not feeling very clever right about now, but the following code works as expected. For reference, the set_trap is private in DK but the public trap method overrides the defaults that come with the DK startup.
DaemonKit::Application.running! do |config|
surprise = Surprise.new(interval, frequency, false)
config.trap("TERM") { surprise.stop }
config.trap( "INT" ) { surprise.stop }
surprise.start
end
Interestingly I saw this line on startup that I hadn't noticed before
[daemon-kit]: Trapping SIGINT signals not supported on this platform
INT and TERM both work though
I am using the open4 gem to wrap system calls to a potentially long-running third-party command line tool. The tool may sometimes fail, keeping two processes busy, and partially blocking a pipeline, as the parent process is part of a pool of worker scripts (serving a Beanstalk queue). From outside of the system, I can identify a stuck worker script and its process id programatically, based on the data model of what is being processed. Inside the Open4.open4 block, I can identify the child process id.
I'd like to set up the Open4 block so that when I send a SIGTERM to the parent worker process, it forwards on the SIGTERM to the child. In addition, if the child process has still failed to exit after a short wait, I want to send a SIGKILL to the child process. In both cases, I'd then like the parent process to respond as normal to the SIGTERM it was sent.
This is all being done so I can expose a "stop" button in a customer services app, so non-technical team members have a tool to manage their way out of a situation with a blocked queue.
I have found some related questions in SO - e.g. How to make child process die after parent exits? - but the answers are not really usable for me from Ruby application code.
Here is a current implementation in Ruby that I have tested on my Mac:
Test stand-in for "bad" process that won't always respond to SIGTERM:
# Writing to a log file shows whether or not a detached process continues
# once the parent has closed IO to it.
$f = open( 'log.txt', 'w' );
def say m
begin
$f.puts m
$f.flush
$stderr.puts m
rescue Exception => e
# When the parent process closes, we get
# #<Errno::EPIPE: Broken pipe - <STDERR>> in this
# test, but with a stuck child process, this is not
# guaranteed to happen or cause the child to exit.
$f.puts e.inspect
$f.flush
end
end
Signal.trap( "TERM" ) { say "Received and ignored TERM" }
# Messages get logged, and sleep allows test of manual interrupts
say "Hello"
sleep 3
say "Foo Bar Baz"
sleep 3
say "Doo Be Doo"
sleep 3
say "Goodbye"
$f.close
Test Open4 block (part of a "worker" test script):
Open4.open4(#command) do | pid, stdin, stdout, stderr |
begin
stderr.each { |l|
puts "[#{pid}] STDERR: #{l}" if l
}
rescue SignalException => e
puts "[#{$$}] Received signal (#{e.signo} #{e.signm}) in Open4 block"
# Forward a SIGTERM to child, upgrade to SIGKILL if it doesn't work
if e.signo == 15
begin
puts "[#{$$}] Sending TERM to child process"
Process.kill( 'TERM', pid )
timeout(3.0) { Process.waitpid( pid ) }
rescue Timeout::Error
puts "[#{$$}] Sending KILL to child process"
Process.kill( 'KILL', pid )
end
end
raise e
end
end
Typical output if I start this up, and run e.g. kill -15 16854:
[16855] STDERR: Hello
[16854] Received signal (15 SIGTERM) in Open4 block
[16854] Sending TERM to child process
[16854] Sending KILL to child process
Contents of log file for same test:
Hello
Received and ignored TERM
Foo Bar Baz
The code is IMO a bit unwieldy, although it appears to work as I want. My questions:
Is the above attempt ok, or fatally flawed in the use case I need it for?
Have I missed a cleaner way of doing the same thing using existing Open4 and core Ruby methods?
Here's the code:
while 1
input = gets
puts input
end
Here's what I want to do but I have no idea how to do it:
I want to create multiple instances of the code to run in the background and be able to pass input to a specific instance.
Q1: How do I run multiple instances of the script in the background?
Q2: How do I refer to an individual instance of the script so I can pass input to the instance (Q3)?
Q3: The script is using the cmd "gets" to take input, how would I pass input into an indivdual's script's gets?
e.g
Let's say I'm running threes instances of the code in the background and I refer to the instance as #1, #2, and #3 respectively.
I pass "hello" to #1, #1 puts "hello" to the screen.
Then I pass "world" to #3 and #3 puts "hello" to the screen.
Thanks!
UPDATE:
Answered my own question. Found this awesome tut: http://rubylearning.com/satishtalim/ruby_threads.html and resource here: http://www.ruby-doc.org/core/classes/Thread.html#M000826.
puts Thread.main
x = Thread.new{loop{puts 'x'; puts gets; Thread.stop}}
y = Thread.new{loop{puts 'y'; puts gets; Thread.stop}}
z = Thread.new{loop{puts 'z'; puts gets; Thread.stop}}
while x.status != "sleep" and y.status != "sleep" and z.status !="sleep"
sleep(1)
end
Thread.list.each {|thr| p thr }
x.run
x.join
Thank you for all the help guys! Help clarified my thinking.
I assume that you mean that you want multiple bits of Ruby code running concurrently. You can do it the hard way using Ruby threads (which have their own gotchas) or you can use the job control facilities of your OS. If you're using something UNIX-y, you can just put the code for each daemon in separate .rb files and run them at the same time.
E.g.,
# ruby daemon1.rb &
# ruby daemon2.rb &
There are many ways to "handle input and output" in a Ruby program. Pipes, sockets, etc. Since you asked about daemons, I assume that you mean network I/O. See Net::HTTP.
Ignoring what you think will happen with multiple daemons all fighting over STDIN at the same time:
(1..3).map{ Thread.new{ loop{ puts gets } } }.each(&:join)
This will create three threads that loop indefinitely, asking for input and then outputting it. Each thread is "joined", preventing the main program from exiting until each thread is complete (which it never will be).
You could try using multi_daemons gem which has capability to run multiple daemons and control them.
# this is server.rb
proc_code = Proc do
loop do
sleep 5
end
end
scheduler = MultiDaemons::Daemon.new('scripts/scheduler', name: 'scheduler', type: :script, options: {})
looper = MultiDaemons::Daemon.new(proc_code, name: 'looper', type: :proc, options: {})
MultiDaemons.runner([scheduler, looper], { force_kill_timeout: 60 })
To start and stop
ruby server.rb start
ruby server.rb stop