I want to offload a block of code in my main process to child process to make it run concurrently. I also want to have the PID of the spawned child process so I can monitor and kill it if necessary.
In addition to Chris' great answer, remember to call Process.wait from your master in order to reap your child process, else you'll leave zombies behind.
Example as requested in comments:
pid = Process.fork do
puts "child, pid #{Process.pid} sleeping..."
sleep 5
puts "child exiting"
end
puts "parent, pid #{Process.pid}, waiting on child pid #{pid}"
Process.wait
puts "parent exiting"
You can use the fork kernel method. Here is an example:
#!/usr/bin/env ruby
puts "This is the master process."
child_pid = fork do
puts "This is the child process"
exit
end
puts "The PID of the child process is #{child_pid}"
The fork method returns the PID of the process it forks and executes any code in the block passed. Like regular Ruby blocks it keeps the bindings of the parent process.
It is a good idea to make your forked process exit.
In 1.9 you can use Process.spawn command.
See also http://en.wikibooks.org/wiki/Ruby_Programming/Running_Multiple_Processes
If you are happy to use Threads, rather than Processes, then something like this may be a bit more scaleable to more-than-one fork:
def doit(x)
sleep(rand(10))
puts "Done... #{x}"
end
thingstodo = ["a","b","c","d","e","f","g"]
tasklist = []
# Set the threads going
thingstodo.each { |thing|
task = Thread.new(thing) { |this| doit(this) }
tasklist << task
}
# Wait for the threads to finish
tasklist.each { |task|
task.join
}
Please see John Topley's excellent comments and reference, below, regarding the Ruby execution model and its restrictions.
Just edited to correct a glaring error (no assignment to task), and to follow #(Jason King)'s advice.
Related
My below code example runs a command, e.g., ls <file> and captures stdout and stderr ( as well as the process exit status ).
However, if for example the command were to hang, then the ruby program will be "stuck" waiting for the command to finish, ( this can be seen for example if running sleep).
To avoid that possibility, I think what I need to do is fork a child process, ( so any "stuck" child process will not keep the ruby program "waiting" ).
However, I'm not sure how to capture the stdout and stderr from a forked child process, is this even possible ?
( for reasons I'd also like to be able to do this within the ruby std libs and not have dependency on any extra gem/s.
Also, this is just for ruby, not rails )
edit: To help clarify -
Trying to understand if there is a way to fork a child process, ( so there is no blocking until the child is done ), and still have the ruby program capture the stdout, stderr when the child process exits.
#!/bin/env ruby
require 'open3'
require 'logger'
logger = Logger.new('./test_open3.log')
files = ['file1','file2','file3']
files.each do |f|
stdout, stderr, status = Open3.capture3("ls #{f}")
logger.info(stdout)
logger.info(stderr)
logger.info(status)
end
Following the suggestion in the comments to use threads, I found that this gave me what I was looking for:
#!/bin/env ruby
require 'open3'
require 'logger'
require 'thread'
logger = Logger.new('./test_threads_open3.log')
files = ['file1','file2','file3']
threads = []
files.each_with_index do |f, i|
threads << Thread.new(f, i) do
puts "Thread #{i} is running"
stdout, stderr, status = Open3.capture3("ls #{f}")
logger.info(stdout)
logger.info(stderr)
logger.info(status)
end
end
threads.each { |t| t.join }
This creates an array of threads each with the block of code, and then last line, each thread in the array is run.
It probably requires some extra code to manage and limit the number of threads that can be run at a time, so as to be safer, maybe by using a queue/worker feature.
( This post also touches on the topic of the join method - Thread.join blocks the main thread )
I am developing a long-running program in Ruby. I am writing some integration tests for this. These tests need to kill or stop the program after starting it; otherwise the tests hang.
For example, with a file bin/runner
#!/usr/bin/env ruby
while true do
puts "Hello World"
sleep 10
end
The (integration) test would be:
class RunReflectorTest < TestCase
test "it prints a welcome message over and over" do
out, err = capture_subprocess_io do
system "bin/runner"
end
assert_empty err
assert_includes out, "Hello World"
end
end
Only, obviously, this will not work; the test starts and never stops, because the system call never ends.
How should I tackle this? Is the problem in system itself, and would Kernel#spawn provide a solution? If so, how? Somehow the following keeps the out empty:
class RunReflectorTest < TestCase
test "it prints a welcome message over and over" do
out, err = capture_subprocess_io do
pid = spawn "bin/runner"
sleep 2
Process.kill pid
end
assert_empty err
assert_includes out, "Hello World"
end
end
. This direction also seems like it will cause a lot of timing-issues (and slow tests). Ideally, a reader would follow the stream of STDOUT and let the test pass as soon as the string is encountered and then immediately kill the subprocess. I cannot find how to do this with Process.
Test Behavior, Not Language Features
First, what you're doing is a TDD anti-pattern. Tests should focus on behaviors of methods or objects, not on language features like loops. If you must test a loop, construct a test that checks for a useful behavior like "entering an invalid response results in a re-prompt." There's almost no utility in checking that a loop loops forever.
However, you might decide to test a long-running process by checking to see:
If it's still running after t time.
If it's performed at least i iterations.
If a loop exits properly given certain input or upon reaching a boundary condition.
Use Timeouts or Signals to End Testing
Second, if you decide to do it anyway, you can just escape the block with Timeout::timeout. For example:
require 'timeout'
# Terminates block
Timeout::timeout(3) { `sleep 300` }
This is quick and easy. However, note that using timeout doesn't actually signal the process. If you run this a few times, you'll notice that sleep is still running multiple times as a system process.
It's better is to signal the process when you want to exit with Process::kill, ensuring that you clean up after yourself. For example:
pid = spawn 'sleep 300'
Process::kill 'TERM', pid
sleep 3
Process::wait pid
Aside from resource issues, this is a better approach when you're spawning something stateful and don't want to pollute the independence of your tests. You should almost always kill long-running (or infinite) processes in your test teardown whenever you can.
Ideally, a reader would follow the stream of STDOUT and let the test pass as soon as the string is encountered and then immediately kill the subprocess. I cannot find how to do this with Process.
You can redirect stdout of spawned process to any file descriptor by specifying out option
pid = spawn(command, :out=>"/dev/null") # write mode
Documentation
Example of redirection
With the answer from CodeGnome on how to use Timeout::timeout and the answer from andyconhin on how to redirect Process::spawn IO, I came up with two Minitest helpers that can be used as follows:
it "runs a deamon" do
wait_for(timeout: 2) do
wait_for_spawned_io(regexp: /Hello World/, command: ["bin/runner"])
end
end
The helpers are:
def wait_for(timeout: 1, &block)
Timeout::timeout(timeout) do
yield block
end
rescue Timeout::Error
flunk "Test did not pass within #{timeout} seconds"
end
def wait_for_spawned_io(regexp: //, command: [])
buffer = ""
begin
read_pipe, write_pipe = IO.pipe
pid = Process.spawn(command.shelljoin, out: write_pipe, err: write_pipe)
loop do
buffer << read_pipe.readpartial(1000)
break if regexp =~ buffer
end
ensure
read_pipe.close
write_pipe.close
Process.kill("INT", pid)
end
buffer
end
These can be used in a test which allows me to start a subprocess, capture the STDOUT and as soon as it matches the test Regular Expression, it passes, else it will wait 'till timeout and flunk (fail the test).
The loop will capture output and pass the test once it sees matching output. It uses a IO.pipe because that is most transparant for subprocesses (and their children) to write to.
I doubt this will work on Windows. And it needs some cleaning up of the wait_for_spawned_io which is doing slightly too much IMO. Antoher problem is that the Process.kill('INT') might not reach the children which are orphaned but still running after this test has ran. I need to find a way to ensure the entire subtree of processes is killed.
How do you get the parent process id of a process that is not the current process in Ruby?
I've checked Ruby's Process module, but it only seems to provide a means to access the PPID of the current process.
I also checked google for anything on the subject, but the first two pages seemed to only contain links regarding how to use the aforementioned Process module.
I was hoping to do this without having to rely too much on the underlying OS, but whatever works.
Shell out:
1.9.3p429 :001 > `ps -p 7544 -o ppid=`.strip
=> "7540"
Process.ppid returns the parent process id.
http://ruby-doc.org/core-2.4.1/Process.html#method-c-ppid
You can just remember it in a variable:
parent_pid = Process.pid
Process.fork do
child_pid = Process.pid
puts parent_pid, child_pid
# do stuff
exit
end
Process.wait
# 94791
# 94798
alternatively, if you need the information on the level of the parent process:
parent_pid = Process.pid
child_pid = Process.fork do
# do stuff
exit
end
Process.wait
puts parent_pid, child_pid
# 6361
# 6362
I am using the open4 gem to wrap system calls to a potentially long-running third-party command line tool. The tool may sometimes fail, keeping two processes busy, and partially blocking a pipeline, as the parent process is part of a pool of worker scripts (serving a Beanstalk queue). From outside of the system, I can identify a stuck worker script and its process id programatically, based on the data model of what is being processed. Inside the Open4.open4 block, I can identify the child process id.
I'd like to set up the Open4 block so that when I send a SIGTERM to the parent worker process, it forwards on the SIGTERM to the child. In addition, if the child process has still failed to exit after a short wait, I want to send a SIGKILL to the child process. In both cases, I'd then like the parent process to respond as normal to the SIGTERM it was sent.
This is all being done so I can expose a "stop" button in a customer services app, so non-technical team members have a tool to manage their way out of a situation with a blocked queue.
I have found some related questions in SO - e.g. How to make child process die after parent exits? - but the answers are not really usable for me from Ruby application code.
Here is a current implementation in Ruby that I have tested on my Mac:
Test stand-in for "bad" process that won't always respond to SIGTERM:
# Writing to a log file shows whether or not a detached process continues
# once the parent has closed IO to it.
$f = open( 'log.txt', 'w' );
def say m
begin
$f.puts m
$f.flush
$stderr.puts m
rescue Exception => e
# When the parent process closes, we get
# #<Errno::EPIPE: Broken pipe - <STDERR>> in this
# test, but with a stuck child process, this is not
# guaranteed to happen or cause the child to exit.
$f.puts e.inspect
$f.flush
end
end
Signal.trap( "TERM" ) { say "Received and ignored TERM" }
# Messages get logged, and sleep allows test of manual interrupts
say "Hello"
sleep 3
say "Foo Bar Baz"
sleep 3
say "Doo Be Doo"
sleep 3
say "Goodbye"
$f.close
Test Open4 block (part of a "worker" test script):
Open4.open4(#command) do | pid, stdin, stdout, stderr |
begin
stderr.each { |l|
puts "[#{pid}] STDERR: #{l}" if l
}
rescue SignalException => e
puts "[#{$$}] Received signal (#{e.signo} #{e.signm}) in Open4 block"
# Forward a SIGTERM to child, upgrade to SIGKILL if it doesn't work
if e.signo == 15
begin
puts "[#{$$}] Sending TERM to child process"
Process.kill( 'TERM', pid )
timeout(3.0) { Process.waitpid( pid ) }
rescue Timeout::Error
puts "[#{$$}] Sending KILL to child process"
Process.kill( 'KILL', pid )
end
end
raise e
end
end
Typical output if I start this up, and run e.g. kill -15 16854:
[16855] STDERR: Hello
[16854] Received signal (15 SIGTERM) in Open4 block
[16854] Sending TERM to child process
[16854] Sending KILL to child process
Contents of log file for same test:
Hello
Received and ignored TERM
Foo Bar Baz
The code is IMO a bit unwieldy, although it appears to work as I want. My questions:
Is the above attempt ok, or fatally flawed in the use case I need it for?
Have I missed a cleaner way of doing the same thing using existing Open4 and core Ruby methods?
I'm looking for a reliable way of implementing a callback on a forked process, once it has finished.
I tried using trap (see the code below), but it appears to fail from time to time.
trap("CLD") {
pid = Process.wait
# do stuff
}
pid = fork {
# do stuff
}
While I did found (via google) possible explanations why this may be happening, I'm having a hard time figuring out a possible solution.
The only solution I see so far is to open a pipe between processes (parent - read end, child - write end). Then put the parent processes thread on blocking read and trap "broken pipe" or "pipe closed" exceptions.
Any of these exceptions will obviously mean that child process is dead.
UPDATE: If I'm not wrong normally closed pipe will result into EOF result of blocking read, and broken pipe (if child process crashed) will result into Errno::EPIPE exception.
#Openning a pipe
p_read, p_write = IO.pipe
pid = fork {
#We are only "using" write end here, thus closing read end in child process
#and let the write end hang open in the process
p_read.close
}
#We are only reading in parent, thus closing write end here
p_write.close
Thread.new {
begin
p_write.read
#Should never get here
rescue EOFError
#Child normally closed its write end
#do stuff
rescue Errno::EPIPE
#Chiled didn't normally close its write end
#do stuff (maybe the same stuff as in EOFError handling)
end
#Should never get here
}
#Do stuff in parents main thread