Say I have a function like below, how do I capture the output of the Process.spawn call? I should also be able to kill the process if it takes longer than a specified timeout.
Note that the function must also be cross-platform (Windows/Linux).
def execute_with_timeout!(command)
begin
pid = Process.spawn(command) # How do I capture output of this process?
status = Timeout::timeout(5) {
Process.wait(pid)
}
rescue Timeout::Error
Process.kill('KILL', pid)
end
end
Thanks.
You can use IO.pipe and tell Process.spawn to use the redirected output without the need of external gem.
Of course, only starting with Ruby 1.9.2 (and I personally recommend 1.9.3)
The following is a simple implementation used by Spinach BDD internally to capture both out and err outputs:
# stdout, stderr pipes
rout, wout = IO.pipe
rerr, werr = IO.pipe
pid = Process.spawn(command, :out => wout, :err => werr)
_, status = Process.wait2(pid)
# close write ends so we could read them
wout.close
werr.close
#stdout = rout.readlines.join("\n")
#stderr = rerr.readlines.join("\n")
# dispose the read ends of the pipes
rout.close
rerr.close
#last_exit_status = status.exitstatus
The original source is in features/support/filesystem.rb
Is highly recommended you read Ruby's own Process.spawn documentation.
Hope this helps.
PS: I left the timeout implementation as homework for you ;-)
I followed Anselm's advice in his post on the Ruby forum here.
The function looks like this -
def execute_with_timeout!(command)
begin
pipe = IO.popen(command, 'r')
rescue Exception => e
raise "Execution of command #{command} unsuccessful"
end
output = ""
begin
status = Timeout::timeout(timeout) {
Process.waitpid2(pipe.pid)
output = pipe.gets(nil)
}
rescue Timeout::Error
Process.kill('KILL', pipe.pid)
end
pipe.close
output
end
This does the job, but I'd rather use a third-party gem that wraps this functionality. Anyone have any better ways of doing this? I have tried Terminator, it does exactly what I want but it does not seem to work on Windows.
Related
I'm using a command line program, it works as mentioned below:
$ ROUTE_TO_FOLDER/app < "long text"
If "long text" is written using the parameters "app" needs, then it will fill a text file with results. If not, it will fill the text file with dots continuously (I can't handle or modify the code of "app" in order to avoid this).
In a ruby script there's a line like this:
text = "long text that will be used by app"
output = system("ROUTE_TO_FOLDER/app < #{text}")
Now, if text is well written, there won't be problems and I will get an output file as mentioned before. The problem comes when text is not well written. What happens next is that my ruby script hangs and I'm not sure how to kill it.
I've found Open3 and I've used the method like this:
irb> cmd = "ROUTE_TO_FOLDER/app < #{text}"
irb> stdin, stdout, stderr, wait_thr = Open3.popen3(cmd)
=> [#<IO:fd 10>, #<IO:fd 11>, #<IO:fd 13>, #<Thread:0x007f3a1a6f8820 run>]
When I do:
irb> wait_thr.value
it also hangs, and :
irb> wait_thr.status
=> "sleep"
How can I avoid these problems? Is it not recognizing that "app" has failed?
wait_thr.pid provides you the pid of the started process. Just do
Process.kill("KILL",wait_thr.pid)
when you need to kill it.
You can combine it with detecting if the process is hung (continuously outputs dots) in one of the two ways.
1) Set a timeout for waiting for the process:
get '/process' do
text = "long text that will be used by app"
cmd = "ROUTE_TO_FOLDER/app < #{text}"
Open3.popen3(cmd) do |i,o,e,w|
begin
Timeout.timeout(10) do # timeout set to 10 sec, change if needed
# process output of the process. it will produce EOF when done.
until o.eof? do
# o.read_nonblock(N) ...
end
end
rescue Timeout::Error
# here you know that the process took longer than 10 seconds
Process.kill("KILL", w.pid)
# do whatever other error processing you need
end
end
end
2) Check the process output. (The code below is simplified - you probably don't want to read the output of your process into a single String buf first and then process, but I guess you get the idea).
get '/process' do
text = "long text that will be used by app"
cmd = "ROUTE_TO_FOLDER/app < #{text}"
Open3.popen3(cmd) do |i,o,e,w|
# process output of the process. it will produce EOF when done.
# If you get 16 dots in a row - the process is in the continuous loop
# (you may want to deal with stderr instead - depending on where these dots are sent to)
buf = ""
error = false
until o.eof? do
buf << o.read_nonblock(16)
if buf.size>=16 && buf[-16..-1] == '.'*16
# ok, the process is hung
Process.kill("KILL", w.pid)
error = true
# you should also get o.eof? the next time you check (or after flushing the pipe buffer),
# so you will get out of the until o.eof? loop
end
end
if error
# do whatever error processing you need
else
# process buf, it contains all the output
end
end
end
In ruby18 I sometimes did the following to get a subprocess with full control:
stdin, #stdin= IO.pipe
#stdout, stdout= IO.pipe
#stderr, stderr= IO.pipe
#pid= fork do
#stdin.close
STDIN.close
stdin.dup
#stdout.close
STDOUT.close
stdout.dup
#stderr.close
STDERR.close
stderr.dup
exec(...)
end
This does not work in ruby19. The close method for STDIN, STDOUT, STDERR does not close the underlying filedescriptor in ruby19. How do I do this in ruby19.
Check out Process.spawn, Open3, and the childprocess gem.
I can't tell exactly what you're trying to do there, but you can take control of a child process's IO in many ways.
Using Unix pipes:
readme, writeme = IO.pipe
pid = fork {
$stdout.reopen writeme
readme.close
exec(...)
}
Juggling the IOs with Process.spawn:
pid = spawn(command, :err=>:out)
Or wrapping the process in POpen3:
require 'open3'
include Open3
popen3(RUBY, '-r', THIS_FILE, '-e', 'hello("Open3", true)') do
|stdin, stdout, stderr|
stdin.write("hello from parent")
stdin.close_write
stdout.read.split("\n").each do |line|
puts "[parent] stdout: #{line}"
end
stderr.read.split("\n").each do |line|
puts "[parent] stderr: #{line}"
end
You might also consider Jesse Storimer's Working With Unix Processes. It has a lot of information and his writing style is very easy to read and understand. The book doubles as a reference guide that is somehow more useful than a lot of the actual documentation.
references:
http://pleac.sourceforge.net/pleac_ruby/processmanagementetc.html
http://rubydoc.info/stdlib/core/1.9.3/Process.spawn
http://devver.wordpress.com/2009/10/12/ruby-subprocesses-part_3/
This post shows one way to temporarily replace stdin in Ruby:
begin
save_stdin = $stdin # a dup by any other name
$stdin.reopen('/dev/null') # dup2, essentially
# do stuff
ensure
$stdin.reopen(save_stdin) # restore original $stdout
save_stdin.close # and dispose of the copy
end
Since this question is one of the top google hits for “ruby replace stdin,” I hope this will help others looking for how to do that.
I'm creating a script to wrap jdb (java debugger). I essentially want to wrap this process and proxy the user interaction. So I want it to:
start jdb from my script
send the output of jdb to stdout
pause and wait for input when jdb does
when the user enters commands, pass it to jdb
At the moment I really want a pass thru to jdb. The reason for this is to initialize the process with specific parameters and potentially add more commands in the future.
Update:
Here's the shell of what ended up working for me using expect:
PTY.spawn("jdb -attach 1234") do |read,write,pid|
write.sync = true
while (true) do
read.expect(/\r\r\n> /) do |s|
s = s[0].split(/\r\r\n/)
s.pop # get rid of prompt
s.each { |line| puts line }
print '> '
STDOUT.flush
write.print(STDIN.gets)
end
end
end
Use Open3.popen3(). e.g.:
Open3.popen3("jdb args") { |stdin, stdout, stderr|
# stdin = jdb's input stream
# stdout = jdb's output stream
# stderr = jdb's stderr stream
threads = []
threads << Thread.new(stderr) do |terr|
while (line = terr.gets)
puts "stderr: #{line}"
end
end
threads << Thread.new(stdout) do |terr|
while (line = terr.gets)
puts "stdout: #{line}"
end
end
stdin.puts "blah"
threads.each{|t| t.join()} #in order to cleanup when you're done.
}
I've given you examples for threads, but you of course want to be responsive to what jdb is doing. The above is merely a skeleton for how you open the process and handle communication with it.
The Ruby standard library includes expect, which is designed for just this type of problem. See the documentation for more information.
Similar to Getting output of system() calls in Ruby , I am running a system command, but in this case I need to output the STDOUT from the command as it runs.
As in the linked question, the answer is again not to use system at all as system does not support this.
However this time the solution isn't to use backticks, but IO.popen, which returns an IO object that you can use to read the input as it is being generated.
In case someone might want to read stdout and stderr:
It is important to read them in parallel, not first one then the other. Because programs are allowed to output to stdout and stderr by turns and even in parallel. So, you need threads. This fact isn't even Ruby-specific.
Stolen from here.
require 'open3'
cmd = './packer_mock.sh'
data = {:out => [], :err => []}
# see: http://stackoverflow.com/a/1162850/83386
Open3.popen3(cmd) do |stdin, stdout, stderr, thread|
# read each stream from a new thread
{ :out => stdout, :err => stderr }.each do |key, stream|
Thread.new do
until (raw_line = stream.gets).nil? do
parsed_line = Hash[:timestamp => Time.now, :line => "#{raw_line}"]
# append new lines
data[key].push parsed_line
puts "#{key}: #{parsed_line}"
end
end
end
thread.join # don't exit until the external process is done
end
here is my solution
def io2stream(shell, &block)
Open3.popen3(shell) do |_, stdout, stderr|
while line = stdout.gets
block.call(line)
end
while line = stderr.gets
block.call(line)
end
end
end
io2stream("ls -la", &lambda { |str| puts str })
With following you can capture stdout of a system command:
output = capture(:stdout) do
system("pwd") # your system command goes here
end
puts output
shortened version:
output = capture(:stdout) { system("pwd") }
Similarly we can also capture standard errors too with :stderr
capture method is provided by active_support/core_ext/kernel/reporting.rb
Looking at that library's code comments, capture is going to be deprecated, so not sure what is the current supported method name is.
I am trying to use IO.popen in order to put (with .puts method) and to get (with .gets method) messages from a process to its sub-process.
I am not very experimented and I have a question about. Having the following code, I have an error because it is not possible to write in a closed stream.
class Interface
def initialize(path)
#sub_process = IO.popen(path, 'w+')
end
def start!
if ok?
#sub_process.puts 'Hello', 'my name is ...'
# and more...
end
end
protected
def ok?
is_ready?(#sub_process) && is_cool?(#sub_process)
end
def is_ready?(sub_process)
reply = process_command(sub_process, 'are u ready?')
reply.chomp.match(/yes_i_am_ready$/)
end
def is_cool?(sub_process)
reply = process_command(sub_process, 'are u cool?')
reply.chomp.match(/yes_i_am_cool$/)
end
def process_command(sub_process, command)
rdr = Thread.new { sub_process.read } # alternative: io.readlines
sub_process.puts "#{command}"
sub_process.close_write
rdr.value # joins and fetches the result
end
end
a = Interface.new("./program")
a.start!
(...) in `write': not opened for writing (IOError)
As we can see, this error occur during is_cool? test (as explained at: http://ruby-doc.org/core/classes/IO.html#M002289).
But if I try to comment in process_command method the line:
# sub_process.close_write
the script seems to sleep... infinitely :s
I believe that it is not possible to open again a closed stream. And I can't create an other IO.popen instance of my program "./program" because it needs to be initialized with some command (like 'are u ready?' and 'are u cool?') at the beginning, before I use it (by sending and receiving messages like a simple discussion).
How changes can I do over the current code in order to solve this problem?
Edit: in other words, I would like to establish a such communication (according to a given protocol):
Parent message: Child answer:
-------------- ------------
'are u ready?' 'yes_i_am_ready'
'are u cool?' 'yes_i_am_cool'
'Hello' 'foo'
'my name is ...' 'bar'
Many thanks for any help.
Perhaps it will help to have a working example. Here's one, tested and known to work in MRI 1.8.7 on Linux.
bar.rb
#!/usr/bin/ruby1.8
begin
loop do
puts "You said: #{gets}"
$stdout.flush
end
rescue Errno::EPIPE
end
foo.rb
#!/usr/bin/ruby1.8
class Parent
def initialize
#pipe = IO.popen(CHILD_COMMAND, 'w+')
end
def talk(message)
#pipe.puts(message)
response = #pipe.gets
if response.nil?
$stderr.puts "Failed: #{CHILD_COMMAND}"
exit(1)
end
response.chomp
end
private
CHILD_COMMAND = './bar.rb'
end
parent = Parent.new
puts parent.talk('blah blah blah')
puts parent.talk('foo bar baz')
foo.rb output
You said: blah blah blah
You said: foo bar baz
A closed IO can not be used anymore. You should not close an IO if you intend on still using it.
If you remove the IO#close_write there still remains the problem with your code in the following line.
rdr = Thread.new { sub_process.read }
IO#read read's until EOF. So until the stream get's closed it never terminates. You mentioned IO#readline in your code, this would be the better alternative. Using IO#readline your program would only hang if the popend process never send's a newline.
Another problem with popen is the following. IO#popen create's a new process. Process's may be killed by you, other users, memory shortages, …. Don't expect your process to always run all the time. If the process is killed IO#readline will throw an EOFError, IO#read will return imidiatley. You can determine the termination reason with the following code.
Process::wait(io.pid)
status= $?
status.class # => Process::Status
status.signaled? # killed by signal?
status.stopsig # the signal which killed it
status.exited # terminated normal
status.exitstatus # the return value
status.ki
Does it help to use this form of Thread.new?
rdr = Thread.new(sub_process) {|x| x.readlines }