Ruby - Using Kernel.exec with custom STDOUT - ruby

I'm trying to exec a shell process such that its standard output is prefixed with an identifier.
My approach is to write a custom IO object that re-implements write, passing it as the :out argument to exec (documented under Process::spawn).
require "delegate"
class PrefixedStdout < DelegateClass(IO)
def initialize(prefix, io)
#prefix = prefix
super(io)
end
def write(str)
super("#{#prefix}: #{str}")
end
end
pr_stdout = PrefixedStdout.new("my_prefix", $stdout)
pr_stdout.write("hello\n") # outputs "my_prefix: hello"
exec("echo hello", out: pr_stdout) # outputs "hello"
Somehow exec is bypassing PrefixedStdout#write and calling $stdout.write directly. How do I force exec to use my prefixed output stream as its stdout?

What gets preserved in the other process is the underlying file descriptor (or rather they are hooked up under the hood), so as I commented I don't think you'll ever get writes to that descriptor to be funnelled through your write method - exec replaces the running process with a new one.
A possible approach is to create a pipe, pass one end to your child process and then read from the other end, inserting prefixes as needed,
For example you might do
IO.pipe do |read_pipe, write_pipe|
fork do
exec("echo hello", out: write_pipe)
end
write_pipe.close
while line = read_pipe.gets
puts "prefix: #{line}"
end
end
You might also be interested in IO.popen which wraps some of this up.

Somehow exec is bypassing PrefixedStdout#write and calling
$stdout.write
Take a look at this example:
class MyIO < IO
def initialize(fd)
super
end
def write(str)
STDOUT.puts 'write called'
super
end
end
fd = IO.sysopen("data.txt", "w")
io = MyIO.new(fd)
io.write "goodbye\n"
puts '---now with exec()...'
exec("echo hello", :out => io)
--output:--
write called
---now with exec()...
Now, what do you think is in the file data.txt?
spoiler:
$cat data.txt
hello
So passing an IO object to exec() 'works', but not the way you expected: exec() never calls io.write() to write the output of the child process to io. Instead, I assume exec() obtains the file descriptor for io, then passes it to some C code, which does some system level redirection of the output from the child process to the file data.txt.
Do you have to use exec()? If not:
prefix = "prefix: "
cmd = 'echo hello'
output = `#{cmd}`
puts "#{prefix}#{output}"
--output:--
prefix: hello

Related

Kill a process called using open3 in ruby

I'm using a command line program, it works as mentioned below:
$ ROUTE_TO_FOLDER/app < "long text"
If "long text" is written using the parameters "app" needs, then it will fill a text file with results. If not, it will fill the text file with dots continuously (I can't handle or modify the code of "app" in order to avoid this).
In a ruby script there's a line like this:
text = "long text that will be used by app"
output = system("ROUTE_TO_FOLDER/app < #{text}")
Now, if text is well written, there won't be problems and I will get an output file as mentioned before. The problem comes when text is not well written. What happens next is that my ruby script hangs and I'm not sure how to kill it.
I've found Open3 and I've used the method like this:
irb> cmd = "ROUTE_TO_FOLDER/app < #{text}"
irb> stdin, stdout, stderr, wait_thr = Open3.popen3(cmd)
=> [#<IO:fd 10>, #<IO:fd 11>, #<IO:fd 13>, #<Thread:0x007f3a1a6f8820 run>]
When I do:
irb> wait_thr.value
it also hangs, and :
irb> wait_thr.status
=> "sleep"
How can I avoid these problems? Is it not recognizing that "app" has failed?
wait_thr.pid provides you the pid of the started process. Just do
Process.kill("KILL",wait_thr.pid)
when you need to kill it.
You can combine it with detecting if the process is hung (continuously outputs dots) in one of the two ways.
1) Set a timeout for waiting for the process:
get '/process' do
text = "long text that will be used by app"
cmd = "ROUTE_TO_FOLDER/app < #{text}"
Open3.popen3(cmd) do |i,o,e,w|
begin
Timeout.timeout(10) do # timeout set to 10 sec, change if needed
# process output of the process. it will produce EOF when done.
until o.eof? do
# o.read_nonblock(N) ...
end
end
rescue Timeout::Error
# here you know that the process took longer than 10 seconds
Process.kill("KILL", w.pid)
# do whatever other error processing you need
end
end
end
2) Check the process output. (The code below is simplified - you probably don't want to read the output of your process into a single String buf first and then process, but I guess you get the idea).
get '/process' do
text = "long text that will be used by app"
cmd = "ROUTE_TO_FOLDER/app < #{text}"
Open3.popen3(cmd) do |i,o,e,w|
# process output of the process. it will produce EOF when done.
# If you get 16 dots in a row - the process is in the continuous loop
# (you may want to deal with stderr instead - depending on where these dots are sent to)
buf = ""
error = false
until o.eof? do
buf << o.read_nonblock(16)
if buf.size>=16 && buf[-16..-1] == '.'*16
# ok, the process is hung
Process.kill("KILL", w.pid)
error = true
# you should also get o.eof? the next time you check (or after flushing the pipe buffer),
# so you will get out of the until o.eof? loop
end
end
if error
# do whatever error processing you need
else
# process buf, it contains all the output
end
end
end

Ruby: Printing system output in real time?

I have a ruby rake task that calls a bash script via:
Open3.popen('/path/file_converter.sh', file_list, output_format)
That bash script outputs logs to the command line as it processes (which takes from 30 secs to 5 hours)
When I call the rake task, the output from bash is returned to the command line, but only as one large message after the entire script has run. Anyone know of a way to pipe command line output direct to ruby output as it occurs?
According to the documentation you should be able to use the output stream given in the block:
Open3.popen3('/path/file_converter.sh', file_list, output_format) do |_,out,_,_|
out.each_line do |line|
puts line
end
end
Put the output into a file. And run the process in the background creating a new thread. After it you can parse the file.
class FileConverter
def initialize
#output_file = '/tmp/something.txt'
output_format = 'foo'
file_list = 'bar foo something'
#child = Thread.new do
`/path/file_converter.sh #{file_list} #{output_format} 2>&1 >#{#output_file}`
end
end
def data
File.readlines(#output_file)
end
def parse
while #child.alive?
# parse data # TODO: need to implement real parsing
sleep 0.5
end
end
end
fc = FileConverter.new
fc.parse

How can I capture STDOUT to a string?

puts "hi"
puts "bye"
I want to store the STDOUT of the code so far (in this case hi \nbye into a variable say 'result' and print it )
puts result
The reason I am doing this is I have integrate an R code into my Ruby code, output of which is given to the STDOUT as the R code runs , but the ouput cannot be accessed inside the code to do some evaluations. Sorry if this is confusing. So the "puts result" line should give me hi and bye.
A handy function for capturing stdout into a string...
The following method is a handy general purpose tool to capture stdout and return it as a string. (I use this frequently in unit tests where I want to verify something printed to stdout.) Note especially the use of the ensure clause to restore $stdout (and avoid astonishment):
def with_captured_stdout
original_stdout = $stdout # capture previous value of $stdout
$stdout = StringIO.new # assign a string buffer to $stdout
yield # perform the body of the user code
$stdout.string # return the contents of the string buffer
ensure
$stdout = original_stdout # restore $stdout to its previous value
end
So, for example:
>> str = with_captured_stdout { puts "hi"; puts "bye"}
=> "hi\nbye\n"
>> print str
hi
bye
=> nil
Redirect Standard Output to a StringIO Object
You can certainly redirect standard output to a variable. For example:
# Set up standard output as a StringIO object.
foo = StringIO.new
$stdout = foo
# Send some text to $stdout.
puts 'hi'
puts 'bye'
# Access the data written to standard output.
$stdout.string
# => "hi\nbye\n"
# Send your captured output to the original output stream.
STDOUT.puts $stdout.string
In practice, this is probably not a great idea, but at least now you know it's possible.
You can do this by making a call to your R script inside backticks, like this:
result = `./run-your-script`
puts result # will contain STDOUT from run-your-script
For more information on running subprocesses in Ruby, check out this Stack Overflow question.
If activesupport is available in your project you may do the following:
output = capture(:stdout) do
run_arbitrary_code
end
More info about Kernel.capture can be found here
For most practical purposes you can put anything into $stdout that responds to write, flush, sync, sync= and tty?.
In this example I use a modified Queue from the stdlib.
class Captor < Queue
alias_method :write, :push
def method_missing(meth, *args)
false
end
def respond_to_missing?(*args)
true
end
end
stream = Captor.new
orig_stdout = $stdout
$stdout = stream
puts_thread = Thread.new do
loop do
puts Time.now
sleep 0.5
end
end
5.times do
STDOUT.print ">> #{stream.shift}"
end
puts_thread.kill
$stdout = orig_stdout
You need something like this if you want to actively act on the data and not just look at it after the task has finished. Using StringIO or a file will have be problematic with multiple threads trying to sync reads and writes simultaneously.
Capture stdout (or stderr) for both Ruby code and subprocesses
# capture_stream(stream) { block } -> String
#
# Captures output on +stream+ for both Ruby code and subprocesses
#
# === Example
#
# capture_stream($stdout) { puts 1; system("echo 2") }
#
# produces
#
# "1\n2\n"
#
def capture_stream(stream)
raise ArgumentError, 'missing block' unless block_given?
orig_stream = stream.dup
IO.pipe do |r, w|
# system call dup2() replaces the file descriptor
stream.reopen(w)
# there must be only one write end of the pipe;
# otherwise the read end does not get an EOF
# by the final `reopen`
w.close
t = Thread.new { r.read }
begin
yield
ensure
stream.reopen orig_stream # restore file descriptor
end
t.value # join and get the result of the thread
end
end
I got inspiration from Zhon.
Minitest versions:
assert_output if you need to ensure if some output is generated:
assert_output "Registrars processed: 1\n" do
puts 'Registrars processed: 1'
end
assert_output
or use capture_io if you really need to capture it:
out, err = capture_io do
puts "Some info"
warn "You did a bad thing"
end
assert_match %r%info%, out
assert_match %r%bad%, err
capture_io
Minitest itself is available in any Ruby version starting from 1.9.3
For RinRuby, please know that R has capture.output:
R.eval <<EOF
captured <- capture.output( ... )
EOF
puts R.captured
Credit to #girasquid's answer. I modified it to a single file version:
def capture_output(string)
`echo #{string.inspect}`.chomp
end
# example usage
response_body = "https:\\x2F\\x2Faccounts.google.com\\x2Faccounts"
puts response_body #=> https:\x2F\x2Faccounts.google.com\x2Faccounts
capture_output(response_body) #=> https://accounts.google.com/accounts

How to proxy a shell process in ruby

I'm creating a script to wrap jdb (java debugger). I essentially want to wrap this process and proxy the user interaction. So I want it to:
start jdb from my script
send the output of jdb to stdout
pause and wait for input when jdb does
when the user enters commands, pass it to jdb
At the moment I really want a pass thru to jdb. The reason for this is to initialize the process with specific parameters and potentially add more commands in the future.
Update:
Here's the shell of what ended up working for me using expect:
PTY.spawn("jdb -attach 1234") do |read,write,pid|
write.sync = true
while (true) do
read.expect(/\r\r\n> /) do |s|
s = s[0].split(/\r\r\n/)
s.pop # get rid of prompt
s.each { |line| puts line }
print '> '
STDOUT.flush
write.print(STDIN.gets)
end
end
end
Use Open3.popen3(). e.g.:
Open3.popen3("jdb args") { |stdin, stdout, stderr|
# stdin = jdb's input stream
# stdout = jdb's output stream
# stderr = jdb's stderr stream
threads = []
threads << Thread.new(stderr) do |terr|
while (line = terr.gets)
puts "stderr: #{line}"
end
end
threads << Thread.new(stdout) do |terr|
while (line = terr.gets)
puts "stdout: #{line}"
end
end
stdin.puts "blah"
threads.each{|t| t.join()} #in order to cleanup when you're done.
}
I've given you examples for threads, but you of course want to be responsive to what jdb is doing. The above is merely a skeleton for how you open the process and handle communication with it.
The Ruby standard library includes expect, which is designed for just this type of problem. See the documentation for more information.

How do I get the STDOUT of a ruby system() call while it is being run?

Similar to Getting output of system() calls in Ruby , I am running a system command, but in this case I need to output the STDOUT from the command as it runs.
As in the linked question, the answer is again not to use system at all as system does not support this.
However this time the solution isn't to use backticks, but IO.popen, which returns an IO object that you can use to read the input as it is being generated.
In case someone might want to read stdout and stderr:
It is important to read them in parallel, not first one then the other. Because programs are allowed to output to stdout and stderr by turns and even in parallel. So, you need threads. This fact isn't even Ruby-specific.
Stolen from here.
require 'open3'
cmd = './packer_mock.sh'
data = {:out => [], :err => []}
# see: http://stackoverflow.com/a/1162850/83386
Open3.popen3(cmd) do |stdin, stdout, stderr, thread|
# read each stream from a new thread
{ :out => stdout, :err => stderr }.each do |key, stream|
Thread.new do
until (raw_line = stream.gets).nil? do
parsed_line = Hash[:timestamp => Time.now, :line => "#{raw_line}"]
# append new lines
data[key].push parsed_line
puts "#{key}: #{parsed_line}"
end
end
end
thread.join # don't exit until the external process is done
end
here is my solution
def io2stream(shell, &block)
Open3.popen3(shell) do |_, stdout, stderr|
while line = stdout.gets
block.call(line)
end
while line = stderr.gets
block.call(line)
end
end
end
io2stream("ls -la", &lambda { |str| puts str })
With following you can capture stdout of a system command:
output = capture(:stdout) do
system("pwd") # your system command goes here
end
puts output
shortened version:
output = capture(:stdout) { system("pwd") }
Similarly we can also capture standard errors too with :stderr
capture method is provided by active_support/core_ext/kernel/reporting.rb
Looking at that library's code comments, capture is going to be deprecated, so not sure what is the current supported method name is.

Resources