I have a ruby rake task that calls a bash script via:
Open3.popen('/path/file_converter.sh', file_list, output_format)
That bash script outputs logs to the command line as it processes (which takes from 30 secs to 5 hours)
When I call the rake task, the output from bash is returned to the command line, but only as one large message after the entire script has run. Anyone know of a way to pipe command line output direct to ruby output as it occurs?
According to the documentation you should be able to use the output stream given in the block:
Open3.popen3('/path/file_converter.sh', file_list, output_format) do |_,out,_,_|
out.each_line do |line|
puts line
end
end
Put the output into a file. And run the process in the background creating a new thread. After it you can parse the file.
class FileConverter
def initialize
#output_file = '/tmp/something.txt'
output_format = 'foo'
file_list = 'bar foo something'
#child = Thread.new do
`/path/file_converter.sh #{file_list} #{output_format} 2>&1 >#{#output_file}`
end
end
def data
File.readlines(#output_file)
end
def parse
while #child.alive?
# parse data # TODO: need to implement real parsing
sleep 0.5
end
end
end
fc = FileConverter.new
fc.parse
Related
I have a ruby script I'm making to upload a directory of roles to chef server. Doing this 1 at a time with a .each loop is slow. So I added parallelism by running each command in separate threads. Now I'm trying to figure out how to store the output of the commands so I can read them back in order of the threads that were created. The roles array is already in alphabetical order. We also use bash_profile aliases for running the knife command with different configuration files for dev and prod.
I've tried many different ways to run the bash command and trying to store the output in an array or to a file etc... Currently this displays the output from each thread as it runs or finishes so out put is hard to read or tell if everything finished correctly and the files that the bash command output is supposed to be redirecting to get created but have empty content.
Sorry if this script isn't the easiest to read. I've only been doing ruby for a little over a year now and self taught myself it when we started to get into chef. I didn't have a programming background before that.
#!/opt/chefdk/embedded/bin/ruby
def print_usage_and_exit
puts 'Need to specify 1 or more role.json files or no arguments to upload all roles'
puts "ruby #{__FILE__} or ruby #{__FILE__} [role1.json] [role2.json] [...]"
exit(1)
end
def fetch_roles
roles = []
current_dir = File.dirname(__FILE__)
Dir.foreach("#{current_dir}/roles") do |role|
next if role == '.' || role == '..' || role == 'README.md'
roles.push(role)
end
roles
end
upload = []
i = 0
roles = (ARGV.empty? ? fetch_roles : ARGV[0..-1])
# Probably redundant, but a cheap check to make sure we're only looking at json files
roles.keep_if { |b| b.end_with?('.json') }
print_usage_and_exit if roles.empty?
print "\nSpecify new knife command if you have seperate knife command for dev and prod created with .bash_profile function."
print "\nLeave blank to use default 'knife' command"
print "\nWhich knife command to use: "
knife = ($stdin.gets.chomp('') ? 'knife' : $stdin.gets.chomp)
print "\n**** Starting upload of roles to chef server ****\n"
roles.each do |role|
upload[i] = Thread.new{
system("bash", "-cl", "#{knife} role from file #{role} > /tmp/#{role}.log")
}
i += 1
end
upload.each {|t| t.join}
roles.each do |role|
logfile = "/tmp/#{role}.log"
logmsg = open(logfile)
print "\n#{logmsg.read}\n"
#FileUtils.rm("/tmp/#{role}.log")
end
print "\n**** Finished uploading roles to chef server ****\n"
The right way to do this is knife upload roles/. That doesn't actually answer your question per se, but I think you'll find it a lot simpler.
I prefer to use Open3's caputure3 function to execute subprocesses, as it makes it easy to handle all the various details ( stdin, stdout, stderr, environment variables, etc ).
Pair that with the use of thread-local data, a built in feature of ruby threads, and you have a pretty easy method of running subprocesses. I'm a big fan of using threads for this kind of concurrency. The GIL prevents ruby from running all the threads concurrently, but the capture3 subprocesses run concurrently anyway, so it doesn't really matter.
require 'open3'
commands = [
'true',
'echo "a more complex command from `pwd`" 1>&2 && echo "and stdout"',
]
threads = []
commands.each_with_index do |cmd, i|
threads[i] = Thread.new do
stdout, stderr, status = Open3.capture3("bash", stdin_data: cmd)
Thread.current['stdout'] = stdout
Thread.current['stderr'] = stderr
Thread.current['status'] = status
end
end
threads.each_with_index do |th,i|
th.join
puts "Thread # #{i}:"
%w( stdout stderr status ).each do |s|
puts "\t#{s}: #{th[s]}"
end
puts
end
The results are exactly what you'd expect:
$ ruby ./t.rb
Thread # 0:
stdout:
stderr:
status: pid 34244 exit 0
Thread # 1:
stdout: and stdout
stderr: a more complex command from /Users/dfarrell/t
status: pid 34243 exit 0
You can use the exit status to give a final summary of how many commands failed or succeeded.
I'm using a command line program, it works as mentioned below:
$ ROUTE_TO_FOLDER/app < "long text"
If "long text" is written using the parameters "app" needs, then it will fill a text file with results. If not, it will fill the text file with dots continuously (I can't handle or modify the code of "app" in order to avoid this).
In a ruby script there's a line like this:
text = "long text that will be used by app"
output = system("ROUTE_TO_FOLDER/app < #{text}")
Now, if text is well written, there won't be problems and I will get an output file as mentioned before. The problem comes when text is not well written. What happens next is that my ruby script hangs and I'm not sure how to kill it.
I've found Open3 and I've used the method like this:
irb> cmd = "ROUTE_TO_FOLDER/app < #{text}"
irb> stdin, stdout, stderr, wait_thr = Open3.popen3(cmd)
=> [#<IO:fd 10>, #<IO:fd 11>, #<IO:fd 13>, #<Thread:0x007f3a1a6f8820 run>]
When I do:
irb> wait_thr.value
it also hangs, and :
irb> wait_thr.status
=> "sleep"
How can I avoid these problems? Is it not recognizing that "app" has failed?
wait_thr.pid provides you the pid of the started process. Just do
Process.kill("KILL",wait_thr.pid)
when you need to kill it.
You can combine it with detecting if the process is hung (continuously outputs dots) in one of the two ways.
1) Set a timeout for waiting for the process:
get '/process' do
text = "long text that will be used by app"
cmd = "ROUTE_TO_FOLDER/app < #{text}"
Open3.popen3(cmd) do |i,o,e,w|
begin
Timeout.timeout(10) do # timeout set to 10 sec, change if needed
# process output of the process. it will produce EOF when done.
until o.eof? do
# o.read_nonblock(N) ...
end
end
rescue Timeout::Error
# here you know that the process took longer than 10 seconds
Process.kill("KILL", w.pid)
# do whatever other error processing you need
end
end
end
2) Check the process output. (The code below is simplified - you probably don't want to read the output of your process into a single String buf first and then process, but I guess you get the idea).
get '/process' do
text = "long text that will be used by app"
cmd = "ROUTE_TO_FOLDER/app < #{text}"
Open3.popen3(cmd) do |i,o,e,w|
# process output of the process. it will produce EOF when done.
# If you get 16 dots in a row - the process is in the continuous loop
# (you may want to deal with stderr instead - depending on where these dots are sent to)
buf = ""
error = false
until o.eof? do
buf << o.read_nonblock(16)
if buf.size>=16 && buf[-16..-1] == '.'*16
# ok, the process is hung
Process.kill("KILL", w.pid)
error = true
# you should also get o.eof? the next time you check (or after flushing the pipe buffer),
# so you will get out of the until o.eof? loop
end
end
if error
# do whatever error processing you need
else
# process buf, it contains all the output
end
end
end
I'm trying to exec a shell process such that its standard output is prefixed with an identifier.
My approach is to write a custom IO object that re-implements write, passing it as the :out argument to exec (documented under Process::spawn).
require "delegate"
class PrefixedStdout < DelegateClass(IO)
def initialize(prefix, io)
#prefix = prefix
super(io)
end
def write(str)
super("#{#prefix}: #{str}")
end
end
pr_stdout = PrefixedStdout.new("my_prefix", $stdout)
pr_stdout.write("hello\n") # outputs "my_prefix: hello"
exec("echo hello", out: pr_stdout) # outputs "hello"
Somehow exec is bypassing PrefixedStdout#write and calling $stdout.write directly. How do I force exec to use my prefixed output stream as its stdout?
What gets preserved in the other process is the underlying file descriptor (or rather they are hooked up under the hood), so as I commented I don't think you'll ever get writes to that descriptor to be funnelled through your write method - exec replaces the running process with a new one.
A possible approach is to create a pipe, pass one end to your child process and then read from the other end, inserting prefixes as needed,
For example you might do
IO.pipe do |read_pipe, write_pipe|
fork do
exec("echo hello", out: write_pipe)
end
write_pipe.close
while line = read_pipe.gets
puts "prefix: #{line}"
end
end
You might also be interested in IO.popen which wraps some of this up.
Somehow exec is bypassing PrefixedStdout#write and calling
$stdout.write
Take a look at this example:
class MyIO < IO
def initialize(fd)
super
end
def write(str)
STDOUT.puts 'write called'
super
end
end
fd = IO.sysopen("data.txt", "w")
io = MyIO.new(fd)
io.write "goodbye\n"
puts '---now with exec()...'
exec("echo hello", :out => io)
--output:--
write called
---now with exec()...
Now, what do you think is in the file data.txt?
spoiler:
$cat data.txt
hello
So passing an IO object to exec() 'works', but not the way you expected: exec() never calls io.write() to write the output of the child process to io. Instead, I assume exec() obtains the file descriptor for io, then passes it to some C code, which does some system level redirection of the output from the child process to the file data.txt.
Do you have to use exec()? If not:
prefix = "prefix: "
cmd = 'echo hello'
output = `#{cmd}`
puts "#{prefix}#{output}"
--output:--
prefix: hello
I want to be able to stream the output of a child process in Ruby
e.g.
p `ping google.com`
I want to see the ping responses immediately; I don't want to wait for the process to complete.
You can do the following instead of using backticks:
IO.popen('ping google.com') do |io|
io.each { |s| print s }
end
Cheers!
You should use IO#popen:
IO.popen("ping -c 3 google.com") do |data|
while line = data.gets
puts line
end
end
If you'd like to capture both the stdout and stderr you can use popen2e:
require 'open3'
Open3.popen2e('do something') do |_stdin, stdout_err, _wait_thr|
stdout_err.each { |line| puts line }
end
I'm creating a script to wrap jdb (java debugger). I essentially want to wrap this process and proxy the user interaction. So I want it to:
start jdb from my script
send the output of jdb to stdout
pause and wait for input when jdb does
when the user enters commands, pass it to jdb
At the moment I really want a pass thru to jdb. The reason for this is to initialize the process with specific parameters and potentially add more commands in the future.
Update:
Here's the shell of what ended up working for me using expect:
PTY.spawn("jdb -attach 1234") do |read,write,pid|
write.sync = true
while (true) do
read.expect(/\r\r\n> /) do |s|
s = s[0].split(/\r\r\n/)
s.pop # get rid of prompt
s.each { |line| puts line }
print '> '
STDOUT.flush
write.print(STDIN.gets)
end
end
end
Use Open3.popen3(). e.g.:
Open3.popen3("jdb args") { |stdin, stdout, stderr|
# stdin = jdb's input stream
# stdout = jdb's output stream
# stderr = jdb's stderr stream
threads = []
threads << Thread.new(stderr) do |terr|
while (line = terr.gets)
puts "stderr: #{line}"
end
end
threads << Thread.new(stdout) do |terr|
while (line = terr.gets)
puts "stdout: #{line}"
end
end
stdin.puts "blah"
threads.each{|t| t.join()} #in order to cleanup when you're done.
}
I've given you examples for threads, but you of course want to be responsive to what jdb is doing. The above is merely a skeleton for how you open the process and handle communication with it.
The Ruby standard library includes expect, which is designed for just this type of problem. See the documentation for more information.