I want to run multiple time-consuming shell commands from Ruby in a non-blocking (asynchronous) way.
I want to pass options to commands, receive output in Ruby, and (ideally) handle errors.
The script below will naturally take 15 seconds to execute:
test.rb
3.times do |i|
puts `sleep 5; echo #{i} | tail -n 1` # some time-consuming complex command
end
$ /usr/bin/time ruby test.rb
0
1
2
15.29 real 0.13 user 0.09 sys
With Thread, it can apparently be executed in parallel, and it takes only 5 seconds, as expected:
threads = []
3.times do |i|
threads << Thread.new {
puts `sleep 5; echo #{i} | tail -n 1`
}
end
threads.each {|t| t.join() }
$ /usr/bin/time ruby test.rb
2
0
1
5.17 real 0.12 user 0.06 sys
But is this the best approach? Is there any other way?
I have also written using Open3.popen2, but this seems to take 15 seconds to execute as in the first example (unless wrapped in a Thread):
require 'open3'
3.times do |i|
Open3.popen2("sleep 5; echo #{i} | tail -n 1") do |stdin, stdout|
puts stdout.read()
end
end
The documentation describes "block form" and "non-block form", but this "block" refers to anonymous functions, and has nothing to do with concurrency, correct?
Is the Open3 class alone only capable of blocking execution?
The problem with your code is that stdout.read is a blocking call.
You could defer the reading until the command is finished.
At first, create the commands:
commands = Array.new(3) { |i| Open3.popen2("sleep 5; echo hello from #{i}") }
Then, wait for each command to finish:
commands.each { |stdin, stdout, wait_thr| wait_thr.join }
Finally, gather the output and close the IO streams:
commands.each do |stdin, stdout, wait_thr|
puts stdout.read
stdin.close
stdout.close
end
Output: (after 5 seconds)
hello from 0
hello from 1
hello from 2
Related
I'm writing a Thor script to run some tests from a different tool i.e. running a shell command. I'd like the stdout and stderr from the command to continuously stream out into my console.
First attempt was to just use backticks, but naturally the stdout/stderr are not printed (rather, stdout is captured in the return value).
desc "mytask", "my description"
def mytask
`run-my-tests.sh`
end
My next approach was to use Open3 as in:
require "open3"
desc "mytask", "my description"
def mytask
Open3.popen3("run-my-tests.sh") do |stdin, stdout, stderr|
STDOUT.puts(stdout.read())
STDERR.puts(stderr.read())
end
end
However, the above approach will get the whole output from both stdout and stderr and only print at the end. Un my use case, I'd rather see the output of failing and passing tests as it becomes available.
From http://blog.bigbinary.com/2012/10/18/backtick-system-exec-in-ruby.html, I saw that we can read the streams by chunks i.e. with gets() instead of read(). For example:
require "open3"
desc "mytask", "my description"
def mytask
Open3.popen3(command) do |stdin, stdout, stderr|
while (out = stdout.gets()) || err = (stderr.gets())
STDOUT.print(out) if out
STDERR.print(err) if err
end
exit_code = wait_thr.value
unless exit_code.success?
raise "Failure"
end
end
end
Does it look like the best and cleanest approach? Is it an issue that I have to manually try to print stdout before stderr?
I'm using IO.popen for similar task, like so:
IO.popen([env, *command]) do |io|
io.each { |line| puts ">>> #{line}" }
end
To capture stderr I'd just redirect it to stdout command = %w(run-my-tests.sh 2>&1)
Update
I've constructed a script using Open3::popen3 to capture stdout and stderr separately. It obviously has a lot of room form improvement, but basic idea hopefully is clear.
require 'open3'
command = 'for i in {1..5}; do echo $i; echo "$i"err >&2; sleep 0.5; done'
stdin, stdout, stderr, _command_thread = Open3.popen3(command)
reading_thread = Thread.new do
kilobyte = 1024
loop do
begin
stdout.read_nonblock(kilobyte).lines { |line| puts "stdout >>> #{line}" }
stderr.read_nonblock(kilobyte).lines { |line| puts "stderr >>> #{line}" }
rescue IO::EAGAINWaitReadable
next
rescue EOFError
break
end
sleep 1
end
end
reading_thread.join
stdin.close
stdout.close
stderr.close
Seems to me like the simplest way to run a shell command and not try to capture the stdout or stderr (instead, let them bubble up as they come) was something like:
def run *args, **options
pid = spawn(*args, options)
pid, status = Process.wait2(pid)
exit(status.exitstatus) unless status.success?
end
The problem with backticks or system() is that the former captures the stdout and the latter only returns whether the command succeeded or not. spawn() is a more informative alternative to system(). I'd rather have my Thor script tool fail as if it was merely a wrapper for those shell commands.
Goal: I am writing a workflow command-line program in ruby that sequentially executes other programs on the UNIX shell, some of which require the user to enter input.
Problem: Although I can successfully handle the stdout and stderr thanks to this helpful blog post by Nick Charlton, I am however stuck on capturing user input and passing it into the sub-processes via the command line. The code is as follows:
Method
module CMD
def run(cmd, &block)
Open3.popen3(cmd) do |stdin, stdout, stderr, thread|
Thread.new do # STDOUT
until (line = stdout.gets).nil? do
yield nil, line, nil, thread if block_given?
end
end
Thread.new do # STDERR
until (line = stderr.gets).nil? do
yield nil, nil, line, thread if block_given?
end
end
Thread.new do # STDIN
# ????? How to handle
end
thread.join
end
end
end
Calling the method
This example calls the shell command units which prompts the user to enter a unit of measurement and then prompts for a unit to convert to. This is how it would look in the shell
> units
586 units, 56 prefixes # stdout
You have: 1 litre # user input
You want: gallons # user input
* 0.26417205 # stdout
/ 3.7854118 # stdout
When I run this from my program I expect to be able to interact with it in exactly the same way.
unix_cmd = 'units'
run unix_cmd do | stdin, stdout, stderr, thread|
puts "stdout #{stdout.strip}" if stdout
puts "stderr #{stderr.strip}" if stderr
# I'm unsure how I would allow the user to
# interact with STDIN here?
end
Note: Calling the run method this way allows the user to be able to parse the output, control process flow and add custom logging.
From what I've gathered about STDIN, the snippet below is as close as I've come in understanding how to handle STDIN, there are clearly some gaps in my knowledge because I'm still unsure how to integrate this into my run method above and pass the input into the child process.
# STDIN: Constant declared in ruby
# stdin: Parameter declared in Open3.popen3
Thread.new do
# Read each line from the console
STDIN.each_line do |line|
puts "STDIN: #{line}" # print captured input
stdin.write line # write input into stdin
stdin.sync # sync the input into the sub process
break if line == "\n"
end
end
Summary: I wish to understand how to handle user input from the command-line via the Open3.popen3 method so that I can allow users to enter data into various sequence of sub-commands called from my program.
Here's something that should work:
module CMD
def run(cmd, &block)
Open3.popen3(cmd) do |stdin, stdout, stderr, thread|
Thread.new do # STDOUT
until (line = stdout.gets).nil? do
yield nil, line, nil, thread if block_given?
end
end
Thread.new do # STDERR
until (line = stderr.gets).nil? do
yield nil, nil, line, thread if block_given?
end
end
t = Thread.new { loop { stdin.puts gets } }
thread.join
t.kill
end
end
end
I've just added two lines to your original run method: t = Thread.new { loop { stdin.puts gets } }, and t.kill.
After a lot of reading about STDIN as well as some good old trial and error, I discovered an implementation not to dissimilar to Charles Finkel's answer but with some subtle differences.
require "open3"
module Cmd
def run(cmd, &block)
Open3.popen3(cmd) do |stdin, stdout, stderr, thread|
# We only need to check if the block is provided once
# rather than every cycle of the loop as we were doing
# in the original question.
if block_given?
Thread.new do
until (line = stdout.gets).nil? do
yield line, nil, thread
end
end
Thread.new do
until (line = stderr.gets).nil? do
yield nil, line, thread
end
end
end
# $stdin.gets reads from the console
#
# stdin.puts writes to child process
#
# while thread.alive? means that we keep on
# reading input until the child process ends
Thread.new do
stdin.puts $stdin.gets while thread.alive?
end
thread.join
end
end
end
include Cmd
Calling the method like so:
run './test_script.sh' do | stdout, stderr, thread|
puts "#{thread.pid} stdout: #{stdout}" if stdout
puts "#{thread.pid} stderr: #{stderr}" if stderr
end
Where test_script.sh is as follows:
echo "Message to STDOUT"
>&2 echo "Message to STDERR"
echo "enter username: "
read username
echo "enter a greeting"
read greeting
echo "$greeting $username"
exit 0
Produces the following successful output:
25380 stdout: Message to STDOUT
25380 stdout: enter username:
25380 stderr: Message to STDERR
> Wayne
25380 stdout: enter a greeting
> Hello
25380 stdout: Hello Wayne
Note: You will notice the stdout and stderr don't appear in order, this is a limitation I'm yet to solve.
If you're interested in knowing more about stdin it's worth reading the following answer to the question - What is the difference between STDIN and $stdin in Ruby?
I'm running a long-running shell command inside ruby script like this:
Open3.popen2(command) {|i,o,t|
while line = o.gets
MyRubyProgram.read line
puts line
end
}
So I would be able to look at the command output in the shell window.
How do I attach STDIN to command input?
You need to:
Wait for the user input from STDIN
Wait for the output of the command from the popen3 -- o
You may need IO.select or other IO scheduler, or some other multi-task scheduler, such as Thread.
Here is a demo for the Thread approach:
require 'open3'
Open3.popen3('ruby -e "while line = gets; print line; end"') do |i, o, t|
tin = Thread.new do
# here you can manipulate the standard input of the child process
i.puts "Hello"
i.puts "World"
i.close
end
tout = Thread.new do
# here you can fetch and process the standard output of the child process
while line = o.gets
print "COMMAND => #{line}"
end
end
tin.join # wait for the input thread
tout.join # wait for the output thread
end
I'm looking for something equivalent of the backticks operator (``) with the capability to display output during shell command execution.
I saw a solution in another post:
(Running a command from Ruby displaying and capturing the output)
output = []
IO.popen("ruby -e '3.times{|i| p i; sleep 1}'").each do |line|
p line.chomp
output << line.chomp
end
p output
This solution doesn't fit my needs since $? remains nil after the shell command execution. The solution I'm looking for should also set $? (returning the value of $?.exitstatus in another way is also sufficient)
Thanks!
First, I'd recommend using one of the methods in Open3.
I use capture3 for one of my systems where we need to grab the output of STDOUT and STDERR of a lot of command-line applications.
If you need a piped sub-process, try popen3 or one of the other "pipeline" commands.
Here's some code to illustrate how to use popen2, which ignores the STDERR channel. If you want to track that also use popen3:
require 'open3'
output = []
exit_status = Open3.popen2(ENV, "ruby -e '3.times{|i| p i; sleep 1}'") { |stdin, stdout, thr|
stdin.close
stdout.each_line do |o|
o.chomp!
output << o
puts %Q(Read from pipe: "#{ o }")
end
thr.value
}
puts "Output array: #{ output.join(', ') }"
puts "Exit status: #{ exit_status }"
Running that outputs:
Read from pipe: "0"
Read from pipe: "1"
Read from pipe: "2"
Output array: 0, 1, 2
Exit status: pid 43413 exit 0
The example code shows one way to do it.
It's not necessary to use each_line, but that demonstrates how you can read line-by-line until the sub-process closes its STDOUT.
capture3 doesn't accept a block; It waits until the child has closed its output and exits, then it returns the content, which is great when you want a blocking process. popen2 and popen3 have blocking and non-blocking versions, but I show only the non-blocking version here to demonstrate how to read and output the content as it comes in from the sub-process.
Try following:
output = []
IO.popen("ruby -e '3.times{|i| p i; sleep 1 }'") do |f|
f.each do |line|
p line.chomp
output << line.chomp
end
end
p $?
prints
"0"
"1"
"2"
#<Process::Status: pid 2501 exit 0>
Using open3
require 'open3'
output = []
Open3.popen2("ruby -e '3.times{|i| p i; sleep 1}'") do |stdin,stdout,wait_thr|
stdout.each do |line|
p line.chomp
output << line.chomp
end
p wait_thr.value
end
Given this Unix shell script:
test.sh:
#!/bin/sh
sleep 2 &
sleep 5 &
sleep 1 &
wait
time ./test.sh
real 0m5.008s
user 0m0.040s
sys 0m0.000s
How would you accomplish the same thing in Ruby on a Unix machine?
The sleep commands are just an example, just assume that they are long running external commands instead.
Straight from Process#waitall documentation:
fork { sleep 0.2; exit 2 } #=> 27432
fork { sleep 0.1; exit 1 } #=> 27433
fork { exit 0 } #=> 27434
p Process.waitall
Of course, instead of using Ruby's sleep, you can call whichever external command using Kernel#system, or backtick operator.
#!/usr/bin/env ruby
pids = []
pids << Kernel.fork { `sleep 2` }
pids << Kernel.fork { `sleep 5` }
pids << Kernel.fork { `sleep 1` }
pids.each { |pid| Process.wait(pid) }
To answer my own question (just found out about this):
#!/usr/bin/ruby
spawn 'sleep 2'
spawn 'sleep 5'
spawn 'sleep 1'
Process.waitall
On ruby 1.8 you need to install the sfl gem and also require this:
require 'rubygems'
require 'sfl'