When I do something like the following:
output = `identify some_file`
output == "Output of identify"
But when...
output = `identify non_existant_file`
output != "Error output of identify"
How can I get the error output of system calls?
I found out the answer. The output is being sent to stderr. So I can just add the following at the end of the command to redirect stderr to stdout:
output = `identify any_file 2>&1`
output == "Error or output of identify"
Here is the explanation of this witchcraft
You may use Open3.popen3.
http://www.ruby-doc.org/stdlib-1.9.3/libdoc/open3/rdoc/Open3.html#method-c-popen3
popen3(*cmd, &block) click to toggle source
Open stdin, stdout, and stderr streams and start external executable.
Open3.popen3([env,] cmd... [, opts]) {|stdin, stdout, stderr, wait_thr|
pid = wait_thr.pid # pid of the started process.
...
exit_status = wait_thr.value # Process::Status object returned.
}
Related
I'm writing a Thor script to run some tests from a different tool i.e. running a shell command. I'd like the stdout and stderr from the command to continuously stream out into my console.
First attempt was to just use backticks, but naturally the stdout/stderr are not printed (rather, stdout is captured in the return value).
desc "mytask", "my description"
def mytask
`run-my-tests.sh`
end
My next approach was to use Open3 as in:
require "open3"
desc "mytask", "my description"
def mytask
Open3.popen3("run-my-tests.sh") do |stdin, stdout, stderr|
STDOUT.puts(stdout.read())
STDERR.puts(stderr.read())
end
end
However, the above approach will get the whole output from both stdout and stderr and only print at the end. Un my use case, I'd rather see the output of failing and passing tests as it becomes available.
From http://blog.bigbinary.com/2012/10/18/backtick-system-exec-in-ruby.html, I saw that we can read the streams by chunks i.e. with gets() instead of read(). For example:
require "open3"
desc "mytask", "my description"
def mytask
Open3.popen3(command) do |stdin, stdout, stderr|
while (out = stdout.gets()) || err = (stderr.gets())
STDOUT.print(out) if out
STDERR.print(err) if err
end
exit_code = wait_thr.value
unless exit_code.success?
raise "Failure"
end
end
end
Does it look like the best and cleanest approach? Is it an issue that I have to manually try to print stdout before stderr?
I'm using IO.popen for similar task, like so:
IO.popen([env, *command]) do |io|
io.each { |line| puts ">>> #{line}" }
end
To capture stderr I'd just redirect it to stdout command = %w(run-my-tests.sh 2>&1)
Update
I've constructed a script using Open3::popen3 to capture stdout and stderr separately. It obviously has a lot of room form improvement, but basic idea hopefully is clear.
require 'open3'
command = 'for i in {1..5}; do echo $i; echo "$i"err >&2; sleep 0.5; done'
stdin, stdout, stderr, _command_thread = Open3.popen3(command)
reading_thread = Thread.new do
kilobyte = 1024
loop do
begin
stdout.read_nonblock(kilobyte).lines { |line| puts "stdout >>> #{line}" }
stderr.read_nonblock(kilobyte).lines { |line| puts "stderr >>> #{line}" }
rescue IO::EAGAINWaitReadable
next
rescue EOFError
break
end
sleep 1
end
end
reading_thread.join
stdin.close
stdout.close
stderr.close
Seems to me like the simplest way to run a shell command and not try to capture the stdout or stderr (instead, let them bubble up as they come) was something like:
def run *args, **options
pid = spawn(*args, options)
pid, status = Process.wait2(pid)
exit(status.exitstatus) unless status.success?
end
The problem with backticks or system() is that the former captures the stdout and the latter only returns whether the command succeeded or not. spawn() is a more informative alternative to system(). I'd rather have my Thor script tool fail as if it was merely a wrapper for those shell commands.
This code is not working properly:
my_command = %x{java -version &>/dev/null}
if $? != 0
How do I see what the STDERR/STDOUT status code is?
I tried using puts:
puts "values ===> $? or or '$?' my_command or #{my_command} %x{echo $?}"
but it doesn't work.
Using %x or something equivalent is an easy way of doing this but doesn't give you a lot of control. Instead use the Open3 library and popen3:
require 'open3'
Open3.popen3("java", "-version") do |stdin, stdout, stderr, wait_thr|
version = stdout.chomp
status = wait_thr.value
end
To get the status code :
stdout = %x(ls -l)
exit_code = $?.exitstatus
and then
if exit_code == 0 then
puts "Success"
else
puts "Problem"
end
Goal: I am writing a workflow command-line program in ruby that sequentially executes other programs on the UNIX shell, some of which require the user to enter input.
Problem: Although I can successfully handle the stdout and stderr thanks to this helpful blog post by Nick Charlton, I am however stuck on capturing user input and passing it into the sub-processes via the command line. The code is as follows:
Method
module CMD
def run(cmd, &block)
Open3.popen3(cmd) do |stdin, stdout, stderr, thread|
Thread.new do # STDOUT
until (line = stdout.gets).nil? do
yield nil, line, nil, thread if block_given?
end
end
Thread.new do # STDERR
until (line = stderr.gets).nil? do
yield nil, nil, line, thread if block_given?
end
end
Thread.new do # STDIN
# ????? How to handle
end
thread.join
end
end
end
Calling the method
This example calls the shell command units which prompts the user to enter a unit of measurement and then prompts for a unit to convert to. This is how it would look in the shell
> units
586 units, 56 prefixes # stdout
You have: 1 litre # user input
You want: gallons # user input
* 0.26417205 # stdout
/ 3.7854118 # stdout
When I run this from my program I expect to be able to interact with it in exactly the same way.
unix_cmd = 'units'
run unix_cmd do | stdin, stdout, stderr, thread|
puts "stdout #{stdout.strip}" if stdout
puts "stderr #{stderr.strip}" if stderr
# I'm unsure how I would allow the user to
# interact with STDIN here?
end
Note: Calling the run method this way allows the user to be able to parse the output, control process flow and add custom logging.
From what I've gathered about STDIN, the snippet below is as close as I've come in understanding how to handle STDIN, there are clearly some gaps in my knowledge because I'm still unsure how to integrate this into my run method above and pass the input into the child process.
# STDIN: Constant declared in ruby
# stdin: Parameter declared in Open3.popen3
Thread.new do
# Read each line from the console
STDIN.each_line do |line|
puts "STDIN: #{line}" # print captured input
stdin.write line # write input into stdin
stdin.sync # sync the input into the sub process
break if line == "\n"
end
end
Summary: I wish to understand how to handle user input from the command-line via the Open3.popen3 method so that I can allow users to enter data into various sequence of sub-commands called from my program.
Here's something that should work:
module CMD
def run(cmd, &block)
Open3.popen3(cmd) do |stdin, stdout, stderr, thread|
Thread.new do # STDOUT
until (line = stdout.gets).nil? do
yield nil, line, nil, thread if block_given?
end
end
Thread.new do # STDERR
until (line = stderr.gets).nil? do
yield nil, nil, line, thread if block_given?
end
end
t = Thread.new { loop { stdin.puts gets } }
thread.join
t.kill
end
end
end
I've just added two lines to your original run method: t = Thread.new { loop { stdin.puts gets } }, and t.kill.
After a lot of reading about STDIN as well as some good old trial and error, I discovered an implementation not to dissimilar to Charles Finkel's answer but with some subtle differences.
require "open3"
module Cmd
def run(cmd, &block)
Open3.popen3(cmd) do |stdin, stdout, stderr, thread|
# We only need to check if the block is provided once
# rather than every cycle of the loop as we were doing
# in the original question.
if block_given?
Thread.new do
until (line = stdout.gets).nil? do
yield line, nil, thread
end
end
Thread.new do
until (line = stderr.gets).nil? do
yield nil, line, thread
end
end
end
# $stdin.gets reads from the console
#
# stdin.puts writes to child process
#
# while thread.alive? means that we keep on
# reading input until the child process ends
Thread.new do
stdin.puts $stdin.gets while thread.alive?
end
thread.join
end
end
end
include Cmd
Calling the method like so:
run './test_script.sh' do | stdout, stderr, thread|
puts "#{thread.pid} stdout: #{stdout}" if stdout
puts "#{thread.pid} stderr: #{stderr}" if stderr
end
Where test_script.sh is as follows:
echo "Message to STDOUT"
>&2 echo "Message to STDERR"
echo "enter username: "
read username
echo "enter a greeting"
read greeting
echo "$greeting $username"
exit 0
Produces the following successful output:
25380 stdout: Message to STDOUT
25380 stdout: enter username:
25380 stderr: Message to STDERR
> Wayne
25380 stdout: enter a greeting
> Hello
25380 stdout: Hello Wayne
Note: You will notice the stdout and stderr don't appear in order, this is a limitation I'm yet to solve.
If you're interested in knowing more about stdin it's worth reading the following answer to the question - What is the difference between STDIN and $stdin in Ruby?
I'm using IO.popen to start a subprocess, but I only get the result of everything that happened in the time it took for the subprocess to run (sometimes 5 minutes or whatever) when the subprocess exits. I really need to be able to see everything the subprocess writes to stderr and stdout as-and-when it happens.
So far I could not find anything that works like this, but I'm sure it's possible.
if you need to get output in real time i would recommend to use stdlib PTY instead of popen
something like this:
require 'pty'
cmd = 'echo a; sleep 1; cat /some/file; sleep 1; echo b'
PTY.spawn cmd do |r, w, pid|
begin
r.sync
r.each_line { |l| puts "#{Time.now.strftime('%M:%S')} - #{l.strip}" }
rescue Errno::EIO => e
# simply ignoring this
ensure
::Process.wait pid
end
end
exit "#{cmd} failed" unless $? && $?.exitstatus == 0
> 33:36 - a
> 33:37 - cat: /some/file: No such file or directory
> 33:38 - b
this way you get output instantly, just as in terminal
You might want to use Open3.popen3 from standard library, it gives access to stdin, stdout, and stderr as streams.
In Ruby's popen/spawn, how do I merge both STDOUT and STDERR as a single stream wihthout resorting to using >2&1?
In Python, this would be:
>>> import subprocess
>>> subprocess.check_output('my_prog args', stderr=subprocess.STDOUT, shell=True)
Note the stderr argument.
I use Open3 - as I don't want just stdout - but it already separates them into two streams.
Using the code from your other question, here you go:
cmd = 'a_prog --arg ... --arg2 ...'
Open3.popen2({"MYVAR" => "a_value"}, "#{cmd}", {:err => [:child, :out]}) { |i,o|
# This output should include stderr as well
output = o.read()
repr = "$ #{cmd}\n#{output}"
}
A couple changes:
The third parameter to popen2 will redirect stderr to stdoutl. Note that it needs to be the spawned process's stdout, not the system-wide stdout, so you need to specify :child's :out
You need to use .popen2 instead of .popen3 as it seems the redirection is ignored if you include the 3rd e option for stderr
Because you're using .popen2, you only pass |i,o| to the block:
A bit late, but take a look at Open3.popen2e - docs.
This behaves exactly as popen3, but merges stderr stdout as the second argument to the block.
So you can simply do
cmd = 'a_prog --arg ... --arg2 ...'
Open3.popen2e(cmd) { |input,output|
# Process as desired, with output containing stdout and stderr
}