I'm using IO.popen to start a subprocess, but I only get the result of everything that happened in the time it took for the subprocess to run (sometimes 5 minutes or whatever) when the subprocess exits. I really need to be able to see everything the subprocess writes to stderr and stdout as-and-when it happens.
So far I could not find anything that works like this, but I'm sure it's possible.
if you need to get output in real time i would recommend to use stdlib PTY instead of popen
something like this:
require 'pty'
cmd = 'echo a; sleep 1; cat /some/file; sleep 1; echo b'
PTY.spawn cmd do |r, w, pid|
begin
r.sync
r.each_line { |l| puts "#{Time.now.strftime('%M:%S')} - #{l.strip}" }
rescue Errno::EIO => e
# simply ignoring this
ensure
::Process.wait pid
end
end
exit "#{cmd} failed" unless $? && $?.exitstatus == 0
> 33:36 - a
> 33:37 - cat: /some/file: No such file or directory
> 33:38 - b
this way you get output instantly, just as in terminal
You might want to use Open3.popen3 from standard library, it gives access to stdin, stdout, and stderr as streams.
Related
I'm writing a Thor script to run some tests from a different tool i.e. running a shell command. I'd like the stdout and stderr from the command to continuously stream out into my console.
First attempt was to just use backticks, but naturally the stdout/stderr are not printed (rather, stdout is captured in the return value).
desc "mytask", "my description"
def mytask
`run-my-tests.sh`
end
My next approach was to use Open3 as in:
require "open3"
desc "mytask", "my description"
def mytask
Open3.popen3("run-my-tests.sh") do |stdin, stdout, stderr|
STDOUT.puts(stdout.read())
STDERR.puts(stderr.read())
end
end
However, the above approach will get the whole output from both stdout and stderr and only print at the end. Un my use case, I'd rather see the output of failing and passing tests as it becomes available.
From http://blog.bigbinary.com/2012/10/18/backtick-system-exec-in-ruby.html, I saw that we can read the streams by chunks i.e. with gets() instead of read(). For example:
require "open3"
desc "mytask", "my description"
def mytask
Open3.popen3(command) do |stdin, stdout, stderr|
while (out = stdout.gets()) || err = (stderr.gets())
STDOUT.print(out) if out
STDERR.print(err) if err
end
exit_code = wait_thr.value
unless exit_code.success?
raise "Failure"
end
end
end
Does it look like the best and cleanest approach? Is it an issue that I have to manually try to print stdout before stderr?
I'm using IO.popen for similar task, like so:
IO.popen([env, *command]) do |io|
io.each { |line| puts ">>> #{line}" }
end
To capture stderr I'd just redirect it to stdout command = %w(run-my-tests.sh 2>&1)
Update
I've constructed a script using Open3::popen3 to capture stdout and stderr separately. It obviously has a lot of room form improvement, but basic idea hopefully is clear.
require 'open3'
command = 'for i in {1..5}; do echo $i; echo "$i"err >&2; sleep 0.5; done'
stdin, stdout, stderr, _command_thread = Open3.popen3(command)
reading_thread = Thread.new do
kilobyte = 1024
loop do
begin
stdout.read_nonblock(kilobyte).lines { |line| puts "stdout >>> #{line}" }
stderr.read_nonblock(kilobyte).lines { |line| puts "stderr >>> #{line}" }
rescue IO::EAGAINWaitReadable
next
rescue EOFError
break
end
sleep 1
end
end
reading_thread.join
stdin.close
stdout.close
stderr.close
Seems to me like the simplest way to run a shell command and not try to capture the stdout or stderr (instead, let them bubble up as they come) was something like:
def run *args, **options
pid = spawn(*args, options)
pid, status = Process.wait2(pid)
exit(status.exitstatus) unless status.success?
end
The problem with backticks or system() is that the former captures the stdout and the latter only returns whether the command succeeded or not. spawn() is a more informative alternative to system(). I'd rather have my Thor script tool fail as if it was merely a wrapper for those shell commands.
I'm running a long-running shell command inside ruby script like this:
Open3.popen2(command) {|i,o,t|
while line = o.gets
MyRubyProgram.read line
puts line
end
}
So I would be able to look at the command output in the shell window.
How do I attach STDIN to command input?
You need to:
Wait for the user input from STDIN
Wait for the output of the command from the popen3 -- o
You may need IO.select or other IO scheduler, or some other multi-task scheduler, such as Thread.
Here is a demo for the Thread approach:
require 'open3'
Open3.popen3('ruby -e "while line = gets; print line; end"') do |i, o, t|
tin = Thread.new do
# here you can manipulate the standard input of the child process
i.puts "Hello"
i.puts "World"
i.close
end
tout = Thread.new do
# here you can fetch and process the standard output of the child process
while line = o.gets
print "COMMAND => #{line}"
end
end
tin.join # wait for the input thread
tout.join # wait for the output thread
end
I'm looking for something equivalent of the backticks operator (``) with the capability to display output during shell command execution.
I saw a solution in another post:
(Running a command from Ruby displaying and capturing the output)
output = []
IO.popen("ruby -e '3.times{|i| p i; sleep 1}'").each do |line|
p line.chomp
output << line.chomp
end
p output
This solution doesn't fit my needs since $? remains nil after the shell command execution. The solution I'm looking for should also set $? (returning the value of $?.exitstatus in another way is also sufficient)
Thanks!
First, I'd recommend using one of the methods in Open3.
I use capture3 for one of my systems where we need to grab the output of STDOUT and STDERR of a lot of command-line applications.
If you need a piped sub-process, try popen3 or one of the other "pipeline" commands.
Here's some code to illustrate how to use popen2, which ignores the STDERR channel. If you want to track that also use popen3:
require 'open3'
output = []
exit_status = Open3.popen2(ENV, "ruby -e '3.times{|i| p i; sleep 1}'") { |stdin, stdout, thr|
stdin.close
stdout.each_line do |o|
o.chomp!
output << o
puts %Q(Read from pipe: "#{ o }")
end
thr.value
}
puts "Output array: #{ output.join(', ') }"
puts "Exit status: #{ exit_status }"
Running that outputs:
Read from pipe: "0"
Read from pipe: "1"
Read from pipe: "2"
Output array: 0, 1, 2
Exit status: pid 43413 exit 0
The example code shows one way to do it.
It's not necessary to use each_line, but that demonstrates how you can read line-by-line until the sub-process closes its STDOUT.
capture3 doesn't accept a block; It waits until the child has closed its output and exits, then it returns the content, which is great when you want a blocking process. popen2 and popen3 have blocking and non-blocking versions, but I show only the non-blocking version here to demonstrate how to read and output the content as it comes in from the sub-process.
Try following:
output = []
IO.popen("ruby -e '3.times{|i| p i; sleep 1 }'") do |f|
f.each do |line|
p line.chomp
output << line.chomp
end
end
p $?
prints
"0"
"1"
"2"
#<Process::Status: pid 2501 exit 0>
Using open3
require 'open3'
output = []
Open3.popen2("ruby -e '3.times{|i| p i; sleep 1}'") do |stdin,stdout,wait_thr|
stdout.each do |line|
p line.chomp
output << line.chomp
end
p wait_thr.value
end
When I do something like the following:
output = `identify some_file`
output == "Output of identify"
But when...
output = `identify non_existant_file`
output != "Error output of identify"
How can I get the error output of system calls?
I found out the answer. The output is being sent to stderr. So I can just add the following at the end of the command to redirect stderr to stdout:
output = `identify any_file 2>&1`
output == "Error or output of identify"
Here is the explanation of this witchcraft
You may use Open3.popen3.
http://www.ruby-doc.org/stdlib-1.9.3/libdoc/open3/rdoc/Open3.html#method-c-popen3
popen3(*cmd, &block) click to toggle source
Open stdin, stdout, and stderr streams and start external executable.
Open3.popen3([env,] cmd... [, opts]) {|stdin, stdout, stderr, wait_thr|
pid = wait_thr.pid # pid of the started process.
...
exit_status = wait_thr.value # Process::Status object returned.
}
I can do something like this in bash:
myCommand arg1 arg2 2>&1 >> myLogFolder/myLogFile.log | tee -a myLogFolder/myLogFile.log
I would like to be able to say this instead:
log.rb myLogFolder/myLogFile.log myCommand arg1 arg2
Using the log.rb script would accomplish two things:
Result in a simpler statement with less redirection tricks and only a single specification of the log file.
Create the log folder, if necessary.
I was looking at Ruby's popen and spawn options, but I don't see a way to split the STDERR stream to two destinations.
This Ruby script satisfies my needs, but maybe there is a better way.
logPath = ARGV[0]
logFolder = File.dirname(logPath)
command = ARGV.slice(1..-1).join(" ")
`mkdir -p #{logFolder}`
exec "#{command} 2>&1 >> #{logPath} | tee -a #{logPath}"
Try this article about implementing functionality similar to tee using Ruby. You should be able to use that as a starting point for a pure-ruby (or at least exec-free) implementation of your shell code.
You can use Open3 module (manual) It returns 3 objects: stdin, stdout, stderr
However, you are not able to preserve the order between stdout and stderr.
Example:
#include <stdio.h>
int main(int argc, char *argv[]) {
fprintf(stdout, "Normal output\n");
fprintf(stderr, "Error output\n");
fprintf(stdout, "Normal output\n");
}
It is captured as:
Normal output
Normal output
Error output
The only way how to preserve order is to run program twice. :-(
Sample ruby code:
#!/usr/bin/env ruby
require 'open3'
require 'pathname'
logfile = ARGV.first()
cmd = ARGV.drop(1).join(" ")
dir = Pathname(logfile).dirname
if not File.directory?(dir)
Dir.mkdir(dir)
end
file = File.open(logfile, "w")
stdin, stdout, stderr = Open3.popen3(cmd)
stdin.close()
file.puts(stdout.read())
error = stderr.read()
file.puts(error)
puts error
file.close