Calling a Perl script from Ruby - ruby

I am currently attempting to figure out a way to call a Perl script from Ruby and have it output as if I was in the terminal and would allow me to provide input if it is needed.
I have figured out how I can do this and get the input after the fact but because the Perl script is still running, I am not able to run anything else.
I should note that I can not edit the Perl scripts. These scripts are being provided and this Ruby script is being made to make the process of running all of the Perl scripts easier and ensuring they are in the right order.
upgradestatus = `#{upgradearray[arraylocation]}`
This would be the relevant part my code for this. I have attempted a few other variations of how to do this but I get the same situation every time. When the script starts running it requires input so it just sits there.

You can't do what you want using backticks, %x or as a normal sub-shell, because they lack the ability to watch the output of the sub-command's output.
You could do it using Open3's popen2 or popen3 methods. They let you send to the STDIN stream for the called program, and receive data from the STDOUT. popen3 also lets you see/capture the STDOUT stream too. Unfortunately, often you have to send, then close the STDIN stream before the called program will return its information, which might be the case of the Perl scripts.
If you need more control, look into using Ruby's built-in Pty module. It's designed to let you talk to a running app through a scripting mechanism. You have to set up code to look for prompts, then respond to them by sending back the appropriate data. It can be simple, or it can be a major PITA, depending on the code you're talking to.
This is the example for the open command:
PTY.open {|m, s|
p m #=> #<IO:masterpty:/dev/pts/1>
p s #=> #<File:/dev/pts/1>
p s.path #=> "/dev/pts/1"
}
# Change the buffering type in factor command,
# assuming that factor uses stdio for stdout buffering.
# If IO.pipe is used instead of PTY.open,
# this code deadlocks because factor's stdout is fully buffered.
require 'io/console' # for IO#raw!
m, s = PTY.open
s.raw! # disable newline conversion.
r, w = IO.pipe
pid = spawn("factor", :in=>r, :out=>s)
r.close
s.close
w.puts "42"
p m.gets #=> "42: 2 3 7\n"
w.puts "144"
p m.gets #=> "144: 2 2 2 2 3 3\n"
w.close
# The result of read operation when pty slave is closed is platform
# dependent.
ret = begin
m.gets # FreeBSD returns nil.
rescue Errno::EIO # GNU/Linux raises EIO.
nil
end
p ret #=> nil

Related

Why is Ruby STDOUT buffering when I don't expect it to?

The following code is a simplification of my current situation. I have a JSON log source which I continuously fetch and write to stdout with puts.
#!/usr/bin/env ruby
require "json"
loop do
puts({ value: "foobar" }.to_json)
sleep 1
end
I want to be able to pipe the output of this script into jq for further processing, but in a 'stream'-friendly way, using unix pipes. Running the above code like so:
./my_script | jq
Results in an empty output. However, if I place an exit statement after the sleep call, the output is sent through the pipe to jq as expected. I was able to solve this problem by calling $stdout.flush following the puts call. While it's working now, I'm not sure why. $stdout.sync is set to true by default (see IO#sync). It seems to me that if sync was enabled, then Ruby should be doing no output buffering, and calling $stdout.flush should not be required - yet it is.
My follow-up question is about using tail instead of jq. It seems to me that I should be able to pipe a text stream into tail the same way I pipe it into jq, but neither method (with the $stdout.flush call or without it) works - the output is just empty.
As #Ry points out in the comments, $stdout.sync is true by default in IRB, but this is not necessarily the same for scripts.
So you should set $stdout.sync = true to be sure to prevent buffering.

How do I pass an angle bracket ("<") to IO.popen in Ruby?

This command works fine when using backticks to make a system call:
aspell -a < /path/textfile
However, if I try this it does not work:
result = IO.popen(["aspell", "-a", "<", "/path/textfile"]).read
It seems to be an issue with the angle bracket, because this works fine:
result = IO.popen([ "aspell", "--help"]).read
What am I doing wrong here?
That's a shell operation, and as popen is interfacing directly with your command, you can't do it. Instead you assume the responsibility for doing this, which is why the Open3 library exists and things like the popen2 method in particular:
Adapting your code:
require 'open3'
Open3.popen2('aspell', '-a') do |stdin, stdout, wait_thr|
# Feed the contents of /path/textfile into the STDIN of this
# subprocess.
stdin.write(File.read('/path/textfile'))
stdin.close
# Read the results back
result = stdout.read
end
This might be advantageous since you no longer need to write to a temporary file to do operations like this, you can just feed in data directly through the stdin pipe.

Why can't open4 read from stdout when the program is waiting for stdin?

I am using the open4 gem and having problems reading from the spawned processes stdout. I have a ruby program, test1.rb:
print 'hi.' # 3 characters
$stdin.read(1) # block
And another ruby program in the same directory, test2.rb:
require 'open4'
pid, stdin, stdout, stderr = Open4.popen4 'ruby test1.rb'
p stdout.read(2) # 2 characters
When I run the second program:
$ ruby test2.rb
It just sits there forever without printing anything. Why does this happen, and what can I do to stop it?
I needed to change test1.rb to this. I don't know why.
print 'hi.' # 3 characters
$stdout.flush
$stdin.read(1) # block
By default, everything that you printto stdout or to another file is written into a buffer of Ruby (or the standard C library, which is underneath Ruby). The content of the buffer is forwarded to the OS if one of the following events occurs:
The buffer gets full.
You close stdout.
You have printed a newline sequence (`\n')
You call flush explicitly.
For other files, a flush is done on other occasions, too, like ftell.
If you put stdout in unbuffered mode ($stdout.sync = true), the buffer will not be used.
stderr is unbuffered by default.
The reason for doing buffering is efficiency: Aggregating output data in a buffer can save many system call (calls to operating system). System calls are very expensive: They take many hundreds or even thousands of CPU cycles. Avoiding them with a little bit of code and some buffers in user space results in a good speedup.
A good reading on buffering: Why does printf not flush after the call unless a newline is in the format string?
I'm not an expert in process.
From my first sight of API document, the sequence of using open4 is like this:
first send text to stdin, then close stdin and lastly read text from stdout.
So. You can the test2.rb like this
require 'open4'
pid, stdin, stdout, stderr = Open4.popen4 'ruby test1.rb'
stdin.puts "something" # This line is important
stdin.close # It might be optional, open4 might close itself.
p stdout.read(2) # 2 characters

How to get a single character without pressing enter?

How can I get a single keyboard character from the terminal with Ruby without pressing enter?
I tried Curses::getch, but that didn't really work for me.
Since ruby 2.0.0, there is a 'io/console' in the stdlib with this feature
require 'io/console'
STDIN.getch
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/2999
#!/usr/bin/ruby
begin
system("stty raw -echo")
str = STDIN.getc
ensure
system("stty -raw echo")
end
p str.chr
(Tested on my OS X system, may not be portable to all Ruby platforms). See http://www.rubyquiz.com/quiz5.html for some additional suggestions, including for Windows.
#Jay gave a great answer, but there are two problems:
You can mess up default tty state;
You ignore control characters (^C for SIGINT, etc).
A simple fix for that is to save previous tty state and use following parameters:
-icanon - disable canonical input (ERASE and KILL processing);
isig - enable the checking of characters against the special control characters INTR, QUIT, and SUSP.
In the end you would have a function like this:
def get_char
state = `stty -g`
`stty raw -echo -icanon isig`
STDIN.getc.chr
ensure
`stty #{state}`
end
Raw mode (stty raw -echo) unfortunately causes control-C to get sent in as a character, not as a SIGINT. So if you want blocking input like above, but allow the user to hit control-C to stop the program while it's waiting, make sure to do this:
Signal.trap("INT") do # SIGINT = control-C
exit
end
And if you want non-blocking input -- that is, periodically check if the user has pressed a key, but in the meantime, go do other stuff -- then you can do this:
require 'io/wait'
def char_if_pressed
begin
system("stty raw -echo") # turn raw input on
c = nil
if $stdin.ready?
c = $stdin.getc
end
c.chr if c
ensure
system "stty -raw echo" # turn raw input off
end
end
while true
c = char_if_pressed
puts "[#{c}]" if c
sleep 1
puts "tick"
end
Note that you don't need a special SIGINT handler for the non-blocking version since the tty is only in raw mode for a brief moment.
Note: This is and old answer and the solution no longer works on most systems.
But the answer could still be useful for some environments, where the other methods don't work. Please read the comments below.
First you have to install highline:
gem install highline
Then try if the highline method works for you:
require "highline/system_extensions"
include HighLine::SystemExtensions
print "Press any key:"
k = get_character
puts k.chr
And if you are building curses application, you need to call
nocbreak
http://www.ruby-doc.org/stdlib-1.9.3/libdoc/curses/rdoc/Curses.html#method-c-cbreak

How to call shell commands from Ruby

How do I call shell commands from inside of a Ruby program? How do I then get output from these commands back into Ruby?
This explanation is based on a commented Ruby script from a friend of mine. If you want to improve the script, feel free to update it at the link.
First, note that when Ruby calls out to a shell, it typically calls /bin/sh, not Bash. Some Bash syntax is not supported by /bin/sh on all systems.
Here are ways to execute a shell script:
cmd = "echo 'hi'" # Sample string that can be used
Kernel#` , commonly called backticks – `cmd`
This is like many other languages, including Bash, PHP, and Perl.
Returns the result (i.e. standard output) of the shell command.
Docs: http://ruby-doc.org/core/Kernel.html#method-i-60
value = `echo 'hi'`
value = `#{cmd}`
Built-in syntax, %x( cmd )
Following the x character is a delimiter, which can be any character.
If the delimiter is one of the characters (, [, {, or <,
the literal consists of the characters up to the matching closing delimiter,
taking account of nested delimiter pairs. For all other delimiters, the
literal comprises the characters up to the next occurrence of the
delimiter character. String interpolation #{ ... } is allowed.
Returns the result (i.e. standard output) of the shell command, just like the backticks.
Docs: https://docs.ruby-lang.org/en/master/syntax/literals_rdoc.html#label-Percent+Strings
value = %x( echo 'hi' )
value = %x[ #{cmd} ]
Kernel#system
Executes the given command in a subshell.
Returns true if the command was found and run successfully, false otherwise.
Docs: http://ruby-doc.org/core/Kernel.html#method-i-system
wasGood = system( "echo 'hi'" )
wasGood = system( cmd )
Kernel#exec
Replaces the current process by running the given external command.
Returns none, the current process is replaced and never continues.
Docs: http://ruby-doc.org/core/Kernel.html#method-i-exec
exec( "echo 'hi'" )
exec( cmd ) # Note: this will never be reached because of the line above
Here's some extra advice:
$?, which is the same as $CHILD_STATUS, accesses the status of the last system executed command if you use the backticks, system() or %x{}.
You can then access the exitstatus and pid properties:
$?.exitstatus
For more reading see:
http://www.elctech.com/blog/i-m-in-ur-commandline-executin-ma-commands
http://blog.jayfields.com/2006/06/ruby-kernel-system-exec-and-x.html
http://tech.natemurray.com/2007/03/ruby-shell-commands.html
Here's a flowchart based on "When to use each method of launching a subprocess in Ruby". See also, "Trick an application into thinking its stdout is a terminal, not a pipe".
The way I like to do this is using the %x literal, which makes it easy (and readable!) to use quotes in a command, like so:
directorylist = %x[find . -name '*test.rb' | sort]
Which, in this case, will populate file list with all test files under the current directory, which you can process as expected:
directorylist.each do |filename|
filename.chomp!
# work with file
end
Here's the best article in my opinion about running shell scripts in Ruby: "6 Ways to Run Shell Commands in Ruby".
If you only need to get the output use backticks.
I needed more advanced stuff like STDOUT and STDERR so I used the Open4 gem. You have all the methods explained there.
My favourite is Open3
require "open3"
Open3.popen3('nroff -man') { |stdin, stdout, stderr| ... }
Some things to think about when choosing between these mechanisms are:
Do you just want stdout or do you
need stderr as well? Or even
separated out?
How big is your output? Do you want
to hold the entire result in memory?
Do you want to read some of your
output while the subprocess is still
running?
Do you need result codes?
Do you need a Ruby object that
represents the process and lets you
kill it on demand?
You may need anything from simple backticks (``), system(), and IO.popen to full-blown Kernel.fork/Kernel.exec with IO.pipe and IO.select.
You may also want to throw timeouts into the mix if a sub-process takes too long to execute.
Unfortunately, it very much depends.
I'm definitely not a Ruby expert, but I'll give it a shot:
$ irb
system "echo Hi"
Hi
=> true
You should also be able to do things like:
cmd = 'ls'
system(cmd)
One more option:
When you:
need stderr as well as stdout
can't/won't use Open3/Open4 (they throw exceptions in NetBeans on my Mac, no idea why)
You can use shell redirection:
puts %x[cat bogus.txt].inspect
=> ""
puts %x[cat bogus.txt 2>&1].inspect
=> "cat: bogus.txt: No such file or directory\n"
The 2>&1 syntax works across Linux, Mac and Windows since the early days of MS-DOS.
The answers above are already quite great, but I really want to share the following summary article: "6 Ways to Run Shell Commands in Ruby"
Basically, it tells us:
Kernel#exec:
exec 'echo "hello $HOSTNAME"'
system and $?:
system 'false'
puts $?
Backticks (`):
today = `date`
IO#popen:
IO.popen("date") { |f| puts f.gets }
Open3#popen3 -- stdlib:
require "open3"
stdin, stdout, stderr = Open3.popen3('dc')
Open4#popen4 -- a gem:
require "open4"
pid, stdin, stdout, stderr = Open4::popen4 "false" # => [26327, #<IO:0x6dff24>, #<IO:0x6dfee8>, #<IO:0x6dfe84>]
If you really need Bash, per the note in the "best" answer.
First, note that when Ruby calls out to a shell, it typically calls /bin/sh, not Bash. Some Bash syntax is not supported by /bin/sh on all systems.
If you need to use Bash, insert bash -c "your Bash-only command" inside of your desired calling method:
quick_output = system("ls -la")
quick_bash = system("bash -c 'ls -la'")
To test:
system("echo $SHELL")
system('bash -c "echo $SHELL"')
Or if you are running an existing script file like
script_output = system("./my_script.sh")
Ruby should honor the shebang, but you could always use
system("bash ./my_script.sh")
to make sure, though there may be a slight overhead from /bin/sh running /bin/bash, you probably won't notice.
You can also use the backtick operators (`), similar to Perl:
directoryListing = `ls /`
puts directoryListing # prints the contents of the root directory
Handy if you need something simple.
Which method you want to use depends on exactly what you're trying to accomplish; check the docs for more details about the different methods.
Using the answers here and linked in Mihai's answer, I put together a function that meets these requirements:
Neatly captures STDOUT and STDERR so they don't "leak" when my script is run from the console.
Allows arguments to be passed to the shell as an array, so there's no need to worry about escaping.
Captures the exit status of the command so it is clear when an error has occurred.
As a bonus, this one will also return STDOUT in cases where the shell command exits successfully (0) and puts anything on STDOUT. In this manner, it differs from system, which simply returns true in such cases.
Code follows. The specific function is system_quietly:
require 'open3'
class ShellError < StandardError; end
#actual function:
def system_quietly(*cmd)
exit_status=nil
err=nil
out=nil
Open3.popen3(*cmd) do |stdin, stdout, stderr, wait_thread|
err = stderr.gets(nil)
out = stdout.gets(nil)
[stdin, stdout, stderr].each{|stream| stream.send('close')}
exit_status = wait_thread.value
end
if exit_status.to_i > 0
err = err.chomp if err
raise ShellError, err
elsif out
return out.chomp
else
return true
end
end
#calling it:
begin
puts system_quietly('which', 'ruby')
rescue ShellError
abort "Looks like you don't have the `ruby` command. Odd."
end
#output: => "/Users/me/.rvm/rubies/ruby-1.9.2-p136/bin/ruby"
We can achieve it in multiple ways.
Using Kernel#exec, nothing after this command is executed:
exec('ls ~')
Using backticks or %x
`ls ~`
=> "Applications\nDesktop\nDocuments"
%x(ls ~)
=> "Applications\nDesktop\nDocuments"
Using Kernel#system command, returns true if successful, false if unsuccessful and returns nil if command execution fails:
system('ls ~')
=> true
Don't forget the spawn command to create a background process to execute the specified command. You can even wait for its completion using the Process class and the returned pid:
pid = spawn("tar xf ruby-2.0.0-p195.tar.bz2")
Process.wait pid
pid = spawn(RbConfig.ruby, "-eputs'Hello, world!'")
Process.wait pid
The doc says: This method is similar to #system but it doesn't wait for the command to finish.
The easiest way is, for example:
reboot = `init 6`
puts reboot
The backticks (`) method is the easiest one to call shell commands from Ruby. It returns the result of the shell command:
url_request = 'http://google.com'
result_of_shell_command = `curl #{url_request}`
Given a command like attrib:
require 'open3'
a="attrib"
Open3.popen3(a) do |stdin, stdout, stderr|
puts stdout.read
end
I've found that while this method isn't as memorable as
system("thecommand")
or
`thecommand`
in backticks, a good thing about this method compared to other methods is
backticks don't seem to let me puts the command I run/store the command I want to run in a variable, and system("thecommand") doesn't seem to let me get the output whereas this method lets me do both of those things, and it lets me access stdin, stdout and stderr independently.
See "Executing commands in ruby" and Ruby's Open3 documentation.
If you have a more complex case than the common case that can not be handled with ``, then check out Kernel.spawn(). This seems to be the most generic/full-featured provided by stock Ruby to execute external commands.
You can use it to:
create process groups (Windows).
redirect in, out, error to files/each-other.
set env vars, umask.
change the directory before executing a command.
set resource limits for CPU/data/etc.
Do everything that can be done with other options in other answers, but with more code.
The Ruby documentation has good enough examples:
env: hash
name => val : set the environment variable
name => nil : unset the environment variable
command...:
commandline : command line string which is passed to the standard shell
cmdname, arg1, ... : command name and one or more arguments (no shell)
[cmdname, argv0], arg1, ... : command name, argv[0] and zero or more arguments (no shell)
options: hash
clearing environment variables:
:unsetenv_others => true : clear environment variables except specified by env
:unsetenv_others => false : dont clear (default)
process group:
:pgroup => true or 0 : make a new process group
:pgroup => pgid : join to specified process group
:pgroup => nil : dont change the process group (default)
create new process group: Windows only
:new_pgroup => true : the new process is the root process of a new process group
:new_pgroup => false : dont create a new process group (default)
resource limit: resourcename is core, cpu, data, etc. See Process.setrlimit.
:rlimit_resourcename => limit
:rlimit_resourcename => [cur_limit, max_limit]
current directory:
:chdir => str
umask:
:umask => int
redirection:
key:
FD : single file descriptor in child process
[FD, FD, ...] : multiple file descriptor in child process
value:
FD : redirect to the file descriptor in parent process
string : redirect to file with open(string, "r" or "w")
[string] : redirect to file with open(string, File::RDONLY)
[string, open_mode] : redirect to file with open(string, open_mode, 0644)
[string, open_mode, perm] : redirect to file with open(string, open_mode, perm)
[:child, FD] : redirect to the redirected file descriptor
:close : close the file descriptor in child process
FD is one of follows
:in : the file descriptor 0 which is the standard input
:out : the file descriptor 1 which is the standard output
:err : the file descriptor 2 which is the standard error
integer : the file descriptor of specified the integer
io : the file descriptor specified as io.fileno
file descriptor inheritance: close non-redirected non-standard fds (3, 4, 5, ...) or not
:close_others => false : inherit fds (default for system and exec)
:close_others => true : dont inherit (default for spawn and IO.popen)
This is not really an answer but maybe someone will find it useful:
When using TK GUI on Windows, and you need to call shell commands from rubyw, you will always have an annoying CMD window popping up for less then a second.
To avoid this you can use:
WIN32OLE.new('Shell.Application').ShellExecute('ipconfig > log.txt','','','open',0)
or
WIN32OLE.new('WScript.Shell').Run('ipconfig > log.txt',0,0)
Both will store the ipconfig output inside log.txt, but no windows will come up.
You will need to require 'win32ole' inside your script.
system(), exec() and spawn() will all pop up that annoying window when using TK and rubyw.
Not sure about shell commands. I used following for capturing system command's output into a variable val:
val = capture(:stdout) do
system("pwd")
end
puts val
shortened version:
val = capture(:stdout) { system("pwd") }
capture method is provided by active_support/core_ext/kernel/reporting.rb
Simlarly we can also capture standard errors too with :stderr
Here's a cool one that I use in a ruby script on OS X (so that I can start a script and get an update even after toggling away from the window):
cmd = %Q|osascript -e 'display notification "Server was reset" with title "Posted Update"'|
system ( cmd )
You can use format method as below to print some information:
puts format('%s', `ps`)
puts format('%d MB', (`ps -o rss= -p #{Process.pid}`.to_i / 1024))

Resources