The following code is a simplification of my current situation. I have a JSON log source which I continuously fetch and write to stdout with puts.
#!/usr/bin/env ruby
require "json"
loop do
puts({ value: "foobar" }.to_json)
sleep 1
end
I want to be able to pipe the output of this script into jq for further processing, but in a 'stream'-friendly way, using unix pipes. Running the above code like so:
./my_script | jq
Results in an empty output. However, if I place an exit statement after the sleep call, the output is sent through the pipe to jq as expected. I was able to solve this problem by calling $stdout.flush following the puts call. While it's working now, I'm not sure why. $stdout.sync is set to true by default (see IO#sync). It seems to me that if sync was enabled, then Ruby should be doing no output buffering, and calling $stdout.flush should not be required - yet it is.
My follow-up question is about using tail instead of jq. It seems to me that I should be able to pipe a text stream into tail the same way I pipe it into jq, but neither method (with the $stdout.flush call or without it) works - the output is just empty.
As #Ry points out in the comments, $stdout.sync is true by default in IRB, but this is not necessarily the same for scripts.
So you should set $stdout.sync = true to be sure to prevent buffering.
Related
My question is a follow up to this question: No return on command line when running Ruby script because the answer doesn't offer an explanation:
Take the following file, script.rb:
def hello(names)
names.map {|name| "#{name} is awesome!"}
end
hello(["mark", "tony", "scott"])
When executing the file on the command line with ruby script.rb the return value of the following function does not appear. However, testing the method in IRB or by dropping into the code with PRY outputs an explicit return value. Why isn't the return value visible via script execution?
P.S. I am aware that using puts will output code into the terminal but I'm interested in why the return value doesn't output.
Because both IRB or Pry are REPL's
REPL stands for: read, evaluate, print and loop. And that's exactly what both Pry and IRB are doing.
They will first read your input, evaluate your code, print the result of the code execution and then start over.
A Ruby script can't return a value directly like you want it to, the Bash shell works in the same way, a Bash function can't return a string directly. You can return a string (with stdout) and assign it to the variable.
~:$~ cat scr.rb
~:$~ puts "Something here"
~:$~ echo $(ruby ./scr.rb)
Something here
~:$~ myvar=$(echo $(ruby ./scr.rb))
~:$~ echo $myvar
Something here
It's really simple: Bash (or whatever shell you are using) and Ruby are different programming languages. Bash doesn't know anything about Ruby. It doesn't know what a " Ruby return" is, it doesn't know what a "Ruby array" is, it doesn't know what a "Ruby string" is. Therefore, you simply cannot possibly return a Ruby object to Bash.
In fact, the shell usually just uses some operating system function to execute the Ruby script (e.g. the classical fork and exec or something like vfork or clone). If you wanted to return values this way, the operating system kernel would have to know about the semantics of every programming language ever invented plus every programming language that is going to be invented in the future. That is just not feasible.
Note that a command can return a value to the shell, namely an integer between 0 and 255 intended as a status code (with 0 meaning "success" and nonzero meaning "error"), and you can set that return value by calling Kernel#exit.
I used to have the same question myself when I started coding. If you have a closer look at your code you can see why it doesn't print anything. You are actually no asking it in your code. Imagine having a huge script of thousands of lines and you want to execute it. You would have millions of pointless outputs if ruby myscript.rb worked the same way as the REPLs.
In addition, if you do want it to work that way, you can just do require the script inside the REPL session ( require_relative 'your_script' ) and then if you call your function hello it will work the way you describe.
I can use the ruval gem. It evaluates each statement and returns its value.
$ ruval names.rb
def hello(names)
names.map {|name| "#{name} is awesome!"}
end
=> hello
hello(["mark", "tony", "scott"])
=> ["mark is awesome!", "tony is awesome!", "scott is awesome!"]
This command works fine when using backticks to make a system call:
aspell -a < /path/textfile
However, if I try this it does not work:
result = IO.popen(["aspell", "-a", "<", "/path/textfile"]).read
It seems to be an issue with the angle bracket, because this works fine:
result = IO.popen([ "aspell", "--help"]).read
What am I doing wrong here?
That's a shell operation, and as popen is interfacing directly with your command, you can't do it. Instead you assume the responsibility for doing this, which is why the Open3 library exists and things like the popen2 method in particular:
Adapting your code:
require 'open3'
Open3.popen2('aspell', '-a') do |stdin, stdout, wait_thr|
# Feed the contents of /path/textfile into the STDIN of this
# subprocess.
stdin.write(File.read('/path/textfile'))
stdin.close
# Read the results back
result = stdout.read
end
This might be advantageous since you no longer need to write to a temporary file to do operations like this, you can just feed in data directly through the stdin pipe.
I try to run an external command in Ruby, and parse its output .
IO.popen(command, :err=>[:child, :out]) {|ls_io|
ls_io.each do |line|
print line
end
}
This way of doing it works wonders… except when I parse the progress-output of a c-program that shows it progress to stdout with \r.
As long as the c-program has not outputted a \n (that is as long as it has not finished some long-operation), Ruby waits and sees nothing. Then when a \n is outputted, Ruby sees it all
1%\r2%\r3%\r…100%
task finished
I tried all of the many ways to call external commands (eg Calling shell commands from Ruby) ; but none seem to capture the progress. I also tried every opeartor such as STDOUT.sync = true, and the c-program does call fflush(stdout)
I finally found a workaroud. I do :
IO.popen(commande, :err=>[:child, :out]) {|ls_io|
while true
byte=ls_io.read(1)
if byte.nil?
break
end
print byte
end
}
It's stupid… but it works.
Any more elegant way, and much more efficient way to do this ? Performance is terrible, as if the "refresh rate" was slow.
Set the input record separator to "\r" right before your block (provided you know it in advance):
$/ = "\r"
Reference of global preset variables: http://www.zenspider.com/Languages/Ruby/QuickRef.html#pre-defined-variables
I am currently attempting to figure out a way to call a Perl script from Ruby and have it output as if I was in the terminal and would allow me to provide input if it is needed.
I have figured out how I can do this and get the input after the fact but because the Perl script is still running, I am not able to run anything else.
I should note that I can not edit the Perl scripts. These scripts are being provided and this Ruby script is being made to make the process of running all of the Perl scripts easier and ensuring they are in the right order.
upgradestatus = `#{upgradearray[arraylocation]}`
This would be the relevant part my code for this. I have attempted a few other variations of how to do this but I get the same situation every time. When the script starts running it requires input so it just sits there.
You can't do what you want using backticks, %x or as a normal sub-shell, because they lack the ability to watch the output of the sub-command's output.
You could do it using Open3's popen2 or popen3 methods. They let you send to the STDIN stream for the called program, and receive data from the STDOUT. popen3 also lets you see/capture the STDOUT stream too. Unfortunately, often you have to send, then close the STDIN stream before the called program will return its information, which might be the case of the Perl scripts.
If you need more control, look into using Ruby's built-in Pty module. It's designed to let you talk to a running app through a scripting mechanism. You have to set up code to look for prompts, then respond to them by sending back the appropriate data. It can be simple, or it can be a major PITA, depending on the code you're talking to.
This is the example for the open command:
PTY.open {|m, s|
p m #=> #<IO:masterpty:/dev/pts/1>
p s #=> #<File:/dev/pts/1>
p s.path #=> "/dev/pts/1"
}
# Change the buffering type in factor command,
# assuming that factor uses stdio for stdout buffering.
# If IO.pipe is used instead of PTY.open,
# this code deadlocks because factor's stdout is fully buffered.
require 'io/console' # for IO#raw!
m, s = PTY.open
s.raw! # disable newline conversion.
r, w = IO.pipe
pid = spawn("factor", :in=>r, :out=>s)
r.close
s.close
w.puts "42"
p m.gets #=> "42: 2 3 7\n"
w.puts "144"
p m.gets #=> "144: 2 2 2 2 3 3\n"
w.close
# The result of read operation when pty slave is closed is platform
# dependent.
ret = begin
m.gets # FreeBSD returns nil.
rescue Errno::EIO # GNU/Linux raises EIO.
nil
end
p ret #=> nil
I have a simple Ruby script that looks like this
require 'csv'
while line = STDIN.gets
array = CSV.parse_line(line)
puts array[2]
end
But when I try using this script in a Unix pipeline like this, I get 10 lines of output, followed by an error:
ruby lib/myscript.rb < data.csv | head
12080450
12080451
12080517
12081046
12081048
12081050
12081051
12081052
12081054
lib/myscript.rb:4:in `write': Broken pipe - <STDOUT> (Errno::EPIPE)
Is there a way to write the Ruby script in a way that prevents the broken pipe exception from being raised?
head is closing the standard output stream after it has read all the data it needs. You should handle the exception and stop writing to standard output. The following code will abort the loop once standard output has been closed:
while line = STDIN.gets
array = CSV.parse_line(line)
begin
puts array[2]
rescue Errno::EPIPE
break
end
end
The trick I use is to replace head with sed -n 1,10p.
This keeps the pipe open so ruby (or any other program that tests for broken pipes and complains) doesn't get the broken pipe and therefore doesn't complain. Choose the value you want for the number of lines.
Clearly, this is not attempting to modify your Ruby script. There almost certainly is a way to do it in the Ruby code. However, the 'sed instead of head' technique works even where you don't have the option of modifying the program that generates the message.