My Ruby script is running a shell command and parsing the output from it. However, it seems the command is first executed and output saved in an array. I would like to be able to access the output lines in real time just as they are printed. I've played around with threads, but haven't got it to work. Any suggestions?
You are looking for pipes. Here is an example:
# This example runs the netstat command via a pipe
# and processes the data in Ruby as it come back
pipe = IO.popen("netstat 3")
while (line = pipe.gets)
print line
print "and"
end
When call methods/functions to run system/shell commands, your interpreter spawns another process to run it and waits for it to finish, then gives you the output.
Even if you use threads, the only thing that you would accomplish is not letting your program to hang while the command is run, but you still won't get the output till its done.
I think you can accomplish that with pipes, but I am not sure how.
#Marcel got it.
Related
Say you have a shell command like
cat file1 | ./my_script
Is there any way from inside the 'my_script' command to detect the command run first as the pipe input (in the above example cat file1)?
I've been digging into it and so far I've not found any possibilities.
I've been unable to find any environment variables set in the process space of the second command recording the full command line, the command data the my_script commands sees (via /proc etc) is just _./my_script_ and doesn't include any information about it being run as part of a pipe. Checking the process list from inside the second command even doesn't seem to provide any data since the first process seems to exit before the second starts.
The best information I've been able to find suggests in bash in some cases you can get the exit codes of processes in the pipe via PIPESTATUS, unfortunately nothing similar seems to be present for the name of commands/files in the pipe. My research seems to be saying it's impossible to do in a generic manner (I can't control how people decide to run my_script so I can't force 3rd party pipe replacement tools to be used over build in shell pipes) but it just at the same time doesn't seem like it should be impossible since the shell has the full command line present as the command is run.
(update adding in later information following on from comments below)
I am on Linux.
I've investigated the /proc/$$/fd data and it almost does the job. If the first command doesn't exit for several seconds while piping data to the second command can you read /proc/$$/fd/0 to see the value pipe:[PIPEID] that it symlinks to. That can then be used to search through the rest of the /proc//fd/ data for other running processes to find another process with a pipe open using the same PIPEID which gives you the first process pid.
However in most real world tests I've done of piping you can't trust that the first command will stay running long enough for the second one to have time to locate it's pipe fd in /proc before it exits (which removes the proc data preventing it being read). So if this method will return any information is something I can't rely on.
What I would like to do is:
run a ruby script...
that executes a shell command
and redirects it to a named pipe accessible outside the script
from the system shell, read from that pipe
That is, have the Ruby script capture some command output and redirect it in such a way that it's connectable to from outside the script?
I want to mention that the script cannot simply start and exit, since it's a REPL. The idea is that using the REPL you would be able to run a command and redirect its output elsewhere to consume it.
Using abort and an exit message, will pass the message to STDERR (and the script will fail with exit code 1). You can pass this shell command output in this way.
This is possibly not the only (or best) way, but it has worked for me in the past.
[edit]
You can also redirect the output to a file (using standard methods), and read that file outside the ruby script.
require 'open3'
stdin, stderr, status = Open3.capture3(commandline)
stdin.chomp #Here, you should ge
Incase, if someone wanted to use you can get the output via stdin.chomp
I'm using a bash script to automatically run a simulation program. This program periodically prints the current status of the simulation in the console, like "Iteration step 42 ended normally".
Is it possible to abort the script, if the console output is something like "warning: parameter xyz outside range of validity"?
And what can I do, if the console output is piped to a text file?
Sorry if this sounds stupid, I'm new to this :-)
Thanks in advance
This isn't an ideal job for Bash. However, you can certainly capture and test STDOUT inside a Bash iteration loop using an admixture of conditionals, grep-like tools, and command substitution.
On the other hand, if Bash isn't doing the looping (e.g. it's just waiting for an external command to finish) then you need to use something like expect. Expect is purpose-built to monitor output streams for regular expressions, and perform branching based on expression matches.
I have an app that runs continuously, dumping output from a server and sending strings to stdout. I want to process this output with a Ruby script. The strings are \n-terminated.
For example, I'm trying to run this on the command line:
myapp.exe | my_script.rb
...with my_script.rb defined as:
while $stdin.gets
puts $_
end
I ultimately am going to process the strings using regexes and display some summary data, but for now I'm just trying to get the basic functionality hooked up. When I run the above, I get the following error:
my_script.rb:1:in `gets': Bad file descriptor (Errno::EBADF)
from my_script.rb:1
I am running this on Windows Server 2003 R2 SP2 and Ruby 1.8.6.
How do I continuously process stdin in a Ruby script? (Continuously as in not processing a file, but running until I kill it.)
EDIT:
I was able to make this work, sort of. There were several problems standing in my way. For one thing, it may be that using Ruby to process the piped-in stdin from another process doesn't work on Windows 2003R2. Another direction, suggested by Adrian below, was to run my script as the parent process and use popen to connect to myapp.exe as a forked child process. Unfortunately, fork isn't implemented in Windows, so this didn't work either.
Finally I was able to download POpen4, a RubyGem that does implement popen on Windows. Using this in combination with Adrian's suggestion, I was able to write this script which does what I really want -- processes the output from myapp.exe:
file: my_script.rb
require 'rubygems'
require 'popen4'
status =
POpen4::popen4("myapp.exe") do |stdout, stderr, stdin, pid|
puts pid
while s = stdout.gets
puts s
end
end
This script echoes the output from myapp.exe, which is exactly what I want.
Try just plain gets, without the $stdin. If that doesn't work, you might have to examine the output of myapp.exe for non-printable characters with another ruby script, using IO.popen.
gets doesn't always use stdin but instead tries to open a file.
See SO.
Try executing your Ruby script by explicitly calling ruby:
myapp.exe | ruby my_script.rb
I've experienced some odd behavior using stdin in Ruby when relying on Windows to invoke the correct program based on the file associations.
My $SHELL is tcsh. I want to run a C shell script that will call a program many times with some arguments changed each time. The program I need to call is in Fortran. I do not want to edit it. The program only takes arguments once it is executed, but not on the command line. Upon calling the program in the script, the program takes control (this is where I am stuck currently, I can never get out because the script will not execute anything until after the program process stops). At this point I need to pass it some variables, then after several iterations I will need to Ctrl+C out of the program and continue with the script.
How can this be done?
To add to what #Toybuilder said, you can use a "here document". I.e. your script could have
./myfortranprogram << EOF
first line of input
second line of input
EOF
Everything between the "<<EOF" and the "EOF" will be fed to the program's standard input (does Fortran still use "read (5,*)" to read from standard input?)
And because I think #ephemient's comment deserves to be in the answer:
Some more tips: <<'EOF' prevents
interpolation in the here-doc body;
<<-EOF removes all leading tabs (so
you can indent the here-doc to match
its surroundings), and EOF can be
replaced by any token. An empty token
(<<"") indicates a here-doc that stops
at the first empty line.
I'm not sure how portable those ones are, or if they're just tcsh extensions - I've only used the <<EOF type "here document" myself.
What you want to use is Expect.
Uhm, can you feed your Fortran code with a redirection? You can create a temporary file with your inputs, and then pipe it in with the stdin redirect (<).
This is a job for the unix program expect, which can nicely and easily interactively command programs and respond to their prompts.
I was sent here after being told my question was close to being a duplicate of this one.
FWIW, I had a similar problem with a csh C shell script.
This bit of code was allowing the custom_command to execute without getting ANY input arguments:
foreach f ($forecastTimes)
custom_command << EOF
arg1=x$f;2
arg2=ya
arg3=z,z$f
run
exit
EOF
end
It didn't work the first time I tried it, but after I backspaced out all of the white space in that section of the code I removed the space between the "<<" and the "EOF". I also backspaced the closing "EOF" all the way to the left margin. After that it worked:
foreach f ($forecastTimes)
custom_command <<EOF
arg1=x$f;2
arg2=ya
arg3=z,z$f
run
exit
EOF
end
Not a tcsh user, but if the program runs then reads in commands via stdin then you can use shell redirection < to feed it the required commands. If you run it in the background with & you will not block when it is executed. Then you can sleep for a bit, then use whatever tools you have (ps, grep, awk, etc) to discover the program's PID, then use kill to send it SIGTERM which is the same as doing a Ctrl-C.