When you use a system call in a Ruby script, you can get the output of that command like this:
output = `ls`
puts output
That's what this question was about.
But is there a way to show the continuous output of a system call? For example, if you run this secure copy command, to get a file from a server over SSH:
scp user#someserver:remoteFile /some/local/folder/
... it shows continuous output with the progress of the download. But this:
output = `scp user#someserver:remoteFile /some/local/folder/`
puts output
... doesn't capture that output.
How can I show the ongoing progress of the download from inside my Ruby script?
Try:
IO.popen("scp -v user#server:remoteFile /local/folder/").each do |fd|
puts(fd.readline)
end
I think you would have better luck using the ruby standard library to handle SCP (as opposed to forking a shell process). The Net::SCP library (as well as the entire Net::* libraries) are full featured and used with Capistrano to handle remote commands.
Checkout http://net-ssh.rubyforge.org/ for a rundown of what is available.
Tokland answered the question as I asked it, but Adam's approach was what I ended up using. Here was my completed script, which does show a running count of bytes downloaded, and also a percentage complete.
require 'rubygems'
require 'net/scp'
puts "Fetching file"
# Establish the SSH session
ssh = Net::SSH.start("IP Address", "username on server", :password => "user's password on server", :port => 12345)
# Use that session to generate an SCP object
scp = ssh.scp
# Download the file and run the code block each time a new chuck of data is received
scp.download!("path/to/file/on/server/fileName", "/Users/me/Desktop/") do |ch, name, received, total|
# Calculate percentage complete and format as a two-digit percentage
percentage = format('%.2f', received.to_f / total.to_f * 100) + '%'
# Print on top of (replace) the same line in the terminal
# - Pad with spaces to make sure nothing remains from the previous output
# - Add a carriage return without a line feed so the line doesn't move down
print "Saving to #{name}: Received #{received} of #{total} bytes" + " (#{percentage}) \r"
# Print the output immediately - don't wait until the buffer fills up
STDOUT.flush
end
puts "Fetch complete!"
have you tried with IO.popen ?
you should be able to read the output while the process is still running and parse it accordingly.
Redirecting stderr to stdout may work for you:
output = `scp user#someserver:remoteFile /some/local/folder/ 2>&1`
puts output
That should capture both stderr and stdout. You can capture stderr only by throwing away stdout:
output = `scp user#someserver:remoteFile /some/local/folder/ 2>&1 >/dev/null`
puts output
You can then use IO.popen.
Related
I have a ruby script on a remote server that I'm running via Net:SSH on my local pc.
The remote script takes a few minutes to run and outputs it's progress to stdout.
The problem I have is the block in my exec command only gets called when the packet/chunk is full.
So I get the progress all in one hit about each minute.
Here is some cut down examples that illustrate my problem:
Server Script:
(0.999).each do |i|
puts i
sleep 1
end
puts 1000
Local Script:
Net::SSH.start('ip.v.4.addr', 'user', :keys => ['my_key']) do |ssh|
ssh.exec("ruby count_to_1000.rb") do |ch, stream, data|
puts data if stream == :stdout
end
ssh.loop(1)
end
Is there any way from the remote script to force the sending of the packet/chunk?
Or is there a way to set a limit of say a second (or n bits) before it's flushed? (within Net:SSH)
Thanks for all your help!
Try flush:
http://www.ruby-doc.org/core-2.1.5/IO.html#method-i-flush
(0.999).each do |i|
puts i
STDOUT.flush
sleep 1
end
Or sync:
http://www.ruby-doc.org/core-2.1.5/IO.html#method-i-sync
STDOUT.sync = true
(0.999).each do |i|
puts i
sleep 1
end
(Untested, btw. Maybe they need to be used on the client-side instead, or on some other IO stream. But those are the two methods that immediately come to mind.)
In my test setup this works as expected (tested with localhost). However, there might be some issues with the STDOUT flush.
You can try to to write to STDOUT in stead of using puts (I have heard that there is some difference that I don't really understand).
Thus, you can on your server use:
(0.999).each do |i|
STDOUT.puts i
sleep 1
end
STDOUT.puts 1000
#You could possibly also use "STDOUT.write 1000", but it will not append a newline, like puts does.
If that does not work, then you can try to force-flush the STDOUT by using STDOUT.flush(). I believe the same can be achieved by writing an empty string to STDOUT, but I am not 1000% sure.
It might also happen that the exec command actually waits for the entire process to terminate for some reason(I was not able to figure out from the docs). In which case, you won't be able to achieve what you want. Then you can consider setting up websockets, use DRB, or some other means to pass the data.
Suppose I have a method which computes hard problems (maybe a depth-first search in a huge graph), we can call this method dfs(graph).
This method also output to stdout each result reached, using puts result.
def dfs(graph)
while true
# lots of computation
result = something_reached
puts result
end
end
I want to display a progressbar in shell to show that computation is running, so I add an instance like this:
pbar = ProgressBar.create(title: "Computing", starting_at: 1, total: nil)
and progressbar status must be updated (pbar.increment) as computation is running.
In a shell, I execute my program like this:
ruby dfs.rb > dfs_results.txt
Issues:
With pbar is flushed to stdout so prograss-bar is redirected to dfs_results.txt and computation results are not store in this file.
Without pbar result data is stored in file as is expected, but obviously without progress-bar.
I know that results could be stored with File.open usage, but it is desirable done with shell redirections.
Question:
How should I be implemented to flush computation resulta to dfs_results.txt file and show a progress-bar to keep a executing progress for user?
Just open a file to log into:
log = File.open('dfs_results.txt', 'w')
# write into log file
log.write(result)
And your progressbar still writes to stdout. When you open a second screen you can follow the output with:
tails -f dfs_results.txt
Update: Or you can use stdout for the progressbar and stderr for the output of the script. Instead of puts result write:
$stderr.puts results
and start the script with:
ruby dfs.rb 2> dfs_results.txt
I still think the first version is better because results are not errors...
I'm trying to download a file using net/sftp and pass its contents as the stdin for a command-line app. I can do it by first writing the file to disk but I'd rather avoid that step.
Is there any way to control the input to a program invoked with system() in ruby?
Don't use system at all for this sort of thing, system is best for running an external command that you don't need to talk to.
Use Open3.open3 or Open3.open2 to open up some pipes to your external process then write to the stdin pipe just like writing to any other IO channel; if there is any output to deal with, then you can read it straight from the stdout pipe just like reading from any other input IO channel.
Something like this perhaps (using open as mu suggested)?
contents = "Hello, World!"
open('|echo', 'w') { puts contents }
This can also be accomplished with IO.expect
require 'pty'
require 'expect'
str = "RUBY_VERSION"
PTY.spawn("irb") do |reader, writer|
reader.expect(/0> /)
writer.puts(str)
reader.expect(/=> /)
answer = reader.gets
puts "Ruby version from irb: #{answer}"
end
This waits for the spawned process to display "0> " (the end of an irb prompt) and when it sees that prints a defined string. It then looks for the irb to return by waiting for it to display "=> " and grabs the data returned.
I want to be able to read a currently open file. The test.rb is sending its output to test.log which I want to be able to read and ultimately send via email.
I am running this using cron:
*/5 * * * /tmp/test.rb > /tmp/log/test.log 2>&1
I have something like this in test.rb:
#!/usr/bin/ruby
def read_file(file_name)
file = File.open(file_name, "r")
data = file.read
file.close
return data
end
puts "Start"
puts read_file("/tmp/log/test.log")
puts "End"
When I run this code, it only gives me this output:
Start
End
I would expect the output to be something like this:
Start
Start (from the reading of the test.log since it should have the word start already)
End
Ok, you're trying to do several things at once, and I suspect you didn't systematically test before moving from one step to the next.
First we're going to clean up your code:
def read_file(file_name)
file = File.open(file_name, "r")
data = file.read
file.close
return data
end
puts "Start"
puts read_file("/tmp/log/test.log")
puts "End"
can be replaced with:
puts "Start"
puts File.read("./test.log")
puts "End"
It's plain and simple; There's no need for a method or anything complicated... yet.
Note that for ease of testing I'm working with a file in the current directory. To put some content in it I'll simply do:
echo "foo" > ./test.log
Running the test code gives me...
Greg:Desktop greg$ ruby test.rb
Start
foo
End
so I know the code is reading and printing correctly.
Now we can test what would go into the crontab, before we deal with its madness:
Greg:Desktop greg$ ruby test.rb > ./test.log
Greg:Desktop greg$
Hmm. No output. Something is broken with that. We knew there was content in the file previously, so what happened?
Greg:Desktop greg$ cat ./test.log
Start
End
Cat'ing the file shows it has the "Start" and "End" output of the code, but the part that should have been read and output is now missing.
What happening is that the shell truncated "test.log" just before it passed control to Ruby, which then opened and executed the code, which opened the now empty file to print it. In other words, you're asking the shell to truncate (empty) it just before you read it.
The fix is to read from a different file than you're going to write to, if you're trying to do something with the contents of it. If you're not trying to do something with its contents then there's no point in reading it with Ruby just to write it to a different file: We have cp and/or mv to do those things for us witout Ruby being involved. So, this makes more sense if we're going to do something with the contents:
ruby test.rb > ./test.log.out
I'll reset the file contents using echo "foo" > ./test.log, and cat'ing it showed 'foo', so I'm ready to try the redirection test again:
Greg:Desktop greg$ ruby test.rb > ./test.log.out
Greg:Desktop greg$ cat test.log.out
Start
foo
End
That time it worked. Trying it again has the same result, so I won't show the results here.
If you're going to email the file you could add that code at this point. Replacing the puts in the puts File.read('./test.log') line with an assignment to a variable will store the file's content:
contents = File.read('./test.log')
Then you can use contents as the body of a email. (And, rather than use Ruby for all of this I'd probably do it using mail or mailx or pipe it directly to sendmail, using the command-line and shell, but that's your call.)
At this point things are in a good position to add the command to crontab, using the same command as used on the command-line. Because it's running in cron, and errors can happen that we'd want to know about, we'd add the 2>&1 redirect to capture STDERR also, just as you did before. Just remember that you can NOT write to the same file you're going to read from or you'll have an empty file to read.
That's enough to get your app working.
class FileLineRead
File.open("file_line_read.txt") do |file|
file.each do |line|
phone_number = line.gsub(/\n/,'')
user = User.find_by_phone_number(line)
user.destroy unless user.nil?
end
end
end
open file
read line
DB Select
DB Update
In the cron job you have already opened and cleared test.log (via redirection) before you have read it in the Ruby script.
Why not do both the read and write in Ruby?
It may be a permissions issue or the file may not exist.
f = File.open("test","r")
puts f.read()
f.close()
The above will read the file test. If the file exists in the current directory
The problem is, as I can see, already solved by Slomojo. I'll only add:
to read and print a text file in Ruby, just:
puts File.read("/tmp/log/test.log")
I'm looking to expose an interactive command line program via JSON or another RPC style service using Ruby. I've found a couple tricks to do this, but im missing something when redirecting the output and input.
One method at least on linux is to redirect the stdin and stdout to a file then read and write to that file asynchronously with file reads and writes. Another method ive been trying after googling around was to use open4. Here is the code I wrote so far, but its getting stuck after reading a few lines from standard output.
require "open4"
include Open4
status = popen4("./srcds_run -console -game tf +map ctf_2fort -maxplayers 6") do |pid, stdin, stdout, stderr|
puts "PID #{pid}"
lines=""
while (line=stdout.gets)
lines+=line
puts line
end
while (line=stderr.gets)
lines+=line
puts line
end
end
Any help on this or some insight would be appreciated!
What I would recommend is using Xinetd (or similar) to run the command on some socket and then using the ruby network code. One of the problems you've already run into in your code here is that your two while loops are sequential, which can cause problems.
Another trick you might try is to re-direct stderr to stdout in your command, so that your program only has to read the stdout. Something like this:
popen4("./srcds_run -console -game tf +map ctf_2fort -maxplayers 6 2>&1")
The other benefit of this is that you get all the messages/errors in the order they happen during the program run.
EDIT
Your should consider integrating with AnyTerm. You can then either expose AnyTerm directly e.g. via Apache mod_proxy, or have your Rails controller act as the reverse proxy (handling authentication/session validation, then playing back controller.request minus any cookies to localhost:<AnyTerm-daemon-port>, and sending back as a response whatever AnyTerm replies with.)
class ConsoleController < ApplicationController
# AnyTerm speaks via HTTP POST only
def update
# validate session
...
# forward request to AnyTerm
response = Net::HTTP.post_form(URI.parse('http://localhost:#{AnyTermPort}/', request.params))
headers['Content-Type'] = response['Content-Type']
render_text response.body, response.status
end
Otherwise, you'd need to use IO::Select or IO::read_noblock to know when data is available to be read (from either network or subprocess) so you don't deadlock. See this too. Also check that either your Rails is used in a multi-threaded environment or that your Ruby version is not affected by this IO::Select bug.
You can start with something along these lines:
status = POpen4::popen4("ping localhost") do |stdout, stderr, stdin, pid|
puts "PID #{pid}"
# our buffers
stdout_lines=""
stderr_lines=""
begin
loop do
# check whether stdout, stderr or both are
# ready to be read from without blocking
IO.select([stdout,stderr]).flatten.compact.each { |io|
# stdout, if ready, goes to stdout_lines
stdout_lines += io.readpartial(1024) if io.fileno == stdout.fileno
# stderr, if ready, goes to stdout_lines
stderr_lines += io.readpartial(1024) if io.fileno == stderr.fileno
}
break if stdout.closed? && stderr.closed?
# if we acumulated any complete lines (\n-terminated)
# in either stdout/err_lines, output them now
stdout_lines.sub!(/.*\n/m) { puts $& ; '' }
stderr_lines.sub!(/.*\n/m) { puts $& ; '' }
end
rescue EOFError
puts "Done"
end
end
To also handle stdin, change to:
IO.select([stdout,stderr],[stdin]).flatten.compact.each { |io|
# program ready to get stdin? do we have anything for it?
if io.fileno == stdin.fileno && <got data from client?>
<write a small chunk from client to stdin>
end
# stdout, if ready, goes to stdout_lines