Suppose I have a method which computes hard problems (maybe a depth-first search in a huge graph), we can call this method dfs(graph).
This method also output to stdout each result reached, using puts result.
def dfs(graph)
while true
# lots of computation
result = something_reached
puts result
end
end
I want to display a progressbar in shell to show that computation is running, so I add an instance like this:
pbar = ProgressBar.create(title: "Computing", starting_at: 1, total: nil)
and progressbar status must be updated (pbar.increment) as computation is running.
In a shell, I execute my program like this:
ruby dfs.rb > dfs_results.txt
Issues:
With pbar is flushed to stdout so prograss-bar is redirected to dfs_results.txt and computation results are not store in this file.
Without pbar result data is stored in file as is expected, but obviously without progress-bar.
I know that results could be stored with File.open usage, but it is desirable done with shell redirections.
Question:
How should I be implemented to flush computation resulta to dfs_results.txt file and show a progress-bar to keep a executing progress for user?
Just open a file to log into:
log = File.open('dfs_results.txt', 'w')
# write into log file
log.write(result)
And your progressbar still writes to stdout. When you open a second screen you can follow the output with:
tails -f dfs_results.txt
Update: Or you can use stdout for the progressbar and stderr for the output of the script. Instead of puts result write:
$stderr.puts results
and start the script with:
ruby dfs.rb 2> dfs_results.txt
I still think the first version is better because results are not errors...
Related
I have a ruby script on a remote server that I'm running via Net:SSH on my local pc.
The remote script takes a few minutes to run and outputs it's progress to stdout.
The problem I have is the block in my exec command only gets called when the packet/chunk is full.
So I get the progress all in one hit about each minute.
Here is some cut down examples that illustrate my problem:
Server Script:
(0.999).each do |i|
puts i
sleep 1
end
puts 1000
Local Script:
Net::SSH.start('ip.v.4.addr', 'user', :keys => ['my_key']) do |ssh|
ssh.exec("ruby count_to_1000.rb") do |ch, stream, data|
puts data if stream == :stdout
end
ssh.loop(1)
end
Is there any way from the remote script to force the sending of the packet/chunk?
Or is there a way to set a limit of say a second (or n bits) before it's flushed? (within Net:SSH)
Thanks for all your help!
Try flush:
http://www.ruby-doc.org/core-2.1.5/IO.html#method-i-flush
(0.999).each do |i|
puts i
STDOUT.flush
sleep 1
end
Or sync:
http://www.ruby-doc.org/core-2.1.5/IO.html#method-i-sync
STDOUT.sync = true
(0.999).each do |i|
puts i
sleep 1
end
(Untested, btw. Maybe they need to be used on the client-side instead, or on some other IO stream. But those are the two methods that immediately come to mind.)
In my test setup this works as expected (tested with localhost). However, there might be some issues with the STDOUT flush.
You can try to to write to STDOUT in stead of using puts (I have heard that there is some difference that I don't really understand).
Thus, you can on your server use:
(0.999).each do |i|
STDOUT.puts i
sleep 1
end
STDOUT.puts 1000
#You could possibly also use "STDOUT.write 1000", but it will not append a newline, like puts does.
If that does not work, then you can try to force-flush the STDOUT by using STDOUT.flush(). I believe the same can be achieved by writing an empty string to STDOUT, but I am not 1000% sure.
It might also happen that the exec command actually waits for the entire process to terminate for some reason(I was not able to figure out from the docs). In which case, you won't be able to achieve what you want. Then you can consider setting up websockets, use DRB, or some other means to pass the data.
I'm seeing some confusing behavior with a Ruby script while running under nohup. Basically I'm doing this:
require 'logger'
logger_file = open('/mnt/dbsdata/output.log', File::WRONLY | File::APPEND | File::CREAT)
LOGGER = Logger.new(logger_file)
LOGGER.level = Logger::INFO
def run_command(cmd,display=true)
if display
LOGGER.info "Executing: #{cmd}"
end
output = `#{cmd} 2>&1` ; results=$?.success?
if ! results
LOGGER.error "FAILED to execute #{cmd}"
LOGGER.error output
return false
end
return true
end
begin
run_command("some_longrunning_command", true)
run_command("some_other_longrunning_command",true)
# etc...
end
What's weird here is that both when using Logger as above, and just normal puts to STDOUT (nohup.out), the output timing is way off. I'd think I could just tail the logfile and see real time logging messages (log message, execute command, repeat), however what's happening is a bulk flush of messages to log of many stale messages all at once, long after their command was already executed and completed.
When executing the script like so from the shell:
`nohup ruby myscript.rb &`
This behaves as expected when not run under nohup.
Anyone experienced this and know a good workaround?
I can't say for sure, but it sounds like an IO sync issue. I/O is generally buffered for speed. RAM is a lot faster than disks, so reads and writes are temporarily stored until the data is requested by the code, or the disk is able to receive it. Ruby's IO class has the sync= method that lets you tell Ruby to immediately flush the buffer on a write.
This should work:
logger_file = open('/mnt/dbsdata/output.log', File::WRONLY | File::APPEND | File::CREAT)
logger_file.sync = true
LOGGER = Logger.new(logger_file)
LOGGER.level = Logger::INFO
For simplicity, you can create the correct mode you want for your output using:
logger_file = File.open('/mnt/dbsdata/output.log', 'a')
Sounds like buffering. Try logging to stderr and see if that helps.
I'm trying to download a file using net/sftp and pass its contents as the stdin for a command-line app. I can do it by first writing the file to disk but I'd rather avoid that step.
Is there any way to control the input to a program invoked with system() in ruby?
Don't use system at all for this sort of thing, system is best for running an external command that you don't need to talk to.
Use Open3.open3 or Open3.open2 to open up some pipes to your external process then write to the stdin pipe just like writing to any other IO channel; if there is any output to deal with, then you can read it straight from the stdout pipe just like reading from any other input IO channel.
Something like this perhaps (using open as mu suggested)?
contents = "Hello, World!"
open('|echo', 'w') { puts contents }
This can also be accomplished with IO.expect
require 'pty'
require 'expect'
str = "RUBY_VERSION"
PTY.spawn("irb") do |reader, writer|
reader.expect(/0> /)
writer.puts(str)
reader.expect(/=> /)
answer = reader.gets
puts "Ruby version from irb: #{answer}"
end
This waits for the spawned process to display "0> " (the end of an irb prompt) and when it sees that prints a defined string. It then looks for the irb to return by waiting for it to display "=> " and grabs the data returned.
When you use a system call in a Ruby script, you can get the output of that command like this:
output = `ls`
puts output
That's what this question was about.
But is there a way to show the continuous output of a system call? For example, if you run this secure copy command, to get a file from a server over SSH:
scp user#someserver:remoteFile /some/local/folder/
... it shows continuous output with the progress of the download. But this:
output = `scp user#someserver:remoteFile /some/local/folder/`
puts output
... doesn't capture that output.
How can I show the ongoing progress of the download from inside my Ruby script?
Try:
IO.popen("scp -v user#server:remoteFile /local/folder/").each do |fd|
puts(fd.readline)
end
I think you would have better luck using the ruby standard library to handle SCP (as opposed to forking a shell process). The Net::SCP library (as well as the entire Net::* libraries) are full featured and used with Capistrano to handle remote commands.
Checkout http://net-ssh.rubyforge.org/ for a rundown of what is available.
Tokland answered the question as I asked it, but Adam's approach was what I ended up using. Here was my completed script, which does show a running count of bytes downloaded, and also a percentage complete.
require 'rubygems'
require 'net/scp'
puts "Fetching file"
# Establish the SSH session
ssh = Net::SSH.start("IP Address", "username on server", :password => "user's password on server", :port => 12345)
# Use that session to generate an SCP object
scp = ssh.scp
# Download the file and run the code block each time a new chuck of data is received
scp.download!("path/to/file/on/server/fileName", "/Users/me/Desktop/") do |ch, name, received, total|
# Calculate percentage complete and format as a two-digit percentage
percentage = format('%.2f', received.to_f / total.to_f * 100) + '%'
# Print on top of (replace) the same line in the terminal
# - Pad with spaces to make sure nothing remains from the previous output
# - Add a carriage return without a line feed so the line doesn't move down
print "Saving to #{name}: Received #{received} of #{total} bytes" + " (#{percentage}) \r"
# Print the output immediately - don't wait until the buffer fills up
STDOUT.flush
end
puts "Fetch complete!"
have you tried with IO.popen ?
you should be able to read the output while the process is still running and parse it accordingly.
Redirecting stderr to stdout may work for you:
output = `scp user#someserver:remoteFile /some/local/folder/ 2>&1`
puts output
That should capture both stderr and stdout. You can capture stderr only by throwing away stdout:
output = `scp user#someserver:remoteFile /some/local/folder/ 2>&1 >/dev/null`
puts output
You can then use IO.popen.
I'm building a daemon that will help me manage my server(s). Webmin works fine, as does just opening a shell to the server, but I'd prefer to be able to control server operations from a UI I design, and also expose some functionality to end users.
The daemon will pick up actions from a queue and execute them. However, since I'll be accepting input from users, I want to make sure they're not permitted to inject something dangerous into a privileged shell command.
Here's a fragment that exemplifies my problem:
def perform
system "usermod -p #{#options['shadow']} #{#options['username']}"
end
A gist that explains more: https://gist.github.com/773292
I'm not positive if typical escaping and sanitizing of inputs is enough for this case, and being a designer, I don't have a ton of security-related experience. I know that this is something that should probably be obvious to me, but its not!
How can I ensure that the web application that will create and serialize the actions can't pass dangerous text into the privileged process that receives the actions?
Thanks for the help
arb
It doesn't look like you need a shell for what you're doing. See the documentation for system here: http://ruby-doc.org/core/classes/Kernel.html#M001441
You should use the second form of system. Your example above would become:
system 'usermod', '-p', #options['shadow'], #options['username']
A nicer (IMO) way to write this is:
system *%W(usermod -p #{#options['shadow']} #{#options['username']})
The arguments this way are passed directly into the execve call, so you don't have to worry about sneaky shell tricks.
If you need not just the exit status but also the result you probably want to use Open3.popen3:
require 'open3'
stdin, stdout, stderr = Open3.popen3('usermod', '-p', #options['shadow'], #options['username'])
stdout.gets
sterr.gets
More information here: Getting output of system() calls in Ruby
I'd suggest looking into the 'shellwords' module. This script:
require 'shellwords'
parts = ['echo', "'hello world'; !%& some stuff", 'and another argument']
command = Shellwords.shelljoin( parts )
puts command
output = `#{ command }`
puts output
outputs the escaped text and the expected output:
echo \'hello\ world\'\;\ \!\%\&\ some\ stuff and\ another\ argument
'hello world'; !%& some stuff and another argument
This is an old question, but since it's pretty much the only real answer you'll find when googling I thought I'd add a caveat. The multi argument version of system seems reasonably safe on Linux, but it is NOT on Windows.
Try system "dir", "&", "echo", "hi!"
on a Windows system. Both dir and echo will be run. Echo could of course just as well be something far less innocuous.
I know this is an old thread, but there is another option that was lightly touched on by Simon Hürlimann.
There is not a lot of information about this topic and I think this might help others in need.
For this example we'll use Open3 which gives you the ability to run commands synchronously or asynchronously, and provides stdout, stderr, exit codes, and PID.
Open3 grants you access to stdout, stderr, exit codes and a thread to wait for the child process when running another program. You can specify various attributes, redirections, current directory, etc., of the program in the same way as for Process.spawn. (Source: Open3 Docs)
I chose to format the output as a CommandStatus object. This contains our stdout, stderr, pid (Of the worker thread) and exitstatus.
class Command
require 'open3'
class CommandStatus
#stdout = nil
#stderr = nil
#pid = nil
#exitstatus = nil
def initialize(stdout, stderr, process)
#stdout = stdout
#stderr = stderr
#pid = process.pid
#exitstatus = process.exitstatus
end
def stdout
#stdout
end
def stderr
#stderr
end
def exit_status
#exitstatus
end
def pid
#pid
end
end
def self.execute(command)
command_stdout = nil
command_stderr = nil
process = Open3.popen3(ENV, command + ';') do |stdin, stdout, stderr, thread|
stdin.close
stdout_buffer = stdout.read
stderr_buffer = stderr.read
command_stdout = stdout_buffer if stdout_buffer.length > 0
command_stderr = stderr_buffer if stderr_buffer.length > 0
thread.value # Wait for Process::Status object to be returned
end
return CommandStatus.new(command_stdout, command_stderr, process)
end
end
cmd = Command::execute("echo {1..10}")
puts "STDOUT: #{cmd.stdout}"
puts "STDERR: #{cmd.stderr}"
puts "EXIT: #{cmd.exit_status}"
While reading the STDOUT/ERR buffers, I use command_stdout = stdout_buffer if stdout_buffer.length > 0 to control whether the command_stdout variable is assigned or not. You should pass nil instead of "" when no data is present. It's more clear when handing data later on.
You probably noticed me using command + ';'. The reason for this is based on the documentation from Kernel.exec (Which is what popen3 uses):
If the string from the first form (exec("command")) follows these
simple rules:
no meta characters
no shell reserved word and no special built-in
Ruby invokes the command directly without shell
You can force shell invocation by adding ";" to the string (because
";" is a meta character)
This simply prevents a Ruby from throwing a 'spawn': No such file or directory error if you pass a malformed command. Instead it will pass it straight to the kernel where the error will be resolved gracefully and appear as STDERR instead of an uncaught exception.
Modern, secure and simple solution (popen will escape arguments for you):
IO.popen(['usermod', '-p', #options['shadow'], #options['username']]).read
(#read will close the IO before returning)