My below code example runs a command, e.g., ls <file> and captures stdout and stderr ( as well as the process exit status ).
However, if for example the command were to hang, then the ruby program will be "stuck" waiting for the command to finish, ( this can be seen for example if running sleep).
To avoid that possibility, I think what I need to do is fork a child process, ( so any "stuck" child process will not keep the ruby program "waiting" ).
However, I'm not sure how to capture the stdout and stderr from a forked child process, is this even possible ?
( for reasons I'd also like to be able to do this within the ruby std libs and not have dependency on any extra gem/s.
Also, this is just for ruby, not rails )
edit: To help clarify -
Trying to understand if there is a way to fork a child process, ( so there is no blocking until the child is done ), and still have the ruby program capture the stdout, stderr when the child process exits.
#!/bin/env ruby
require 'open3'
require 'logger'
logger = Logger.new('./test_open3.log')
files = ['file1','file2','file3']
files.each do |f|
stdout, stderr, status = Open3.capture3("ls #{f}")
logger.info(stdout)
logger.info(stderr)
logger.info(status)
end
Following the suggestion in the comments to use threads, I found that this gave me what I was looking for:
#!/bin/env ruby
require 'open3'
require 'logger'
require 'thread'
logger = Logger.new('./test_threads_open3.log')
files = ['file1','file2','file3']
threads = []
files.each_with_index do |f, i|
threads << Thread.new(f, i) do
puts "Thread #{i} is running"
stdout, stderr, status = Open3.capture3("ls #{f}")
logger.info(stdout)
logger.info(stderr)
logger.info(status)
end
end
threads.each { |t| t.join }
This creates an array of threads each with the block of code, and then last line, each thread in the array is run.
It probably requires some extra code to manage and limit the number of threads that can be run at a time, so as to be safer, maybe by using a queue/worker feature.
( This post also touches on the topic of the join method - Thread.join blocks the main thread )
Related
I am developing a long-running program in Ruby. I am writing some integration tests for this. These tests need to kill or stop the program after starting it; otherwise the tests hang.
For example, with a file bin/runner
#!/usr/bin/env ruby
while true do
puts "Hello World"
sleep 10
end
The (integration) test would be:
class RunReflectorTest < TestCase
test "it prints a welcome message over and over" do
out, err = capture_subprocess_io do
system "bin/runner"
end
assert_empty err
assert_includes out, "Hello World"
end
end
Only, obviously, this will not work; the test starts and never stops, because the system call never ends.
How should I tackle this? Is the problem in system itself, and would Kernel#spawn provide a solution? If so, how? Somehow the following keeps the out empty:
class RunReflectorTest < TestCase
test "it prints a welcome message over and over" do
out, err = capture_subprocess_io do
pid = spawn "bin/runner"
sleep 2
Process.kill pid
end
assert_empty err
assert_includes out, "Hello World"
end
end
. This direction also seems like it will cause a lot of timing-issues (and slow tests). Ideally, a reader would follow the stream of STDOUT and let the test pass as soon as the string is encountered and then immediately kill the subprocess. I cannot find how to do this with Process.
Test Behavior, Not Language Features
First, what you're doing is a TDD anti-pattern. Tests should focus on behaviors of methods or objects, not on language features like loops. If you must test a loop, construct a test that checks for a useful behavior like "entering an invalid response results in a re-prompt." There's almost no utility in checking that a loop loops forever.
However, you might decide to test a long-running process by checking to see:
If it's still running after t time.
If it's performed at least i iterations.
If a loop exits properly given certain input or upon reaching a boundary condition.
Use Timeouts or Signals to End Testing
Second, if you decide to do it anyway, you can just escape the block with Timeout::timeout. For example:
require 'timeout'
# Terminates block
Timeout::timeout(3) { `sleep 300` }
This is quick and easy. However, note that using timeout doesn't actually signal the process. If you run this a few times, you'll notice that sleep is still running multiple times as a system process.
It's better is to signal the process when you want to exit with Process::kill, ensuring that you clean up after yourself. For example:
pid = spawn 'sleep 300'
Process::kill 'TERM', pid
sleep 3
Process::wait pid
Aside from resource issues, this is a better approach when you're spawning something stateful and don't want to pollute the independence of your tests. You should almost always kill long-running (or infinite) processes in your test teardown whenever you can.
Ideally, a reader would follow the stream of STDOUT and let the test pass as soon as the string is encountered and then immediately kill the subprocess. I cannot find how to do this with Process.
You can redirect stdout of spawned process to any file descriptor by specifying out option
pid = spawn(command, :out=>"/dev/null") # write mode
Documentation
Example of redirection
With the answer from CodeGnome on how to use Timeout::timeout and the answer from andyconhin on how to redirect Process::spawn IO, I came up with two Minitest helpers that can be used as follows:
it "runs a deamon" do
wait_for(timeout: 2) do
wait_for_spawned_io(regexp: /Hello World/, command: ["bin/runner"])
end
end
The helpers are:
def wait_for(timeout: 1, &block)
Timeout::timeout(timeout) do
yield block
end
rescue Timeout::Error
flunk "Test did not pass within #{timeout} seconds"
end
def wait_for_spawned_io(regexp: //, command: [])
buffer = ""
begin
read_pipe, write_pipe = IO.pipe
pid = Process.spawn(command.shelljoin, out: write_pipe, err: write_pipe)
loop do
buffer << read_pipe.readpartial(1000)
break if regexp =~ buffer
end
ensure
read_pipe.close
write_pipe.close
Process.kill("INT", pid)
end
buffer
end
These can be used in a test which allows me to start a subprocess, capture the STDOUT and as soon as it matches the test Regular Expression, it passes, else it will wait 'till timeout and flunk (fail the test).
The loop will capture output and pass the test once it sees matching output. It uses a IO.pipe because that is most transparant for subprocesses (and their children) to write to.
I doubt this will work on Windows. And it needs some cleaning up of the wait_for_spawned_io which is doing slightly too much IMO. Antoher problem is that the Process.kill('INT') might not reach the children which are orphaned but still running after this test has ran. I need to find a way to ensure the entire subtree of processes is killed.
Here's the code:
while 1
input = gets
puts input
end
Here's what I want to do but I have no idea how to do it:
I want to create multiple instances of the code to run in the background and be able to pass input to a specific instance.
Q1: How do I run multiple instances of the script in the background?
Q2: How do I refer to an individual instance of the script so I can pass input to the instance (Q3)?
Q3: The script is using the cmd "gets" to take input, how would I pass input into an indivdual's script's gets?
e.g
Let's say I'm running threes instances of the code in the background and I refer to the instance as #1, #2, and #3 respectively.
I pass "hello" to #1, #1 puts "hello" to the screen.
Then I pass "world" to #3 and #3 puts "hello" to the screen.
Thanks!
UPDATE:
Answered my own question. Found this awesome tut: http://rubylearning.com/satishtalim/ruby_threads.html and resource here: http://www.ruby-doc.org/core/classes/Thread.html#M000826.
puts Thread.main
x = Thread.new{loop{puts 'x'; puts gets; Thread.stop}}
y = Thread.new{loop{puts 'y'; puts gets; Thread.stop}}
z = Thread.new{loop{puts 'z'; puts gets; Thread.stop}}
while x.status != "sleep" and y.status != "sleep" and z.status !="sleep"
sleep(1)
end
Thread.list.each {|thr| p thr }
x.run
x.join
Thank you for all the help guys! Help clarified my thinking.
I assume that you mean that you want multiple bits of Ruby code running concurrently. You can do it the hard way using Ruby threads (which have their own gotchas) or you can use the job control facilities of your OS. If you're using something UNIX-y, you can just put the code for each daemon in separate .rb files and run them at the same time.
E.g.,
# ruby daemon1.rb &
# ruby daemon2.rb &
There are many ways to "handle input and output" in a Ruby program. Pipes, sockets, etc. Since you asked about daemons, I assume that you mean network I/O. See Net::HTTP.
Ignoring what you think will happen with multiple daemons all fighting over STDIN at the same time:
(1..3).map{ Thread.new{ loop{ puts gets } } }.each(&:join)
This will create three threads that loop indefinitely, asking for input and then outputting it. Each thread is "joined", preventing the main program from exiting until each thread is complete (which it never will be).
You could try using multi_daemons gem which has capability to run multiple daemons and control them.
# this is server.rb
proc_code = Proc do
loop do
sleep 5
end
end
scheduler = MultiDaemons::Daemon.new('scripts/scheduler', name: 'scheduler', type: :script, options: {})
looper = MultiDaemons::Daemon.new(proc_code, name: 'looper', type: :proc, options: {})
MultiDaemons.runner([scheduler, looper], { force_kill_timeout: 60 })
To start and stop
ruby server.rb start
ruby server.rb stop
I'm building a daemon that will help me manage my server(s). Webmin works fine, as does just opening a shell to the server, but I'd prefer to be able to control server operations from a UI I design, and also expose some functionality to end users.
The daemon will pick up actions from a queue and execute them. However, since I'll be accepting input from users, I want to make sure they're not permitted to inject something dangerous into a privileged shell command.
Here's a fragment that exemplifies my problem:
def perform
system "usermod -p #{#options['shadow']} #{#options['username']}"
end
A gist that explains more: https://gist.github.com/773292
I'm not positive if typical escaping and sanitizing of inputs is enough for this case, and being a designer, I don't have a ton of security-related experience. I know that this is something that should probably be obvious to me, but its not!
How can I ensure that the web application that will create and serialize the actions can't pass dangerous text into the privileged process that receives the actions?
Thanks for the help
arb
It doesn't look like you need a shell for what you're doing. See the documentation for system here: http://ruby-doc.org/core/classes/Kernel.html#M001441
You should use the second form of system. Your example above would become:
system 'usermod', '-p', #options['shadow'], #options['username']
A nicer (IMO) way to write this is:
system *%W(usermod -p #{#options['shadow']} #{#options['username']})
The arguments this way are passed directly into the execve call, so you don't have to worry about sneaky shell tricks.
If you need not just the exit status but also the result you probably want to use Open3.popen3:
require 'open3'
stdin, stdout, stderr = Open3.popen3('usermod', '-p', #options['shadow'], #options['username'])
stdout.gets
sterr.gets
More information here: Getting output of system() calls in Ruby
I'd suggest looking into the 'shellwords' module. This script:
require 'shellwords'
parts = ['echo', "'hello world'; !%& some stuff", 'and another argument']
command = Shellwords.shelljoin( parts )
puts command
output = `#{ command }`
puts output
outputs the escaped text and the expected output:
echo \'hello\ world\'\;\ \!\%\&\ some\ stuff and\ another\ argument
'hello world'; !%& some stuff and another argument
This is an old question, but since it's pretty much the only real answer you'll find when googling I thought I'd add a caveat. The multi argument version of system seems reasonably safe on Linux, but it is NOT on Windows.
Try system "dir", "&", "echo", "hi!"
on a Windows system. Both dir and echo will be run. Echo could of course just as well be something far less innocuous.
I know this is an old thread, but there is another option that was lightly touched on by Simon Hürlimann.
There is not a lot of information about this topic and I think this might help others in need.
For this example we'll use Open3 which gives you the ability to run commands synchronously or asynchronously, and provides stdout, stderr, exit codes, and PID.
Open3 grants you access to stdout, stderr, exit codes and a thread to wait for the child process when running another program. You can specify various attributes, redirections, current directory, etc., of the program in the same way as for Process.spawn. (Source: Open3 Docs)
I chose to format the output as a CommandStatus object. This contains our stdout, stderr, pid (Of the worker thread) and exitstatus.
class Command
require 'open3'
class CommandStatus
#stdout = nil
#stderr = nil
#pid = nil
#exitstatus = nil
def initialize(stdout, stderr, process)
#stdout = stdout
#stderr = stderr
#pid = process.pid
#exitstatus = process.exitstatus
end
def stdout
#stdout
end
def stderr
#stderr
end
def exit_status
#exitstatus
end
def pid
#pid
end
end
def self.execute(command)
command_stdout = nil
command_stderr = nil
process = Open3.popen3(ENV, command + ';') do |stdin, stdout, stderr, thread|
stdin.close
stdout_buffer = stdout.read
stderr_buffer = stderr.read
command_stdout = stdout_buffer if stdout_buffer.length > 0
command_stderr = stderr_buffer if stderr_buffer.length > 0
thread.value # Wait for Process::Status object to be returned
end
return CommandStatus.new(command_stdout, command_stderr, process)
end
end
cmd = Command::execute("echo {1..10}")
puts "STDOUT: #{cmd.stdout}"
puts "STDERR: #{cmd.stderr}"
puts "EXIT: #{cmd.exit_status}"
While reading the STDOUT/ERR buffers, I use command_stdout = stdout_buffer if stdout_buffer.length > 0 to control whether the command_stdout variable is assigned or not. You should pass nil instead of "" when no data is present. It's more clear when handing data later on.
You probably noticed me using command + ';'. The reason for this is based on the documentation from Kernel.exec (Which is what popen3 uses):
If the string from the first form (exec("command")) follows these
simple rules:
no meta characters
no shell reserved word and no special built-in
Ruby invokes the command directly without shell
You can force shell invocation by adding ";" to the string (because
";" is a meta character)
This simply prevents a Ruby from throwing a 'spawn': No such file or directory error if you pass a malformed command. Instead it will pass it straight to the kernel where the error will be resolved gracefully and appear as STDERR instead of an uncaught exception.
Modern, secure and simple solution (popen will escape arguments for you):
IO.popen(['usermod', '-p', #options['shadow'], #options['username']]).read
(#read will close the IO before returning)
I'm looking to expose an interactive command line program via JSON or another RPC style service using Ruby. I've found a couple tricks to do this, but im missing something when redirecting the output and input.
One method at least on linux is to redirect the stdin and stdout to a file then read and write to that file asynchronously with file reads and writes. Another method ive been trying after googling around was to use open4. Here is the code I wrote so far, but its getting stuck after reading a few lines from standard output.
require "open4"
include Open4
status = popen4("./srcds_run -console -game tf +map ctf_2fort -maxplayers 6") do |pid, stdin, stdout, stderr|
puts "PID #{pid}"
lines=""
while (line=stdout.gets)
lines+=line
puts line
end
while (line=stderr.gets)
lines+=line
puts line
end
end
Any help on this or some insight would be appreciated!
What I would recommend is using Xinetd (or similar) to run the command on some socket and then using the ruby network code. One of the problems you've already run into in your code here is that your two while loops are sequential, which can cause problems.
Another trick you might try is to re-direct stderr to stdout in your command, so that your program only has to read the stdout. Something like this:
popen4("./srcds_run -console -game tf +map ctf_2fort -maxplayers 6 2>&1")
The other benefit of this is that you get all the messages/errors in the order they happen during the program run.
EDIT
Your should consider integrating with AnyTerm. You can then either expose AnyTerm directly e.g. via Apache mod_proxy, or have your Rails controller act as the reverse proxy (handling authentication/session validation, then playing back controller.request minus any cookies to localhost:<AnyTerm-daemon-port>, and sending back as a response whatever AnyTerm replies with.)
class ConsoleController < ApplicationController
# AnyTerm speaks via HTTP POST only
def update
# validate session
...
# forward request to AnyTerm
response = Net::HTTP.post_form(URI.parse('http://localhost:#{AnyTermPort}/', request.params))
headers['Content-Type'] = response['Content-Type']
render_text response.body, response.status
end
Otherwise, you'd need to use IO::Select or IO::read_noblock to know when data is available to be read (from either network or subprocess) so you don't deadlock. See this too. Also check that either your Rails is used in a multi-threaded environment or that your Ruby version is not affected by this IO::Select bug.
You can start with something along these lines:
status = POpen4::popen4("ping localhost") do |stdout, stderr, stdin, pid|
puts "PID #{pid}"
# our buffers
stdout_lines=""
stderr_lines=""
begin
loop do
# check whether stdout, stderr or both are
# ready to be read from without blocking
IO.select([stdout,stderr]).flatten.compact.each { |io|
# stdout, if ready, goes to stdout_lines
stdout_lines += io.readpartial(1024) if io.fileno == stdout.fileno
# stderr, if ready, goes to stdout_lines
stderr_lines += io.readpartial(1024) if io.fileno == stderr.fileno
}
break if stdout.closed? && stderr.closed?
# if we acumulated any complete lines (\n-terminated)
# in either stdout/err_lines, output them now
stdout_lines.sub!(/.*\n/m) { puts $& ; '' }
stderr_lines.sub!(/.*\n/m) { puts $& ; '' }
end
rescue EOFError
puts "Done"
end
end
To also handle stdin, change to:
IO.select([stdout,stderr],[stdin]).flatten.compact.each { |io|
# program ready to get stdin? do we have anything for it?
if io.fileno == stdin.fileno && <got data from client?>
<write a small chunk from client to stdin>
end
# stdout, if ready, goes to stdout_lines
I want to offload a block of code in my main process to child process to make it run concurrently. I also want to have the PID of the spawned child process so I can monitor and kill it if necessary.
In addition to Chris' great answer, remember to call Process.wait from your master in order to reap your child process, else you'll leave zombies behind.
Example as requested in comments:
pid = Process.fork do
puts "child, pid #{Process.pid} sleeping..."
sleep 5
puts "child exiting"
end
puts "parent, pid #{Process.pid}, waiting on child pid #{pid}"
Process.wait
puts "parent exiting"
You can use the fork kernel method. Here is an example:
#!/usr/bin/env ruby
puts "This is the master process."
child_pid = fork do
puts "This is the child process"
exit
end
puts "The PID of the child process is #{child_pid}"
The fork method returns the PID of the process it forks and executes any code in the block passed. Like regular Ruby blocks it keeps the bindings of the parent process.
It is a good idea to make your forked process exit.
In 1.9 you can use Process.spawn command.
See also http://en.wikibooks.org/wiki/Ruby_Programming/Running_Multiple_Processes
If you are happy to use Threads, rather than Processes, then something like this may be a bit more scaleable to more-than-one fork:
def doit(x)
sleep(rand(10))
puts "Done... #{x}"
end
thingstodo = ["a","b","c","d","e","f","g"]
tasklist = []
# Set the threads going
thingstodo.each { |thing|
task = Thread.new(thing) { |this| doit(this) }
tasklist << task
}
# Wait for the threads to finish
tasklist.each { |task|
task.join
}
Please see John Topley's excellent comments and reference, below, regarding the Ruby execution model and its restrictions.
Just edited to correct a glaring error (no assignment to task), and to follow #(Jason King)'s advice.