Double fork and stdin - ruby

I wrote this code to run my process in a daemon. The goal is to make this process running even if I close its parent. Now, i would like to be able to write something in its stdin. What should I do ? Here's the code.
def daemonize(cmd, options = {})
rd, wr = IO.pipe
p1 = Process.fork {
Process.setsid
p2 = Process.fork {
$0 = cmd #Name of the command
pidfile = File.new(options[:pid_file], 'w')
pidfile.chmod( 0644 )
pidfile.puts "#{Process.pid}"
pidfile.close
Dir.chdir(ENV["PWD"] = options[:working_dir].to_s) if options[:working_dir]
File.umask 0000
STDIN.reopen '/dev/null'
STDOUT.reopen '/dev/null', 'a'
STDERR.reopen STDOUT
Signal.trap("USR1") do
Console.show 'I just received a USR1', 'warning'
end
::Kernel.exec(*Shellwords.shellwords(cmd)) #Executing the command in the parent process
exit
}
raise 'Fork failed!' if p2 == -1
Process.detach(p2) # divorce p2 from parent process (p1)
rd.close
wr.write p2
wr.close
exit
}
raise 'Fork failed!' if p1 == -1
Process.detach(p1) # divorce p1 from parent process (shell)
wr.close
daemon_id = rd.read.to_i
rd.close
daemon_id
end
Is there a way to reopen stdin in something like a pipe instead of /dev/null in which I would be able to write ?

How about a fifo? In linux, you can use the mkfifo command:
$ mkfifo /tmp/mypipe
Then you can reopen STDIN on that pipe:
STDIN.reopen '/tmp/mypipe'
# Do read-y things
Anything else can write to that pipe:
$ echo "roflcopter" > /tmp/mypipe
allowing that data to be read by the ruby process.
(Update) Caveat with blocking
Since fifos block until there's a read and write (e.g. a read is blocked unless there's a write, and vice-versa), it's best handled with multiple threads. One thread should do the reading, passing the data to a queue, and another should handle that input. Here's an example of that situation:
require 'thread'
input = Queue.new
threads = []
# Read from the fifo and add to an input queue (glorified array)
threads << Thread.new(input) do |ip|
STDIN.reopen 'mypipe'
loop do
if line = STDIN.gets
puts "Read: #{line}"
ip.push line
end
end
end
# Handle the input passed by the reader thread
threads << Thread.new(input) do |ip|
loop do
puts "Ouput: #{ip.pop}"
end
end
threads.map(&:join)

Related

How to terminate a child process as part of terminating the thread that created it

I have a Ruby application that spawns a thread on-demand which in turn does a system call to execute a native binary.
I want to abort this call before the native call completes.
I tried using all options the Thread documentation provided, like kill, raise and terminate, but nothing seems to help.
This is what I'm trying to do:
class Myserver < Grape::API
##thr = nil
post "/start" do
puts "Starting script"
##thr = Thread.new do
op=`sh chumma_sh.sh`
puts op
end
puts ##thr.status
end
put "/stop" do
##thr.terminate
##thr.raise
Thread.kill(##thr)
puts ##thr.status
end
end
The thread appears to enter a sleep state as an IO operation is in process, but how do I kill the thread so that all child processes it created are terminated and not attached to root.
Doing ps-ef | grep for the script returns the pid, and I could try Process.kill pid but wanted to know if there are better options.
I don't have the option at this moment of modifying how the script is executed as it is part of an inherited library.
Using ps is the only approach I've found that works. If you also want to kill child threads, you could use something like this:
def child_pids_recursive(pid)
# get children
pipe = IO.popen("ps -ef | grep #{pid}")
child_pids = pipe.readlines.map do |line|
parts = line.split(/\s+/)
parts[2] if parts[3] == pid.to_s && parts[2] != pipe.pid.to_s
end.compact
pipe.close
# get grandchildren
grandchild_pids = child_pids.map do |cpid|
child_pids_recursive(cpid)
end.flatten
child_pids + grandchild_pids
end
def kill_all(pid)
child_pids_recursive(pid).reverse.each do |p|
begin
Process.kill('TERM', p.to_i)
rescue
# ignore
end
end
end

Is it possible to read the last line of STDOUT from an open process?

I am using Stockfish to analyze a chess position, but the specific context isn't that important. I want to read the output from the program in real-time without having to wait until the process is finished. In particular I want to read the last or last few lines. Current I have:
require 'pty'
master, slave = PTY.open
read, write = IO.pipe
pid = spawn("stockfish", :in=>read, :out=>slave)
calculating=0
slave.close
read.close
write.puts('uci')
write.puts("setoption name Threads value 1")
write.puts("position fen rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR")
write.puts('go infinite')
while calculating<10
sleep(1)
master.each do |line|
puts line
end
calculating+=1
end
write.puts('quit')
write.close
puts "done"
The problem with this is that the program isn't seeing an EOF marker until all 10 loops are finished so it's getting stuck in the master.each line block indefinitely. I can read the next line from the beginning one at a time with master.gets, but how would I read the last line instead?
I have a working solution now:
fen="rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR"
overseer=fork do
require 'pty'
master, slave = PTY.open
read, write = IO.pipe
pid = spawn("stockfish", :in=>read, :out=>slave)
slave.close
read.close
write.puts('uci')
write.puts("setoption name Threads value 4")
write.puts("position fen #{fen}")
write.puts('go infinite')
loop do
puts master.gets
end
end
if $stdin.gets
Process.kill("KILL",overseer)
end
This will print out a stream of stream until the user kills it.

Continuously read from STDOUT and STDERR in real time

Say we have a small application (example.rb) which behave like the following ruby code:
#!/usr/bin/env ruby
done = false
out =Thread.new do
cnt = 0
while !done
STDOUT.puts "out #{cnt}"
cnt += 1
sleep 1
end
end
err = Thread.new do
cnt = 0
while !done
STDERR.puts "err #{cnt}"
cnt += 1
sleep 1
end
end
while true
i = STDIN.gets
if i == "q\n"
puts "Quiting"
done = true
break
end
end
out.join
err.join
exit 42
It print something to stdout and to stderr, it must be quited by writing "q\n" to stdin, and when it exit a value is returned in the return code.
Now, I would like to write a small ruby script which can run this program in an external process, where stdout and stdin are captured, and when the external process should be terminated this is done by writing "q\n" to its stdin. This program is called monitor.rb.
This is what I have tried here:
#!/usr/bin/env ruby
require 'open3'
class Monitor
#cmd
attr_accessor :return_code
def initialize cmd
#cmd = cmd
end
def run
#runner_thread = Thread.new do
Open3::popen3(#cmd) do |stdin, stdout, stderr, thread|
puts "#{Time.now} #{#cmd} is running as pid: #{thread.pid}"
stdin.sync = true;
stdout.sync = true;
stderr.sync = true;
#stdin = stdin
t_out = Thread.new do
stdout.readlines do |l|
puts "#{Time.now} STDOUT> #{l}"
end
end
t_err = Thread.new do
stderr.readlines do |l|
puts "#{Time.now} STDERR> #{l}"
end
end
thread.join
t_err.join
t_out.join
#return_code = thread.value
end
end
end
def quit
puts "Quiting"
#stdin.puts "q"
#stdin.close
#runner_thread.join
end
end
mon = Monitor.new "./example.rb"
mon.run
sleep 5
mon.quit
puts "Return code: #{mon.return_code}"
Question 1: What is wrong with my code since the output of the external process is not being printed?
Question 2: Can this be done in a more elegant way, and what would that look like?
The code must be able to run on Linux and portability is not a priority, I uses ruby 2.0.
When run example.rb in a terminal I get:
$ ./example.rb
out 0
err 0
out 1
err 1
out 2
err 2
q
Quiting
When I run the monitor application I get:
$ ./monitor.rb
2013-11-19 14:39:20 +0100 ./example.rb is running as pid: 7228
Quiting
Return code: pid 7228 exit 42
I expected the monitor.rb to print the output from example.rb
Try changing your t_out and t_err threads to use the following code. readlines will read the entire file at once and stdout and stderr will block until your script exits. I think this is why you were not getting any output.
while l = stdout.gets
puts "#{Time.now} STDOUT> #{l}"
end
This should print out to the screen as soon as any output is available.

Why is IO::WaitReadable being raised differently for STDOUT than STDERR?

Given that I wish to test non-blocking reads from a long command, I created the following script, saved it as long, made it executable with chmod 755, and placed it in my path (saved as ~/bin/long where ~/bin is in my path).
I am on a *nix variant with ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-darwin11.0.0] compiled with RVM defaults. I do not use Windows, and am therefore unsure if the test script will work for you if you do.
#!/usr/bin/env ruby
3.times do
STDOUT.puts 'message on stdout'
STDERR.puts 'message on stderr'
sleep 1
end
Why does long_err produce each STDERR message as it is printed by "long"
def long_err( bash_cmd = 'long', maxlen = 4096)
stdin, stdout, stderr = Open3.popen3(bash_cmd)
begin
begin
puts 'err -> ' + stderr.read_nonblock(maxlen)
end while true
rescue IO::WaitReadable
IO.select([stderr])
retry
rescue EOFError
puts 'EOF'
end
end
while long_out remains blocked until all STDOUT messages are printed?
def long_out( bash_cmd = 'long', maxlen = 4096)
stdin, stdout, stderr = Open3.popen3(bash_cmd)
begin
begin
puts 'out -> ' + stdout.read_nonblock(maxlen)
end while true
rescue IO::WaitReadable
IO.select([stdout])
retry
rescue EOFError
puts 'EOF'
end
end
I assume you will require 'open3' before testing either function.
Why is IO::WaitReadable being raised differently for STDOUT than STDERR?
Workarounds using other ways to start subprocesses also appreciated if you have them.
In most OS's STDOUT is buffered while STDERR is not. What popen3 does is basically open a pipe between the exeutable you launch and Ruby.
Any output that is in buffered mode is not sent through this pipe until either:
The buffer is filled (thereby forcing a flush).
The sending application exits (EOF is reached, forcing a flush).
The stream is explicitly flushed.
The reason STDERR is not buffered is that it's usually considered important for error messages to appear instantly, rather than go for for efficiency through buffering.
So, knowing this, you can emulate STDERR behaviour with STDOUT like this:
#!/usr/bin/env ruby
3.times do
STDOUT.puts 'message on stdout'
STDOUT.flush
STDERR.puts 'message on stderr'
sleep 1
end
and you will see the difference.
You might also want to check "Understanding Ruby and OS I/O buffering".
Here's the best I've got so far for starting subprocesses. I launch a lot of network commands so I needed a way to time them out if they take too long to come back. This should be handy in any situation where you want to remain in control of your execution path.
I adapted this from a Gist, adding code to test the exit status of the command for 3 outcomes:
Successful completion (exit status 0)
Error completion (exit status is non-zero) - raises an exception
Command timed out and was killed - raises an exception
Also fixed a race condition, simplified parameters, added a few more comments, and added debug code to help me understand what was happening with exits and signals.
Call the function like this:
output = run_with_timeout("command that might time out", 15)
output will contain the combined STDOUT and STDERR of the command if it completes successfully. If the command doesn't complete within 15 seconds it will be killed and an exception raised.
Here's the function (2 constants you'll need defined at the top):
DEBUG = false # change to true for some debugging info
BUFFER_SIZE = 4096 # in bytes, this should be fine for many applications
def run_with_timeout(command, timeout)
output = ''
tick = 1
begin
# Start task in another thread, which spawns a process
stdin, stderrout, thread = Open3.popen2e(command)
# Get the pid of the spawned process
pid = thread[:pid]
start = Time.now
while (Time.now - start) < timeout and thread.alive?
# Wait up to `tick' seconds for output/error data
Kernel.select([stderrout], nil, nil, tick)
# Try to read the data
begin
output << stderrout.read_nonblock(BUFFER_SIZE)
puts "we read some data..." if DEBUG
rescue IO::WaitReadable
# No data was ready to be read during the `tick' which is fine
print "." # give feedback each tick that we're waiting
rescue EOFError
# Command has completed, not really an error...
puts "got EOF." if DEBUG
# Wait briefly for the thread to exit...
# We don't want to kill the process if it's about to exit on its
# own. We decide success or failure based on whether the process
# completes successfully.
sleep 1
break
end
end
if thread.alive?
# The timeout has been reached and the process is still running so
# we need to kill the process, because killing the thread leaves
# the process alive but detached.
Process.kill("TERM", pid)
end
ensure
stdin.close if stdin
stderrout.close if stderrout
end
status = thread.value # returns Process::Status when process ends
if DEBUG
puts "thread.alive?: #{thread.alive?}"
puts "status: #{status}"
puts "status.class: #{status.class}"
puts "status.exited?: #{status.exited?}"
puts "status.exitstatus: #{status.exitstatus}"
puts "status.signaled?: #{status.signaled?}"
puts "status.termsig: #{status.termsig}"
puts "status.stopsig: #{status.stopsig}"
puts "status.stopped?: #{status.stopped?}"
puts "status.success?: #{status.success?}"
end
# See how process ended: .success? => true, false or nil if exited? !true
if status.success? == true # process exited (0)
return output
elsif status.success? == false # process exited (non-zero)
raise "command `#{command}' returned non-zero exit status (#{status.exitstatus}), see below output\n#{output}"
elsif status.signaled? # we killed the process (timeout reached)
raise "shell command `#{command}' timed out and was killed (timeout = #{timeout}s): #{status}"
else
raise "process didn't exit and wasn't signaled. We shouldn't get to here."
end
end
Hope this is useful.

How to proxy a shell process in ruby

I'm creating a script to wrap jdb (java debugger). I essentially want to wrap this process and proxy the user interaction. So I want it to:
start jdb from my script
send the output of jdb to stdout
pause and wait for input when jdb does
when the user enters commands, pass it to jdb
At the moment I really want a pass thru to jdb. The reason for this is to initialize the process with specific parameters and potentially add more commands in the future.
Update:
Here's the shell of what ended up working for me using expect:
PTY.spawn("jdb -attach 1234") do |read,write,pid|
write.sync = true
while (true) do
read.expect(/\r\r\n> /) do |s|
s = s[0].split(/\r\r\n/)
s.pop # get rid of prompt
s.each { |line| puts line }
print '> '
STDOUT.flush
write.print(STDIN.gets)
end
end
end
Use Open3.popen3(). e.g.:
Open3.popen3("jdb args") { |stdin, stdout, stderr|
# stdin = jdb's input stream
# stdout = jdb's output stream
# stderr = jdb's stderr stream
threads = []
threads << Thread.new(stderr) do |terr|
while (line = terr.gets)
puts "stderr: #{line}"
end
end
threads << Thread.new(stdout) do |terr|
while (line = terr.gets)
puts "stdout: #{line}"
end
end
stdin.puts "blah"
threads.each{|t| t.join()} #in order to cleanup when you're done.
}
I've given you examples for threads, but you of course want to be responsive to what jdb is doing. The above is merely a skeleton for how you open the process and handle communication with it.
The Ruby standard library includes expect, which is designed for just this type of problem. See the documentation for more information.

Resources