I'm attempting to add some more automated tests to the ruby-newt module. The code I have seems to work, but still requires manually hitting ENTER at the terminal in order for it to complete.
For example in the following code, \t will switch to the next button and \r will press the button, and both commands execute successfully, but the ENTER key still needs to be manually pressed at the terminal otherwise the program just hangs indefinitely.
If the line wr.write "\t\r" is commented out, then the program will time out and exit successfully after 10 seconds. I've tried wr.flush, but that does not help. I've also tried including \n in the command.
Is there anything additional I should include in the write command to ensure the child program successfully receives it?
require 'newt'
require 'pty'
def newt_run
begin
Newt::Screen.new
Newt::Screen.centered_window(20, 15, 'Button')
b1 = Newt::Button.new(1, 1, 'Button1')
b2 = Newt::Button.new(1, 6, 'Button2')
b = Newt::Button.new(1, 11, 'Exit')
f = Newt::Form.new
f.set_timer(10000)
f.add(b1, b2, b)
rv = f.run
ensure
Newt::Screen.finish
end
end
master, slave = PTY.open
rd, wr = IO.pipe
if fork.nil? then
master.close
wr.close
$stdin.reopen(rd)
$stdout.reopen(slave)
$stderr.reopen(slave)
newt_run
else
slave.close
rd.close
wr.write "\t\r"
Process.wait
end
The problem is that the newt C-library opens /dev/tty by default for input. It does not use stdin. This is why nothing you send it seems to work. It's not reading your pipe, it is reading /dev/tty.
Here is the problem in more detail:
Ruby newt calls newtInit() in the C-lib
newtInit() calls SLang_init_tty
SLang_init_tty attaches input to /dev/tty by default always.
If you read the documentation of SLang_init_tty you will find that the variable SLang_TT_Read_FD determines if /dev/tty is used or not.
Solution 1:
You need to set SLang_TT_Read_FD to stdin, before calling newtInit() from Ruby.
Solution 2:
Use setsid and ioctl(TIOCSCTTY) to reassign the controlling terminal in the forked process (see docs for ioctl here).
Working example:
TIOCSCTTY = 0x540E
master, slave = PTY.open
if fork.nil? then
# Child process
# Make group leader. Required for aquiring controlling TTY.
Process.setsid
master.close # Close master side
STDIN.reopen(slave) # Reassign STDIN
STDIN.ioctl(TIOCSCTTY, 0) # Reassign controlling TTY (important part)
# Ensure master is ready
slave.gets
# Now we can run the UI
newt_run
else
# Parent process
slave.close
# Sync up with slave
master.puts 'hello'
# Allow for UI setup
sleep 1
master.write "\e[B" # Arrow down
master.write "\e[B" # Arrow down
master.write "\t" # Tab
master.write "\r" # Enter
Process.wait
end
Related
What I'm trying to achieve:
From a Ruby process, spawning a subprocess
The subprocess should print as normal back to the terminal. By "normal", I mean the process shouldn't miss out color output, or ignore user input (STDIN).
For that subprocess, capturing STDOUT/STDERR (jointly) e.g. into a String variable that can be accessed after the subprocess is dead. Escape characters and all.
Capturing STDOUT/STDERR is possible by passing a different IO pipe, however the subprocess can then detect that it's not in a tty. For example git log will not print characters that influence text color, nor use it's pager.
Using a pty to launch the process essentially "tricks" the subprocess into thinking it's being launched by a user. As far as I can tell, this is exactly what I want, and the result of this essentially ticks all the boxes.
My general tests to test if a solution fits my needs is:
Does it run ls -al normally?
Does it run vim normally?
Does it run irb normally?
The following Ruby code is able to check all the above:
to_execute = "vim"
output = ""
require 'pty'
require 'io/console'
master, slave = PTY.open
slave.raw!
pid = ::Process.spawn(to_execute, :in => STDIN, [:out, :err] => slave)
slave.close
master.winsize = $stdout.winsize
Signal.trap(:WINCH) { master.winsize = $stdout.winsize }
Signal.trap(:SIGINT) { ::Process.kill("INT", pid) }
master.each_char do |char|
STDOUT.print char
output.concat(char)
end
::Process.wait(pid)
master.close
This works for the most part but it turns out it's not perfect. For some reason, certain applications seem to fail to switch into a raw state. Even though vim works perfectly fine, it turned out neovim did not. At first I thought it was a bug in neovim but I have since been able to reproduce the problem using the Termion crate for the Rust language.
By setting to raw manually (IO.console.raw!) before executing, applications like neovim behave as expected, but then applications like irb do not.
Oddly spawning another pty in Python, within this pty, allows the application to work as expected (using python -c 'import pty; pty.spawn("/usr/local/bin/nvim")'). This obviously isn't a real solution, but interesting nonetheless.
For my actual question I guess I'm looking towards any help to resolving the weird raw issue or, say if I've completely misunderstood tty/pty, any different direction to where/how I should look at the problem.
[edited: see the bottom for the amended update]
Figured it out :)
To really understand the problem I read up a lot on how a PTY works. I don't think I really understood it properly until I drew it out. Basically PTY could be used for a Terminal emulator, and that was the simplest way to think of the data flow for it:
keyboard -> OS -> terminal -> master pty -> termios -> slave pty -> shell
|
v
monitor <- OS <- terminal <- master pty <- termios
(note: this might not be 100% correct, I'm definitely no expert on the subject, just posting it incase it helps anybody else understand it)
So the important bit in the diagram that I hadn't really realised was that when you type, the only reason you see your input on screen is because it's passed back (left-wards) to the master.
So first thing's first - this ruby script should first set the tty to raw (IO.console.raw!), it can restore it after execution is finished (IO.console.cooked!). This'll make sure the keyboard inputs aren't printed by this parent Ruby script.
Second thing is the slave itself should not be raw, so the slave.raw! call is removed. To explain this, I originally added this because it removes extra return carriages from the output: running echo hello results in "hello\r\n". What I missed was that this return carriage is a key instruction to the terminal emulator (whoops).
Third thing, the process should only be talking to the slave. Passing STDIN felt convenient, but it upsets the flow shown in the diagram.
This brings up a new problem on how to pass user input through, so I tried this. So we basically pass STDIN to the master:
input_thread = Thread.new do
STDIN.each_char do |char|
master.putc(char) rescue nil
end
end
that kind of worked, but it has its own issues in terms of some interactive processes weren't receiving a key some of the time. Time will tell, but using IO.copy_stream instead appears to solve that issue (and reads much nicer of course).
input_thread = Thread.new { IO.copy_stream(STDIN, master) }
update 21st Aug:
So the above example mostly worked, but for some reason keys like CTRL+c still wouldn't behave correctly. I even looked up other people's approach to see what I could be doing wrong, and effectively it seemed the same approach - as IO.copy_stream(STDIN, master) was successfully sending 3 to the master. None of the following seemed to help at all:
master.putc 3
master.putc "\x03"
master.putc "\003"
Before I went and delved into trying to achieve this in a lower level language I tried out 1 more thing - the block syntax. Apparently the block syntax magically fixes this problem.
To prevent this answer getting a bit too verbose, the following appears to work:
require 'pty'
require 'io/console'
def run
output = ""
IO.console.raw!
input_thread = nil
PTY.spawn('bash') do |read, write, pid|
Signal.trap(:WINCH) { write.winsize = STDOUT.winsize }
input_thread = Thread.new { IO.copy_stream(STDIN, write) }
read.each_char do |char|
STDOUT.print char
output.concat(char)
end
Process.wait(pid)
end
input_thread.kill if input_thread
IO.console.cooked!
end
Bundler.send(:with_env, Bundler.clean_env) do
run
end
I am using the open4 gem to wrap system calls to a potentially long-running third-party command line tool. The tool may sometimes fail, keeping two processes busy, and partially blocking a pipeline, as the parent process is part of a pool of worker scripts (serving a Beanstalk queue). From outside of the system, I can identify a stuck worker script and its process id programatically, based on the data model of what is being processed. Inside the Open4.open4 block, I can identify the child process id.
I'd like to set up the Open4 block so that when I send a SIGTERM to the parent worker process, it forwards on the SIGTERM to the child. In addition, if the child process has still failed to exit after a short wait, I want to send a SIGKILL to the child process. In both cases, I'd then like the parent process to respond as normal to the SIGTERM it was sent.
This is all being done so I can expose a "stop" button in a customer services app, so non-technical team members have a tool to manage their way out of a situation with a blocked queue.
I have found some related questions in SO - e.g. How to make child process die after parent exits? - but the answers are not really usable for me from Ruby application code.
Here is a current implementation in Ruby that I have tested on my Mac:
Test stand-in for "bad" process that won't always respond to SIGTERM:
# Writing to a log file shows whether or not a detached process continues
# once the parent has closed IO to it.
$f = open( 'log.txt', 'w' );
def say m
begin
$f.puts m
$f.flush
$stderr.puts m
rescue Exception => e
# When the parent process closes, we get
# #<Errno::EPIPE: Broken pipe - <STDERR>> in this
# test, but with a stuck child process, this is not
# guaranteed to happen or cause the child to exit.
$f.puts e.inspect
$f.flush
end
end
Signal.trap( "TERM" ) { say "Received and ignored TERM" }
# Messages get logged, and sleep allows test of manual interrupts
say "Hello"
sleep 3
say "Foo Bar Baz"
sleep 3
say "Doo Be Doo"
sleep 3
say "Goodbye"
$f.close
Test Open4 block (part of a "worker" test script):
Open4.open4(#command) do | pid, stdin, stdout, stderr |
begin
stderr.each { |l|
puts "[#{pid}] STDERR: #{l}" if l
}
rescue SignalException => e
puts "[#{$$}] Received signal (#{e.signo} #{e.signm}) in Open4 block"
# Forward a SIGTERM to child, upgrade to SIGKILL if it doesn't work
if e.signo == 15
begin
puts "[#{$$}] Sending TERM to child process"
Process.kill( 'TERM', pid )
timeout(3.0) { Process.waitpid( pid ) }
rescue Timeout::Error
puts "[#{$$}] Sending KILL to child process"
Process.kill( 'KILL', pid )
end
end
raise e
end
end
Typical output if I start this up, and run e.g. kill -15 16854:
[16855] STDERR: Hello
[16854] Received signal (15 SIGTERM) in Open4 block
[16854] Sending TERM to child process
[16854] Sending KILL to child process
Contents of log file for same test:
Hello
Received and ignored TERM
Foo Bar Baz
The code is IMO a bit unwieldy, although it appears to work as I want. My questions:
Is the above attempt ok, or fatally flawed in the use case I need it for?
Have I missed a cleaner way of doing the same thing using existing Open4 and core Ruby methods?
I'm building a daemon that will help me manage my server(s). Webmin works fine, as does just opening a shell to the server, but I'd prefer to be able to control server operations from a UI I design, and also expose some functionality to end users.
The daemon will pick up actions from a queue and execute them. However, since I'll be accepting input from users, I want to make sure they're not permitted to inject something dangerous into a privileged shell command.
Here's a fragment that exemplifies my problem:
def perform
system "usermod -p #{#options['shadow']} #{#options['username']}"
end
A gist that explains more: https://gist.github.com/773292
I'm not positive if typical escaping and sanitizing of inputs is enough for this case, and being a designer, I don't have a ton of security-related experience. I know that this is something that should probably be obvious to me, but its not!
How can I ensure that the web application that will create and serialize the actions can't pass dangerous text into the privileged process that receives the actions?
Thanks for the help
arb
It doesn't look like you need a shell for what you're doing. See the documentation for system here: http://ruby-doc.org/core/classes/Kernel.html#M001441
You should use the second form of system. Your example above would become:
system 'usermod', '-p', #options['shadow'], #options['username']
A nicer (IMO) way to write this is:
system *%W(usermod -p #{#options['shadow']} #{#options['username']})
The arguments this way are passed directly into the execve call, so you don't have to worry about sneaky shell tricks.
If you need not just the exit status but also the result you probably want to use Open3.popen3:
require 'open3'
stdin, stdout, stderr = Open3.popen3('usermod', '-p', #options['shadow'], #options['username'])
stdout.gets
sterr.gets
More information here: Getting output of system() calls in Ruby
I'd suggest looking into the 'shellwords' module. This script:
require 'shellwords'
parts = ['echo', "'hello world'; !%& some stuff", 'and another argument']
command = Shellwords.shelljoin( parts )
puts command
output = `#{ command }`
puts output
outputs the escaped text and the expected output:
echo \'hello\ world\'\;\ \!\%\&\ some\ stuff and\ another\ argument
'hello world'; !%& some stuff and another argument
This is an old question, but since it's pretty much the only real answer you'll find when googling I thought I'd add a caveat. The multi argument version of system seems reasonably safe on Linux, but it is NOT on Windows.
Try system "dir", "&", "echo", "hi!"
on a Windows system. Both dir and echo will be run. Echo could of course just as well be something far less innocuous.
I know this is an old thread, but there is another option that was lightly touched on by Simon Hürlimann.
There is not a lot of information about this topic and I think this might help others in need.
For this example we'll use Open3 which gives you the ability to run commands synchronously or asynchronously, and provides stdout, stderr, exit codes, and PID.
Open3 grants you access to stdout, stderr, exit codes and a thread to wait for the child process when running another program. You can specify various attributes, redirections, current directory, etc., of the program in the same way as for Process.spawn. (Source: Open3 Docs)
I chose to format the output as a CommandStatus object. This contains our stdout, stderr, pid (Of the worker thread) and exitstatus.
class Command
require 'open3'
class CommandStatus
#stdout = nil
#stderr = nil
#pid = nil
#exitstatus = nil
def initialize(stdout, stderr, process)
#stdout = stdout
#stderr = stderr
#pid = process.pid
#exitstatus = process.exitstatus
end
def stdout
#stdout
end
def stderr
#stderr
end
def exit_status
#exitstatus
end
def pid
#pid
end
end
def self.execute(command)
command_stdout = nil
command_stderr = nil
process = Open3.popen3(ENV, command + ';') do |stdin, stdout, stderr, thread|
stdin.close
stdout_buffer = stdout.read
stderr_buffer = stderr.read
command_stdout = stdout_buffer if stdout_buffer.length > 0
command_stderr = stderr_buffer if stderr_buffer.length > 0
thread.value # Wait for Process::Status object to be returned
end
return CommandStatus.new(command_stdout, command_stderr, process)
end
end
cmd = Command::execute("echo {1..10}")
puts "STDOUT: #{cmd.stdout}"
puts "STDERR: #{cmd.stderr}"
puts "EXIT: #{cmd.exit_status}"
While reading the STDOUT/ERR buffers, I use command_stdout = stdout_buffer if stdout_buffer.length > 0 to control whether the command_stdout variable is assigned or not. You should pass nil instead of "" when no data is present. It's more clear when handing data later on.
You probably noticed me using command + ';'. The reason for this is based on the documentation from Kernel.exec (Which is what popen3 uses):
If the string from the first form (exec("command")) follows these
simple rules:
no meta characters
no shell reserved word and no special built-in
Ruby invokes the command directly without shell
You can force shell invocation by adding ";" to the string (because
";" is a meta character)
This simply prevents a Ruby from throwing a 'spawn': No such file or directory error if you pass a malformed command. Instead it will pass it straight to the kernel where the error will be resolved gracefully and appear as STDERR instead of an uncaught exception.
Modern, secure and simple solution (popen will escape arguments for you):
IO.popen(['usermod', '-p', #options['shadow'], #options['username']]).read
(#read will close the IO before returning)
I want to write a Ruby program that will always be running in the background (a daemon) on my Mac.
Can someone point me in the right direction on how this would be done?
Ruby 1.9.x has now the following:
Process.daemon
Put it in your code and that's it.
Taken from "Daemon Processes in Ruby."
Use Daemonize.rb
require 'daemons'
Daemons.daemonize
Very simple sample: http://github.com/utkarsh2012/backitup/blob/master/backitup.rb
How to install daemons gem:
gem install daemons
Ah, Google to the rescue! Check out
http://fitzgeraldsteele.wordpress.com/2009/05/04/launchd-example-start-web-server-at-boot-time/
wherein a helpful blogger provides an example of writing a launchd plist to launch a ruby Web application server.
This is a module to daemonize your code. Here's an offshoot that wraps an existing script.
Essentially it boils down to this (from Travis Whitton's Daemonize.rb, the first link above, modified for some program I wrote ages ago):
private
# This method causes the current running process to become a daemon
# If closefd is true, all existing file descriptors are closed
def daemonize(pathStdErr, oldmode=0, closefd=false)
srand # Split rand streams between spawning and daemonized process
safefork and exit# Fork and exit from the parent
# Detach from the controlling terminal
unless sess_id = Process.setsid
raise 'Cannot detach from controlled terminal'
end
# Prevent the possibility of acquiring a controlling terminal
if oldmode.zero?
trap 'SIGHUP', 'IGNORE'
exit if pid = safefork
end
Dir.chdir "/" # Release old working directory
File.umask 0000 # Insure sensible umask
if closefd
# Make sure all file descriptors are closed
ObjectSpace.each_object(IO) do |io|
unless [STDIN, STDOUT, STDERR].include?(io)
io.close rescue nil
end
end
end
STDIN.reopen "/dev/null" # Free file descriptors and
STDOUT.reopen "/dev/null" # point them somewhere sensible
STDERR.reopen pathStdErr, "w" # STDOUT/STDERR should go to a logfile
return oldmode ? sess_id : 0 # Return value is mostly irrelevant
end
# Try to fork if at all possible retrying every 5 sec if the
# maximum process limit for the system has been reached
def safefork
tryagain = true
while tryagain
tryagain = false
begin
if pid = fork
return pid
end
rescue Errno::EWOULDBLOCK
sleep 5
tryagain = true
end
end
end
Need to see the daemons-rails gem for Rails 3 (based on rails_generator):
https://github.com/mirasrael/daemons-rails
Possible to generate daemon stub like this:
rails generate daemon <name>
Features:
individual control script per daemon
rake:daemon command per daemon
capistrano friendly
app-wide control script
monitoring API
possible multiple daemon sets
I'm looking to expose an interactive command line program via JSON or another RPC style service using Ruby. I've found a couple tricks to do this, but im missing something when redirecting the output and input.
One method at least on linux is to redirect the stdin and stdout to a file then read and write to that file asynchronously with file reads and writes. Another method ive been trying after googling around was to use open4. Here is the code I wrote so far, but its getting stuck after reading a few lines from standard output.
require "open4"
include Open4
status = popen4("./srcds_run -console -game tf +map ctf_2fort -maxplayers 6") do |pid, stdin, stdout, stderr|
puts "PID #{pid}"
lines=""
while (line=stdout.gets)
lines+=line
puts line
end
while (line=stderr.gets)
lines+=line
puts line
end
end
Any help on this or some insight would be appreciated!
What I would recommend is using Xinetd (or similar) to run the command on some socket and then using the ruby network code. One of the problems you've already run into in your code here is that your two while loops are sequential, which can cause problems.
Another trick you might try is to re-direct stderr to stdout in your command, so that your program only has to read the stdout. Something like this:
popen4("./srcds_run -console -game tf +map ctf_2fort -maxplayers 6 2>&1")
The other benefit of this is that you get all the messages/errors in the order they happen during the program run.
EDIT
Your should consider integrating with AnyTerm. You can then either expose AnyTerm directly e.g. via Apache mod_proxy, or have your Rails controller act as the reverse proxy (handling authentication/session validation, then playing back controller.request minus any cookies to localhost:<AnyTerm-daemon-port>, and sending back as a response whatever AnyTerm replies with.)
class ConsoleController < ApplicationController
# AnyTerm speaks via HTTP POST only
def update
# validate session
...
# forward request to AnyTerm
response = Net::HTTP.post_form(URI.parse('http://localhost:#{AnyTermPort}/', request.params))
headers['Content-Type'] = response['Content-Type']
render_text response.body, response.status
end
Otherwise, you'd need to use IO::Select or IO::read_noblock to know when data is available to be read (from either network or subprocess) so you don't deadlock. See this too. Also check that either your Rails is used in a multi-threaded environment or that your Ruby version is not affected by this IO::Select bug.
You can start with something along these lines:
status = POpen4::popen4("ping localhost") do |stdout, stderr, stdin, pid|
puts "PID #{pid}"
# our buffers
stdout_lines=""
stderr_lines=""
begin
loop do
# check whether stdout, stderr or both are
# ready to be read from without blocking
IO.select([stdout,stderr]).flatten.compact.each { |io|
# stdout, if ready, goes to stdout_lines
stdout_lines += io.readpartial(1024) if io.fileno == stdout.fileno
# stderr, if ready, goes to stdout_lines
stderr_lines += io.readpartial(1024) if io.fileno == stderr.fileno
}
break if stdout.closed? && stderr.closed?
# if we acumulated any complete lines (\n-terminated)
# in either stdout/err_lines, output them now
stdout_lines.sub!(/.*\n/m) { puts $& ; '' }
stderr_lines.sub!(/.*\n/m) { puts $& ; '' }
end
rescue EOFError
puts "Done"
end
end
To also handle stdin, change to:
IO.select([stdout,stderr],[stdin]).flatten.compact.each { |io|
# program ready to get stdin? do we have anything for it?
if io.fileno == stdin.fileno && <got data from client?>
<write a small chunk from client to stdin>
end
# stdout, if ready, goes to stdout_lines