I've got a simple test case for reading from a pipe that passes in MRI and fails in JRuby 1.7+. It's reported as a critical bug in JRUBY-6986.
tl;dr Why would a 16kb (16384 bytes) file be read perfectly from a pipe and a 16kb + 1 byte file fail? On another system, the threshold was found to be 72kb (73728 bytes).
$ ruby testpipe.rb 16384
Starting...didn't hang!
$ ruby testpipe.rb 16385
Starting...^C
There, the script hung at 16kb + 1 byte.
require 'open3'
path = "test.txt"
File.open(path, 'w') { |f| f.write('Z' * ARGV[0].to_i) }
STDOUT.write "Starting..."
Open3.popen3('cat', path) do |stdin, stdout, stderr|
stdin.close
readers = [stdout, stderr]
while readers.any?
ready = IO.select(readers, [], readers)
# no writers
# ready[1].each { ... }
ready[0].each do |fd|
if fd.eof?
fd.close
readers.delete fd
else
# read from pipe one byte at a time
fd.readpartial 1
end
end
end
end
puts "didn't hang!"
Could some operating system or JVM buffer size be causing different signals to be sent to IO.select or something?
PS. I doubled-checked with Jesse Storimer, author of Working With UNIX processes and he believes my usage of IO.select, etc. is correct.
Related
I'm using a command line program, it works as mentioned below:
$ ROUTE_TO_FOLDER/app < "long text"
If "long text" is written using the parameters "app" needs, then it will fill a text file with results. If not, it will fill the text file with dots continuously (I can't handle or modify the code of "app" in order to avoid this).
In a ruby script there's a line like this:
text = "long text that will be used by app"
output = system("ROUTE_TO_FOLDER/app < #{text}")
Now, if text is well written, there won't be problems and I will get an output file as mentioned before. The problem comes when text is not well written. What happens next is that my ruby script hangs and I'm not sure how to kill it.
I've found Open3 and I've used the method like this:
irb> cmd = "ROUTE_TO_FOLDER/app < #{text}"
irb> stdin, stdout, stderr, wait_thr = Open3.popen3(cmd)
=> [#<IO:fd 10>, #<IO:fd 11>, #<IO:fd 13>, #<Thread:0x007f3a1a6f8820 run>]
When I do:
irb> wait_thr.value
it also hangs, and :
irb> wait_thr.status
=> "sleep"
How can I avoid these problems? Is it not recognizing that "app" has failed?
wait_thr.pid provides you the pid of the started process. Just do
Process.kill("KILL",wait_thr.pid)
when you need to kill it.
You can combine it with detecting if the process is hung (continuously outputs dots) in one of the two ways.
1) Set a timeout for waiting for the process:
get '/process' do
text = "long text that will be used by app"
cmd = "ROUTE_TO_FOLDER/app < #{text}"
Open3.popen3(cmd) do |i,o,e,w|
begin
Timeout.timeout(10) do # timeout set to 10 sec, change if needed
# process output of the process. it will produce EOF when done.
until o.eof? do
# o.read_nonblock(N) ...
end
end
rescue Timeout::Error
# here you know that the process took longer than 10 seconds
Process.kill("KILL", w.pid)
# do whatever other error processing you need
end
end
end
2) Check the process output. (The code below is simplified - you probably don't want to read the output of your process into a single String buf first and then process, but I guess you get the idea).
get '/process' do
text = "long text that will be used by app"
cmd = "ROUTE_TO_FOLDER/app < #{text}"
Open3.popen3(cmd) do |i,o,e,w|
# process output of the process. it will produce EOF when done.
# If you get 16 dots in a row - the process is in the continuous loop
# (you may want to deal with stderr instead - depending on where these dots are sent to)
buf = ""
error = false
until o.eof? do
buf << o.read_nonblock(16)
if buf.size>=16 && buf[-16..-1] == '.'*16
# ok, the process is hung
Process.kill("KILL", w.pid)
error = true
# you should also get o.eof? the next time you check (or after flushing the pipe buffer),
# so you will get out of the until o.eof? loop
end
end
if error
# do whatever error processing you need
else
# process buf, it contains all the output
end
end
end
I am using Stockfish to analyze a chess position, but the specific context isn't that important. I want to read the output from the program in real-time without having to wait until the process is finished. In particular I want to read the last or last few lines. Current I have:
require 'pty'
master, slave = PTY.open
read, write = IO.pipe
pid = spawn("stockfish", :in=>read, :out=>slave)
calculating=0
slave.close
read.close
write.puts('uci')
write.puts("setoption name Threads value 1")
write.puts("position fen rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR")
write.puts('go infinite')
while calculating<10
sleep(1)
master.each do |line|
puts line
end
calculating+=1
end
write.puts('quit')
write.close
puts "done"
The problem with this is that the program isn't seeing an EOF marker until all 10 loops are finished so it's getting stuck in the master.each line block indefinitely. I can read the next line from the beginning one at a time with master.gets, but how would I read the last line instead?
I have a working solution now:
fen="rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR"
overseer=fork do
require 'pty'
master, slave = PTY.open
read, write = IO.pipe
pid = spawn("stockfish", :in=>read, :out=>slave)
slave.close
read.close
write.puts('uci')
write.puts("setoption name Threads value 4")
write.puts("position fen #{fen}")
write.puts('go infinite')
loop do
puts master.gets
end
end
if $stdin.gets
Process.kill("KILL",overseer)
end
This will print out a stream of stream until the user kills it.
I use Net::SSHv2 to connect to a server and start a script on that server. So far it is working, but I want to interrupt the script when it is running for over 10 minutes or the output file gets too big. After receiving the interrupt the script will shutdown and output some statistics.
My current code looks like this:
File.open(filename, "w") do |f|
Net::SSH.start(host, user, password: password) do |ssh|
ssh.exec! "do_work" do |channel, stream, data|
f << "data"
#break if f.size > 1024 * 1024 * 100 #file size > 100 MB
#channel.send_data "^C" if f.size > 1024 * 1024 * 100 #file size > 100 MB
end
end
end
I tried a couple of other things with opening a block for the channel and requesting a shell, but it didn't work.
send_data is the correct approach, but:
You need a PTY in order to send control codes to the server, which also means you need to open a channel before you exec on it.
There is nothing in the code to understand the human display convention of a caret character (^) followed by a capital C character to mean "send the character that is generated by a keyboard BIOS when you press CTRL+C". You have to send the ASCII character itself, which is the ETX (end of text) control character and is represented in C hexadecimal escape sequence as \x03.
This is not technically the SIGINT signal - its the terminal break signal which is unrelated to the POSIX signal IPC, but shells often interpret an ETX as "the user wants me to send a SIGINT to the process" - which is the main reason why you need a PTY - it instructs the system shell to honor this "keyboard generated control characters should be converted to signals" convention. There is currently no way to send actual signals over the SSH session with common SSH implementations. See this answer to a similar (though not Ruby specific) question for more details.
Your code should probably look like this:
File.open(filename, "w") do |f|
Net::SSH.start(host, user, password: password) do |ssh|
ssh.open_channel do |channel|
channel.request_pty
channel.exec "do_work" do |ch, success|
raise "could not execute command: #{command.inspect}" unless success
channel.on_data do |ch2, data|
f << data
end
#break if f.size > 1024 * 1024 * 100 #file size > 100 MB
channel.send_data "\x03" if f.size > 1024 * 1024 * 100 #file size > 100 MB
end
end.wait
end
end
The channel working code was copied more or less verbatim from session.rb source code. Please refer to that for more information.
I'm creating a script to wrap jdb (java debugger). I essentially want to wrap this process and proxy the user interaction. So I want it to:
start jdb from my script
send the output of jdb to stdout
pause and wait for input when jdb does
when the user enters commands, pass it to jdb
At the moment I really want a pass thru to jdb. The reason for this is to initialize the process with specific parameters and potentially add more commands in the future.
Update:
Here's the shell of what ended up working for me using expect:
PTY.spawn("jdb -attach 1234") do |read,write,pid|
write.sync = true
while (true) do
read.expect(/\r\r\n> /) do |s|
s = s[0].split(/\r\r\n/)
s.pop # get rid of prompt
s.each { |line| puts line }
print '> '
STDOUT.flush
write.print(STDIN.gets)
end
end
end
Use Open3.popen3(). e.g.:
Open3.popen3("jdb args") { |stdin, stdout, stderr|
# stdin = jdb's input stream
# stdout = jdb's output stream
# stderr = jdb's stderr stream
threads = []
threads << Thread.new(stderr) do |terr|
while (line = terr.gets)
puts "stderr: #{line}"
end
end
threads << Thread.new(stdout) do |terr|
while (line = terr.gets)
puts "stdout: #{line}"
end
end
stdin.puts "blah"
threads.each{|t| t.join()} #in order to cleanup when you're done.
}
I've given you examples for threads, but you of course want to be responsive to what jdb is doing. The above is merely a skeleton for how you open the process and handle communication with it.
The Ruby standard library includes expect, which is designed for just this type of problem. See the documentation for more information.
How can we lock an IO that has been shared by multiple ruby process?
Consider this script:
#!/usr/bin/ruby -w
# vim: ts=2 sw=2 et
if ARGV.length != 2
$stderr.puts "Usage: test-io-fork.rb num_child num_iteration"
exit 1
end
CHILD = ARGV[0].to_i
ITERATION = ARGV[1].to_i
def now
t = Time.now
"#{t.strftime('%H:%M:%S')}.#{t.usec}"
end
MAP = %w(nol satu dua tiga empat lima enam tujuh delapan sembilan)
IO.popen('-', 'w') {|pipe|
unless pipe
# Logger child
File.open('test-io-fork.log', 'w') {|log|
log.puts "#{now} Program start"
$stdin.each {|line|
log.puts "#{now} #{line}"
}
log.puts "#{now} Program end"
}
exit!
end
pipe.sync = true
pipe.puts "Before fork"
CHILD.times {|c|
fork {
pid = Process.pid
srand
ITERATION.times {|i|
n = rand(9)
sleep(n / 100000.0)
pipe.puts "##{c}:#{i} #{MAP[n]} => #{n}, #{n} => #{MAP[n]} ##{c}:#{i}"
}
}
}
}
And try it like this:
./test-io-fork.rb 200 50
Like expected, the test-io-fork.log files would contains sign of IO race condition.
What I want to achieve is to make a TCP server for custom GPS protocol that will save the GPS points to database. Because this server would handle 1000 concurrent clients, I would like to restrict database connection to only one child instead opening 1000 database connection simultaneously. This server would run on linux.
UPDATE
It may be bad form to update after the answer was accepted, but the original is a bit misleading. Whether or not ruby makes a separate write(2) call for the automatically-appended newline is dependent upon the buffering state of the output IO object.
$stdout (when connected to a tty) is generally line-buffered, so the effect of a puts() -- given reasonably sized string -- with implicitly added newline is a single call to write(2). Not so, however, with IO.pipe and $stderr, as the OP discovered.
ORIGINAL ANSWER
Change your chief pipe.puts() argument to be a newline terminated string:
pipe.puts "##{c} ... #{i}\n" # <-- note the newline
Why? You set pipe.sync hoping that the pipe writes would be atomic and non-interleaved, since they are (presumably) less than PIPE_BUF bytes. But it didn't work, because ruby's pipe puts() implementation makes a separate call to write(2) to append the trailing newline, and that's why your writes are sometimes interleaved where you expected a newline.
Here's a corroborating excerpt from a fork-following strace of your script:
$ strace -s 2048 -fe trace=write ./so-1326067.rb
....
4574 write(4, "#0:12 tiga => 3, 3 => tiga #0:12", 32) = 32
4574 write(4, "\n", 1)
....
But putting in your own newline solves the problem, making sure that your entire record is transmitted in one syscall:
....
5190 write(4, "#194:41 tujuh => 7, 7 => tujuh #194:41\n", 39 <unfinished ...>
5179 write(4, "#183:38 enam => 6, 6 => enam #183:38\n", 37 <unfinished ...>
....
If for some reason that cannot work for you, you'll have to coordinate an interprocess mutex (like File.flock()).