In Ruby, is it possible to prevent the standard input of a spawned child process from being attached to the terminal without having to capture the STDOUT or STDERR of that same process?
Backticks and x-strings (`...`, %x{...}) don't work because they capture STDIN.
Kernel#system doesn't work because it leaves STDIN attached to the
terminal (which intercepts signals like ^C and prevents them from
reaching my program, which is what I'm trying to avoid).
Open3 doesn't work because its methods capture either STDOUT or
both STDOUT and STDERR.
So what should I use?
If you’re on a platform that supports it, you could do this with pipe, fork and exec:
# create a pipe
read_io, write_io = IO.pipe
child = fork do
# in child
# close the write end of the pipe
write_io.close
# change our stdin to be the read end of the pipe
STDIN.reopen(read_io)
# exec the desired command which will keep the stdin just set
exec 'the_child_process_command'
end
# in parent
# close read end of pipe
read_io.close
# write what we want to the pipe, it will be sent to childs stdin
write_io.write "this will go to child processes stdin"
write_io.close
Process.wait child
Related
With python I can do proc.communicate(lines) to a command process and "lines" is available instantly as soon as the program starts.
So now I'm trying to do the same with ruby using Open3:
stdin, stdout, stderr, wait_thr = Open3.popen3(cmd)
stdin.puts(lines)
The problem is that there's a slight delay between when the program starts, and when the stdin gets supplied, that delay does not exist in the python version.
How can I run a command, inject stdin, process stdin instantly/at-start, and receive the exitcode (wait_thr in this case) ?
edit: I think I fixed the issue: https://gist.github.com/niuage/c0637b8dd10549a12b6a223dbd5f158a
I might have been missing the Process.wait, hence creating a lot of zombie processes.
I have a piece of code that's working most of the time, but "locks" itself after a while, probably because of a race condition.
My code
pipe = "goals.png"
(1..100).each do |time|
fork do
# This runs ffmpeg, and gets it to write 1 frame of a video to the pipe 'goals.png'
print "before ffmpeg"
`#{ffmpeg(time, score_crop_options, pipe)}`
exit
end
# Reads from 'pipe'
print "before read"
blob = File.read(pipe)
image = Rocket::Image.from_blob(blob)
# do stuff with image
end
Things to note:
#{ffmpeg(time, pipe)} writes to pipe, and is blocking until something reads from pipe
File.read(pipe) is blocking until something writes to pipe
My issue
edit: when the script is locked, and I try to read the pipe from another script, I get zsh: fork failed: resource temporarily unavailable. That's probably a good clue...
Most of the time, File.read(pipe) gets executed before the code in fork, so it works great, but after a little while the script just stops: it prints "before ffmpeg" and never gets to "before read"...
First, should I use threads instead of fork? And can I control the order the 2 statements (read and write) get run, to avoid a race condition? Or maybe it's not even about the race condition and I'm missing something?
The issue wasn't caused by a race condition, but too many zombie processes, since I wasn't calling Process.wait
The parent process should use Process.wait to collect the termination statuses of its children or use Process.detach to register disinterest in their status; otherwise, the operating system may accumulate zombie processes.
That's why I was getting zsh: fork failed: resource temporarily unavailable when trying to read from the pipe from another script probably.
Here's something that works:
(1..100) do
if fork
# Parent
image = read_image(pipe)
# do stuff with image
Process.wait # I think that's what was missing previously
else
# Child
Open3.popen3(command(time, score_crop_options, pipe)) do |stdin, stdout, stderr, wait_thr|
# stuff
end
exit!(0)
end
end
I am getting unexpected behaviour using popen3, which I want to use to run a command like tool ala cmd < file1 > file2. The below example hangs, so that stdout done is never reached. Using other tools than cat may cause hanging, so that stdin done is never reached. I suspect, I am suffering from buffering, but how do I fix this?
#!/usr/bin/env ruby
require 'open3'
Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
stdin.puts "foobar"
puts "stdin done"
stdout.each_line { |line| puts line }
puts "stdout done"
puts wait_thr.value
end
puts "all done"
stdout.each_line is waiting for further output from cat because cat's output stream is still open. It's still open because cat is still waiting for input from the user because its input stream hasn't been closed yet (you'll notice that when you open cat in a terminal and type in foobar, it will still be running and waiting for input until you press ^d to close the stream).
So to fix this, simply call stdin.close before you print the output.
Your code is hanging, because stdin is still open!
You need to close it with IO#close or with IO#close_write if you use popen3.
If you use popen then you need to use IO#close_write because it only uses one file descriptor.
#!/usr/bin/env ruby
require 'open3'
Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
stdin.puts "foobar"
stdin.close # close stdin like this! or with stdin.close_write
stdout.each_line { |line| puts line }
puts wait_thr.value
end
See also:
Ruby 1.8.7 IO#close_write
Ruby 1.9.2 IO#close_write
Ruby 2.3.1 IO#close_write
The answers by Tilo and by sepp2k are right: If you close stdin, your simple test will end. Problem solved.
Though in your comment to the answer of sepp2k, you indicate that you still experience hangs.
Well, there are some traps that you might have overlooked.
Stuck on full buffer for stderr
If you call a program that prints more to stderr than the buffer of an anonymous pipe can hold (64KiB for current Linuxes), the program gets suspended. A suspended program neither exits nor closes stdout. Consequently, reading from its stdout will hang. So if you want to do it right, you have to use threads or IO.select, non-blocking, unbuffered reads in order to read from both stdout and stderr in parallel or by turns without getting stuck.
Stuck on full buffer for stdin
If you try to feed more (much more) than "foobar" to your program (cat), the buffer of the anonymous pipe for stdout will get full. The OS will suspend cat. If you write even more to stdin, the buffer of the anonymous pipe for stdin will get full. Then your call to stdin.write will get stuck. This means: You need to write to stdin, read from stdout and read from stderr in parallel or by turns.
Conclusion
Read a good book (Richards Stevens, "UNIX Network Programming: Interprocess communications") and use good library functions. IPC (interprocess communications) is just too complicated and prone to indeterministic run-time behavior. It is for too much hassle to try to get it right by try-and-error.
Use Open3.capture3.
I am getting unexpected behaviour using popen3, which I want to use to run a command like tool ala cmd < file1 > file2. The below example hangs, so that stdout done is never reached. Using other tools than cat may cause hanging, so that stdin done is never reached. I suspect, I am suffering from buffering, but how do I fix this?
#!/usr/bin/env ruby
require 'open3'
Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
stdin.puts "foobar"
puts "stdin done"
stdout.each_line { |line| puts line }
puts "stdout done"
puts wait_thr.value
end
puts "all done"
stdout.each_line is waiting for further output from cat because cat's output stream is still open. It's still open because cat is still waiting for input from the user because its input stream hasn't been closed yet (you'll notice that when you open cat in a terminal and type in foobar, it will still be running and waiting for input until you press ^d to close the stream).
So to fix this, simply call stdin.close before you print the output.
Your code is hanging, because stdin is still open!
You need to close it with IO#close or with IO#close_write if you use popen3.
If you use popen then you need to use IO#close_write because it only uses one file descriptor.
#!/usr/bin/env ruby
require 'open3'
Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
stdin.puts "foobar"
stdin.close # close stdin like this! or with stdin.close_write
stdout.each_line { |line| puts line }
puts wait_thr.value
end
See also:
Ruby 1.8.7 IO#close_write
Ruby 1.9.2 IO#close_write
Ruby 2.3.1 IO#close_write
The answers by Tilo and by sepp2k are right: If you close stdin, your simple test will end. Problem solved.
Though in your comment to the answer of sepp2k, you indicate that you still experience hangs.
Well, there are some traps that you might have overlooked.
Stuck on full buffer for stderr
If you call a program that prints more to stderr than the buffer of an anonymous pipe can hold (64KiB for current Linuxes), the program gets suspended. A suspended program neither exits nor closes stdout. Consequently, reading from its stdout will hang. So if you want to do it right, you have to use threads or IO.select, non-blocking, unbuffered reads in order to read from both stdout and stderr in parallel or by turns without getting stuck.
Stuck on full buffer for stdin
If you try to feed more (much more) than "foobar" to your program (cat), the buffer of the anonymous pipe for stdout will get full. The OS will suspend cat. If you write even more to stdin, the buffer of the anonymous pipe for stdin will get full. Then your call to stdin.write will get stuck. This means: You need to write to stdin, read from stdout and read from stderr in parallel or by turns.
Conclusion
Read a good book (Richards Stevens, "UNIX Network Programming: Interprocess communications") and use good library functions. IPC (interprocess communications) is just too complicated and prone to indeterministic run-time behavior. It is for too much hassle to try to get it right by try-and-error.
Use Open3.capture3.
I have a set of tasks that I need to run from a Ruby script, however one particular task always waits for EOF on STDIN before quitting.
Obviously this causes the script to hang while waiting for the child process to end.
I have the process ID of the child process, but not a pipe or any kind of handle to it. How could I open a handle to the STDIN of a process to send EOF to it?
EDIT: Given that you aren't starting the script, a solution that occurs to me is to put $stdin under your control while using your gem. I suggest something like:
old_stdin = $stdin.dup
# note that old_stdin.fileno is non-0.
# create a file handle you can use to signal EOF
new_stdin = File::open('/dev/null', 'r')
# and make $stdin use it, instead.
$stdin.reopen(new_stdin)
new_stdin.close
# note that $stdin.fileno is still 0, though now it's using /dev/null for input.
# replace with the call that runs the external program
system('/bin/cat')
# "cat" will now exit. restore the old state.
$stdin.reopen(old_stdin)
old_stdin.close
If your ruby script is creating the tasks, it can use IO::popen. For example, cat, when run with no arguments, will wait for EOF on stdin before it exits, but you can run the following:
f = IO::popen('cat', 'w')
f.puts('hello')
# signals EOF to "cat"
f.close