With python I can do proc.communicate(lines) to a command process and "lines" is available instantly as soon as the program starts.
So now I'm trying to do the same with ruby using Open3:
stdin, stdout, stderr, wait_thr = Open3.popen3(cmd)
stdin.puts(lines)
The problem is that there's a slight delay between when the program starts, and when the stdin gets supplied, that delay does not exist in the python version.
How can I run a command, inject stdin, process stdin instantly/at-start, and receive the exitcode (wait_thr in this case) ?
Related
When I use Go's exec.Command{} to execute some bash script that contains nohup, it will hang forever.
I don't know what are the differences between ping and ifconfig. I tried to redirect the stdin (< /dev/null), the stdout(> /dev/null) and the stderr(2> /dev/null), and their combination, some of them work some don't.
When I use sh to execute the script, it just ends up immediately.
The Go code:
package main
import (
"fmt"
"os/exec"
)
func main() {
cmd := exec.Command("sh", "a.sh")
out, err := cmd.Output() // Or cmd.CombinedOutput()
fmt.Println(string(out), err)
}
The Bash script (a.sh):
#!/bin/bash
# hangs
#nohup ping localhost &
# dot not hang
nohup ifconfig &
(Converting comments, with glitches fixed, to answer)
The use of nohup here is mostly a red herring. The real problem is that ping never finishes. However, nohup has some extra weirdness, which you can see if you run, from an interactive terminal, these two sets of commands:
$ nohup echo foo
nohup: ignoring input and appending output to 'nohup.out'
$ cat nohup.out
foo
$
vs:
$ nohup echo foo </dev/null 2>&1 | cat
foo
$
Note how the first one printed a weird message, and then the output foo went to a file; the second did not, and then the output foo showed up on the regular output stream. This is because POSIX says that nohup should do these redirections if appropriate.1 When run with exec.Cmd and cmd.Output, the redirections are not performed.
At the OS level, on a Linux- or other Unix-like system, the exec code creates an OS pipe object by which the invoked command can send output back to the Go runtime. (There may be a separate pipe for its stderr output, or the two may both be directed to a single pipe, depending on how you run the command; see https://golang.org/src/os/exec/exec.go#L280.) This pipe winds up being passed to ping, so that ping can keep writing output there as long as it likes.
The shell itself exits, because the command nohup ping localhost & is backgrounded. However, ping still has write access to the pipe object, so the Go runtime continues calling the OS read code until the pipe is closed—which is never. If the pipe were ever closed, the Go runtime would receive EOF and call the wait system call to collect the shell's exit status, but that never happens.
Redirecting ping's output, such that the shell itself has the only write access to the pipe, should result in the pipe being closed as soon as the shell itself exits.
(Some shells may have a builtin nohup that may behave weirdly, especially in the presence of redirection. This is true of some particularly ancient shells.)
1See https://pubs.opengroup.org/onlinepubs/9699919799/utilities/nohup.html for complete details. The Linux variant redirects stdin as well as stdout and stderr, if the input is a terminal, and if the output and stderr are terminals. The FreeBSD variant redirects only stdout and/or stderr. The "is a terminal" test is based on the C language isatty function, which does the same thing as https://godoc.org/golang.org/x/crypto/ssh/terminal#IsTerminal.
I am getting unexpected behaviour using popen3, which I want to use to run a command like tool ala cmd < file1 > file2. The below example hangs, so that stdout done is never reached. Using other tools than cat may cause hanging, so that stdin done is never reached. I suspect, I am suffering from buffering, but how do I fix this?
#!/usr/bin/env ruby
require 'open3'
Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
stdin.puts "foobar"
puts "stdin done"
stdout.each_line { |line| puts line }
puts "stdout done"
puts wait_thr.value
end
puts "all done"
stdout.each_line is waiting for further output from cat because cat's output stream is still open. It's still open because cat is still waiting for input from the user because its input stream hasn't been closed yet (you'll notice that when you open cat in a terminal and type in foobar, it will still be running and waiting for input until you press ^d to close the stream).
So to fix this, simply call stdin.close before you print the output.
Your code is hanging, because stdin is still open!
You need to close it with IO#close or with IO#close_write if you use popen3.
If you use popen then you need to use IO#close_write because it only uses one file descriptor.
#!/usr/bin/env ruby
require 'open3'
Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
stdin.puts "foobar"
stdin.close # close stdin like this! or with stdin.close_write
stdout.each_line { |line| puts line }
puts wait_thr.value
end
See also:
Ruby 1.8.7 IO#close_write
Ruby 1.9.2 IO#close_write
Ruby 2.3.1 IO#close_write
The answers by Tilo and by sepp2k are right: If you close stdin, your simple test will end. Problem solved.
Though in your comment to the answer of sepp2k, you indicate that you still experience hangs.
Well, there are some traps that you might have overlooked.
Stuck on full buffer for stderr
If you call a program that prints more to stderr than the buffer of an anonymous pipe can hold (64KiB for current Linuxes), the program gets suspended. A suspended program neither exits nor closes stdout. Consequently, reading from its stdout will hang. So if you want to do it right, you have to use threads or IO.select, non-blocking, unbuffered reads in order to read from both stdout and stderr in parallel or by turns without getting stuck.
Stuck on full buffer for stdin
If you try to feed more (much more) than "foobar" to your program (cat), the buffer of the anonymous pipe for stdout will get full. The OS will suspend cat. If you write even more to stdin, the buffer of the anonymous pipe for stdin will get full. Then your call to stdin.write will get stuck. This means: You need to write to stdin, read from stdout and read from stderr in parallel or by turns.
Conclusion
Read a good book (Richards Stevens, "UNIX Network Programming: Interprocess communications") and use good library functions. IPC (interprocess communications) is just too complicated and prone to indeterministic run-time behavior. It is for too much hassle to try to get it right by try-and-error.
Use Open3.capture3.
In Ruby, is it possible to prevent the standard input of a spawned child process from being attached to the terminal without having to capture the STDOUT or STDERR of that same process?
Backticks and x-strings (`...`, %x{...}) don't work because they capture STDIN.
Kernel#system doesn't work because it leaves STDIN attached to the
terminal (which intercepts signals like ^C and prevents them from
reaching my program, which is what I'm trying to avoid).
Open3 doesn't work because its methods capture either STDOUT or
both STDOUT and STDERR.
So what should I use?
If you’re on a platform that supports it, you could do this with pipe, fork and exec:
# create a pipe
read_io, write_io = IO.pipe
child = fork do
# in child
# close the write end of the pipe
write_io.close
# change our stdin to be the read end of the pipe
STDIN.reopen(read_io)
# exec the desired command which will keep the stdin just set
exec 'the_child_process_command'
end
# in parent
# close read end of pipe
read_io.close
# write what we want to the pipe, it will be sent to childs stdin
write_io.write "this will go to child processes stdin"
write_io.close
Process.wait child
I am getting unexpected behaviour using popen3, which I want to use to run a command like tool ala cmd < file1 > file2. The below example hangs, so that stdout done is never reached. Using other tools than cat may cause hanging, so that stdin done is never reached. I suspect, I am suffering from buffering, but how do I fix this?
#!/usr/bin/env ruby
require 'open3'
Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
stdin.puts "foobar"
puts "stdin done"
stdout.each_line { |line| puts line }
puts "stdout done"
puts wait_thr.value
end
puts "all done"
stdout.each_line is waiting for further output from cat because cat's output stream is still open. It's still open because cat is still waiting for input from the user because its input stream hasn't been closed yet (you'll notice that when you open cat in a terminal and type in foobar, it will still be running and waiting for input until you press ^d to close the stream).
So to fix this, simply call stdin.close before you print the output.
Your code is hanging, because stdin is still open!
You need to close it with IO#close or with IO#close_write if you use popen3.
If you use popen then you need to use IO#close_write because it only uses one file descriptor.
#!/usr/bin/env ruby
require 'open3'
Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
stdin.puts "foobar"
stdin.close # close stdin like this! or with stdin.close_write
stdout.each_line { |line| puts line }
puts wait_thr.value
end
See also:
Ruby 1.8.7 IO#close_write
Ruby 1.9.2 IO#close_write
Ruby 2.3.1 IO#close_write
The answers by Tilo and by sepp2k are right: If you close stdin, your simple test will end. Problem solved.
Though in your comment to the answer of sepp2k, you indicate that you still experience hangs.
Well, there are some traps that you might have overlooked.
Stuck on full buffer for stderr
If you call a program that prints more to stderr than the buffer of an anonymous pipe can hold (64KiB for current Linuxes), the program gets suspended. A suspended program neither exits nor closes stdout. Consequently, reading from its stdout will hang. So if you want to do it right, you have to use threads or IO.select, non-blocking, unbuffered reads in order to read from both stdout and stderr in parallel or by turns without getting stuck.
Stuck on full buffer for stdin
If you try to feed more (much more) than "foobar" to your program (cat), the buffer of the anonymous pipe for stdout will get full. The OS will suspend cat. If you write even more to stdin, the buffer of the anonymous pipe for stdin will get full. Then your call to stdin.write will get stuck. This means: You need to write to stdin, read from stdout and read from stderr in parallel or by turns.
Conclusion
Read a good book (Richards Stevens, "UNIX Network Programming: Interprocess communications") and use good library functions. IPC (interprocess communications) is just too complicated and prone to indeterministic run-time behavior. It is for too much hassle to try to get it right by try-and-error.
Use Open3.capture3.
I have a set of tasks that I need to run from a Ruby script, however one particular task always waits for EOF on STDIN before quitting.
Obviously this causes the script to hang while waiting for the child process to end.
I have the process ID of the child process, but not a pipe or any kind of handle to it. How could I open a handle to the STDIN of a process to send EOF to it?
EDIT: Given that you aren't starting the script, a solution that occurs to me is to put $stdin under your control while using your gem. I suggest something like:
old_stdin = $stdin.dup
# note that old_stdin.fileno is non-0.
# create a file handle you can use to signal EOF
new_stdin = File::open('/dev/null', 'r')
# and make $stdin use it, instead.
$stdin.reopen(new_stdin)
new_stdin.close
# note that $stdin.fileno is still 0, though now it's using /dev/null for input.
# replace with the call that runs the external program
system('/bin/cat')
# "cat" will now exit. restore the old state.
$stdin.reopen(old_stdin)
old_stdin.close
If your ruby script is creating the tasks, it can use IO::popen. For example, cat, when run with no arguments, will wait for EOF on stdin before it exits, but you can run the following:
f = IO::popen('cat', 'w')
f.puts('hello')
# signals EOF to "cat"
f.close