Are there any differences between the following two uses of popen3?
html = ''
stdin, stdout, stderr = Open3.popen3("curl #{url}")
html << stdout.read
and
html = ''
Open3.popen3("curl #{url}") do |stdin, stdout, stderr, wait_thr|
result << stdout.read
end
I'm wondering if the second syntax causes some thread to block. I'm fairly new to asynchronous code so any insights are greatly appreciated!
In the first form you should explicitly close stdin, stdout and stderr.
The reason you are experiencing a blocking behavior is because you did not close stdin to the program (curl) you opened via popen3 -- so curl is still waiting for your input.
You should close stdin explicitly via stdin.close after you are done sending data to the program, otherwise it will keep expecting input on stdin, and popen3 will hang.
stdin.close # always close your stdin after you are done sending commands/data
# or popen3 will appear to hang
Related
I am trying to take input from git hook execution code(commit-msg hook). But the ruby is not able to stop at the input point. And its executing the code as if input is like a puts statement. Here is the code I tried and failed.
#!/usr/bin/env ruby
require 'open3'
def take_input_here
Open3.popen3("pwd", :chdir=>"/") {|stdin, stdout, stderr, thread|
p stdout.read.chomp #=> "/"
}
input_val = gets.chomp
puts input_val
puts 'Hellow World!'
end
take_input_here
puts "Commit Aborted."
Process.exit(1)
Somebody please help my take this interactive input or else suggest me a good language for writing git hooks. Thanks in advance.
Most Git hooks are run with stdin either coming from a pipe to which Git writes information, or with stdin disconnected from the terminal entirely. The commit-msg hook falls into this second category.
It won't matter which language you use: reading stdin in a commit-msg hook will see EOF immediately, as stdin is connected to /dev/null (Linux/Unix) or NUL: (Windows).
On Unix-like systems, you can try opening /dev/tty. Note that if Git is being run from something that doesn't have a /dev/tty (some detached process, e.g., via cron) or where reading /dev/tty is bad for some other reason, this may cause other issues, so be careful with this.
I'm using:
- Ruby 1.9.3-p448
- Windows Server 2008
I have a file that contains commands that is used by a program, I'm using it in this way
C:\> PATH_TO_FOLDER/program.exe file.txt
File.txt have some commands so program.exe will do the following:
- Execute commands
- Reads from a DB using an ODBC method used by program
- Outputs result in a txt file
Using powershell this command works fine and as expected.
Now I have this in a file (app.rb):
require 'sinatra'
require 'open3'
get '/process' do
program_path = "path to program.exe"
file_name = "file.txt"
Open3.popen3(program_path, file_name) do |i, o, e, w|
# I have some commands here to execute but just as an example I'm using o.read
puts o.read
end
end
Now when using this by accessing http://localhost/process, Open3 works by doing this (I'm not 100% sure but after trying several times I think is the only option)
Reads commands and executes them (this is ok)
Tries to read from DB by using ODBC method (Here is my problem. I
need to receive some output from Open3 so I can show it in a browser, but I guess when it tries to read it starts another process that Open3 is not aware of, so Open3 goes on and finish without waiting for it)
Exits
Exits
I've found about following:
Use Thread.join (in this case, w.join) in order to wait for process to finish, but it doesn't work
Open4 seems to handle child process but doesn't work on Windows
Process.wait(pid), in this case pid = w.pid, but also doesn't work
Timeout.timeout(n), the problem here is that I'm not sure how long
will it take.
Is there any way of handling this? (waiting for Open3 subprocess so I get proper output).
We had a similar problem getting the exit status and this is what we did
Open3.popen3(*cmd) do |stdin, stdout, stderr, wait_thr|
# print stdout and stderr as it comes in
threads = [stdout, stderr].collect do |output|
Thread.new do
while ((line = output.gets rescue '') != nil) do
unless line.blank?
puts line
end
end
end
end
# get exit code as a Process::Status object
process_status = wait_thr.value #.exitstatus
# wait for logging threads to finish before continuing
# so we don't lose any logging output
threads.each(&:join)
# wait up to 5 minutes to make sure the process has really exited
Timeout::timeout(300) do
while !process_status.exited?
sleep(1)
end
end rescue nil
process_status.exitstatus.to_i
end
Using Open3.popen3 is easy only for trivial cases. I do not know the real code for handling the input, output and error channels of your subprocess. Neither do I know the exact behaviour of your subprocesses: Does it write on stdout? Does it write on stderr? Does it try to read from stdin?
This is why I assume that there are problems in the code that you replaced by puts o.read.
A good summary about the problems you can run into is on http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/63.
Though I disagree with the author of the article, Pavel Shved, when it comes to finding a solution. He recommends his own solution. I just use one of the wrapper functions for popen3 in my projects: Open3.capture*. They do all the difficult things like waiting for stdout and stderr at the same time.
I have a little ruby script which does a mysql import in the way: mysql -u <user> -p<pass> -h <host> <db> < file.sql, but utilizes Open3.popen3 to do so. That is what I have so far:
mysqlimp = "mysql -u #{mysqllocal['user']} "
mysqlimp << "-h #{mysqllocal['host']} "
mysqlimp << "-p#{mysqllocal['pass']} "
mysqlimp << "#{mysqllocal['db']}"
Open3.popen3(mysqlimp) do |stdin, stdout, stderr, wthr|
stdin.write "DROP DATABASE IF EXISTS #{mysqllocal['db']};\n"
stdin.write "CREATE DATABASE #{mysqllocal['db']};\n"
stdin.write "USE #{mysqllocal['db']};\n"
stdin.write mysqldump #a string containing the database data
stdin.close
stdout.each_line { |line| puts line }
stdout.close
stderr.each_line { |line| puts line }
stderr.close
end
That is actually doing the Job, but there is one thing that bothers me, concerned to the output I would like to see.
If I change the first line to:
mysqlimp = "mysql -v -u #{mysqllocal['user']} " #note the -v
then the whole script hangs forever.
I guess, that happens because the read- and write-stream block each other and I also guess that the stdout needs to be flushed regularly so that stdin will go on to be consumed. In other words, as long as the buffer of the stdout is full, the process will wait until its flushed, but since this is done at the very bottom first, that never happens.
I hope someone can verify my theory? How could I write code that does prints out everything from the stdout and writes everything to the stdin as well?
Thanks in ahead!
Since you are only writing to stdout, you can simply use Open3#popen2e which consolidates stdout and stderr into a single stream.
To write newline terminated strings to a stream, you can use puts as you would with $stdout in a simple hello world program.
You must use waith_thread.join or wait_thread.value to wait until the child process terminates.
In any case, you will have to start a separate thread for reading from the stream, if you want to see the results immediately.
Example:
require 'open3'
cmd = 'sh'
Open3.popen2e(cmd) do |stdin, stdout_stderr, wait_thread|
Thread.new do
stdout_stderr.each {|l| puts l }
end
stdin.puts 'ls'
stdin.close
wait_thread.value
end
Your code, fixed:
require 'open3'
mysqldump = # ...
mysqlimp = "mysql -u #{mysqllocal['user']} "
mysqlimp << "-h #{mysqllocal['host']} "
mysqlimp << "-p#{mysqllocal['pass']} "
mysqlimp << "#{mysqllocal['db']}"
Open3.popen2e(mysqlimp) do |stdin, stdout_stderr, wait_thread|
Thread.new do
stdout_stderr.each {|l| puts l }
end
stdin.puts "DROP DATABASE IF EXISTS #{mysqllocal['db']};"
stdin.puts "CREATE DATABASE #{mysqllocal['db']};"
stdin.puts "USE #{mysqllocal['db']};"
stdin.close
wait_thread.value
end
Whenever you start a process from the command line or via fork, the process inherits stdin, stdout and stderr from the father process. This means, if your command line runs in a terminal, stdin, stdout and stderr of the new process are connected to the terminal.
Open3.popen3, on the other hand, does not connect stdin, stdout and stderr to the terminal, because you do not want direct user interaction. So we need something else.
For stdin, we need something with two abilities:
The father process needs something to enqueue data that the subprocess is supposed to get when it reads from stdin.
The subprocess needs something that offers a read function like stdin does.
For stdout and stderr, we need something similar:
The subprocess needs something to write to. puts and print should enqueue the data, that the father process is supposed to read.
The father process needs something that offers a read function in order to get the stdout and stderr data of the subprocess.
This means, for stdin, stdout and stderr, we need three queues (FIFO) for communication between father process and subprocess. These queues have to act a little bit like files as they have to provide read, write (for puts and print), close and select (is data available?).
Therefore, both Linux and Windows provide anonymous pipes. This is one of the conventional (local) interprocess communication mechanisms. And, well, Open3.popen3 really wants to do communication between two different processes. This is why Open3.popen3 connects stdin, stdout and stderr to anonymous pipes.
Each pipe, be it anonymous or named, does have a buffer of limited size. This size depends on operation system. The catch is: If the buffer is full and a processes tries to write to the pipe, the operating system suspends the process until another processes reads from the pipe.
This may be your problem:
You keep feeding data to your subprocess, but you do not read what your subprocess writes to stdout.
Consequently, the output of our subprocess keeps accumulating in a buffer until the buffer is full.
This is when the operation system suspends your subprocess (puts or print blocks).
Now you can still feed data to the anonymous pipe that is connected to the stdin of your subprocesses until too much of stdin data has accumulated. The buffer of the stdin pipe got full. Then the operating system will suspend the father processes (stdin.write will block).
I advise you to use Open3.capture2e or a similar wrapper around Open3.popen3. You can pass data to the subprocess with the keyword argument :stdin_data.
If you insist on communicating with your subprocess "interactively", you need to learn about IO.select or using multi-threading. Both of them are quite a challenge. Better use Open3.capture*.
I am using lftp to download some files. Below is the code I'm using and I'd like to capture the stdout. It doesn't show anything though.
If I just run the command in bash I get streaming progress to stdout. I know its stdout because if I redirect 1 > /dev/null then the output stops.
Can anyone help please tell me if it's possible to capture stdout of this program?
Open3.popen3("lftp -vvv -c 'pget -n 10 ftp://mirror.us.leaseweb.net/debian-cd/7.4.0-live/i386/iso-hybrid/debian-live-7.4-i386-xfce-desktop.iso'") do |stdin, stdout, stderr|
stdout.sync = true
stdout.each { |line| puts line }
stderr.each { |line| puts line }
end
UPDATE: looking through the lftp code it uses a '\r' carriage return to have the stdout rewind to start of same line and update rather then take a new line for each. Maybe open3 can't cope well with this.
I am trying to run a bash script (#command) that requires user input and I'm trying to feed that script input using the following code:
Open3.popen3(#command) do |stdin, stdout, stderr|
stdin.write("y")
stdout.gets
end
Here is an idea of the script:
exec sudo su -c "some command" $username
If anyone could tell me what I am doing wrong or has any suggestions on how to implement this a different way, that would be much appreciated.
Also, I can run the script like this:
#output = `#{#command}`
In this case, I can see the contents of the script output in the console I am running my app from. If there is anyway to feed input to that location that would work too.
Got my solution here:
How to fix hanging popen3 in Ruby?
Open3.popen3(#command) do |stdin, stdout, stderr|
stdin.puts "y\r\n"
stdout.each_line { |line| puts line }
stdin.close
end
out_err, status = Open3.capture2e(#command, :stdin_data => "y\r\n")
print out_err