I am using lftp to download some files. Below is the code I'm using and I'd like to capture the stdout. It doesn't show anything though.
If I just run the command in bash I get streaming progress to stdout. I know its stdout because if I redirect 1 > /dev/null then the output stops.
Can anyone help please tell me if it's possible to capture stdout of this program?
Open3.popen3("lftp -vvv -c 'pget -n 10 ftp://mirror.us.leaseweb.net/debian-cd/7.4.0-live/i386/iso-hybrid/debian-live-7.4-i386-xfce-desktop.iso'") do |stdin, stdout, stderr|
stdout.sync = true
stdout.each { |line| puts line }
stderr.each { |line| puts line }
end
UPDATE: looking through the lftp code it uses a '\r' carriage return to have the stdout rewind to start of same line and update rather then take a new line for each. Maybe open3 can't cope well with this.
Related
I have a little ruby script which does a mysql import in the way: mysql -u <user> -p<pass> -h <host> <db> < file.sql, but utilizes Open3.popen3 to do so. That is what I have so far:
mysqlimp = "mysql -u #{mysqllocal['user']} "
mysqlimp << "-h #{mysqllocal['host']} "
mysqlimp << "-p#{mysqllocal['pass']} "
mysqlimp << "#{mysqllocal['db']}"
Open3.popen3(mysqlimp) do |stdin, stdout, stderr, wthr|
stdin.write "DROP DATABASE IF EXISTS #{mysqllocal['db']};\n"
stdin.write "CREATE DATABASE #{mysqllocal['db']};\n"
stdin.write "USE #{mysqllocal['db']};\n"
stdin.write mysqldump #a string containing the database data
stdin.close
stdout.each_line { |line| puts line }
stdout.close
stderr.each_line { |line| puts line }
stderr.close
end
That is actually doing the Job, but there is one thing that bothers me, concerned to the output I would like to see.
If I change the first line to:
mysqlimp = "mysql -v -u #{mysqllocal['user']} " #note the -v
then the whole script hangs forever.
I guess, that happens because the read- and write-stream block each other and I also guess that the stdout needs to be flushed regularly so that stdin will go on to be consumed. In other words, as long as the buffer of the stdout is full, the process will wait until its flushed, but since this is done at the very bottom first, that never happens.
I hope someone can verify my theory? How could I write code that does prints out everything from the stdout and writes everything to the stdin as well?
Thanks in ahead!
Since you are only writing to stdout, you can simply use Open3#popen2e which consolidates stdout and stderr into a single stream.
To write newline terminated strings to a stream, you can use puts as you would with $stdout in a simple hello world program.
You must use waith_thread.join or wait_thread.value to wait until the child process terminates.
In any case, you will have to start a separate thread for reading from the stream, if you want to see the results immediately.
Example:
require 'open3'
cmd = 'sh'
Open3.popen2e(cmd) do |stdin, stdout_stderr, wait_thread|
Thread.new do
stdout_stderr.each {|l| puts l }
end
stdin.puts 'ls'
stdin.close
wait_thread.value
end
Your code, fixed:
require 'open3'
mysqldump = # ...
mysqlimp = "mysql -u #{mysqllocal['user']} "
mysqlimp << "-h #{mysqllocal['host']} "
mysqlimp << "-p#{mysqllocal['pass']} "
mysqlimp << "#{mysqllocal['db']}"
Open3.popen2e(mysqlimp) do |stdin, stdout_stderr, wait_thread|
Thread.new do
stdout_stderr.each {|l| puts l }
end
stdin.puts "DROP DATABASE IF EXISTS #{mysqllocal['db']};"
stdin.puts "CREATE DATABASE #{mysqllocal['db']};"
stdin.puts "USE #{mysqllocal['db']};"
stdin.close
wait_thread.value
end
Whenever you start a process from the command line or via fork, the process inherits stdin, stdout and stderr from the father process. This means, if your command line runs in a terminal, stdin, stdout and stderr of the new process are connected to the terminal.
Open3.popen3, on the other hand, does not connect stdin, stdout and stderr to the terminal, because you do not want direct user interaction. So we need something else.
For stdin, we need something with two abilities:
The father process needs something to enqueue data that the subprocess is supposed to get when it reads from stdin.
The subprocess needs something that offers a read function like stdin does.
For stdout and stderr, we need something similar:
The subprocess needs something to write to. puts and print should enqueue the data, that the father process is supposed to read.
The father process needs something that offers a read function in order to get the stdout and stderr data of the subprocess.
This means, for stdin, stdout and stderr, we need three queues (FIFO) for communication between father process and subprocess. These queues have to act a little bit like files as they have to provide read, write (for puts and print), close and select (is data available?).
Therefore, both Linux and Windows provide anonymous pipes. This is one of the conventional (local) interprocess communication mechanisms. And, well, Open3.popen3 really wants to do communication between two different processes. This is why Open3.popen3 connects stdin, stdout and stderr to anonymous pipes.
Each pipe, be it anonymous or named, does have a buffer of limited size. This size depends on operation system. The catch is: If the buffer is full and a processes tries to write to the pipe, the operating system suspends the process until another processes reads from the pipe.
This may be your problem:
You keep feeding data to your subprocess, but you do not read what your subprocess writes to stdout.
Consequently, the output of our subprocess keeps accumulating in a buffer until the buffer is full.
This is when the operation system suspends your subprocess (puts or print blocks).
Now you can still feed data to the anonymous pipe that is connected to the stdin of your subprocesses until too much of stdin data has accumulated. The buffer of the stdin pipe got full. Then the operating system will suspend the father processes (stdin.write will block).
I advise you to use Open3.capture2e or a similar wrapper around Open3.popen3. You can pass data to the subprocess with the keyword argument :stdin_data.
If you insist on communicating with your subprocess "interactively", you need to learn about IO.select or using multi-threading. Both of them are quite a challenge. Better use Open3.capture*.
I am trying to run a bash script (#command) that requires user input and I'm trying to feed that script input using the following code:
Open3.popen3(#command) do |stdin, stdout, stderr|
stdin.write("y")
stdout.gets
end
Here is an idea of the script:
exec sudo su -c "some command" $username
If anyone could tell me what I am doing wrong or has any suggestions on how to implement this a different way, that would be much appreciated.
Also, I can run the script like this:
#output = `#{#command}`
In this case, I can see the contents of the script output in the console I am running my app from. If there is anyway to feed input to that location that would work too.
Got my solution here:
How to fix hanging popen3 in Ruby?
Open3.popen3(#command) do |stdin, stdout, stderr|
stdin.puts "y\r\n"
stdout.each_line { |line| puts line }
stdin.close
end
out_err, status = Open3.capture2e(#command, :stdin_data => "y\r\n")
print out_err
Here comes a sample ruby code:
r = gets
puts r
if the script is executed standalone from console, it work fine. But if i ran it via pipeline:
echo 'testtest' | ruby test.rb
gets seem is redirected to pipeline inputs, but i need some user input.
How?
Stdin has been attached to the receiving end of the pipe by the invoking shell. If you really need interactive input you have a couple choices. You can open the tty input directly, leavng stdin bound to the pipe:
tty_input = open('/dev/tty') {|f| f.gets }
/dev/tty works under linux and OS/x, but might not work everywhere.
Alternatively, you can use a different form of redirection, process substitution, under bash to supply the (formerly piped) input as a psuedo-file passed as an argument and leave stdin bound to your terminal:
ruby test.rb <(echo 'testtest')
# test.rb
input = open(ARGV[0])
std_input = gets
input.readlines { |line| process_line(line) }
Are there any differences between the following two uses of popen3?
html = ''
stdin, stdout, stderr = Open3.popen3("curl #{url}")
html << stdout.read
and
html = ''
Open3.popen3("curl #{url}") do |stdin, stdout, stderr, wait_thr|
result << stdout.read
end
I'm wondering if the second syntax causes some thread to block. I'm fairly new to asynchronous code so any insights are greatly appreciated!
In the first form you should explicitly close stdin, stdout and stderr.
The reason you are experiencing a blocking behavior is because you did not close stdin to the program (curl) you opened via popen3 -- so curl is still waiting for your input.
You should close stdin explicitly via stdin.close after you are done sending data to the program, otherwise it will keep expecting input on stdin, and popen3 will hang.
stdin.close # always close your stdin after you are done sending commands/data
# or popen3 will appear to hang
I have a problem.
I want to run a ruby script from another ruby script and capture it's output information while letting it output to the screen too.
runner
#!/usr/bin/env ruby
print "Enter your password: "
password = gets.chomp
puts "Here is your password: #{password}"
The script file that I run:
start.rb
output = `runner`
puts output.match(/Here is your (password: .*)/).captures[0].to_s
As you see here there is a problem.
In the first line of start.rb the screen is empty.
I cannot see the "Enter your password: " in runner.
Is there a way to display the output of the runner script before it's finished, and still let me capture it to a string so I can process the information, eg. using match like in this example?
runner.rb
STDOUT.print "Enter your password: "
password = gets.chomp
puts "Here is your password: #{password}"
Note STDOUT.print
start.rb
require "stringio"
buffer = StringIO.new
$stdout = buffer
require "runner"
$stdout = STDOUT
buffer.rewind
puts buffer.read.match(/Here is your (password: .*)/).captures[0].to_s
output
Enter your password: hello
password: hello
Read more...
I recently did a write-up on this here: Output Buffering with Ruby
Try this:
rd, wr = IO::pipe
pid = Process.fork do
$stdout.reopen(wr)
rd.close
exec("command")
end
wr.close
rd.each do |line|
puts "line from command: #{line}"
end
Process.wait(pid)
Similar if you want to capture stderr. If you need to capture both it would a bit more difficult (Kernel.select?)
Edit: Some explanation. This is an ancient Unix procedure: pipe + fork + calls to dup2 (reopen) depending on what you want. In a nutshell: you create a pipe as a means of communication between child and parent. After the fork, each peer close the pipe's endpoint it does not use, the child remaps (reopen) the channel you need to the write endpoint of the pipe and finally the parent reads on the read channel of the pipe.
For script independent output logging you might want to enable it from the terminal emulator (shell container).
screen -L
OR
xterm -l
This will capture all output produced by any shell or program running inside the emulator, including output generated by your ruby scripts.
You could use tee to write the contents to a file or a pipe, and read the file afterwards.
Have a look at POpen4.
It claims to be platform independent (but I do not think it works in jruby where you can use IO#popen instead).
Have your script do its prompt output to stderr.
echo "Enter something" >&2
read answer
echo "output that will be captured"
This will be done for you if you use read -p to issue the prompt:
read -p "Enter something" answer
echo "output that will be captured"
io = IO.popen(<your command here>)
log = io.readlines
io.close
Now in log variable you have the output of executed command. Parse it, convert it, or do whatever you want.