I would like to use Sinatra's Streaming capability introduced in 1.3 coupled with some stdout redirection. It would basically be a live streaming output of a long running job. I looked into this question and the Sinatra streaming sample in the README.
Running 1.8.7 on OSX:
require 'stringio'
require 'sinatra'
$stdout.sync = true
module Kernel
def capture_stdout
out = StringIO.new
$stdout = out
yield out
ensure
$stdout = STDOUT
end
end
get '/' do
stream do |out|
out << "Part one of a three part series... <br>\n"
sleep 1
out << "...part two... <br>\n"
sleep 1
out << "...and now the conclusion...\n"
Kernel.capture_stdout do |stream|
Thread.new do
until (line = stream.gets).nil? do
out << line
end
end
method_that_prints_text
end
end
end
def method_that_prints_text
puts "starting long running job..."
sleep 3
puts "almost there..."
sleep 3
puts "work complete!"
end
So this bit of code prints out the first three strings properly, and blocks while the method_that_prints_text executes and does not print anything to the browser. My feeling is that stdout is empty on the first call and it never outputs to the out buffer. I'm not quite sure what the proper ordering would be and would appreciate any suggestions.
I tried a few of the EventMachine implementations mentioned in the question above, but couldn't get them to work.
UPDATE
I tried something slightly different to where I had the method run in a new thread, and override STDOUT for that thread as described here...
Instead of Kernel.capture_stdout above...
s = StringIO.new
Thread.start do
Thread.current[:stdout] = s
method_that_prints_text
end.join
while line = s.gets do
out << line
end
out << s.string
With the ThreadOut module listed in the link above, this seems to work a bit better. However it doesn't stream. The only time something is printed to the browser is on the final line out << s.string. Does StringIO not have the capability to stream?
I ended up solving this by discovering that s.string was updated periodically as time went on, so I just captured the output in a separate thread and grabbed the differences and streamed them out. It appears as though string redirection doesn't behave like a normal IO object.
s = StringIO.new
t = Thread.start do
Thread.current[:stdout] = s
method_that_prints_text
sleep 2
end
displayed_text = ''
while t.alive? do
current_text = s.string
unless (current_text.eql?(displayed_text))
new_text = current_text[displayed_text.length..current_text.length]
out << new_text
displayed_text = current_text * 1
end
sleep 2
end
Related
I was trying to make a script that downloads all images or videos from a thread in my favourite imageboard: 2ch.hk
I was successful until I wanted to download these files asynchronously (for example, to improve performance)
Here is the code http://ideone.com/k2l4Hm
file = http.get(source).body
require 'net/http'
multithreading = false
Net::HTTP.start("2ch.hk", :use_ssl => true) do |http|
thread = http.get("/b/res/133467978.html").body
sources = []
thread.scan(/<a class="desktop" target="_blank" href=".+">.+<\/a>/).each do |a|
source = "/b#{/<a class="desktop" target="_blank" href="\.\.(.+)">.+<\/a>/.match(a).to_a[1]}"
sources << source
end
i = 0
start = Time.now
if multithreading
threads = []
sources.each do |source|
threads << Thread.new(i) do |j|
file = http.get(source).body #breaks everything
# type = /.+\.(.+)/.match(source)[1]
# open("#{j}.#{type}","wb") { |new_file|
# new_file.write(file)
# }
end
i += 1
end
threads.each do |thr|
thr.join
end
# until downloade=sources.size
#
# end
else
sources.each do |source|
file = http.get(source).body
type = /.+\.(.+)/.match(source)[1]
open("#{i}.#{type}","wb") { |new_file|
new_file.write(file)
}
i += 1
print "#{(((i).to_f / sources.size) * 100).round(2)}% "
end
puts
end
puts "Done. #{i} files were downloaded. It took #{Time.now - start} seconds"
end
I suppose that this line crashes everything.
file = http.get(source).body
Or maybe that's the problem.
threads.each do |thr|
thr.join
end
Error messages are always different, from Bad File Descriptor and IO errors to "You may have encountered a bug in the Ruby interpreter or extension libraries."
If you want to try and run my code, please substitute a link to thread in 4th line with a new thread (from 2ch.hk/b), because the one in my code may be deleted by the time you run my code
Version of ruby: 2.3.1, OS Xubuntu 16.10
You'll probably have much better performance using a ruby http lib that supports parallel requests:
https://github.com/typhoeus/typhoeus
e.g.
hydra = Typhoeus::Hydra.new
10.times.map{ hydra.queue(Typhoeus::Request.new("www.example.com", followlocation: true)) }
hydra.run
The problem with my code is that I can't make multiple requests on a Net::HTTP instance at the same time.
The solution is to open an HTTP connection for each thread.
I have a ruby console app that you run with an argument, then once running outputs some text to the screen, asks for some more user input and then outputs some more text to the screen. I want to do an end to end test on this app and I don't know how. If I were writing an end to end test for an REST API, I would just hit the public endpoint, follow the links and then have an expect statement on the output. Easy. But on a console app I have no idea how to do the same thing. Are there any gems for stepping through a console app in the context of a test? I've been looking all day but can't find anything.
ANY help appreciated.
Inspired by this gem which has a fairly simple implementation, I wrote a method which captures console input & output and can, therefore, be used in tests:
require 'stringio'
module Kernel
def emulate_console(console_input)
$stdin = StringIO.new(console_input)
out = StringIO.new
$stdout = out
yield
return out
ensure
$stdout = STDOUT
$stdin = STDIN
end
end
This method captures console output, and also provides as input the string value which you specify in the console_input parameter.
Basic usage
Here's a simple usage of the emulate_console method:
out = emulate_console("abc\n") do
input = gets.chomp
puts "You entered: #{input}!"
end
The return value out is a StringIO object. To access its value, use the #string method:
out.string
=> "You entered: abc!\n"
Note that the input contains a newline character (\n) to simulate pressing the ENTER key.
Testing
Now, let's assume that you want to test this method, that uses both stdin and stdout:
def console_add_numbers
x = Integer(gets)
y = Integer(gets)
puts x + y
end
The following RSpec test tests the happy path of this code:
require 'rspec/autorun'
RSpec.describe '#console_add_numbers' do
it 'computes correct result' do
input = <<-EOS
2
3
EOS
output = emulate_console(input) { console_add_numbers }
expect(output.string.chomp).to eql '5'
end
end
puts "hi"
puts "bye"
I want to store the STDOUT of the code so far (in this case hi \nbye into a variable say 'result' and print it )
puts result
The reason I am doing this is I have integrate an R code into my Ruby code, output of which is given to the STDOUT as the R code runs , but the ouput cannot be accessed inside the code to do some evaluations. Sorry if this is confusing. So the "puts result" line should give me hi and bye.
A handy function for capturing stdout into a string...
The following method is a handy general purpose tool to capture stdout and return it as a string. (I use this frequently in unit tests where I want to verify something printed to stdout.) Note especially the use of the ensure clause to restore $stdout (and avoid astonishment):
def with_captured_stdout
original_stdout = $stdout # capture previous value of $stdout
$stdout = StringIO.new # assign a string buffer to $stdout
yield # perform the body of the user code
$stdout.string # return the contents of the string buffer
ensure
$stdout = original_stdout # restore $stdout to its previous value
end
So, for example:
>> str = with_captured_stdout { puts "hi"; puts "bye"}
=> "hi\nbye\n"
>> print str
hi
bye
=> nil
Redirect Standard Output to a StringIO Object
You can certainly redirect standard output to a variable. For example:
# Set up standard output as a StringIO object.
foo = StringIO.new
$stdout = foo
# Send some text to $stdout.
puts 'hi'
puts 'bye'
# Access the data written to standard output.
$stdout.string
# => "hi\nbye\n"
# Send your captured output to the original output stream.
STDOUT.puts $stdout.string
In practice, this is probably not a great idea, but at least now you know it's possible.
You can do this by making a call to your R script inside backticks, like this:
result = `./run-your-script`
puts result # will contain STDOUT from run-your-script
For more information on running subprocesses in Ruby, check out this Stack Overflow question.
If activesupport is available in your project you may do the following:
output = capture(:stdout) do
run_arbitrary_code
end
More info about Kernel.capture can be found here
For most practical purposes you can put anything into $stdout that responds to write, flush, sync, sync= and tty?.
In this example I use a modified Queue from the stdlib.
class Captor < Queue
alias_method :write, :push
def method_missing(meth, *args)
false
end
def respond_to_missing?(*args)
true
end
end
stream = Captor.new
orig_stdout = $stdout
$stdout = stream
puts_thread = Thread.new do
loop do
puts Time.now
sleep 0.5
end
end
5.times do
STDOUT.print ">> #{stream.shift}"
end
puts_thread.kill
$stdout = orig_stdout
You need something like this if you want to actively act on the data and not just look at it after the task has finished. Using StringIO or a file will have be problematic with multiple threads trying to sync reads and writes simultaneously.
Capture stdout (or stderr) for both Ruby code and subprocesses
# capture_stream(stream) { block } -> String
#
# Captures output on +stream+ for both Ruby code and subprocesses
#
# === Example
#
# capture_stream($stdout) { puts 1; system("echo 2") }
#
# produces
#
# "1\n2\n"
#
def capture_stream(stream)
raise ArgumentError, 'missing block' unless block_given?
orig_stream = stream.dup
IO.pipe do |r, w|
# system call dup2() replaces the file descriptor
stream.reopen(w)
# there must be only one write end of the pipe;
# otherwise the read end does not get an EOF
# by the final `reopen`
w.close
t = Thread.new { r.read }
begin
yield
ensure
stream.reopen orig_stream # restore file descriptor
end
t.value # join and get the result of the thread
end
end
I got inspiration from Zhon.
Minitest versions:
assert_output if you need to ensure if some output is generated:
assert_output "Registrars processed: 1\n" do
puts 'Registrars processed: 1'
end
assert_output
or use capture_io if you really need to capture it:
out, err = capture_io do
puts "Some info"
warn "You did a bad thing"
end
assert_match %r%info%, out
assert_match %r%bad%, err
capture_io
Minitest itself is available in any Ruby version starting from 1.9.3
For RinRuby, please know that R has capture.output:
R.eval <<EOF
captured <- capture.output( ... )
EOF
puts R.captured
Credit to #girasquid's answer. I modified it to a single file version:
def capture_output(string)
`echo #{string.inspect}`.chomp
end
# example usage
response_body = "https:\\x2F\\x2Faccounts.google.com\\x2Faccounts"
puts response_body #=> https:\x2F\x2Faccounts.google.com\x2Faccounts
capture_output(response_body) #=> https://accounts.google.com/accounts
I'm creating a script to wrap jdb (java debugger). I essentially want to wrap this process and proxy the user interaction. So I want it to:
start jdb from my script
send the output of jdb to stdout
pause and wait for input when jdb does
when the user enters commands, pass it to jdb
At the moment I really want a pass thru to jdb. The reason for this is to initialize the process with specific parameters and potentially add more commands in the future.
Update:
Here's the shell of what ended up working for me using expect:
PTY.spawn("jdb -attach 1234") do |read,write,pid|
write.sync = true
while (true) do
read.expect(/\r\r\n> /) do |s|
s = s[0].split(/\r\r\n/)
s.pop # get rid of prompt
s.each { |line| puts line }
print '> '
STDOUT.flush
write.print(STDIN.gets)
end
end
end
Use Open3.popen3(). e.g.:
Open3.popen3("jdb args") { |stdin, stdout, stderr|
# stdin = jdb's input stream
# stdout = jdb's output stream
# stderr = jdb's stderr stream
threads = []
threads << Thread.new(stderr) do |terr|
while (line = terr.gets)
puts "stderr: #{line}"
end
end
threads << Thread.new(stdout) do |terr|
while (line = terr.gets)
puts "stdout: #{line}"
end
end
stdin.puts "blah"
threads.each{|t| t.join()} #in order to cleanup when you're done.
}
I've given you examples for threads, but you of course want to be responsive to what jdb is doing. The above is merely a skeleton for how you open the process and handle communication with it.
The Ruby standard library includes expect, which is designed for just this type of problem. See the documentation for more information.
I've managed to make it play sound but it never gets EOS message.
And thus script never exits.
require 'gst'
main_loop = GLib::MainLoop.new
pipeline = Gst::Pipeline.new "audio-player"
source = Gst::ElementFactory.make "filesrc", "file-source"
source.location = "/usr/share/sounds/gnome/default/alerts/bark.ogg"
decoder = Gst::ElementFactory.make "decodebin", "decoder"
conv = Gst::ElementFactory.make "audioconvert", "converter"
sink = Gst::ElementFactory.make "alsasink", "output"
pipeline.add source, decoder, conv, sink
source >> decoder
conv >> sink
decoder.signal_connect "pad-added" do |element, pad, data|
pad >> conv['sink']
end
pipeline.bus.add_watch do |bus, message|
puts "Message: #{message.inspect}"
case message.type
when Gst::Message::Type::ERROR
puts message.structure["debug"]
main_loop.quit
when Gst::Message::Type::EOS
puts 'End of stream'
main_loop.quit
end
end
pipeline.play
begin
puts 'Running main loop'
main_loop.run
ensure
puts 'Shutting down main loop'
pipeline.stop
end
I've found a solution. Sharing with everybody who may run into same problem.
The real problem was that block didn't return true. It has to return true to be kept in watch. This is documented somewhere deep in GStreamer docs but is not mentioned in ruby-gnome2 docs.
Gst::Message::Type::EOS
should be:
Gst::Message::EOS
same goes for ERROR