Working on a web service that sometimes needs to return large files, and want it to send something to the client quickly so the client doesn't time out waiting for the start of the data. stream seemed perfect for this, but I ran into a problem.
Dumb example:
get '/path' do
status 200
headers 'Content-Type' => 'text/plain'
stream do |out|
sleep 1
out << "Hello,\n"
sleep 1
out << "World!\n"
end
end
This works fine:
$ curl http://localhost:4567/path
Hello,
World!
But I have a side log that the service writes to, and trying to mix File I/O with the streaming API doesn't work at all:
get '/path' do
status 200
headers 'Content-Type' => 'text/plain'
File.open '/tmp/side-log', 'a' do |lf|
stream do |out|
lf.puts "Woo!"
sleep 1
out << "Hello,\n"
sleep 1
out << "World!\n"
end
end
end
Now I get this:
$ curl http://localhost:4567/path
curl: (18) transfer closed with outstanding read data remaining
Puma doesn't indicate any problems on the server side, but Thin exits entirely:
hello2.rb:13:in `write': closed stream (IOError)
from hello2.rb:13:in `puts'
from hello2.rb:13:in `block (3 levels) in <main>'
from vendor/bundle/gems/sinatra-1.4.6/lib/sinatra/base.rb:437:in `block (2 levels) in stream'
from vendor/bundle/gems/sinatra-1.4.6/lib/sinatra/base.rb:628:in `with_params'
from vendor/bundle/gems/sinatra-1.4.6/lib/sinatra/base.rb:437:in `block in stream'
from vendor/bundle/gems/sinatra-1.4.6/lib/sinatra/base.rb:403:in `call'
from vendor/bundle/gems/sinatra-1.4.6/lib/sinatra/base.rb:403:in `block in each'
from vendor/bundle/gems/eventmachine-1.0.8/lib/eventmachine.rb:1062:in `call'
from vendor/bundle/gems/eventmachine-1.0.8/lib/eventmachine.rb:1062:in `block in spawn_threadpool'
[1]+ Exit 1 ruby hello2.rb
So what should I do if I want to write something out to someplace other than the output stream from inside the stream block?
Not sure if this is the best solution, but using the asynchronous em-files gem worked for me, even in Puma (which I understand is not EventMachine-based):
require 'em-files'
get '/path' do
status 200
headers 'Content-Type' => 'text/plain'
EM::File.open '/tmp/side-log', 'a' do |lf|
stream do |out|
lf.write "Woo!\n"
sleep 1
out << "Hello,\n"
sleep 1
out << "World!\n"
end
end
end
Related
I'm trying to write a simple test for a ruby script that uses gets and I'm experiencing some weird behavior.
When I run the test with rspec it passes fine, but as soon as I run rspec with any flags, like rspec -f json or rspec -O rand the test will fail and give me No such file or directory # rb_sysopen - -O, where -O is whatever the flag that was run.
Stranger still is that if I run rspec and specify the test file like rspec spec/gets_spec.rb, I get a completely different error:
Failure/Error: expect { require_relative '../gets' }.to output("Hello, Lena!\n").to_stdout
expected block to output "Hello, Lena!\n" to stdout, but output "Hello, Describe \"gets.rb\" do!\n"
Diff:
## -1,2 +1,2 ##
-Hello, Lena!
+Hello, Describe "gets.rb" do!
# ./spec/gets_spec.rb:14:in `block (2 levels) in <top (required)>'
I'm having trouble figuring out the right way to write my test so that this doesn't happen but I'm not sure what part of my code needs to change.
I'm using ruby 2.6.3
I learned about testing input from this question
The script, gets.rb, looks like
puts "Hello, #{gets.chomp.capitalize}!"
My test looks like
describe "gets.rb" do
before do
$stdin = StringIO.new("lena")
end
after do
$stdin = STDIN
end
it "should output 'Hello, name!'" , points: 1 do
allow($stdin).to receive(:gets).and_return("lena")
expect { require_relative "../gets" }.to output("Hello, Lena!\n").to_stdout
end
end
Here's the full failure message:
Failures:
1) gets.rb should output 'Hello, name!'
Failure/Error: expect { require_relative '../gets' }.to output("Hello, Lena!\n").to_stdout
Errno::ENOENT:
No such file or directory # rb_sysopen - -O
#./gets.rb:1:in `gets'
# ./gets.rb:1:in `gets'
# ./gets.rb:1:in `<top (required)>'
# ./spec/gets_spec.rb:14:in `require_relative'
# ./spec/gets_spec.rb:14:in `block (3 levels) in <top (required)>'
# ./spec/gets_spec.rb:14:in `block (2 levels) in <top (required)>'
You can try to use a little different approach:
describe "gets.rb" do
it "should output 'Hello, name!'" do
allow_any_instance_of(Object).to receive(:gets).and_return("lena")
expect { require_relative "../gets.rb" }.to output("Hello, Lena!\n").to_stdout
end
end
allow_any_instance_of I took from https://makandracards.com/makandra/41096-stubbing-terminal-user-input-in-rspec
Reading user input in console applications is usually done using Kernel#gets.
The page suggests to use Object.any_instance.stub(gets: 'user input') but it's deprecated syntax already and I've updated it.
I have is a deadlock, but I am not using any threads in my program. Plus, the error only happens about once every 1000 to 1500 function calls, making it very difficult to pinpoint and correct.
Here is the complete error message when the issue occurs:
/usr/lib/ruby/2.3.0/timeout.rb:95:in `join': No live threads left. Deadlock? (fatal)
from /usr/lib/ruby/2.3.0/timeout.rb:95:in `ensure in block in timeout'
from /usr/lib/ruby/2.3.0/timeout.rb:95:in `block in timeout'
from /usr/lib/ruby/2.3.0/timeout.rb:101:in `timeout'
from /usr/lib/ruby/2.3.0/net/http.rb:878:in `connect'
from /usr/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
from /usr/lib/ruby/2.3.0/net/http.rb:852:in `start'
from /usr/lib/ruby/2.3.0/open-uri.rb:319:in `open_http'
from /usr/lib/ruby/2.3.0/open-uri.rb:737:in `buffer_open'
from /usr/lib/ruby/2.3.0/open-uri.rb:212:in `block in open_loop'
from /usr/lib/ruby/2.3.0/open-uri.rb:210:in `catch'
from /usr/lib/ruby/2.3.0/open-uri.rb:210:in `open_loop'
from /usr/lib/ruby/2.3.0/open-uri.rb:151:in `open_uri'
from /usr/lib/ruby/2.3.0/open-uri.rb:717:in `open'
from /usr/lib/ruby/2.3.0/open-uri.rb:35:in `open'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/utils.rb:85:in `get_pic'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_download.rb:87:in `page_link'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_download.rb:116:in `chapter_link'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_download.rb:142:in `chapter'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:57:in `block in MF_manga_missing_chapters'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:45:in `reverse_each'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:45:in `MF_manga_missing_chapters'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:80:in `MF_update'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:5:in `update_manga'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:15:in `block in update_all'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:14:in `each'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:14:in `update_all'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:22:in `update'
from ./MangaScrap.rb:28:in `<main>'
The link to the complete program is https://github.com/Hellfire01/MangaScrap
The issue happens to the three different methods that use open. Here is the one that crashed this time:
# conect to link and download picture
def get_pic(link)
safe_link = link.gsub(/[\[\]]/) { '%%%s' % $&.ord.to_s(16) }
tries ||= 20
begin
page = open(safe_link, "User-Agent" => "Ruby/#{RUBY_VERSION}")
rescue URI::InvalidURIError => error
puts "Warning : bad url"
puts link
puts "message is : " + error.message
return nil
rescue => error
if tries > 0
tries -= 1
sleep(0.2)
retry
else
puts 'could not get picture ' + safe_link + ' after ' + $nb_tries.to_s + ' tries'
puts "message is : " + error.message
return nil
end
end
sleep(0.2)
return page
end
Here is the link to the file: https://github.com/Hellfire01/MangaScrap/blob/master/sources/utils.rb
I would like to know:
How can I fix this error?
If I can not fix this error, are there alternatives to OpenUri that I can use?
You're not catching all exceptions here. When nothing is specified after rescue, it means that you're catching StandardError which is not at the root of Exceptions' hierarchy.
If you want to make sure you're catching all exceptions and retry opening a URL (or whatever behavior you'd like), what you want to do is:
rescue Exception => error
I have is a deadlock, but I am not using any threads in my program. Plus, the error only happens about once every 1000 to 1500 function calls, making it very difficult to pinpoint and correct.
Here is the complete error message when the issue occurs:
/usr/lib/ruby/2.3.0/timeout.rb:95:in `join': No live threads left. Deadlock? (fatal)
from /usr/lib/ruby/2.3.0/timeout.rb:95:in `ensure in block in timeout'
from /usr/lib/ruby/2.3.0/timeout.rb:95:in `block in timeout'
from /usr/lib/ruby/2.3.0/timeout.rb:101:in `timeout'
from /usr/lib/ruby/2.3.0/net/http.rb:878:in `connect'
from /usr/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
from /usr/lib/ruby/2.3.0/net/http.rb:852:in `start'
from /usr/lib/ruby/2.3.0/open-uri.rb:319:in `open_http'
from /usr/lib/ruby/2.3.0/open-uri.rb:737:in `buffer_open'
from /usr/lib/ruby/2.3.0/open-uri.rb:212:in `block in open_loop'
from /usr/lib/ruby/2.3.0/open-uri.rb:210:in `catch'
from /usr/lib/ruby/2.3.0/open-uri.rb:210:in `open_loop'
from /usr/lib/ruby/2.3.0/open-uri.rb:151:in `open_uri'
from /usr/lib/ruby/2.3.0/open-uri.rb:717:in `open'
from /usr/lib/ruby/2.3.0/open-uri.rb:35:in `open'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/utils.rb:85:in `get_pic'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_download.rb:87:in `page_link'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_download.rb:116:in `chapter_link'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_download.rb:142:in `chapter'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:57:in `block in MF_manga_missing_chapters'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:45:in `reverse_each'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:45:in `MF_manga_missing_chapters'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:80:in `MF_update'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:5:in `update_manga'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:15:in `block in update_all'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:14:in `each'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:14:in `update_all'
from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:22:in `update'
from ./MangaScrap.rb:28:in `<main>'
The link to the complete program is https://github.com/Hellfire01/MangaScrap
The issue happens to the three different methods that use open. Here is the one that crashed this time:
# conect to link and download picture
def get_pic(link)
safe_link = link.gsub(/[\[\]]/) { '%%%s' % $&.ord.to_s(16) }
tries ||= 20
begin
page = open(safe_link, "User-Agent" => "Ruby/#{RUBY_VERSION}")
rescue URI::InvalidURIError => error
puts "Warning : bad url"
puts link
puts "message is : " + error.message
return nil
rescue => error
if tries > 0
tries -= 1
sleep(0.2)
retry
else
puts 'could not get picture ' + safe_link + ' after ' + $nb_tries.to_s + ' tries'
puts "message is : " + error.message
return nil
end
end
sleep(0.2)
return page
end
Here is the link to the file: https://github.com/Hellfire01/MangaScrap/blob/master/sources/utils.rb
I would like to know:
How can I fix this error?
If I can not fix this error, are there alternatives to OpenUri that I can use?
You're not catching all exceptions here. When nothing is specified after rescue, it means that you're catching StandardError which is not at the root of Exceptions' hierarchy.
If you want to make sure you're catching all exceptions and retry opening a URL (or whatever behavior you'd like), what you want to do is:
rescue Exception => error
I'm trying to download all of my class notes from coursera. I figured that since I'm learning ruby this would be a good practice exercise, downloading all the PDFs they have for future use. Unfortunately though, I'm getting an exception saying ruby can't connect for some reason. Here is my code:
require 'net/http'
module Coursera
class Downloader
attr_accessor :page_url
attr_accessor :destination_directory
attr_accessor :cookie
def initialize(page_url,dest,cookie)
#page_url=page_url
#destination_directory = dest
#cookie=cookie
end
def download
puts #page_url
request = Net::HTTP::Get.new(#page_url)
puts #cookie.encoding
request['Cookie']=#cookie
# the line below is where the exception is thrown
res = Net::HTTP.start(#page_url.hostname, use_ssl=true,#page_url.port) {|http|
http.request(request)
}
html_page = res.body
pattern = /http[^\"]+\.pdf/
i=0
while (match = pattern.match(html_page,i)) != nil do
# 0 is the entire string.
url_string = match[0]
# make sure that 'i' is updated
i = match.begin(0)+1
# we want just the name of the file.
j = url_string.rindex("/")
filename = url_string[j+1..url_string.length]
destination = #destination_directory+"\\"+filename
# I want to download that resource to that file.
uri = URI(url_string)
res = Net::HTTP.get_response(uri)
# write that body to the file
f=File.new(destination,mode="w")
f.print(res.body)
end
end
end
end
page_url_string = 'https://class.coursera.org/datasci-002/lecture'
puts page_url_string.encoding
dest='C:\\Users\\michael\\training material\\data_science'
page_url=URI(page_url_string)
# I copied this from my browsers developer tools, I'm omitting it since
# it's long and has my session key in it
cookie="..."
downloader = Coursera::Downloader.new(page_url,dest,cookie)
downloader.download
At runtime the following is written to console:
Fast Debugger (ruby-debug-ide 0.4.22, debase 0.0.9) listens on 127.0.0.1:65485
UTF-8
https://class.coursera.org/datasci-002/lecture
UTF-8
Uncaught exception: A socket operation was attempted to an unreachable network. - connect(2)
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `initialize'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `open'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `block in connect'
C:/Ruby200-x64/lib/ruby/2.0.0/timeout.rb:52:in `timeout'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:877:in `connect'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:862:in `do_start'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:851:in `start'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:582:in `start'
C:/Users/michael/Documents/Aptana Studio 3 Workspace/practice/CourseraDownloader.rb:20:in `download'
C:/Users/michael/Documents/Aptana Studio 3 Workspace/practice/CourseraDownloader.rb:52:in `<top (required)>'
C:/Ruby200-x64/bin/rdebug-ide:23:in `load'
C:/Ruby200-x64/bin/rdebug-ide:23:in `<main>'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `initialize': A socket operation was attempted to an unreachable network. - connect(2) (Errno::ENETUNREACH)
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `open'
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `block in connect'
from C:/Ruby200-x64/lib/ruby/2.0.0/timeout.rb:52:in `timeout'
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:877:in `connect'
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:862:in `do_start'
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:851:in `start'
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:582:in `start'
from C:/Users/michael/Documents/Aptana Studio 3 Workspace/practice/CourseraDownloader.rb:20:in `download'
from C:/Users/michael/Documents/Aptana Studio 3 Workspace/practice/CourseraDownloader.rb:52:in `<top (required)>'
from C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ruby-debug-ide-0.4.22/lib/ruby-debug-ide.rb:86:in `debug_load'
from C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ruby-debug-ide-0.4.22/lib/ruby-debug-ide.rb:86:in `debug_program'
from C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ruby-debug-ide-0.4.22/bin/rdebug-ide:110:in `<top (required)>'
from C:/Ruby200-x64/bin/rdebug-ide:23:in `load'
from C:/Ruby200-x64/bin/rdebug-ide:23:in `<main>'
I was following instructions here to write all the HTTP code. As far as I can see I'm following them ver-batim.
I'm using Windows 7, ruby 2.0.0p481, and Aptana Studio 3. When I copy the url into my browser it goes straight to the page without a problem. When I look at the request headers in my browser for that url, I don't see anything else I think I'm missing. I also tried setting the Host and Referer request headers, it made no difference.
I am out of ideas, and have already searched Stack Overflow for similar questions but that didn't help. Please let me know what I'm missing.
So, I had this same error message with a different project and the problem was that my machine literally couldn't connect to the IP / Port. Have you tried connecting with curl? If it works in your browser, it could be using a proxy or something to actually get there. Testing the URL with curl solved the problem for me.
The following code tries to list the entries of a remote directory via SFTP and Net::SFTP, but it causes an "closed stream" IOError if the directory contains a large number of files (~ 6000 files):
require 'net/ssh'
require 'net/sftp'
Net::SFTP.start('hostname', 'username', :password => 'password') do |sftp|
# list the entries in a directory
sftp.dir.foreach("/") do |entry|
puts entry.longname
end
end
What is the best way to avoid it? Versions are net-sftp Gem: 2.0.5 and net-ssh Gem: 2.2.1, Ruby: 1.8.7. The full error message reads:
IOError: closed stream
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/ruby_compat.rb:33:in `select'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/ruby_compat.rb:33:in `io_select'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/ruby_compat.rb:32:in `synchronize'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/ruby_compat.rb:32:in `io_select'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/transport/packet_stream.rb:73:in `available_for_read?'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/transport/packet_stream.rb:85:in `next_packet'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/transport/session.rb:170:in `poll_message'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/transport/session.rb:165:in `loop'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/transport/session.rb:165:in `poll_message'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:451:in `dispatch_incoming_packets'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:213:in `preprocess'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:197:in `process'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:161:in `loop'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:161:in `loop_forever'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:161:in `loop'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-ssh-2.2.1/lib/net/ssh/connection/session.rb:110:in `close'
from ~/.rvm/gems/ruby-1.8.7-p330/gems/net-sftp-2.0.5/lib/net/sftp.rb:36:in `start'
The behavior could be deliberate, if we take a look at the dir source code in net-sftp/lib/net/sftp/operations/dir.rb, we see a close operation:
def foreach(path)
..
ensure
sftp.close!(handle) if handle
end
It is possible that this close operation causes the closed stream error. If it does not indicate a bug, it is possible the catch the IOError exception. It also seems to help to run the SSH event loop occasionally:
begin
..
sftp.dir.foreach("/") do |entry|
puts entry.longname
# ...
sftp.loop # Runs the SSH event loop
end
rescue IOError => Ex
puts "*** We are done: "+Ex.message
end