I try this code, but when the proxy is too slow I get connection timed out error. How can I solve this? I tried Exception handling but doesn't work. Can anybody help?
Net::HTTP.new('example.com', nil, '140.113.182.81', '808').start { |http|
begin
response = http.request
p response
rescue Timeout::Error
p 'timed out'
end
}
The Timeout::Error is raised by the Net::HTTP.connect method that is executed by start, not by request.
It means that in order to rescue the timeout, the whole Net::HTTP call should be inside the begin block.
begin
Net::HTTP.new('example.com', nil, '140.113.182.81', '808').start do |http|
response = http.request
p response
end
rescue Timeout::Error
p 'timed out'
end
Related
Using curb gem (https://github.com/taf2/curb) to GET from a REST API.
resp = Curl.get("http://someurl.com/users.json") do |http|
http.headers["API-Key"] = ENV["API_KEY"]
end
# do stuff with resp.body_str
I've started encountering occasional time-outs with the Curl.get.
Would like to add logic where I try to GET: if the request times out, we try it again, i.e.
loop do
resp = Curl.get("http://someurl.com/users.json") do |http|
http.headers["API-Key"] = ENV["API_KEY"]
end
# test result of Curl.get
# if time-out, then then try again
end
Haven't been able to find/figure out how to test for a time-out result.
What am I missing?
UPDATED: added exception details
Curl::Err::TimeoutError: Timeout was reached
/app/vendor/bundle/ruby/2.3.0/gems/curb-0.9.3/lib/curl/easy.rb:73:in `perform'
/app/vendor/bundle/ruby/2.3.0/gems/curb-0.9.3/lib/curl.rb:17:in `http'
/app/vendor/bundle/ruby/2.3.0/gems/curb-0.9.3/lib/curl.rb:17:in `http'
/app/vendor/bundle/ruby/2.3.0/gems/curb-0.9.3/lib/curl.rb:22:in `get'
/app/lib/tasks/redmine.rake:307:in `block (4 levels) in <top (required)>'
Here is the general idea of the rescue approach I mentioned in my comment:
loop do
begin
resp = Curl.get("http://someurl.com/users.json") do |http|
http.headers["API-Key"] = ENV["API_KEY"]
end
# process successful response here
rescue Curl::Err::TimeoutError
# process error here
end
end
You would then need to modify this to do the retries. Here is 1 implementation (not tested though):
# Returns the response on success, nil on TimeoutError
def get1(url)
begin
Curl.get(url) do |http|
http.headers["API-Key"] = ENV["API_KEY"]
end
rescue Curl::Err::TimeoutError
nil
end
end
# Returns the response on success, nil on TimeoutErrors after all retry_count attempts.
def get_with_retries(url, retry_count)
retry_count.times do
result = get1(url)
return result if result
end
nil
end
response = get_with_retries("http://someurl.com/users.json", 3)
if response
# handle success
else
# handle timeout failure
end
We can also do it in block
def handle_timeouts
begin
yield
rescue Curl::Err::TimeoutError
retry
end
end
handle_timeouts do
resp = Curl.get("http://someurl.com/users.json") do |http|
http.headers["API-Key"] = ENV["API_KEY"]
end
end
In Ruby none of the error handling that I do seems to take any effect. For example in this function.
def http (uri)
url = URI.parse(uri)
if Addressable::url.host
if url.scheme=='https'
response = Net::HTTP.start(url.host, use_ssl: true, verify_mode: OpenSSL::SSL::VERIFY_NONE) do |http|
http.get url.request_uri, 'User-Agent' => 'MyLib v1.2'
end
elsif url.scheme=='http'
begin
http = Net::HTTP.new(url.host, url.port)
response = http.request(Net::HTTP::Get.new(url.request_uri))
rescue
response.body = "lol"
end
end
else
response.body = "lol"
end
return response.body
end
Regardless of the error handling, the code would still crash and give me error on the line right after begin.
I know that the url host is not valid, but is the error handling not supposed to fix it?
`initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError)
Why nokogiri waits for couple of secongs (3-5) when the server is busy and I'm requesting pages one by one, but when these request are in a loop, nokogiri does not wait and throws the timeout message.
I'm using timeout block wrapping the request, but nokogiri does not wait for that time at all.
Any suggested procedure on this?
# this is a method from the eng class
def get_page(url,page_type)
begin
timeout(10) do
# Get a Nokogiri::HTML::Document for the page we’re interested in...
##doc = Nokogiri::HTML(open(url))
end
rescue Timeout::Error
puts "Time out connection request"
raise
end
end
# this is a snippet from the main app calling eng class
# receives a hash with urls and goes throgh asking one by one
def retrieve_in_loop(links)
(0..links.length).each do |idx|
url = links[idx]
puts "Visiting link #{idx} of #{links.length}"
puts "link: #{url}"
begin
##eng.get_page(url, product)
rescue Exception => e
puts "Error getting url: #{idx} #{url}"
puts "This link will be skeeped. Continuing with next one"
end
end
end
The timeout block is simply the max time that that code has to execute inside the block without triggering an exception. It does not affect anything inside Nokogiri or OpenURI.
You can set the timeout to a year, but OpenURI can still time out whenever it likes.
So your problem is most likely that OpenURI is timing out on the connection attempt itself. Nokogiri has no timeouts; it's just a parser.
Adjusting read timeout
The only timeout you can adjust on OpenURI is the read timeout. It seems you cannot change the connection timeout through this method:
open(url, :read_timeout => 10)
Adjusting connection timeout
To adjust the connection timeout you would have to go with Net::HTTP directly instead:
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.open_timeout = 10
http.read_timeout = 10
response = http.get(uri.path)
Nokogiri.parse(response.body)
You can also take a look at some additional discussion here:
Ruby Net::HTTP time out
Increase timeout for Net::HTTP
I wrote simple function which handles fetching of the url:
def tender_page_get url, agent
sleep(rand(6)+2)
begin
return agent.get(url).parser
rescue Errno::ETIMEDOUT, Timeout::Error, Net::HTTPNotFound
EYE.debug "--winter sleep #{url}"
puts "-x-#{url}"
sleep(300)
tender_page_get url, agent
rescue => e
puts "-x-#{url}"
EYE.debug "--unknown exception"
EYE.debug "#{url} #{e.inspect}"
end
end
The problem is, even though I am catching Net::HTTPNotFound in my first rescue block, I still see in my log records like:
--unknown exception
{url} 404 => Net::HTTPNotFound
which means that this exception was caught by the second rescue block. What could be the reason for that?
Mechanize raises a Mechanize::ResponseCodeError for a 404 and not a Net::HTTPNotFound. The to_s on Mechanize::ResponseCodeError looks like this:
def to_s
"#{response_code} => #{Net::HTTPResponse::CODE_TO_OBJ[response_code]}"
end
This returns '404 => Net::HTTPNotFound' which makes it look like this is the exception being raised.
I'm using the SystemTimer gem to deal with timeout problems.
https://github.com/ph7/system-timer
I can't find a way to catch the Exception when a Timeout
begin
SystemTimer.timeout_after(10.seconds) do
# facebook api
rest_graph.fql(query)
end
rescue RestGraph::Error::InvalidAccessToken
return nil
rescue Timeout::Error
# never executed
end
But the last Exception Timeout::Error is never triggered.
Why not use Timeout, which comes with 1.9.2 and is designed to do this?
require 'timeout'
status = Timeout::timeout(5) {
# Something that should be interrupted if it takes too much time...
}
Try this: (based on your link)
class TimedOut < StandardError
end
begin
SystemTimer.timeout_after(10.seconds, TimedOut) do
# ...
end
rescue TimedOut
# ...
end