I'm expecting this piece of code to run for no longer than 5 seconds:
require 'httpi'
require 'timeout'
puts Time.new
begin
request,response=nil,nil
Timeout::timeout(5){
request=HTTPI::Request.new(url: "http://example.com")
response=HTTPI.get(request)
}
rescue
puts "except: #{$!}"
ensure
puts Time.new
end
But this is the output I'm getting:
2016-11-04 09:44:55 -0400
D, [2016-11-04T09:44:55.916557 #2476] DEBUG -- : HTTPI GET request to example.com (net_http)
except: execution expired
2016-11-04 09:45:16 -0400
I'm assuming NET's default HTTP timeout is 20 seconds, so Timeout::timeout is just allowing the code to run however long it wants. Why?
As you can see here, here and here, the Ruby's Timeout module is famous for having some problems.
You should consider not using this module, unless it is extremely necessary.
Instead, you can use the read_timeout and/or open_timeout options provided by the HTTPI's API.
request = HTTPI::Request.new(url: "http://example.com")
request.open_timeout = 5 # seconds
request.read_timeout = 5 # seconds
response = HTTPI.get(request)
Related
I’m using Rails 4.2.3 and Nokogiri to get data from a web site. I want to perform an action when I don’t get any response from the server, so I have:
begin
content = open(url).read
if content.lstrip[0] == '<'
doc = Nokogiri::HTML(content)
else
begin
json = JSON.parse(content)
rescue JSON::ParserError => e
content
end
end
rescue Net::OpenTimeout => e
attempts = attempts + 1
if attempts <= max_attempts
sleep(3)
retry
end
end
Note that this is different than getting a 500 from the server. I only want to retry when I get no response at all, either because I get no TCP connection or because the server fails to respond (or some other reason that causes me not to get any response). Is there a more generic way to take account of this situation other than how I have it? I feel like there are a lot of other exception types I’m not thinking of.
This is generic sample how you can define timeout durations for HTTP connection, and perform several retries in case of any error while fetching content (edited)
require 'open-uri'
require 'nokogiri'
url = "http://localhost:3000/r503"
openuri_params = {
# set timeout durations for HTTP connection
# default values for open_timeout and read_timeout is 60 seconds
:open_timeout => 1,
:read_timeout => 1,
}
attempt_count = 0
max_attempts = 3
begin
attempt_count += 1
puts "attempt ##{attempt_count}"
content = open(url, openuri_params).read
rescue OpenURI::HTTPError => e
# it's 404, etc. (do nothing)
rescue SocketError, Net::ReadTimeout => e
# server can't be reached or doesn't send any respones
puts "error: #{e}"
sleep 3
retry if attempt_count < max_attempts
else
# connection was successful,
# content is fetched,
# so here we can parse content with Nokogiri,
# or call a helper method, etc.
doc = Nokogiri::HTML(content)
p doc
end
When it comes to rescuing exceptions, you should aim to have a clear understanding of:
Which lines in your system can raise exceptions
What is going on under the hood when those lines of code run
What specific exceptions could be raised by the underlying code
In your code, the line that's fetching the content is also the one that could see network errors:
content = open(url).read
If you go to the documentation for the OpenURI module you'll see that it uses Net::HTTP & friends to get the content of arbitrary URIs.
Figuring out what Net::HTTP can raise is actually very complicated but, thankfully, others have already done this work for you. Thoughtbot's suspenders project has lists of common network errors that you can use. Notice that some of those errors have to do with different network conditions than what you had in mind, like the connection being reset. I think it's worth rescuing those as well, but feel free to trim the list down to your specific needs.
So here's what your code should look like (skipping the Nokogiri and JSON parts to simplify things a bit):
require 'net/http'
require 'open-uri'
HTTP_ERRORS = [
EOFError,
Errno::ECONNRESET,
Errno::EINVAL,
Net::HTTPBadResponse,
Net::HTTPHeaderSyntaxError,
Net::ProtocolError,
Timeout::Error,
]
MAX_RETRIES = 3
attempts = 0
begin
content = open(url).read
rescue *HTTP_ERRORS => e
if attempts < MAX_RETRIES
attempts += 1
sleep(2)
retry
else
raise e
end
end
I would think about using a Timeout that raises an exception after a short period:
MAX_RESPONSE_TIME = 2 # seconds
begin
content = nil # needs to be defined before the following block
Timeout.timeout(MAX_RESPONSE_TIME) do
content = open(url).read
end
# parsing `content`
rescue Timeout::Error => e
attempts += 1
if attempts <= max_attempts
sleep(3)
retry
end
end
I want to test a http get request. output something if timeout.
begin
url = "#{url}?#{params.to_param}"
Net::HTTP.get_response(URI.parse(url))
rescue Timeout::Error
puts "....."
end
How to raise a timeout error manually? or how to set a shorter timeout number for http request?
For a http request, should I change the default timeout number? How long is appropriate？
Based on http://opensourceconnections.com/blog/2008/04/24/adding-timeout-to-nethttp-get_response/
http = Net::HTTP.new(url.host, url.port)
http.read_timeout = 5
http.open_timeout = 5
resp = http.start() {|http|
http.get(url.path)
}
puts resp.kind_of? Net::HTTPResponse
puts resp.code
puts resp.body
To set timeout use:
http = Net::HTTP.new(host_param)
http.read_timeout = 500
There are few types of timeouts you can set. From docs:
open_timeout:
Number of seconds to wait for the connection to open.
Any number may be used, including Floats for fractional seconds.
If the HTTP object cannot open a connection in this many seconds,
it raises a Net::OpenTimeout exception. The default value is nil.
read_timeout:
Number of seconds to wait for one block to be read (via one read(2) call).
Any number may be used, including Floats for fractional seconds.
If the HTTP object cannot read data in this many seconds,
it raises a Net::ReadTimeout exception. The default value is 60 seconds.
ssl_timeout:
Sets the SSL timeout seconds.
I parse RSS stream with Feedjira.
When I used a fetch_and_parse method it sometimes blocked and doesn't respond.
The same thing happens with manual curb downloading.
I write in a loop:
#my_logger.info "--- Before perform ---"
easy = Curl::Easy.new
easy.follow_location = true
easy.max_redirects = 3
easy.connect_timeout = 120
easy.url = url
easy.useragent = "Ruby/Curb"
easy.perform
#my_logger.info "--- After perform ---"
doc = easy.body_str
easy.close
After some time (it may be a day or an hour), process stops on the easy.perform line and doesn't respond. E.g. process outputs --- Before perform --- and nothing else.
It can be related to a network issue happening randomly.
If you use a timeout you can skip this kind of situations in long running tasks.
require 'timeout'
begin
Timeout.timeout(5) do
easy.perform
end
rescue Timeout::Error
puts 'timeout'
end
I have a function that gets response over http. It runs some tests. Lately it started to happen that the test never finishes. So I introduced a time out. Then I found out that if I stop the database server the test script finishes with a db error that is in fact very good lead why the test didn't finish as expected. So to get the error could help to save me time. Because I wouldn't have to reproduce the whole test again manually.
Q1: Is there any way to let the connection time out but then get the response after the database server is restarted? Note that I cannot send the http request again as it would start the same text again.
Q2: I think that a solution would be to introduce timer while "waiting" for http response. But I don't know how to do that. Any idea?
My function is like
def execute_db2_script(url)
db2_database = 'RATIONAL'
http_read_timeout=$http_read_timeout
uri = URI.parse(url)
start = Time.new
connection = Net::HTTP.new(uri.host, 443)
connection.use_ssl = true
begin
response = connection.start() do |http|
http.open_timeout = 50
http.read_timeout = http_read_timeout
http.request_get(uri.request_uri)
end
rescue Timeout::Error
time_out_message ="security time out - after #{$http_read_timeout} sec"
return time_out_message
end
return response.body.gsub("\n","<BR>")
end
You can use retry keyword
def execute_db2_script(url)
...
begin
...
rescue Timeout::Error
time_out_message ="security time out - after #{$http_read_timeout} sec"
if "the server is going to restart then"
retry # this will restart begin-rescue-end block again
else
return time_out_message
end
end
response.body.gsub("\n","<BR>")
end
I have a ruby code that triggers php script over https.
Use case: The php script usually finishes in 5 minutes so I have set up time out for https request after 10 minutes. I need a timer that would trigger code after let's say 7 minutes after the https request started.
I was thinking of using thread that I created just before I initiate https request. I am not sure if this the correct way to approach this. Maybe there is not need to use threads at all. I am using ruby 1.8.7 (2010-08-16 patchlevel 302) [i386-mingw32]. Also I don't now if I can 'kill' the thread on successful finish of https request.
uri = URI.parse(url)
start = Time.new
http_read_timeout=60*10
connection = Net::HTTP.new(uri.host, 443)
connection.use_ssl = true
begin
response = connection.start() do |http|
http.open_timeout = 50
http.read_timeout = http_read_timeout
http.request_get(uri.request_uri)
# here I need to place a code that is triggered
# in case of custom timeout is reached
end
rescue Timeout::Error
# "Connection failed
time_out_message ="security time out - after #{http_read_timeout} sec"
return time_out_message
end
puts "finished"
The basic structure could be like this:
seconds_timer = MyDelay
counter = 0
test_thread = Thread.new do
run_http_php_test
end
while test_thread.alive?
counter += 1
if counter > seconds_timer
handle_custom_timeout_somehow
# if you want to halt run_http_php_test:
test_thread.kill if test_thread.alive?
# otherwise:
break
end
sleep 1
end
# the below doesn't apply if you kill the run_http_php_test thread
test_thread.join if test_thread.alive?
...but of course you could change that sleep 1 to whatever polling interval you like. Polling is nicer than just forcing your original thread to sleep, because the code will finish faster if run_http_php_test is done before you hit your custom timeout value.
Most or all of your code above can be in the run_http_php_test method, or inserted directly...whichever you'd prefer.
ruby 1.9.3 implements timeout module that has a timeout function. you can see it here. if you scroll down you can click show source and see the definition for timeout method. you can copy it if you dont want to upgrade to ruby 1.9.3 (I recommend upgrade since 1.8.7 is very slow compared to 1.9.3)