I'm developing a small application which posts XML to some webservice.
This is done using Net::HTTP::Post::Post. However, the service provider recommends using a re-connect.
Something like:
1st request fails -> try again after 2 seconds
2nd request fails -> try again after 5 seconds
3rd request fails -> try again after 10 seconds
...
What would be a good approach to do that? Simply running the following piece of code in a loop, catching the exception and run it again after an amount of time? Or is there any other clever way to do that? Maybe the Net package even has some built in functionality that I'm not aware of?
url = URI.parse("http://some.host")
request = Net::HTTP::Post.new(url.path)
request.body = xml
request.content_type = "text/xml"
#run this line in a loop??
response = Net::HTTP.start(url.host, url.port) {|http| http.request(request)}
Thanks very much, always appreciate your support.
Matt
This is one of the rare occasions when Ruby's retry comes in handy. Something along these lines:
retries = [3, 5, 10]
begin
response = Net::HTTP.start(url.host, url.port) {|http| http.request(request)}
rescue SomeException # I'm too lazy to look it up
if delay = retries.shift # will be nil if the list is empty
sleep delay
retry # backs up to just after the "begin"
else
raise # with no args re-raises original error
end
end
I use gem retryable for retry.
With it code transformed from:
retries = [3, 5, 10]
begin
response = Net::HTTP.start(url.host, url.port) {|http| http.request(request)}
rescue SomeException # I'm too lazy to look it up
if delay = retries.shift # will be nil if the list is empty
sleep delay
retry # backs up to just after the "begin"
else
raise # with no args re-raises original error
end
end
To:
retryable( :tries => 10, :on => [SomeException] ) do
response = Net::HTTP.start(url.host, url.port) {|http| http.request(request)}
end
Related
currently I am writing a program that needs to check tons of possible urls searching for any that actually exist. To be precise, I mean exist as in you can visit the url and there's actual content of some sort.. not string parsing to see if it's in url format.
The program generates a list of possible variants for a filename and then checks each one until it gets a url that actually exists, so most of the url remains the same. Examples would be,
https://www.test.com/folder1/FILE.png
https://www.test.com/folder1/File.png
https://www.test.com/folder1/file.png
https://www.test.com/folder1/file1.png
That said, my code currently works fine.. however it ends up taking about 2-4 secods per url check and I don't know of a way to speed it up. Is there any faster or better way to validate urls or am I just out of luck?
This is my function to validate urls:
require "net/http"
def url_exist? url_path
url = URI.parse(url_path)
req = Net::HTTP.new(url.host, url.port)
req.use_ssl = true
res = req.request_head(url.path)
if res.code == "200" || res.code == "403"
return true
end
end
Thank you for taking the time to read this and any help will be much appreciated.
Your code creates a new connection for each URL. It should be faster to send multiple requests over the same connection via HTTP keep-alive.
In Ruby, you can open such connection via Net::HTTP.start, e.g.:
require 'net/http'
class URLChecker
def initialize(base_url)
uri = URI(base_url)
Net::HTTP.start(uri.host, uri.port, use_ssl: uri.is_a?(URI::HTTPS)) do |http|
#http = http
yield self
end
end
def exist?(path)
res = #http.head(path)
res.code == '200' || res.code == '403'
end
end
URLChecker.new('https://stackoverflow.com') do |uc|
p uc.exist?('/questions/tagged/ruby') #=> true
p uc.exist?('/questions/tagged/python') #=> true
p uc.exist?('/questions/tagged/foobar') #=> false
end
I have Sidekiq 5 workers, concurrency 50. My webapp make third party API calls every few minutes, averagely 15000 requests per hour. Each hour I have near 10 errors Net::OpenTimeout: execution expired. Looks like not a big problem, but I want to know how to deal with it. Thanks.
def grabber(url)
response, body = nil
uri = URI(url)
Net::HTTP.start(uri.host, uri.port,
:use_ssl => uri.scheme == 'https', :read_timeout => 1000) do |http|
request = Net::HTTP::Get.new uri
response = http.request request
end
if response.code == '200'
body = JSON.parse(response.body)
end
body
end
Net::HTTP throws an exception when request time out occurs.
You can just catch exceptions and do something with it:
rescue Net::OpenTimeout
#do something
Here you can get some examples.
I have an array of urls. I'm going through each one, sending a get request and printing the response code. Here is part of the code:
arr.each do |url|
res = Faraday.get(link.href)
p res.status
end
However sometimes I get to url, it times out and crashes. Is there a way to tell ruby "if I don't get a response in a certain amount of time then skip to the next url?"
You could add a timeout like this:
require 'timeout'
arr.each do |url|
begin
Timeout.timeout(5) do # a timeout of five seconds
res = Faraday.get(link.href)
p res.status
end
rescue Timeout::Error
# handle error: show user a message?
end
end
I have a function that gets response over http. It runs some tests. Lately it started to happen that the test never finishes. So I introduced a time out. Then I found out that if I stop the database server the test script finishes with a db error that is in fact very good lead why the test didn't finish as expected. So to get the error could help to save me time. Because I wouldn't have to reproduce the whole test again manually.
Q1: Is there any way to let the connection time out but then get the response after the database server is restarted? Note that I cannot send the http request again as it would start the same text again.
Q2: I think that a solution would be to introduce timer while "waiting" for http response. But I don't know how to do that. Any idea?
My function is like
def execute_db2_script(url)
db2_database = 'RATIONAL'
http_read_timeout=$http_read_timeout
uri = URI.parse(url)
start = Time.new
connection = Net::HTTP.new(uri.host, 443)
connection.use_ssl = true
begin
response = connection.start() do |http|
http.open_timeout = 50
http.read_timeout = http_read_timeout
http.request_get(uri.request_uri)
end
rescue Timeout::Error
time_out_message ="security time out - after #{$http_read_timeout} sec"
return time_out_message
end
return response.body.gsub("\n","<BR>")
end
You can use retry keyword
def execute_db2_script(url)
...
begin
...
rescue Timeout::Error
time_out_message ="security time out - after #{$http_read_timeout} sec"
if "the server is going to restart then"
retry # this will restart begin-rescue-end block again
else
return time_out_message
end
end
response.body.gsub("\n","<BR>")
end
I've tried this on a few machines on different networks, all running ruby 1.8.7 and I get the same result after a long wait.
Net::HTTP.get(URI.parse('https://encrypted.google.com/'))
Timeout::Error: execution expired
but HTTP works fine
Net::HTTP.get(URI.parse('http://www.google.com/'))
After the inital timeout I get an EOFError instead
EOFError: end of file reached
It's really got me stumped. If you have any ideas or you can let me know if you get the same results I'd really appreciate it.
I think you need to set use_ssl to true...
example:
uri = URI.parse("https://www.google.com/")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
puts response.body
This is cannibalized from the following Ruby Inside post.