In Ruby none of the error handling that I do seems to take any effect. For example in this function.
def http (uri)
url = URI.parse(uri)
if Addressable::url.host
if url.scheme=='https'
response = Net::HTTP.start(url.host, use_ssl: true, verify_mode: OpenSSL::SSL::VERIFY_NONE) do |http|
http.get url.request_uri, 'User-Agent' => 'MyLib v1.2'
end
elsif url.scheme=='http'
begin
http = Net::HTTP.new(url.host, url.port)
response = http.request(Net::HTTP::Get.new(url.request_uri))
rescue
response.body = "lol"
end
end
else
response.body = "lol"
end
return response.body
end
Regardless of the error handling, the code would still crash and give me error on the line right after begin.
I know that the url host is not valid, but is the error handling not supposed to fix it?
`initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError)
Related
What can cause while making an HTTP connection to return EBADF (Bad file descriptor).
Here is my following code wherein the HTTP connection is made. Although the error is very less now(happening very less) but before I put those error on rescue I need to understand what is the reason for the EBADF
def make_http_request(url, headers={})
uri = URI(url)
Net::HTTP.start(uri.host, uri.port) do |http|
req = Net::HTTP::Get.new(uri, headers)
resp = http.request(req)
if resp.code.to_i != 200
logger.error "Retrieve #{resp.code} with #{url} and #{headers}"
return false
end
return resp.body
end
rescue SocketError, Net::ReadTimeout, Errno::ECONNREFUSED => e
logger.error "make_http_request #{url} with #{headers} resulted in #{e.message} \n #{e.backtrace}"
return false
end
I have a feeling that connect syscall is receiving an FD which ain't valid at that given point in time. But still unable to understand how can that happens.
If it helps the code is used in an application that operates with multiple threads.
In a nutshell, the definition of the above method looks like this...
module Eval
def make_http_request(url, headers={})
...
...
..
end
def request_local_endpoint(url, headers)
response = make_http_request(url, headers)
response && response.fetch('bravo',nil)
end
def request_external_endpoint(url, headers)
response = make_http_request(url, headers)
response && response.fetch('token',nil)
end
end
class RequestBuilder
include Eval
attr_reader :data
def initialize(data)
#data = data
end
def start
token = request_external_endpoint('http://external.com/endpoint1',{'Content-Type'.freeze => 'application/json', 'Authorization' => 'abcdef'})
return unless token
result = request_local_endpoint('http://internal.com/endpoint1',{'Content-Type'.freeze => 'application/json'})
return result
end
end
10.times {
Thread.new { RequestBuilder.new('sample data').start }
}
I am trying to hit a URL in Ruby with the following code:
begin
Net::HTTP.start(uri.host, uri.port, :read_timeout=>5) do |http|
request = Net::HTTP::Get.new uri.request_uri
#response = http.request request
#responsecode = #response.code.to_i
end
rescue Exception => e
::NewRelic::Agent.notice_error(e)
end
The problem I am having is that the URL is hit twice.I don't know why is this happening.Any help is appreciated.If I remove the :read_timeout, it works fine.
EDIT:
The problem I found is if service takes longer than read_timeout to respond, it is hit again.
I have this method in my class:
def self.get(url)
#TODO We could test with https too
if url.match(/^http/)
correct_url = url
else
correct_url = "http://#{url}"
end
uri = URI.parse(correct_url)
if uri.respond_to? 'request_uri'
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Get.new(uri.request_uri)
http.request(request)
else
puts "Incorrect URI"
end
end
Unfortunately it's not following the redirects.
Can someone tell me how to make this method allow a maximum of three redirects?
Try this:
def self.get(url)
# TODO: test with https too
url = "http://#{url}" unless url.match(/^http/)
3.times do
uri = URI.parse(url)
if uri.respond_to?(:request_uri)
response = Net::HTTP.get_response(uri)
case response.code
when '301', '302'
url = response.header['location']
else
return response
end
end
end
end
I am trying to download a binary file via HTTP using the following Ruby script.
#!/usr/bin/env ruby
require 'net/http'
require 'uri'
def http_download(resource, filename, debug = false)
uri = URI.parse(resource)
puts "Starting HTTP download for: #{uri}"
http_object = Net::HTTP.new(uri.host, uri.port)
http_object.use_ssl = true if uri.scheme == 'https'
begin
http_object.start do |http|
request = Net::HTTP::Get.new uri.request_uri
Net::HTTP.get_print(uri) if debug
http.read_timeout = 500
http.request request do |response|
open filename, 'w' do |io|
response.read_body do |chunk|
io.write chunk
end
end
end
end
rescue Exception => e
puts "=> Exception: '#{e}'. Skipping download."
return
end
puts "Stored download as #{filename}."
end
However it downloads the HTML source instead of the binary. When I enter the URL in the browser the binary file is downloaded. Here is a URL with which the script fails:
http://dcatlas.dcgis.dc.gov/catalog/download.asp?downloadID=2175&downloadTYPE=KML
I execute the script as follows
pry> require 'myscript'
pry> resource = "http://dcatlas.dcgis.dc.gov/catalog/download.asp?downloadID=2175&downloadTYPE=KML"
pry> http_download(resource,"StreetTreePt.KML", true)
How can I download the binary?
Redirection experiments
I found this redirection check which looks quite reasonable. When I integrate in the response block it fails with the following error:
Exception: 'undefined method `host' for "save_download.asp?filename=StreetTreePt.KML":String'. Skipping download.
The exception does not occur in the "original" function posted above.
The documentation for Net::HTTP shows how to handle redirects:
Following Redirection
Each Net::HTTPResponse object belongs to a class for its response code.
For example, all 2XX responses are instances of a Net::HTTPSuccess subclass, a 3XX response is an instance of a Net::HTTPRedirection subclass and a 200 response is an instance of the Net::HTTPOK class. For details of response classes, see the section “HTTP Response Classes” below.
Using a case statement you can handle various types of responses properly:
def fetch(uri_str, limit = 10)
# You should choose a better exception.
raise ArgumentError, 'too many HTTP redirects' if limit == 0
response = Net::HTTP.get_response(URI(uri_str))
case response
when Net::HTTPSuccess then
response
when Net::HTTPRedirection then
location = response['location']
warn "redirected to #{location}"
fetch(location, limit - 1)
else
response.value
end
end
print fetch('http://www.ruby-lang.org')
Or, you can use Ruby's OpenURI, which handles it automatically. Or, the Curb gem will do it. Probably Typhoeus and HTTPClient too.
According to the code you show in your question, the exception you are getting can only come from:
http_object = Net::HTTP.new(uri.host, uri.port)
which is hardly likely since uri is a URI object. You need to show the complete code if you want help with that problem.
Suppose that after primitive validation of user submitted URL I have the string that looks like URL:
url = 'http://www.thisdomaindoesntexist.com/dont_even_ask_about/this/uri'
How can I check if its available or not?
I tried this in my is_valid_link function:
require "net/http"
url = URI.parse(url)
req = Net::HTTP.new(url.host, url.port)
res = req.request_head(url.path)
It works if the server exists giving me back the HTTP response, but the problem is that in case of bad url I get an error like this:
SocketError in PostsController#create
getaddrinfo: nodename nor servname provided, or not known
How should I do this kind of validation properly?
Thanks in advance.
You can use rescue to catch errors and do some error handling
begin
require "net/http"
url = URI.parse(url)
req = Net::HTTP.new(url.host, url.port)
res = req.request_head(url.path)
rescue
# error occured, return false
false
else
# valid site
true
end
Use rescue inline:
require "net/http"
url = URI.parse(url)
req = Net::HTTP.new(url.host, url.port)
res = req.request_head(url.path) rescue false
I've looked for a way to check if URL existed for 5 hours and this thread actually helped me.
I'm a newbie in rails and wanted to find something easy.
Here how I integrated the code into controller:
require "net/http"
def url
url = URI.parse('http://www.url that you want to check.com/' + "/")
end
def req
#req = Net::HTTP.new(url.host, url.port)
end
def res #res
#res = req.request_head(url.path)
rescue
false
end
def test
if res == false
"something"
else
"another thing"
end
you have to make sure that URL ends with "/", else the code won't work.