I have the following methods in a Ruby script:
def parse_endpoint(endpoint)
return URI.parse(endpoint)
end
def verify_url(endpoint, fname)
url = “#{endpoint}#{fname}”
req = Net::HTTP.new(url.host, url.port)
res = req.request_head(url.path)
if res.code == “200”
true
else
puts “#{fname} is an invalid file”
false
end
end
Testing the url manually like so works fine (returns true since the url is indeed valid):
endpoint = parse_endpoint('http://mywebsite.com/mySubdirectory/')
verify_url(endpoint, “myFile.json”)
However, when I try to do the following in rspec
describe 'my functionality'
let (:endpoint) { parse_endpoint(“http://mywebsite.com/mySubdirectory/”) }
it 'should verify valid url' do
expect(verify_url(endpoint, “myFile.json”).to eq(true))
end
end
it gives me this error
“NoMethodError:
undefined method `host' for "http://mysebsite.com/mySubdirectory/myFile.json":String”
What am I doing wrong?
url is a String object, and you are trying to access a method called host which does not exist in String:
url = “#{endpoint}#{fname}”
req = Net::HTTP.new(url.host, url.port)
EDIT you probably need an URI object. I think this is what you want:
2.2.1 :004 > require 'uri'
=> true
2.2.1 :001 > url = 'http://mywebsite.com/mySubdirectory/'
=> "http://mywebsite.com/mySubdirectory/"
2.2.1 :005 > parsed_url = URI.parse url
=> #<URI::HTTP http://mywebsite.com/mySubdirectory/>
2.2.1 :006 > parsed_url.host
=> "mywebsite.com"
So just add url = URI.parse url before using url.host.
Testing the url manually like so works fine (returns true since the url is indeed valid):
endpoint = parse_endpoint('http://mywebsite.com/mySubdirectory/')
verify_url(endpoint, “myFile.json”)
It seems you missed something when you tested code above (maybe you tested old version) because it can't work as it is now.
Look at these lines of code:
url = "#{endpoint}#{fname}"
req = Net::HTTP.new(url.host, url.port)
You're creating a string variable url from other two variables endpoint and fname. So far, so good.
But then you're trying to access method host on url variable, which doesn't exist (but it exists on the endpoint variable), that's why you get this error.
You may want to use this code instead:
def verify_url(endpoint, fname)
url = endpoint.merge(fname)
res = Net::HTTP.start(url.host, url.port) do |http|
http.head(url.path)
end
# it's actually a bad idea to puts some text in a query method
# let's just return value instead
res.code == "200"
end
Related
currently I am writing a program that needs to check tons of possible urls searching for any that actually exist. To be precise, I mean exist as in you can visit the url and there's actual content of some sort.. not string parsing to see if it's in url format.
The program generates a list of possible variants for a filename and then checks each one until it gets a url that actually exists, so most of the url remains the same. Examples would be,
https://www.test.com/folder1/FILE.png
https://www.test.com/folder1/File.png
https://www.test.com/folder1/file.png
https://www.test.com/folder1/file1.png
That said, my code currently works fine.. however it ends up taking about 2-4 secods per url check and I don't know of a way to speed it up. Is there any faster or better way to validate urls or am I just out of luck?
This is my function to validate urls:
require "net/http"
def url_exist? url_path
url = URI.parse(url_path)
req = Net::HTTP.new(url.host, url.port)
req.use_ssl = true
res = req.request_head(url.path)
if res.code == "200" || res.code == "403"
return true
end
end
Thank you for taking the time to read this and any help will be much appreciated.
Your code creates a new connection for each URL. It should be faster to send multiple requests over the same connection via HTTP keep-alive.
In Ruby, you can open such connection via Net::HTTP.start, e.g.:
require 'net/http'
class URLChecker
def initialize(base_url)
uri = URI(base_url)
Net::HTTP.start(uri.host, uri.port, use_ssl: uri.is_a?(URI::HTTPS)) do |http|
#http = http
yield self
end
end
def exist?(path)
res = #http.head(path)
res.code == '200' || res.code == '403'
end
end
URLChecker.new('https://stackoverflow.com') do |uc|
p uc.exist?('/questions/tagged/ruby') #=> true
p uc.exist?('/questions/tagged/python') #=> true
p uc.exist?('/questions/tagged/foobar') #=> false
end
Can someone explain to me why I am getting this error when doing this POST? I pulled the snippet from the Ruby-docs page.
undefined method `hostname' for #URI::HTTP:0x10bd441d8 URL:http://ws.mittthetwitapp.com/ws.phpmywebservice (NoMethodError)
Perhaps I am missing a require or something?
require 'net/http'
uri= URI('http://ws.mywebservice.com/ws.php')
req = Net::HTTP::Post.new(uri.path)
req.set_form_data('xmlPayload' => '<TestRequest><Message>Hi Test</Message></TestRequest>')
res = Net::HTTP.start(uri.hostname, uri.port) do |http|
http.request(req)
end
case res
when Net::HTTPSuccess, Net::HTTPRedirection
# OK
else
res.value
end
If you're using a version of Ruby prior to 1.9.3, you should use uri.host.
URI#hostname was added in Ruby 1.9.3. It is different than URI#host in that it removes brackets from IPv6 hostnames. For non-IPv6 hostnames it should behave identically.
The implementation (from APIdock):
def hostname
v = self.host
/\A\[(.*)\]\z/ =~ v ? $1 : v
end
I'm having trouble getting parameters passed in an HTTP Put call, using ruby. Take a look at the "put_data" variable.
When I leave it as a hash, ruby says:
undefined method `bytesize' for #<Hash:0x007fbf41a109e8>
if I convert to a string, I get:
can't convert Net::HTTPUnauthorized into String
I've also tried doing just - '?token=wBsB16NSrfVDpZPoEpM'
def process_activation
uri = URI("http://localhost:3000/api/v1/activation/" + self.member_card_num)
Net::HTTP.start(uri.host, uri.port) do |http|
headers = {'Content-Type' => 'text/plain; charset=utf-8'}
put_data = {:token => "wBsB16NSrfVDpZPoEpM"}
response = http.send_request('PUT', uri.request_uri, put_data, headers)
result = JSON.parse(response)
end
if result['card']['state']['state'] == "active"
return true
else
return false
end
end
I've searched all around, including rubydocs, but can't find an example of how to encode parameters. Any help would be appreciated.
Don't waste your time with NET::HTTP. I used 'rest-client' and had this thing done in minutes...
def process_activation
response = RestClient.put 'http://localhost:3000/api/v1/card_activation/'+ self.member_card_num, :token => "wBsB1pjJNNfiK6NSrfVDpZPoEpM"
result = JSON.parse(response)
return result['card']['state']['state'] == "active"
end
Here is my ruby program
require 'net/http'
require 'uri'
begin
url = URI.parse("http://google.com")
rescue Exception => err
p err
exit
end
http = Net::HTTP.new(url.host, url.port)
res = http.head("/")
p res.code
It works fine, however if I remove http:// from URL.parse(), It gives me this error:
/usr/lib/ruby/1.9.1/net/http.rb:1196:in `addr_port': undefined method `+' for nil:NilClass (NoMethodError) ...
from /usr/lib/ruby/1.9.1/net/http.rb:1094:in `request'
from /usr/lib/ruby/1.9.1/net/http.rb:860:in `head'
Is it the correct way to handle Exception ?
I know maybe the URL is not correct, but It should raise an exception URI::InvalidURIError instead of accepting and continue the program ?
If you say u = URI.parse('http://google.com'), you'll get a URI::HTTP back and the u.port will have a default of 80. If you say u = URI.parse('google.com'), you'll get a URI::Generic back with the u.port will be nil as will u.host.
So, when you do this:
url = URI.parse('google.com')
http = Net::HTTP.new(url.host, url.port)
You're really doing this:
http = Net::HTTP.new(nil, nil)
and Net::HTTP doesn't like that very much at all. You could try something like this instead:
if(str.to_s.empty?)
# complain loudly about a missing str
end
begin
url = URI.parse(str)
url = URI.parse('http://' + str) if !url.scheme
if(url.scheme != 'http' && url.scheme != 'https')
# more complaining about bad input
end
http = Net::HTTP.new(url.host, url.port)
#...
rescue URI::Error => e
# even yet more complaining
end
That sort of thing should bypass the exception completely and cover a few other things that you might be interested in.
You have to specifically catch URI::InvalidURIError, as it is not a descendant of Exception. See:
irb(main):002:0> URI::InvalidURIError.is_a?(Exception)
=> false
So the fix for your code would be:
begin
url = URI.parse("http://google.com")
rescue URI::InvalidURIError => err
p err
exit
end
The correct way is not to let any exception happen at all, but to check your conditions beforehand. Like this:
require 'net/http'
require 'uri'
begin
url = URI.parse("http://google.com")
rescue URI::InvalidURIError => err
p err
exit
end
if url.host && url.port
http = Net::HTTP.new(url.host, url.port)
res = http.head("/")
p res.code
else
p 'Error parsing url'
end
How can I send HTTP GET request with parameters via ruby?
I have tried a lot of examples but all of those failed.
I know this post is old but for the sake of those brought here by google, there is an easier way to encode your parameters in a URL safe manner. I'm not sure why I haven't seen this elsewhere as the method is documented on the Net::HTTP page. I have seen the method described by Arsen7 as the accepted answer on several other questions also.
Mentioned in the Net::HTTP documentation is URI.encode_www_form(params):
# Lets say we have a path and params that look like this:
path = "/search"
params = {q: => "answer"}
# Example 1: Replacing the #path_with_params method from Arsen7
def path_with_params(path, params)
encoded_params = URI.encode_www_form(params)
[path, encoded_params].join("?")
end
# Example 2: A shortcut for the entire example by Arsen7
uri = URI.parse("http://localhost.com" + path)
uri.query = URI.encode_www_form(params)
response = Net::HTTP.get_response(uri)
Which example you choose is very much dependent on your use case. In my current project I am using a method similar to the one recommended by Arsen7 along with the simpler #path_with_params method and without the block format.
# Simplified example implementation without response
# decoding or error handling.
require "net/http"
require "uri"
class Connection
VERB_MAP = {
:get => Net::HTTP::Get,
:post => Net::HTTP::Post,
:put => Net::HTTP::Put,
:delete => Net::HTTP::Delete
}
API_ENDPOINT = "http://dev.random.com"
attr_reader :http
def initialize(endpoint = API_ENDPOINT)
uri = URI.parse(endpoint)
#http = Net::HTTP.new(uri.host, uri.port)
end
def request(method, path, params)
case method
when :get
full_path = path_with_params(path, params)
request = VERB_MAP[method].new(full_path)
else
request = VERB_MAP[method].new(path)
request.set_form_data(params)
end
http.request(request)
end
private
def path_with_params(path, params)
encoded_params = URI.encode_www_form(params)
[path, encoded_params].join("?")
end
end
con = Connection.new
con.request(:post, "/account", {:email => "test#test.com"})
=> #<Net::HTTPCreated 201 Created readbody=true>
I assume that you understand the examples on the Net::HTTP documentation page but you do not know how to pass parameters to the GET request.
You just append the parameters to the requested address, in exactly the same way you type such address in the browser:
require 'net/http'
res = Net::HTTP.start('localhost', 3000) do |http|
http.get('/users?id=1')
end
puts res.body
If you need some generic way to build the parameters string from a hash, you may create a helper like this:
require 'cgi'
def path_with_params(page, params)
return page if params.empty?
page + "?" + params.map {|k,v| CGI.escape(k.to_s)+'='+CGI.escape(v.to_s) }.join("&")
end
path_with_params("/users", :id => 1, :name => "John&Sons")
# => "/users?name=John%26Sons&id=1"