Ruby Mechanize, Nokogiri and Net::HTTP - ruby

I am using Net::HTTP for HTTP requests and getting a response back:
uri = URI("http://www.example.com")
http = Net::HTTP.start(uri.host, uri.port, proxy_host, proxy_port)
request = Net::HTTP::Get.new uri.request_uri
response = http.request request # Net::HTTPResponse object
body = response.body
If I have to use the Nokogiri gem in order to parse this HTML response I will do:
nokogiri_obj = Nokogiri::HTML(body)
But if I want to use Mechanize gem I need to do this:
agent = Mechanize.new
mechanize_obj = agent.get("http://www.example.com")
Is it possible for me to use Net::Http for getting the HTML response and then use the Mechanize gem to convert it into a Mechanize object instead of using agent.get()?
EDIT:
The reason for getting around the agent.get() method is because I am trying to use EventMachine::Iterator to make concurrent EM-HTTP requests.
EventMachine.run do
EM::Iterator.new(urls, 3).each do |url,iter|
puts "giving #{url} to httprequest now"
http = EM::HttpRequest.new(url).get
http.callback { |resp|
uri = resp.send(:URI, url)
puts "inside callback of #{url}"
body = resp.response
page = agent.parse(uri, resp, body)
}
iter.next
end
end
But its not working. I am getting an error:
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5.1/lib/mechanize.rb:1165:in`parse': undefined method `[]' for #<EventMachine::HttpClient:0x0000001c18eb30> (NoMethodError)
when I use the parse method for Net::HTTP it works fine and I get the Mechanize object:
uri = URI("http://www.example.com")
http = Net::HTTP.start(uri.host, uri.port, proxy_host, proxy_port)
request = Net::HTTP::Get.new uri.request_uri
response = http.request request # Net::HTTPResponse object
body = response.body
agent = Mechanize.new
page = agent.parse(uri, response, body)
Am I passing the wrong arguments for the parse method while using em-http?

I'm not sure why you think using Net::HTTP would be better. Mechanize will handle redirects and cookies, plus provides ready access to Nokogiri's parsed document.
require 'mechanize'
agent = Mechanize.new
page = agent.get('http://www.example.com')
# Use Nokogiri to find the content of the <h1> tag...
puts page.at('h1').content # => "Example Domains"
Note, setting the user_agent isn't necessary to reach example.com.
If you want to use a threaded engine to retrieve pages, take a look at Typhoeous and Hydra.

Looks like Mechanize has a parse method, so this could work:
mechanize_obj = Mechanize.parse(uri, response, body)

Related

How to send XML file using Post request in Ruby

I am writing a code that send http post request. Now I write xml body in my code, and its working correctly.
But if I want to send request using xml file I get
undefined method `bytesize' for #
Did you mean? bytes
My code below
require 'net/http'
request_body = <<EOF
<xml_expamle>
EOF
uri = URI.parse('http://example')
post = Net::HTTP::Post.new(uri.path, 'content-type' => 'text/xml; charset=UTF-8')
post.basic_auth 'user','passcode'
Net::HTTP.new(uri.host, uri.port).start {|http|
http.request(post, request_body) {|response|
puts response.body
}
}
**But if I want to make send file**
require 'net/http'
request_body = File.open('example/file.xml')
uri = URI.parse('http://example')
post = Net::HTTP::Post.new(uri.path, 'content-type' => 'application/xml; charset=UTF-8')
post.basic_auth 'user','passcode'
Net::HTTP.new(uri.host, uri.port).start {|http|
http.request(post, request_body) {|response|
puts response.body
}
}
I get
undefined method `bytesize' for #
Did you mean? bytes
You need to load the file content to memory if you want to use it as a request body, use #read method:
request_body = File.open('example/file.xml').read
and it'll work.

Ruby putting token key in requests

I dont know how to put my key into my requests so they are sent back as
{"status"=>"400", "message"=>"Token parameter is required."}
This is the code I have been using
require 'net/http'
require 'json'
token = 'YiwwVvywLngtPT***************'
url = 'https://www.ncdc.noaa.gov/cdo-web/api/v2/stations?locationid=FIPS:23&limit=5&sortfield=mindate'
uri = URI(url)
response = Net::HTTP.get(uri)
response.authorization = token
puts JSON.parse(response)
I have tried a couple different things I've found on the internet but all of them just give errors for
undefined method `methodname' for #<String:0x00007fd97519abd0>
According to the API documentation (based on the URL you referenced), you need to provide the token in a header named token.
So, you should probably try some variation of the below (code untested):
token = 'YiwwVvywLngtPT***************'
url = 'https://www.ncdc.noaa.gov/cdo-web/api/v2/stations?locationid=FIPS:23&limit=5&sortfield=mindate'
uri = URI(url)
request = Net::HTTP::Get.new(uri)
request['token'] = token
response = Net::HTTP.start(uri.hostname, uri.port) do |http|
http.request(request)
end
More information about Net:HTTP headers can be found in this StackOverflow answer.
As a side note, if you are not locked on using Net::HTTP, consider switching to a friendlier HTTP client, perhaps HTTParty. Then, the complete code looks like this:
require 'httparty'
token = 'YiwwVvywLngtPT***************'
url = 'https://www.ncdc.noaa.gov/cdo-web/api/v2/stations?locationid=FIPS:23&limit=5&sortfield=mindate'
response = HTTParty.get url, headers: { token: token }
puts response.body

Ruby httpclient url

I am making a call to a webservice via HTTPClient ruby gem. The location doesn't matter, but let's say it is at: https://mywebservice.com/api/v1/accounts/login/
When I send a post request:
url = "https://mywebservice.com/api/v1/accounts/login/"
client = HTTPClient.new
client.post url
I get a 404 response, that the page was not found. I setup a proxy and checked out exactly what was being sent it seems like the gem is messing up the url, it actually sends the request to:
https://mywebservice.comhttps://mywebservice.com:443/api/v1/accounts/login/
I am completely lost, I have no why it is doing that. Does it have anything to do with the fact that it's https and I should handle https differently?
Thanks
Yes, if you are requesting a https url do this:
require 'net/http'
require 'net/https'
require 'uri'
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.request_uri)
#you need to set this field to true
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
response = http.request(request)
Here is a link that should help:
http://bibwild.wordpress.com/2012/04/30/httpclient-is-a-nice-http-client-forin-ruby/
You should post to an uri with a body of params. Something like
client.post "http://mysite.com:8080", data
where data is a hash with the required parameters for the page.
For instance:
data = {"var" => "var"}

Sending http post request in Ruby by Net::HTTP

I'm sending a request with custom headers to a web service.
require 'uri'
require 'net/http'
uri = URI("https://api.site.com/api.dll")
https = Net::HTTP.new(uri.host, uri.port)
https.use_ssl = true
headers =
{
'HEADER1' => "VALUE1",
'HEADER2' => "HEADER2"
}
response = https.post(uri.path, headers)
puts response
It's not working, I'm receiving an error of:
/usr/lib/ruby/1.9.1/net/http.rb:1932:in `send_request_with_body': undefined method `bytesize' for #<Hash:0x00000001b93a10> (NoMethodError)
How do I solve this?
P.S. Ruby 1.9.3
Try this:
For detailed documentation, take a look at:
http://www.rubyinside.com/nethttp-cheat-sheet-2940.html
require 'uri'
require 'net/http'
uri = URI('https://api.site.com/api.dll')
https = Net::HTTP.new(uri.host, uri.port)
https.use_ssl = true
request = Net::HTTP::Post.new(uri.path)
request['HEADER1'] = 'VALUE1'
request['HEADER2'] = 'VALUE2'
response = https.request(request)
puts response
The second argument of Net::HTTP#post needs to be a String containing the data to post (often form data), the headers would be in the optional third argument.
As qqx mentioned, the second argument of Net::HTTP#post needs to be a String
Luckily there's a neat function that converts a hash into the required string:
response = https.post(uri.path, URI.encode_www_form(headers))

Ruby send JSON request

How do I send a JSON request in ruby? I have a JSON object but I dont think I can just do .send. Do I have to have javascript send the form?
Or can I use the net/http class in ruby?
With header - content type = json and body the json object?
uri = URI('https://myapp.com/api/v1/resource')
body = { param1: 'some value', param2: 'some other value' }
headers = { 'Content-Type': 'application/json' }
response = Net::HTTP.post(uri, body.to_json, headers)
require 'net/http'
require 'json'
def create_agent
uri = URI('http://api.nsa.gov:1337/agent')
http = Net::HTTP.new(uri.host, uri.port)
req = Net::HTTP::Post.new(uri.path, 'Content-Type' => 'application/json')
req.body = {name: 'John Doe', role: 'agent'}.to_json
res = http.request(req)
puts "response #{res.body}"
rescue => e
puts "failed #{e}"
end
HTTParty makes this a bit easier I think (and works with nested json etc, which didn't seem to work in other examples I've seen.
require 'httparty'
HTTParty.post("http://localhost:3000/api/v1/users", body: {user: {email: 'user1#example.com', password: 'secret'}}).body
This works on ruby 2.4 HTTPS Post with JSON object and the response body written out.
require 'net/http' #net/https does not have to be required anymore
require 'json'
require 'uri'
uri = URI('https://your.secure-url.com')
Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
request = Net::HTTP::Post.new(uri, 'Content-Type' => 'application/json')
request.body = {parameter: 'value'}.to_json
response = http.request request # Net::HTTPResponse object
puts "response #{response.body}"
end
real life example, notify Airbrake API about new deployment via NetHttps
require 'uri'
require 'net/https'
require 'json'
class MakeHttpsRequest
def call(url, hash_json)
uri = URI.parse(url)
req = Net::HTTP::Post.new(uri.to_s)
req.body = hash_json.to_json
req['Content-Type'] = 'application/json'
# ... set more request headers
response = https(uri).request(req)
response.body
end
private
def https(uri)
Net::HTTP.new(uri.host, uri.port).tap do |http|
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
end
end
end
project_id = 'yyyyyy'
project_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
url = "https://airbrake.io/api/v4/projects/#{project_id}/deploys?key=#{project_key}"
body_hash = {
"environment":"production",
"username":"tomas",
"repository":"https://github.com/equivalent/scrapbook2",
"revision":"live-20160905_0001",
"version":"v2.0"
}
puts MakeHttpsRequest.new.call(url, body_hash)
Notes:
in case you doing authentication via Authorisation header set header req['Authorization'] = "Token xxxxxxxxxxxx" or http://api.rubyonrails.org/classes/ActionController/HttpAuthentication/Token.html
A simple json POST request example for those that need it even simpler than what Tom is linking to:
require 'net/http'
uri = URI.parse("http://www.example.com/search.json")
response = Net::HTTP.post_form(uri, {"search" => "Berlin"})
I like this light weight http request client called `unirest'
gem install unirest
usage:
response = Unirest.post "http://httpbin.org/post",
headers:{ "Accept" => "application/json" },
parameters:{ :age => 23, :foo => "bar" }
response.code # Status code
response.headers # Response headers
response.body # Parsed body
response.raw_body # Unparsed body
It's 2020 - nobody should be using Net::HTTP any more and all answers seem to be saying so, use a more high level gem such as Faraday - Github
That said, what I like to do is a wrapper around the HTTP api call,something that's called like
rv = Transporter::FaradayHttp[url, options]
because this allows me to fake HTTP calls without additional dependencies, ie:
if InfoSig.env?(:test) && !(url.to_s =~ /localhost/)
response_body = FakerForTests[url: url, options: options]
else
conn = Faraday::Connection.new url, connection_options
Where the faker looks something like this
I know there are HTTP mocking/stubbing frameworks, but at least when I researched last time they didn't allow me to validate requests efficiently and they were just for HTTP, not for example for raw TCP exchanges, this system allows me to have a unified framework for all API communication.
Assuming you just want to quick&dirty convert a hash to json, send the json to a remote host to test an API and parse response to ruby this is probably fastest way without involving additional gems:
JSON.load `curl -H 'Content-Type:application/json' -H 'Accept:application/json' -X POST localhost:3000/simple_api -d '#{message.to_json}'`
Hopefully this goes without saying, but don't use this in production.
The net/http api can be tough to use.
require "net/http"
uri = URI.parse(uri)
Net::HTTP.new(uri.host, uri.port).start do |client|
request = Net::HTTP::Post.new(uri.path)
request.body = "{}"
request["Content-Type"] = "application/json"
client.request(request)
end
data = {a: {b: [1, 2]}}.to_json
uri = URI 'https://myapp.com/api/v1/resource'
https = Net::HTTP.new uri.host, uri.port
https.use_ssl = true
https.post2 uri.path, data, 'Content-Type' => 'application/json'
Using my favourite http request library in ruby:
resp = HTTP.timeout(connect: 15, read: 30).accept(:json).get('https://units.d8u.us/money/1/USD/GBP/', json: {iAmOne: 'Hash'}).parse
resp.class
=> Hash

Resources