I am trying to use the NET:HTTP gem to add an api-key to http header of a client, but it just doesn't seem to be working for some reason when I try and test it out.Basically the server requires the http header of the client or anything to have http_x_api header in order to serve the request.
Server code
require 'sinatra'
before do
halt 400 if (env['API_KEY']) != 'wow'
end
get '/' do
"boo"
end
Client code
require 'net/http'
require 'uri'
port = ENV['PORT'] || '7474'
res = Net::HTTP.start('localhost', port ) { |h| h.get('/')}
res.add_field('api-key', 'wow')
res.each_header do |key, value|
p "#{key} => #{value}"
end
puts (res.code == '200' && res.body == 'boo') ? 'OK' : 'FAIL'
this the response i get back :=>
"x-frame-options => sameorigin"
"x-xss-protection => 1; mode=block"
"content-type => text/html;charset=utf-8"
"content-length => 0"
"connection => keep-alive"
"server => thin 1.5.0 codename Knife"
"api-key => wow"
FAIL
On the server, the HTTP header variables in env are prefixed with HTTP_, so you need to check env['HTTP_API_KEY']. From the documentation:
HTTP_ Variables: Variables corresponding to the client-supplied HTTP request headers (i.e., variables whose names begin with HTTP_). The presence or absence of these variables should correspond with the presence or absence of the appropriate HTTP header in the request.
Related
How to enforce Faraday adapter typhoeus to use HTTP/2 for requests to servers which supported HTTP/2?
I have tested this over service https://http2.pro/doc/api and result was like this:
body="{\"http2\":1,\"protocol\":\"HTTP\\/2.0\",\"push\":0,\"user_agent\":\"Faraday v0.12.2\"}",
\"http2\":1, what means that HTTP/2 not used for request!
There are two things at play here. The first is that the remote API is lying to you in the response body. Their documentation says:
http2: Possible values are 0 (HTTP/2 was used) and 1 (HTTP/2 was not used).
Even though the response body shows 'http2': 1 indicating that HTTP2 was not used, it is being used. You can most easily confirm this using Chrome's dev tools:
So once we know that the API is lying in the response body, how can we independently confirm that Typhoeus is using HTTP2?
(this answer assumes you are using pry as your REPL, not IRB)
First let's confirm that Typhoeus alone will use HTTP2:
require 'typhoeus'
response = Typhoeus.get("https://http2.pro/api/v1", http_version: :httpv2_0)
response.class
=> Typhoeus::Response < Object
response.body
=> "{\"http2\":1,\"protocol\":\"HTTP\\/2.0\",\"push\":0,\"user_agent\":\"Typhoeus - https:\\/\\/github.com\\/typhoeus\\/typhoeus\"}" # this is the lying API response
response.http_version
=> "2" # this is what Typhoeus tells us was actually used
Now let's test it in Faraday:
require 'faraday'
require 'typhoeus'
require 'typhoeus/adapters/faraday'
conn = Faraday.new do |faraday|
faraday.adapter :typhoeus, http_version: :httpv2_0
end
response = conn.get("https://http2.pro/api/v1")
response.body
=> "{\"http2\":1,\"protocol\":\"HTTP\\/2.0\",\"push\":0,\"user_agent\":\"Faraday v0.17.0\"}" # again we get the lying API response
But how can we confirm it was HTTP2? This doesn't work:
response.http_version
NoMethodError: undefined method `http_version' for #<Faraday::Response:0x00007f99935519a8>
Because response isn't a Typhoeus::Response object, it's a Faraday object:
response.class
=> Faraday::Response < Object
So we need to get into the gem itself to figure out where it's creating the Typhoeus::Response object so we can call .http_version on it manually and confirm it's using the protocol we expect. As it turns out, that's right here.
Let's take the easy route and stick binding.pry into our local copy of the gem (you'll need to restart pry to pick up the changes to the gem):
def typhoeus_request(env)
opts = {
:method => env[:method],
:body => env[:body],
:headers => env[:request_headers]
}.merge(#adapter_options)
binding.pry
::Typhoeus::Request.new(env[:url].to_s, opts)
end
Then re-run the request:
require 'faraday'
require 'typhoeus'
require 'typhoeus/adapters/faraday'
conn = Faraday.new do |faraday|
faraday.adapter :typhoeus, http_version: :httpv2_0
end
response = conn.get("https://http2.pro/api/v1")
And you'll see:
Frame number: 0/3
From: /Users/foo/.rvm/gems/ruby-2.6.3/gems/typhoeus-1.3.1/lib/typhoeus/adapters/faraday.rb # line 127 Faraday::Adapter::Typhoeus#typhoeus_request:
120: def typhoeus_request(env)
121: opts = {
122: :method => env[:method],
123: :body => env[:body],
124: :headers => env[:request_headers]
125: }.merge(#adapter_options)
126: binding.pry
=> 127: ::Typhoeus::Request.new(env[:url].to_s, opts)
128: end
Now enter:
response = ::Typhoeus::Request.new(env[:url].to_s, opts).run
And confirm it's a Typhoeus::Response object:
response.class
=> Typhoeus::Response < Object
And confirm it's using HTTP2:
response.http_version
=> "2"
And confirm the API response body is a dirty liar:
response.body
=> "{\"http2\":1,\"protocol\":\"HTTP\\/2.0\",\"push\":0,\"user_agent\":\"Faraday v0.17.0\"}"
And that's how you use Typhoeus as a Faraday adapter to make an HTTP2 request.
The following code works webmock 1.20.4 but not with 2.0.1
stub_request(:get, "http://www.myapi.com/my-endpoint")
.with(headers: {'Authorization' => "Bearer fake_oauth_token"})
.to_return(:body => mock_response)
This is the code I am stubbing.
def get_stuff(oauth_token)
faraday = Faraday.new(:url => "http://www.myapi.com/my-endpoint", :ssl => {verify: false})
response = faraday.get do |req|
req.options[:timeout] = 10
req.headers['Authorization'] = "Bearer #{oauth_token}"
end
if response.status == 200
response.body
else
{error: "failed"}.to_json
end
end
Using assert_requested :get, "http://www.myapi.com/my-endpoint", :headers => {'Authorization' => "Bearer fake_oauth_token"}, :times => 1 and removing the headers from stub_request I get the following output from the assert.
Failure/Error: assert_requested :get, "#{Conf.graphql[:host]}?query=#{graphql_user_details_query}", :headers => headers, :times => 1
The request GET http://www.myapi.com/my-endpoint with headers {'Authorization'=>'Bearer fake_oauth_token'} was expected to execute 1 time but it executed 0 times
The following requests were made:
GET http://www.myapi.com/my-endpoint with headers {'Accept-Encoding'=>'gzip, compressed', 'Authorization'=>'Basic QmVhcmVyIGZha2Vfb2F1dGhfdG9rZW4=', 'User-Agent'=>'Faraday v0.9.2'} was made 1 time
Is there a way to make the stub_request code work with webmock 2?
UPDATE: This issue was fixed in WebMock 2.0.2
The following is now out of date.
WebMock 2.0 was overwriting the Bearer Authorization header with a Basic Autorization header. I have reported the issue on the webmock github page ( https://github.com/bblimke/webmock/issues/617 ). Until the issue is resolved, we are monkey patching to comment out the lines that cause the issue.
We created a file WebMockHttpClient.rb that we require in our spec_helper. This comments out the lines that overwrite the Bearer Authorization header.
require 'em-http-request'
module EventMachine
class WebMockHttpClient
def build_request_signature
headers, body = #req.headers, #req.body
#conn.middleware.select { |m| m.respond_to?(:request) }.each do |m|
headers, body = m.request(self, headers, body)
end
method = #req.method
uri = #req.uri.clone
query = #req.query
uri.query = encode_query(#req.uri, query).slice(/\?(.*)/, 1)
body = form_encode_body(body) if body.is_a?(Hash)
headers = #req.headers
# if headers['authorization']
# headers['Authorization'] = WebMock::Util::Headers.basic_auth_header(headers.delete('authorization'))
# end
WebMock::RequestSignature.new(
method.downcase.to_sym,
uri.to_s,
:body => body || (#req.file && File.read(#req.file)),
:headers => headers
)
end
end
end
With Ruby, my app:
checks if the page status is 200
Parses the PDF files if so
sends via email the result of scraping
Having tested all the parts of the code, everything works fine, except one thing, the mail that is sent doesn't contain the result of my scrpaing;
What is the issue, is it related to the variable #monscrape that may be not recongnised in the final party of the code ?
My code:
require 'open-uri'
require "net/http"
require 'rubygems'
require 'pdf/reader'
require 'mail'
options = { :address => "smtp.gmail.com",
:port => 587,
:domain => 'gmail.com',
:user_name => 'mail#gmail.com',
:password => 'pwd',
:authentication => 'plain',
:enable_starttls_auto => true
}
lien= "http://www.example.com"
url = URI.parse(lien)
req = Net::HTTP.new(url.host, url.port)
res = req.request_head(url.path)
if res.code == "200"
io = open('http://www.example.com')
reader = PDF::Reader.new(io)
reader.pages.each do |page|
res = page.text
#monscrape = res.scan(/text[\s\S]*text/)
end
Mail.defaults do
delivery_method :smtp, options
end
Mail.deliver do
to 'mail#hotmail.com'
from 'Author <mail#gmail.com>'
subject 'testing sendmail'
html_part do
content_type 'text/html; charset=UTF-8'
body '<h1>Please find below the scrape <%= #monscrape %></h1>'
end
end
else
puts "the link doenst work"
end
The problem is the Mail.deliver block is evaluated using instance_eval. Therefore no local instance #variables will be visible to the Mail block.
So #monscrape will always be nil inside the Mail.deliver block.
One solution is to use a local (non-instance) variable instead:
monscrape = "test"
Mail.deliver do
...
body "<h1>Please find below the scrape #{monscrape}</h1>"
...
end
Also note that Mail does not support ERB(!) therefore you cannot use something like <%= monscrape %> in the body. You have to treat it like a normal string using string expansion with double quotes " and not single quotes '.
See further discussion and options here:
Why can't the Mail block see my variable?
You can't use
res = req.request_head(url.path)
when url.path returns "". request_head expects a path of at least "/". That implies you need to fix up the URL being passed so it at least has the root path "/".
url = URI.parse('http://www.example.com')
url.path # => ""
req.request_head(url.path)
*** ArgumentError Exception: HTTP request path is empty
vs.
url = URI.parse('http://www.example.com/')
url.path # => "/"
req.request_head(url.path)
#<Net::HTTPOK 200 OK readbody=true>
The second problem is you're trying to read something as PDF that isn't a PDF file. Example.com returns HTML, which is text. You can't use:
io = open('http://www.example.com')
reader = PDF::Reader.new(io)
Trying to returns "PDF does not contain EOF marker".
It's really important that you understand what types of objects/resources are being returned by a site when you request a URL. You can't declare them willy-nilly and expect code to accept it without errors.
I understand that you could use proxy in the ruby Net::HTTP. However, I have no idea how to do this with a bunch of proxy. I need the Net::HTTP to change to another proxy and send another post request after every post request. Also, is it possible to make the Net::HTTP to change to another proxy if the previous proxy is not working? If so, how?
Code I'm trying to implement the script in:
require 'net/http'
sleep(8)
http = Net::HTTP.new('URLHERE', 80)
http.read_timeout = 5000
http.use_ssl = false
path = 'PATHHERE'
data = '(DATAHERE)'
headers = {
'Referer' => 'REFERER HERE',
'Content-Type' => 'application/x-www-form-urlencoded; charset=UTF-8',
'User-Agent' => '(USERAGENTHERE)'}
resp, data = http.post(path, data, headers)
# Output on the screen -> we should get either a 302 redirect (after a successful login) or an error page
puts 'Code = ' + resp.code
puts 'Message = ' + resp.message
resp.each {|key, val| puts key + ' = ' + val}
puts data
end
Given an array of proxies, the following example will make a request through each proxy in the array until it receives a "302 Found" response. (This isn't actually a working example because Google doesn't accept POST requests, but it should work if you insert your own destination and working proxies.)
require 'net/http'
destination = URI.parse "http://www.google.com/search"
proxies = [
"http://proxy-example-1.net:8080",
"http://proxy-example-2.net:8080",
"http://proxy-example-3.net:8080"
]
# Create your POST request_object once
request_object = Net::HTTP::Post.new(destination.request_uri)
request_object.set_form_data({"q" => "stack overflow"})
proxies.each do |raw_proxy|
proxy = URI.parse raw_proxy
# Create a new http_object for each new proxy
http_object = Net::HTTP.new(destination.host, destination.port, proxy.host, proxy.port)
# Make the request
response = http_object.request(request_object)
# If we get a 302, report it and break
if response.code == "302"
puts "#{proxy.host}:#{proxy.port} responded with #{response.code} #{response.message}"
break
end
end
You should also probably do some error checking with begin ... rescue ... end each time you make a request. If you don't do any error checking and a proxy is down, control will never reach the line that checks for response.code == "302" -- the program will just fail with some type of connection timeout error.
See the Net::HTTPHeader docs for other methods that can be used to customize the Net::HTTP::Post object.
I know some languages have a library that allows you to get the HTTP content for a 404 or 500 message.
Is there a library that allows that for Ruby?
I've tried open-uri but it simply returns an HTTPError exception without the HTML content for the 404 response.
This doesn't seem to be stated clearly enough in the docs, but HttpError has an io attribute, which you can treat as a read only file as far as i know.
require 'open-uri'
begin
response = open('http://google.com/blahblah')
rescue => e
puts e # Error message
puts e.io.status # Http Error code
puts e.io.readlines # Http response body
end
Net::HTTP supports what you need.
You can use the request_get method and it will return a response regardless of the status code.
From script/console:
> http = Net::HTTP.new('localhost', 3000)
=> #<Net::HTTP localhost:3000 open=false>
> resp = http.request_get('/foo') # a page that doesn't exist
=> #<Net::HTTPNotFound 404 Not Found readbody=true>
> resp.code
=> "404"
> resp.body
=> "<html>...</html>"
(If the library is not available to you by default, you can do a require 'net/http'
Works with HTTParty as well https://github.com/jnunemaker/httparty
require 'rubygems'
require 'httparty'
HTTParty.get("http://google.com/blahblah").parsed_response
There are a number of HTTP Clients available, choose one you like from https://www.ruby-toolbox.com/categories/http_clients