I have methods that GETs a resource from an external Rails server via JSON. One uses the net/http library, the other executes curl. They respond differently.
If I use net/http
def get_via_nethttp(endpoint, user, password, id)
uri = URI.parse("#{endpoint}/items/#{id}.json")
http = Net::HTTP.new(uri.host, uri.port)
headers = {
'Content-Type' => 'application/json',
'Accept-Encoding' => 'gzip,deflate',
'Accept' => 'application/json'
}
request = Net::HTTP::Get.new(uri.request_uri, headers)
request.basic_auth(user, password)
http.request(request)
end
then the return code is ASCII-8Bit encoded. Here's a snippet:
\x1F\x8B\b\x00\x00\x00\x00\x00\x00\x03\xBDWmo\xE28\x10\xFE+Vv\xA5...
However, if I use curl
def get_via_curl(endpoint, user, password, id)
%x(curl -u #{user}:#{password} #{endpoint}/items/#{id}.json)
end
then a JSON string returns
"{\"id\":1234,\"title\":\"Foo\"}"
I've tried numerous ways to decode the output of net/http to no avail. In the end, I just want the JSON string, so I ended up going with curl. What's the difference that's causing this?
Setting in headers 'Accept-Encoding' => 'gzip,deflate' can result server sending you a chunked response. Just remove it.
Related
I've been using the code below to call a third party API . This code works fine (i've changed the url and the credentials but the structure of the code is the same) :
require 'base64'
require 'httparty'
require 'json'
######################################################################
# Get the token first
######################################################################
consumer_key = "my_key"
consumer_secret = "my_secret"
credentials = Base64.encode64("#{consumer_key}:#{consumer_secret}").gsub("\n", '')
url = "https://mysite/token"
body = "grant_type=client_credentials"
headers = {
"Authorization" => "Basic #{credentials}",
"Content-Type" => "application/x-www-form-urlencoded;charset=UTF-8"
}
r = HTTParty.post(url, body: body, headers: headers)
bearer_token = JSON.parse(r.body)['access_token']
######################################################################
# Use the token in a call as authorisation header
######################################################################
api_url = "https://apisite/the_value_i_am_looking_for_in_the_api"
url = URI.parse(api_url)
req = Net::HTTP.new(url.host, url.port)
req.use_ssl = true
# If we are just passing a key that doesn't need to be in the token format
headers = {
'Authorization' => "Bearer #{bearer_token}"
}
# Get the response back (he data is in the response body: resp.body )
resp = req.get(url, headers)
My issue is that the API providers have changed their API so you now need to pass an "accept" into the call via the header. I used POSTMAN to make the call, added the accept to the header and was able to get it working without issue. So far so good.
I then changed my ruby code to extend the headers section to include the accept, using the code below:
headers = {
'Authorization' => "Bearer #{bearer_token}",
'Accept' => 'application/vnd.bluebadge-api.v1+json'
}
I've not added an accept to a header before so I may have gotten the syntax wrong.
However, this returns an unauthorised 401 response code:
#<Net::HTTPUnauthorized 401 Unauthorized readbody=true>
I thought I might have the credentials wrong so remove the accept, try again and this changes to a 406 response code:
#<Net::HTTPNotAcceptable 406 Not Acceptable readbody=true>
If I examine the response I get the message I would expect that the accept header is not the supported version. So I know the credentials are correct (and the fact they match the postman credentials which works):
"{\"apiVersion\": \"1\",\"context\": null,\"id\": null,\"method\": null,\"error\": {\"code\": null,\"message\": null,\"reason\": \"Accept header version is not a currently supported version.\",\"errors\": null}}\n"
So I know all my credentials are correct because I've copied them into the postman request which works with no errors. The value for the accept header is correct because I copied that from a working postman request too.
I am at a loss for why this wouldn't work.
I've looked through the NET HTTP library and cant find anything to help me there. I've seen a couple of posts elsewhere which I've tried and they haven't worked either.
I appreciate any help in trying to solve this.
Found the problem. I was using the credentials from the production environment to get the token then trying to query the test environment API. In my defence they look very similar (only 3 characters different). I think I had a case of the code blindness.
The code I posted does work when I put the correct URL for the environments.
I also found that I could use this:
uri = URI.parse("https://myapi/some_text")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Get.new(uri.request_uri)
request["Authorization"] = "Bearer #{bearer_token}"
request["Accept"] = "application/vnd.bluebadge-api.v1+json"
response = http.request(request)
Or using HTTParty like this:
response = HTTParty.get('https://myapi/some_text', {
headers: {"Authorization" => "Bearer #{bearer_token}", "Accept" => "application/vnd.bluebadge-api.v1+json" }
})
I would prefer the format of my orginal code or the HTTparty code because it is easy to see from the code that you're passing headers. Hopefully this will help others to double check their authorization credentials...
I am trying to use ruby with a website's api. The instructions are to send a GET request with a header. These are the instructions from the website and the example php code they give. I am to calculate a HMAC hash and include it under an apisign header.
$apikey='xxx';
$apisecret='xxx';
$nonce=time();
$uri='https://bittrex.com/api/v1.1/market/getopenorders?apikey='.$apikey.'&nonce='.$nonce;
$sign=hash_hmac('sha512',$uri,$apisecret);
$ch = curl_init($uri);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('apisign:'.$sign));
$execResult = curl_exec($ch);
$obj = json_decode($execResult);
I am simply using an .rb file with ruby installed on windows from command prompt. I am using net/http in the ruby file. How can I send a GET request with a header and print the response?
Using net/http as suggested by the question.
References:
Net::HTTP https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP.html
Net::HTTP::get https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP.html#method-c-get
Setting headers: https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP.html#class-Net::HTTP-label-Setting+Headers
Net::HTTP::Get https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP/Get.html
Net::HTTPGenericRequest https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTPGenericRequest.html and Net::HTTPHeader https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTPHeader.html (for methods that you can call on Net::HTTP::Get)
So, for example:
require 'net/http'
uri = URI("http://www.ruby-lang.org")
req = Net::HTTP::Get.new(uri)
req['some_header'] = "some_val"
res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == 'https') { |http|
http.request(req)
}
puts res.body # <!DOCTYPE html> ... </html> => nil
Note: if your response has HTTP result state 301 (Moved permanently), see Ruby Net::HTTP - following 301 redirects
Install httparty gem, it makes requests way easier, then in your script
require 'httparty'
url = 'http://someexample.com'
headers = {
key1: 'value1',
key2: 'value2'
}
response = HTTParty.get(url, headers: headers)
puts response.body
then run your .rb file..
As of Ruby 3.0, Net::HTTP.get_response supports an optional hash for headers:
Net::HTTP.get_response(URI('http://www.example.com/index.html'), { 'Accept' => 'text/html' })
Unfortunately this does not work for Ruby 2 (up to 2.7).
This is the curl request I'd like to perform:
curl --request GET --url 'https://us8.api.mailchimp.com/3.0/' --user 'anystring:<apikey>'
Running this curl command from the command line, I get a bunch of JSON, which is what I want.
I'm trying to perform this same request using Faraday. This is what I've tried:
conn = Faraday.new "https://us8.api.mailchimp.com/3.0/" do |faraday|
faraday.adapter Faraday.default_adapter
end
conn.basic_auth('apikey', <apikey>)
response = conn.get
puts response.body # => "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>301 Moved Permanently</title>\n</head>...
How do I get JSON instead of the html I'm seeing in response.body?
This ended up working for me:
conn = Faraday.new('https://us8.api.mailchimp.com/3.0/') do |c|
c.use FaradayMiddleware::ParseJson, content_type: "application/json"
c.use Faraday::Adapter::NetHttp
end
conn.basic_auth('apikey', <api_key>)
response = conn.get('campaigns')
puts response # => json blob
A much more concise option is to manually encode your credentials into Base64 and set the Authorization header manually. This saves a few lines of code and keeps it in a similar format to regular Faraday requests (i.e. Faraday.post <URL>, <BODY>, <HEADERS>).
i.e.
encoded_credentials = Base64.encode64('apikey:api_secret').chomp
Faraday.post 'http://myhost.local/my_url', {}, authorization: "Basic #{encoded_credentials}"
This is the farday code for curl operations using ruby gem.
def contacts
response = JSON.parse(params)
#conn = Faraday.new('https://domain.example.com/api/v2/') do |farday|
farday.use FaradayMiddleware::ParseJson, content_type: "application/json"
farday.use Faraday::Adapter::NetHttp
end
#conn.basic_auth('apikey', <your_api_key>)
response = #conn.get('contacts')
puts response
render :json => {"success" => true}
end
So here's the request using curl:
curl -XPOST -H content-type:application/json -d "{\"credentials\":{\"username\":\"username\",\"key\":\"key\"}}" https://auth.api.rackspacecloud.com/v1.1/auth
I've been trying to make this same request using ruby, but I can't seem to get it to work.
I tried a couple of libraries also, but I can't get it to work.
Here's what I have so far:
uri = URI.parse("https://auth.api.rackspacecloud.com")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new("/v1.1/auth")
request.set_form_data({'credentials' => {'username' => 'username', 'key' => 'key'}})
response = http.request(request)
I get a 415 unsupported media type error.
You are close, but not quite there. Try something like this instead:
uri = URI.parse("https://auth.api.rackspacecloud.com")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new("/v1.1/auth")
request.add_field('Content-Type', 'application/json')
request.body = {'credentials' => {'username' => 'username', 'key' => 'key'}}.to_json
response = http.request(request)
This will set the Content-Type header as well as post the JSON in the body, rather than in the form data as your code had it. With the sample credentials, it still fails, but I suspect it should work with real data in there.
There's a very good explanation of how to make a JSON POST request with Net::HTTP at this link.
I would recommend using a library like HTTParty. It's well-documented, you can just set up your class like so:
class RackSpaceClient
include HTTParty
base_uri "https://auth.api.rackspacecloud.com/"
format :json
headers 'Accept' => 'application/json'
#methods to do whatever
end
It looks like the main difference between the Ruby code you placed there, and the curl request, is that the curl request is POSTing JSON (content-type application/json) to the endpoint, whereas request.set_form_data is going to send a form in the body of the POST request (content-type application/x-www-form-urlencoded). You have to make sure the content going both ways is of type application/json.
All others are too long here is a ONE LINER:
Net::HTTP.start('auth.api.rackspacecloud.com', :use_ssl => true).post(
'/v1.1/auth', {:credentials => {:username => "username",:key => "key"}}.to_json,
initheader={'Content-Type' => 'application/json'}
)
* to_json needs require 'json'
OR if you want to
NOT verify the hosts
be more readable
ensure the connection is closed once you're done
then:
ssl_opts={:use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE}
Net::HTTP.start('auth.api.rackspacecloud.com', ssl_opts) { |secure_connection|
secure_connection.post(
'/v1.1/auth', {:credentials => {:username => "username",:key => "key"}}.to_json,
initheader={'Content-Type' => 'application/json'}
)
}
In case it's tough to remember what params go where:
SSL options are per connection so you specify them while opening the connection.
You can reuse the connection for multiple REST calls to same base url. Think of thread safety of course.
Header is a "request header" and hence specified per request. I.e. in calls to get/post/patch/....
HTTP.start(): Creates a new Net::HTTP object, then additionally opens the TCP connection and HTTP session.
HTTP.new(): Creates a new Net::HTTP object without opening a TCP connection or HTTP session.
Another example:
#!/usr/bin/ruby
require 'net/http'
require 'json'
require 'uri'
full_url = "http://" + options[:artifactory_url] + "/" + "api/build/promote/" + options[:build]
puts "Artifactory url: #{full_url}"
data = {
status: "staged",
comment: "Tested on all target platforms.",
ciUser: "builder",
#timestamp: "ISO8601",
dryRun: false,
targetRepo: "#{options[:target]}",
copy: true,
artifacts: true,
dependencies: false,
failFast: true,
}
uri = URI.parse(full_url)
headers = {'Content-Type' => "application/json", 'Accept-Encoding'=> "gzip,deflate",'Accept' => "application/json" }
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.request_uri, headers)
request.basic_auth(options[:user], options[:password])
request.body = data.to_json
response = http.request(request)
puts response.code
puts response.body
So here's the request using curl:
curl -XPOST -H content-type:application/json -d "{\"credentials\":{\"username\":\"username\",\"key\":\"key\"}}" https://auth.api.rackspacecloud.com/v1.1/auth
I've been trying to make this same request using ruby, but I can't seem to get it to work.
I tried a couple of libraries also, but I can't get it to work.
Here's what I have so far:
uri = URI.parse("https://auth.api.rackspacecloud.com")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new("/v1.1/auth")
request.set_form_data({'credentials' => {'username' => 'username', 'key' => 'key'}})
response = http.request(request)
I get a 415 unsupported media type error.
You are close, but not quite there. Try something like this instead:
uri = URI.parse("https://auth.api.rackspacecloud.com")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new("/v1.1/auth")
request.add_field('Content-Type', 'application/json')
request.body = {'credentials' => {'username' => 'username', 'key' => 'key'}}.to_json
response = http.request(request)
This will set the Content-Type header as well as post the JSON in the body, rather than in the form data as your code had it. With the sample credentials, it still fails, but I suspect it should work with real data in there.
There's a very good explanation of how to make a JSON POST request with Net::HTTP at this link.
I would recommend using a library like HTTParty. It's well-documented, you can just set up your class like so:
class RackSpaceClient
include HTTParty
base_uri "https://auth.api.rackspacecloud.com/"
format :json
headers 'Accept' => 'application/json'
#methods to do whatever
end
It looks like the main difference between the Ruby code you placed there, and the curl request, is that the curl request is POSTing JSON (content-type application/json) to the endpoint, whereas request.set_form_data is going to send a form in the body of the POST request (content-type application/x-www-form-urlencoded). You have to make sure the content going both ways is of type application/json.
All others are too long here is a ONE LINER:
Net::HTTP.start('auth.api.rackspacecloud.com', :use_ssl => true).post(
'/v1.1/auth', {:credentials => {:username => "username",:key => "key"}}.to_json,
initheader={'Content-Type' => 'application/json'}
)
* to_json needs require 'json'
OR if you want to
NOT verify the hosts
be more readable
ensure the connection is closed once you're done
then:
ssl_opts={:use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE}
Net::HTTP.start('auth.api.rackspacecloud.com', ssl_opts) { |secure_connection|
secure_connection.post(
'/v1.1/auth', {:credentials => {:username => "username",:key => "key"}}.to_json,
initheader={'Content-Type' => 'application/json'}
)
}
In case it's tough to remember what params go where:
SSL options are per connection so you specify them while opening the connection.
You can reuse the connection for multiple REST calls to same base url. Think of thread safety of course.
Header is a "request header" and hence specified per request. I.e. in calls to get/post/patch/....
HTTP.start(): Creates a new Net::HTTP object, then additionally opens the TCP connection and HTTP session.
HTTP.new(): Creates a new Net::HTTP object without opening a TCP connection or HTTP session.
Another example:
#!/usr/bin/ruby
require 'net/http'
require 'json'
require 'uri'
full_url = "http://" + options[:artifactory_url] + "/" + "api/build/promote/" + options[:build]
puts "Artifactory url: #{full_url}"
data = {
status: "staged",
comment: "Tested on all target platforms.",
ciUser: "builder",
#timestamp: "ISO8601",
dryRun: false,
targetRepo: "#{options[:target]}",
copy: true,
artifacts: true,
dependencies: false,
failFast: true,
}
uri = URI.parse(full_url)
headers = {'Content-Type' => "application/json", 'Accept-Encoding'=> "gzip,deflate",'Accept' => "application/json" }
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.request_uri, headers)
request.basic_auth(options[:user], options[:password])
request.body = data.to_json
response = http.request(request)
puts response.code
puts response.body