I have a curl request which works
curl -X GET -k 'https://APIADDRESSHERE' -u 'USERNAME:PASSWORD' -H 'Content-Type: application/json'
I can also get this working in ruby:
require 'net/http'
require 'uri'
require 'openssl'
uri = URI.parse("https://APIADDRESSHERE")
request = Net::HTTP::Get.new(uri)
request.basic_auth("USERNAME", "PASSWORD")
request.content_type = "application/json"
req_options = {
use_ssl: uri.scheme == "https",
verify_mode: OpenSSL::SSL::VERIFY_NONE,
}
response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
http.request(request)
end
However when I try this with httparty I keep getting a 401 error saying:
An Authentication object was not found in the SecurityContext This request requires HTTP authentication.
I have tried a lot of variations of the following but I'm stuck.
response = HTTParty.get('https://APIURLHERE', {"headers": { "Authorization:" => "BASE64ENCODEDUSERNAMEANDPASSWORD", "Content-Type" => "application/json" }})
Have you tried the following?
response = HTTParty.get('https://APIURLHERE', headers: { "Authorization:" => "BASE64ENCODEDUSERNAMEANDPASSWORD", "Content-Type" => "application/json" })
Ok I made a simple mistake somewhere along the way because the original solution from the similar post worked... Dont know how it didnt work 50 times last time I was working on this but it was clearly an error on my part somewhere.
How can I add Authorization Bearer to a POST request with Net::HTTP?
I can only find help for "basic authentication" in the documentation.
req.basic_auth 'user', 'pass'
Source: https://docs.ruby-lang.org/en/2.0.0/Net/HTTP.html#class-Net::HTTP-label-Basic+Authentication
I'm trying to replicate a curl that would look like:
> curl 'http://localhost:8080/places' -d '{"_json":[{"uuid":"0514b...",
> "name":"Athens"}]}' -X POST -H 'Content-Type: application/json' -H
> 'Authorization: Bearer eyJ0eXAiO...'
Currently I've gotten to:
require 'net/http'
require 'net/https'
require 'uri'
uri = URI('http://localhost:8080/places')
res = Net::HTTP.post_form(uri, '_json' => [{'uuid': '0514b...', 'name':'Athens'}])
But I'm having trouble figuring out how to add the Authentication: Bearer... part.
Does anyone have experience with this?
I dont think you can add custom headers with the post_form method. You can add it with post method. Try the code below:
uri = URI("http://localhost:8080/places")
params = [{'uuid': '0514b...', 'name':'Athens'}]
headers = {
'Authorization'=>'Bearer foobar',
'Content-Type' =>'application/json',
'Accept'=>'application/json'
}
http = Net::HTTP.new(uri.host, uri.port)
response = http.post(uri.path, params.to_json, headers)
I am trying to scrape data from Yahoo News as much as I can. And because Yahoo is using infinite pagination, I will need to find a way to simulate that behavior on Ruby. I think it is easier to see which URL is called when more news are loaded when scrolling down. Here are what I found:
URL: http://news.yahoo.com/hjsal?m_mode=multipart&site=news®ion=US&lang=en-US&pagetype=minihome
Header: there are many, but only few of them are important. They includes: Referer, Host, Origin, Cookie, Content-Type
Data: there are 4 keys: _json, _mode, _txnid and _crumb
params = {"_json" => [{"_action"=>"show","batch"=>2,"cat"=>"","catName"=>"","sb"=>0,"ccode"=>"grandSlam_news","woeid"=>28379204,"_subAction"=>"more","items"=>[{"u"=>"18fdc929-d35a-3b57-a253-435b0c9fa1e8","i"=>"4000010183"},{"u"=>"52a2961d-441f-34b7-bfa2-60ee2dae1001","i"=>"4000006954"},{"u"=>"0a6e7572-9eda-3c34-9052-ae58fbf009a2","i"=>"4000006504"},{"u"=>"ba9832a7-30a8-3e58-9a82-667c980db5ef","i"=>"4000009705"},{"u"=>"ff37c89f-4146-310c-ac28-68dea7626396","i"=>"4000007878"},{"u"=>"772cd139-7fec-3107-9bf6-e789c51e333c","i"=>"4000009780"},{"u"=>"20598ae3-7581-3c0d-94b3-55f15e0cace9","i"=>"4000007760"},{"u"=>"a98a5581-a0aa-3e4b-9f7b-b6a8ba16ff2b","i"=>"4000006306"},{"u"=>"00d23e7a-9b03-39ed-b669-f612e051d49f","i"=>"4000005248"},{"u"=>"f5c4f020-c608-32ed-aab2-dca6f20e62e1","i"=>"4000005770"},{"u"=>"ba6ac04c-a94c-3eec-a18a-63c98dabc926","i"=>"4000007324"},{"u"=>"f5a82053-132e-3c5d-abff-1026de72addd","i"=>"4000004708"},{"u"=>"86024916-08d3-3162-b158-9947a7eaeb6e","i"=>"4000009721"},{"u"=>"2486b22c-e8fc-3195-8ad9-26c2c4d1ac4b","i"=>"4000006895"},{"u"=>"9fd886b1-0708-3c75-a70c-5384c3bf2d6e","i"=>"4000008879"},{"u"=>"942dea9b-4686-30a9-b59a-95d4c7c69580","i"=>"4000004912"},{"u"=>"88d6a345-cc86-3c42-a975-729852551041","i"=>"4000004711"},{"u"=>"d2021342-2066-3bdc-8d67-2b07b5051888","i"=>"4000007372"},{"u"=>"b2c80259-6ef9-3622-9016-4625f3d7bf67","i"=>"4000005337"},{"u"=>"071008c4-4fe3-3b98-8695-b8e8ff15c599","i"=>"4000003388"}],"listid"=>"","blogtype"=>"","left"=>130,"key"=>"1","cpos"=>23,"prid"=>"evecfmlai6o1o","_container"=>0,"_id"=>"u_30345786","_txnid"=>1428381836200}],
"_mode" => "json",
"_crumb" => "NugJjVjNCWa",
"_txnid" => 1428381836200
}
I got all above info from the Network tab of Developer tool in Chrome. And the corresponding CURL (generated by Developer Tool) for this request is:
curl 'http://news.yahoo.com/hjsal?m_mode=multipart&site=news®ion=US&lang=en-US&pagetype=minihome' -H 'Cookie: ucs=sfcTs=1425628433&sfc=1; AO=u=1; B=b6dveila7ac17&b=4&d=c_ArSSppYEH6SFcrdazmtHDka6k-&s=b6&i=gQmslnXVPGkRceH.mrGk; F=a=RZpcadUMvSym7INq.f8fhle0_OQ0LB4p7I.8J.56OYwNqahr5jVi6DZsH5vjMWANCHUIoF0-&b=U8Fw; PH=fn=muyarBFoN154nhRrjzs-&i=us; DSS=cnt=1&ts=1425628533; ywandp=10001393120079%3A2763408991; fpc=10001393120079%3AZWeFxmga%7C%7C; ugccmt_comment_usortoptv1=highestRated; ywadp115488662=2349330862; fpms=; yvapF=%7B%22cc%22%3A1%2C%22rcc%22%3A1%2C%22vl%22%3A47.129999999999995%2C%22rvl%22%3A47.129999999999995%7D' -H 'Origin: http://news.yahoo.com' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Referer: http://news.yahoo.com/' --data '_crumb=NugJjVjNCWa&_mode=json&_txnid=1428540838640&_json=%5B%7B%22_action%22%3A%22show%22%2C%22batch%22%3A2%2C%22cat%22%3A%22%22%2C%22catName%22%3A%22%22%2C%22sb%22%3A0%2C%22ccode%22%3A%22grandSlam_news%22%2C%22woeid%22%3A28289488%2C%22_subAction%22%3A%22more%22%2C%22items%22%3A%5B%7B%22u%22%3A%220d9b1c7e-bfe6-3d71-b0fc-e31aa6a4ac78%22%2C%22i%22%3A%224000006886%22%7D%2C%7B%22u%22%3A%22d6291cfc-70e2-331d-b202-7d3e89d43275%22%2C%22i%22%3A%224000009863%22%7D%2C%7B%22u%22%3A%22d3c3af9f-cf8f-3d4e-a642-0070528819e0%22%2C%22i%22%3A%224000006408%22%7D%2C%7B%22u%22%3A%22104e5284-2998-3d7c-aaf6-9ceda966699d%22%2C%22i%22%3A%224000008669%22%7D%2C%7B%22u%22%3A%225f5e58cd-4e73-34f0-b1e3-59b8ed0d6dd2%22%2C%22i%22%3A%224000006338%22%7D%2C%7B%22u%22%3A%228f9ce219-79e9-33f8-baa1-1ddea51aed5a%22%2C%22i%22%3A%224000005997%22%7D%2C%7B%22u%22%3A%22683a22bb-e622-31b9-8c1d-d872db4dbf71%22%2C%22i%22%3A%224000007873%22%7D%2C%7B%22u%22%3A%222bd68c78-9645-34cf-8ba4-08ac328ffc60%22%2C%22i%22%3A%224000005723%22%7D%2C%7B%22u%22%3A%2261ed0f69-1e95-3264-baae-0d635606c9c4%22%2C%22i%22%3A%224000006596%22%7D%2C%7B%22u%22%3A%22e86ec903-edf8-3f46-a27a-4c10cdee8f19%22%2C%22i%22%3A%224000007547%22%7D%2C%7B%22u%22%3A%22d4f5eaea-3d28-36de-89a3-b9bb3fc79834%22%2C%22i%22%3A%224000009036%22%7D%2C%7B%22u%22%3A%228a05f4f4-e162-3586-96f1-0b7212a1ebd9%22%2C%22i%22%3A%224000006790%22%7D%2C%7B%22u%22%3A%22b49da9d0-a9ec-3c49-afbe-da1da9c3dfc5%22%2C%22i%22%3A%224000007305%22%7D%2C%7B%22u%22%3A%22f5fe6bcf-77fb-3e9e-bd56-a62027d604f1%22%2C%22i%22%3A%224000005595%22%7D%2C%7B%22u%22%3A%22b644c626-a6a4-3ab7-9d5d-737ee62fa492%22%2C%22i%22%3A%224000007357%22%7D%2C%7B%22u%22%3A%22bfa50721-be1b-38e2-94d1-e8728d112b0c%22%2C%22i%22%3A%224000008326%22%7D%2C%7B%22u%22%3A%22b9e49778-e5dd-3d84-9eb4-6a2681d81a93%22%2C%22i%22%3A%224000006213%22%7D%2C%7B%22u%22%3A%226073c345-05da-3fba-a6d0-f8699fe0e961%22%2C%22i%22%3A%224000005885%22%7D%2C%7B%22u%22%3A%22104a6bec-1244-39f8-ab7c-0586a1a5bd6d%22%2C%22i%22%3A%224000007645%22%7D%2C%7B%22u%22%3A%221280a6da-4b77-3543-8ded-ee62fcef6f9e%22%2C%22i%22%3A%224000007605%22%7D%5D%2C%22listid%22%3A%22%22%2C%22blogtype%22%3A%22%22%2C%22left%22%3A130%2C%22key%22%3A%221%22%2C%22cpos%22%3A24%2C%22prid%22%3A%223mmkn1daibjbe%22%2C%22_container%22%3A0%2C%22_id%22%3A%22u_30345786%22%2C%22_txnid%22%3A1428540838640%7D%5D' --compressed
Based on this CURL request, I have come up with the following Ruby code to crawl the data:
require 'net/http'
Require 'json'
params = {"_json" => URI.encode([{"_action"=>"show","batch"=>2,"cat"=>"","catName"=>"","sb"=>0,"ccode"=>"grandSlam_news","woeid"=>28379204,"_subAction"=>"more","items"=>[{"u"=>"18fdc929-d35a-3b57-a253-435b0c9fa1e8","i"=>"4000010183"},{"u"=>"52a2961d-441f-34b7-bfa2-60ee2dae1001","i"=>"4000006954"},{"u"=>"0a6e7572-9eda-3c34-9052-ae58fbf009a2","i"=>"4000006504"},{"u"=>"ba9832a7-30a8-3e58-9a82-667c980db5ef","i"=>"4000009705"},{"u"=>"ff37c89f-4146-310c-ac28-68dea7626396","i"=>"4000007878"},{"u"=>"772cd139-7fec-3107-9bf6-e789c51e333c","i"=>"4000009780"},{"u"=>"20598ae3-7581-3c0d-94b3-55f15e0cace9","i"=>"4000007760"},{"u"=>"a98a5581-a0aa-3e4b-9f7b-b6a8ba16ff2b","i"=>"4000006306"},{"u"=>"00d23e7a-9b03-39ed-b669-f612e051d49f","i"=>"4000005248"},{"u"=>"f5c4f020-c608-32ed-aab2-dca6f20e62e1","i"=>"4000005770"},{"u"=>"ba6ac04c-a94c-3eec-a18a-63c98dabc926","i"=>"4000007324"},{"u"=>"f5a82053-132e-3c5d-abff-1026de72addd","i"=>"4000004708"},{"u"=>"86024916-08d3-3162-b158-9947a7eaeb6e","i"=>"4000009721"},{"u"=>"2486b22c-e8fc-3195-8ad9-26c2c4d1ac4b","i"=>"4000006895"},{"u"=>"9fd886b1-0708-3c75-a70c-5384c3bf2d6e","i"=>"4000008879"},{"u"=>"942dea9b-4686-30a9-b59a-95d4c7c69580","i"=>"4000004912"},{"u"=>"88d6a345-cc86-3c42-a975-729852551041","i"=>"4000004711"},{"u"=>"d2021342-2066-3bdc-8d67-2b07b5051888","i"=>"4000007372"},{"u"=>"b2c80259-6ef9-3622-9016-4625f3d7bf67","i"=>"4000005337"},{"u"=>"071008c4-4fe3-3b98-8695-b8e8ff15c599","i"=>"4000003388"}],"listid"=>"","blogtype"=>"","left"=>130,"key"=>"1","cpos"=>23,"prid"=>"evecfmlai6o1o","_container"=>0,"_id"=>"u_30345786","_txnid"=>1428381836200}].to_json),
"_mode" => "json",
"_crumb" => "NugJjVjNCWa",
"_txnid" => 1428381836200
}
cookie = 'ucs=sfcTs=1425628433&sfc=1; AO=u=1; B=b6dveila7ac17&b=4&d=c_ArSSppYEH6SFcrdazmtHDka6k-&s=b6&i=gQmslnXVPGkRceH.mrGk; F=a=RZpcadUMvSym7INq.f8fhle0_OQ0LB4p7I.8J.56OYwNqahr5jVi6DZsH5vjMWANCHUIoF0-&b=U8Fw; PH=fn=muyarBFoN154nhRrjzs-&i=us; DSS=cnt=1&ts=1425628533; ywandp=10001393120079%3A2763408991; fpc=10001393120079%3AZWeFxmga%7C%7C; ugccmt_comment_usortoptv1=highestRated; ywadp115488662=2349330862; fpms=; yvapF=%7B%22cc%22%3A1%2C%22rcc%22%3A1%2C%22vl%22%3A47.129999999999995%2C%22rvl%22%3A47.129999999999995%7D'
uri = URI('http://news.yahoo.com/hjsal?m_mode=multipart&site=news®ion=US&lang=en-US&pagetype=minihome')
req = Net::HTTP::Post.new(uri.path)
req.add_field('Referer', 'http://news.yahoo.com/')
req.add_field('Origin', 'http://news.yahoo.com')
req.add_field('Host', 'news.yahoo.com')
req.add_field('Cookie', cookie)
req.add_field('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8')
req.body = URI.encode(params.to_json)
http = Net::HTTP.new(uri.host,uri.port)
res = http.request(req)
p res.body
But it always returns error message in response when running above Ruby code (the CURL request works perfectly). I am not quite sure what is wrong here and I do hope you guys can give me some hints to resolve this issue.
The response (this error is the same if you copy the URL and paste it on your browser):
<div class=\"oops-msg\" role=\"alertdialog\">\n<span class=\"icon icon-error y-glbl-universal\"></span>\n<h3 class=\"oops\">The module encountered a problem while trying to load</h3>\n <p class=\"oops\">App is currently not available. Please, try again.</p>\n \n </div><!-- hdf50.fp.gq1.yahoo.com compressed/chunked Wed Apr 15 08:15:39 UTC 2015 -->\n
Also, I can not even simulate this request in Postman Client(new version with Interceptor enabled).
Thank you very much
Your Ruby code is specifying a Post request but your curl is not. Try replacing this line:
req = Net::HTTP::Post.new(uri.path)
with this:
req = Net::HTTP::Get.new(uri.path)
I'm trying to pull data from a RESTful JSON web service which uses https. They provided a Curl example which works no problem; however, I'm trying to run the query in Ruby and I'm not a Ruby developer.
Any help much appreciated!
cURL example:
curl -G "https://api.example.com/v1/query/" \
-H "Accept: application/vnd.example.v1+hal+json" \
-u "$API_KEY:$API_SECRET" \
-d "app_id=$APP_ID" \
-d "days=3" \
-d "metrics=users" \
-d "dimensions=day"
My attempt in Ruby which is resulting in a HTTPUnauthorized 401:
require 'net/https'
require 'uri'
# prepare request
uri = URI.parse("https://api.example.com/v1/query/")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Get.new(uri.request_uri, {
'Accept' => 'application/vnd.example.v1+hal+json',
'api_key' => 'api_secret',
'app_id' => 'app_id',
'days' => '3',
'metrics' => 'users',
'dimensions' => 'day'})
response = http.request(request)
response.body
response.status
response["header-here"] # All headers are lowercase
# Analyze the response
if response.code != "200"
puts "Error (status-code: #{response.code})\n#{response.body}"
print 0
else
print 1
end
** Update **
As per feedback below, I've installed Typhoeus and updated the request. Now I'm getting through. Thanks all!
request = Typhoeus::Request.new(
"https://api.example.com/v1/query/",
userpwd: "key:secret",
params: {
app_id: "appid",
days: "3",
metrics: "users",
dimensions: "day"
},
headers: {
Accept: "application/vnd.example.v1+hal+json"
}
)
First you need to realize that:
'Accept' => 'application/vnd.example.v1+hal+json'
is a header, not a parameter.
Also:
$API_KEY:$API_SECRET
is basic HTTP authentication, not a parameter.
Then, take my advice and go with a better Ruby HTTP client:
https://github.com/lostisland/faraday (preferred, a wrapper for the bellow)
https://github.com/typhoeus/typhoeus
https://github.com/geemus/excon
Update:
Try the following from IRB:
Typhoeus::Config.verbose = true # this is useful for debugging, remove it once everything is ok.
request = Typhoeus::Request.get(
"https://api.example.com/v1/query/",
userpwd: "key:secret",
params: {
app_id: "appid",
days: "3",
metrics: "users",
dimensions: "day"
},
headers: {
Accept: "application/vnd.example.v1+hal+json"
}
)
curl's -u sends the Authorization header. so your 'api_key' => 'api_secret', should be replaced with this one(once again, its http header, not parameter).
"Authorization" => "Basic ".Base64.encode64("api_key:api_secret")
## require "base64"
I'm pretty new in Ruby and Rails.
I want to send a HTTP POST request in my rails application, the request can be invoked by command line like:
curl -X POST -u "username:password" \
-H "Content-Type: application/json" \
--data '{"device_tokens": ["0C676037F5FE3194F11709B"], "aps": {"alert": "Hello!"}}' \
https://go.urbanairship.com/api/push/
The ruby code I wrote (actually it's glue code) is:
uri = URI('https://go.urbanairship.com/api/push')
Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
request = Net::HTTP::Post.new(uri.request_uri, initheader = {'Content-Type' =>'application/json'})
request.basic_auth 'username', 'password'
request.body = ActiveSupport::JSON.encode({'device_tokens' => ["4872AAB82341AEE600C6E219AA93BB38B5144176037F2056D65FE3194F11709B"], "aps" => {"alert" => "Hello!"}})
response = http.request request # Net::HTTPResponse object
puts response.body
end
However, running the ruby code in Rails Console didn't give me expected result (the command line does). Can someone give me a hand? I've tried searching relevant posts and Ruby docs, however my knowledge in Ruby is not good enough to solve it.
require 'net/http'
require 'net/https'
https = Net::HTTP.new('go.urbanairship.com', 443)
https.use_ssl = true
path = '/api/push'
It's often tidier to create a little client class. I like HTTParty for that:
require 'httparty'
class UAS
include HTTParty
base_uri "https://go.urbanairship.com"
basic_auth 'username', 'password'
default_params :output => 'json'
#token = "4872AAB82341AEE600C6E219AA93BB38B5144176037F2056D65FE3194F11709B"
def self.alert(message)
post('/api/push/', {'device_tokens' => #token, 'aps' => {"alert" => message}})
end
end
Then you use it like so:
UAS.alert('Hello!')