Can not convert CURL to Ruby request properly - ruby
I am trying to scrape data from Yahoo News as much as I can. And because Yahoo is using infinite pagination, I will need to find a way to simulate that behavior on Ruby. I think it is easier to see which URL is called when more news are loaded when scrolling down. Here are what I found:
URL: http://news.yahoo.com/hjsal?m_mode=multipart&site=news®ion=US&lang=en-US&pagetype=minihome
Header: there are many, but only few of them are important. They includes: Referer, Host, Origin, Cookie, Content-Type
Data: there are 4 keys: _json, _mode, _txnid and _crumb
params = {"_json" => [{"_action"=>"show","batch"=>2,"cat"=>"","catName"=>"","sb"=>0,"ccode"=>"grandSlam_news","woeid"=>28379204,"_subAction"=>"more","items"=>[{"u"=>"18fdc929-d35a-3b57-a253-435b0c9fa1e8","i"=>"4000010183"},{"u"=>"52a2961d-441f-34b7-bfa2-60ee2dae1001","i"=>"4000006954"},{"u"=>"0a6e7572-9eda-3c34-9052-ae58fbf009a2","i"=>"4000006504"},{"u"=>"ba9832a7-30a8-3e58-9a82-667c980db5ef","i"=>"4000009705"},{"u"=>"ff37c89f-4146-310c-ac28-68dea7626396","i"=>"4000007878"},{"u"=>"772cd139-7fec-3107-9bf6-e789c51e333c","i"=>"4000009780"},{"u"=>"20598ae3-7581-3c0d-94b3-55f15e0cace9","i"=>"4000007760"},{"u"=>"a98a5581-a0aa-3e4b-9f7b-b6a8ba16ff2b","i"=>"4000006306"},{"u"=>"00d23e7a-9b03-39ed-b669-f612e051d49f","i"=>"4000005248"},{"u"=>"f5c4f020-c608-32ed-aab2-dca6f20e62e1","i"=>"4000005770"},{"u"=>"ba6ac04c-a94c-3eec-a18a-63c98dabc926","i"=>"4000007324"},{"u"=>"f5a82053-132e-3c5d-abff-1026de72addd","i"=>"4000004708"},{"u"=>"86024916-08d3-3162-b158-9947a7eaeb6e","i"=>"4000009721"},{"u"=>"2486b22c-e8fc-3195-8ad9-26c2c4d1ac4b","i"=>"4000006895"},{"u"=>"9fd886b1-0708-3c75-a70c-5384c3bf2d6e","i"=>"4000008879"},{"u"=>"942dea9b-4686-30a9-b59a-95d4c7c69580","i"=>"4000004912"},{"u"=>"88d6a345-cc86-3c42-a975-729852551041","i"=>"4000004711"},{"u"=>"d2021342-2066-3bdc-8d67-2b07b5051888","i"=>"4000007372"},{"u"=>"b2c80259-6ef9-3622-9016-4625f3d7bf67","i"=>"4000005337"},{"u"=>"071008c4-4fe3-3b98-8695-b8e8ff15c599","i"=>"4000003388"}],"listid"=>"","blogtype"=>"","left"=>130,"key"=>"1","cpos"=>23,"prid"=>"evecfmlai6o1o","_container"=>0,"_id"=>"u_30345786","_txnid"=>1428381836200}],
"_mode" => "json",
"_crumb" => "NugJjVjNCWa",
"_txnid" => 1428381836200
}
I got all above info from the Network tab of Developer tool in Chrome. And the corresponding CURL (generated by Developer Tool) for this request is:
curl 'http://news.yahoo.com/hjsal?m_mode=multipart&site=news®ion=US&lang=en-US&pagetype=minihome' -H 'Cookie: ucs=sfcTs=1425628433&sfc=1; AO=u=1; B=b6dveila7ac17&b=4&d=c_ArSSppYEH6SFcrdazmtHDka6k-&s=b6&i=gQmslnXVPGkRceH.mrGk; F=a=RZpcadUMvSym7INq.f8fhle0_OQ0LB4p7I.8J.56OYwNqahr5jVi6DZsH5vjMWANCHUIoF0-&b=U8Fw; PH=fn=muyarBFoN154nhRrjzs-&i=us; DSS=cnt=1&ts=1425628533; ywandp=10001393120079%3A2763408991; fpc=10001393120079%3AZWeFxmga%7C%7C; ugccmt_comment_usortoptv1=highestRated; ywadp115488662=2349330862; fpms=; yvapF=%7B%22cc%22%3A1%2C%22rcc%22%3A1%2C%22vl%22%3A47.129999999999995%2C%22rvl%22%3A47.129999999999995%7D' -H 'Origin: http://news.yahoo.com' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Referer: http://news.yahoo.com/' --data '_crumb=NugJjVjNCWa&_mode=json&_txnid=1428540838640&_json=%5B%7B%22_action%22%3A%22show%22%2C%22batch%22%3A2%2C%22cat%22%3A%22%22%2C%22catName%22%3A%22%22%2C%22sb%22%3A0%2C%22ccode%22%3A%22grandSlam_news%22%2C%22woeid%22%3A28289488%2C%22_subAction%22%3A%22more%22%2C%22items%22%3A%5B%7B%22u%22%3A%220d9b1c7e-bfe6-3d71-b0fc-e31aa6a4ac78%22%2C%22i%22%3A%224000006886%22%7D%2C%7B%22u%22%3A%22d6291cfc-70e2-331d-b202-7d3e89d43275%22%2C%22i%22%3A%224000009863%22%7D%2C%7B%22u%22%3A%22d3c3af9f-cf8f-3d4e-a642-0070528819e0%22%2C%22i%22%3A%224000006408%22%7D%2C%7B%22u%22%3A%22104e5284-2998-3d7c-aaf6-9ceda966699d%22%2C%22i%22%3A%224000008669%22%7D%2C%7B%22u%22%3A%225f5e58cd-4e73-34f0-b1e3-59b8ed0d6dd2%22%2C%22i%22%3A%224000006338%22%7D%2C%7B%22u%22%3A%228f9ce219-79e9-33f8-baa1-1ddea51aed5a%22%2C%22i%22%3A%224000005997%22%7D%2C%7B%22u%22%3A%22683a22bb-e622-31b9-8c1d-d872db4dbf71%22%2C%22i%22%3A%224000007873%22%7D%2C%7B%22u%22%3A%222bd68c78-9645-34cf-8ba4-08ac328ffc60%22%2C%22i%22%3A%224000005723%22%7D%2C%7B%22u%22%3A%2261ed0f69-1e95-3264-baae-0d635606c9c4%22%2C%22i%22%3A%224000006596%22%7D%2C%7B%22u%22%3A%22e86ec903-edf8-3f46-a27a-4c10cdee8f19%22%2C%22i%22%3A%224000007547%22%7D%2C%7B%22u%22%3A%22d4f5eaea-3d28-36de-89a3-b9bb3fc79834%22%2C%22i%22%3A%224000009036%22%7D%2C%7B%22u%22%3A%228a05f4f4-e162-3586-96f1-0b7212a1ebd9%22%2C%22i%22%3A%224000006790%22%7D%2C%7B%22u%22%3A%22b49da9d0-a9ec-3c49-afbe-da1da9c3dfc5%22%2C%22i%22%3A%224000007305%22%7D%2C%7B%22u%22%3A%22f5fe6bcf-77fb-3e9e-bd56-a62027d604f1%22%2C%22i%22%3A%224000005595%22%7D%2C%7B%22u%22%3A%22b644c626-a6a4-3ab7-9d5d-737ee62fa492%22%2C%22i%22%3A%224000007357%22%7D%2C%7B%22u%22%3A%22bfa50721-be1b-38e2-94d1-e8728d112b0c%22%2C%22i%22%3A%224000008326%22%7D%2C%7B%22u%22%3A%22b9e49778-e5dd-3d84-9eb4-6a2681d81a93%22%2C%22i%22%3A%224000006213%22%7D%2C%7B%22u%22%3A%226073c345-05da-3fba-a6d0-f8699fe0e961%22%2C%22i%22%3A%224000005885%22%7D%2C%7B%22u%22%3A%22104a6bec-1244-39f8-ab7c-0586a1a5bd6d%22%2C%22i%22%3A%224000007645%22%7D%2C%7B%22u%22%3A%221280a6da-4b77-3543-8ded-ee62fcef6f9e%22%2C%22i%22%3A%224000007605%22%7D%5D%2C%22listid%22%3A%22%22%2C%22blogtype%22%3A%22%22%2C%22left%22%3A130%2C%22key%22%3A%221%22%2C%22cpos%22%3A24%2C%22prid%22%3A%223mmkn1daibjbe%22%2C%22_container%22%3A0%2C%22_id%22%3A%22u_30345786%22%2C%22_txnid%22%3A1428540838640%7D%5D' --compressed
Based on this CURL request, I have come up with the following Ruby code to crawl the data:
require 'net/http'
Require 'json'
params = {"_json" => URI.encode([{"_action"=>"show","batch"=>2,"cat"=>"","catName"=>"","sb"=>0,"ccode"=>"grandSlam_news","woeid"=>28379204,"_subAction"=>"more","items"=>[{"u"=>"18fdc929-d35a-3b57-a253-435b0c9fa1e8","i"=>"4000010183"},{"u"=>"52a2961d-441f-34b7-bfa2-60ee2dae1001","i"=>"4000006954"},{"u"=>"0a6e7572-9eda-3c34-9052-ae58fbf009a2","i"=>"4000006504"},{"u"=>"ba9832a7-30a8-3e58-9a82-667c980db5ef","i"=>"4000009705"},{"u"=>"ff37c89f-4146-310c-ac28-68dea7626396","i"=>"4000007878"},{"u"=>"772cd139-7fec-3107-9bf6-e789c51e333c","i"=>"4000009780"},{"u"=>"20598ae3-7581-3c0d-94b3-55f15e0cace9","i"=>"4000007760"},{"u"=>"a98a5581-a0aa-3e4b-9f7b-b6a8ba16ff2b","i"=>"4000006306"},{"u"=>"00d23e7a-9b03-39ed-b669-f612e051d49f","i"=>"4000005248"},{"u"=>"f5c4f020-c608-32ed-aab2-dca6f20e62e1","i"=>"4000005770"},{"u"=>"ba6ac04c-a94c-3eec-a18a-63c98dabc926","i"=>"4000007324"},{"u"=>"f5a82053-132e-3c5d-abff-1026de72addd","i"=>"4000004708"},{"u"=>"86024916-08d3-3162-b158-9947a7eaeb6e","i"=>"4000009721"},{"u"=>"2486b22c-e8fc-3195-8ad9-26c2c4d1ac4b","i"=>"4000006895"},{"u"=>"9fd886b1-0708-3c75-a70c-5384c3bf2d6e","i"=>"4000008879"},{"u"=>"942dea9b-4686-30a9-b59a-95d4c7c69580","i"=>"4000004912"},{"u"=>"88d6a345-cc86-3c42-a975-729852551041","i"=>"4000004711"},{"u"=>"d2021342-2066-3bdc-8d67-2b07b5051888","i"=>"4000007372"},{"u"=>"b2c80259-6ef9-3622-9016-4625f3d7bf67","i"=>"4000005337"},{"u"=>"071008c4-4fe3-3b98-8695-b8e8ff15c599","i"=>"4000003388"}],"listid"=>"","blogtype"=>"","left"=>130,"key"=>"1","cpos"=>23,"prid"=>"evecfmlai6o1o","_container"=>0,"_id"=>"u_30345786","_txnid"=>1428381836200}].to_json),
"_mode" => "json",
"_crumb" => "NugJjVjNCWa",
"_txnid" => 1428381836200
}
cookie = 'ucs=sfcTs=1425628433&sfc=1; AO=u=1; B=b6dveila7ac17&b=4&d=c_ArSSppYEH6SFcrdazmtHDka6k-&s=b6&i=gQmslnXVPGkRceH.mrGk; F=a=RZpcadUMvSym7INq.f8fhle0_OQ0LB4p7I.8J.56OYwNqahr5jVi6DZsH5vjMWANCHUIoF0-&b=U8Fw; PH=fn=muyarBFoN154nhRrjzs-&i=us; DSS=cnt=1&ts=1425628533; ywandp=10001393120079%3A2763408991; fpc=10001393120079%3AZWeFxmga%7C%7C; ugccmt_comment_usortoptv1=highestRated; ywadp115488662=2349330862; fpms=; yvapF=%7B%22cc%22%3A1%2C%22rcc%22%3A1%2C%22vl%22%3A47.129999999999995%2C%22rvl%22%3A47.129999999999995%7D'
uri = URI('http://news.yahoo.com/hjsal?m_mode=multipart&site=news®ion=US&lang=en-US&pagetype=minihome')
req = Net::HTTP::Post.new(uri.path)
req.add_field('Referer', 'http://news.yahoo.com/')
req.add_field('Origin', 'http://news.yahoo.com')
req.add_field('Host', 'news.yahoo.com')
req.add_field('Cookie', cookie)
req.add_field('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8')
req.body = URI.encode(params.to_json)
http = Net::HTTP.new(uri.host,uri.port)
res = http.request(req)
p res.body
But it always returns error message in response when running above Ruby code (the CURL request works perfectly). I am not quite sure what is wrong here and I do hope you guys can give me some hints to resolve this issue.
The response (this error is the same if you copy the URL and paste it on your browser):
<div class=\"oops-msg\" role=\"alertdialog\">\n<span class=\"icon icon-error y-glbl-universal\"></span>\n<h3 class=\"oops\">The module encountered a problem while trying to load</h3>\n <p class=\"oops\">App is currently not available. Please, try again.</p>\n \n </div><!-- hdf50.fp.gq1.yahoo.com compressed/chunked Wed Apr 15 08:15:39 UTC 2015 -->\n
Also, I can not even simulate this request in Postman Client(new version with Interceptor enabled).
Thank you very much
Your Ruby code is specifying a Post request but your curl is not. Try replacing this line:
req = Net::HTTP::Post.new(uri.path)
with this:
req = Net::HTTP::Get.new(uri.path)
Related
Google PeopleApi in Ruby - CreateContact method invalid argument
I'm using the 'google-api-client' gem to connect into Google People API from my Ruby Application. https://github.com/googleapis/google-api-ruby-client I've managed to get other methods working (list_person_connections, get_people, delete_person_contact and update_person_contact) but I can't get the createContact (create_person_contact) method from google people API to work. After I send the post, I get this error: 400 Caught error badRequest: Request contains an invalid argument. This is an example code to create a contact with just the name field (I will want to actually also send email and phoneNumbers parameters on the body, but it also returns the same error, so I'm giving you the simplest example here): require 'google/apis/content_v2' require "google/apis/people_v1" require "googleauth" require "googleauth/stores/file_token_store" require "fileutils" require 'google/apis/people_v1' require 'google/api_client/client_secrets' client_secrets = Google::APIClient::ClientSecrets.load 'credentials.json' auth_client = client_secrets.to_authorization auth_client.update!( :scope => ['https://www.googleapis.com/auth/contacts', 'https://www.googleapis.com/auth/contacts.other.readonly'], :redirect_uri => 'http://localhost:3102/oauth2callback', :client_id => 'MY CLIENT ID', :client_secret => 'MY CLIENT SECRET', #:authorization_uri => 'https://accounts.google.com/o/oauth2/auth', :additional_parameters => {"access_type" => "offline", "include_granted_scopes" => "true"}) auth_uri = auth_client.authorization_uri.to_s auth_client.code = 'THE AUTH CODE' auth_client.fetch_access_token! people = Google::Apis::PeopleV1::PeopleServiceService.new people.authorization = auth_client body = {:names => [{:givenName => "TEST"}]} people.create_person_contact(body, person_fields: 'names') The problem is just those 2 last lines. When called, they send this with the body: Sending HTTP post https://people.googleapis.com/v1/people:createContact?personFields=names And it returns the error above, no matter what I change. In the documentation, you can actually try and test this exact same code and it works. https://developers.google.com/people/api/rest/v1/people/createContact?authuser=2&apix=true&apix_params=%7B%22personFields%22%3A%22names%22%2C%22resource%22%3A%7B%22names%22%3A%5B%7B%22givenName%22%3A%22TEST%22%7D%5D%7D%7D You can fill the request body form exactly as I did, and hit EXECUTE And it will give a 200 OK response. I can't see what the invalid argument is. This is also exactly what I do on the update_person_contact method and that one works. I've searched the internet and can't find anyone with a similar problem and the documentation just doesn't say much: https://www.rubydoc.info/github/google/google-api-ruby-client/Google%2FApis%2FPeopleV1%2FPeopleServiceService:create_person_contact Anyone have any idea to help me? Thank you.
After almost giving up, I decided to try the CURL option. So I copied it from their 'Try this API' page I linked above and translated it to Ruby code. curl --request POST \ 'https://people.googleapis.com/v1/people:createContact?personFields=names' \ --header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --data '{"names":[{"givenName":"TEST"}]}' \ --compressed And you know what? It worked. So I think this gem doesn't work for this particular case (it is google-api-client v: 0.42.2 and 0.43.0 in case you are wondering), but if you came here to find a solution to the same problem, this is what worked for me, and I hope it helps you: (this replace those last 2 lines on my code): require 'net/http' require 'uri' require 'json' uri = URI.parse("https://people.googleapis.com/v1/people:createContact?personFields=names") request = Net::HTTP::Post.new(uri) request.content_type = "application/json" request["Authorization"] = "Bearer #{people.authorization.access_token}" request["Accept"] = "application/json" request.body = JSON.dump({ "names" => [ { "givenName" => "TEST" } ], "emailAddresses" => [ { "value" => "testemail#test.com" } ], "phoneNumbers" => [ { "value" => "12345678" } ] } ) req_options = { use_ssl: uri.scheme == "https", } response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http| http.request(request) end response.code response.body It's not ideal, but it got the job done today. EDIT: After writing this I just realized even the 'personFields' parameter is useless, since the body got the right fields. You can see in my answer, I only called the 'names' fields on the URI, but all those 3 fields were correctly there on my new contact (name, email and phone number) after it got saved. So that's also probably useless/optional.
Post Request from client With Proper Headers Always return 401- UnAuthorized
Am a newbie to ruby. I am trying to invoke a REST API from ruby code. I have tried the code with both as standalone ruby code and inside a application using sinatra. But am facing same issue every where. Below are the options I have tried. My Code to Invoke Post API using RestClient. Have modified the data to make it generic url="https://myapiurl.com/endpointname" person={"name"=>"Sample Name","age"=>"20"} headers={ "Content-Type"=>"application/json", "api-key"=>"ABCDEFGHIJKL" } response=RestClient.post url,person.to_json,headers puts response I wrote the above block of code in a function and tried calling. But I got response as below {"status": 401, "error":"Unauthorized Access"} I tried the same api via postman with below settings and was able to get a proper response from the api. URL : https://myapiurl.com/endpointname Headers : Content-Type: application/json, api-key:"ABCDEFGHIJKL" Body: Raw:application/json: {"name":"Sample Name","age":"20"} When I tried not passing api-key in the headers via postman I got similar response as I got via ruby code. Next I tried generating code from postman option. Below is the code that got generated from postman for Ruby(Net::Http). I removed the post params of cache-control and postman-token from the ruby code and tried running the ruby code. Again I got the same 'Unauthorized' response only ! require 'uri' require 'net/http' url = URI("https://myapiurl.com/endpointname") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true http.verify_mode = OpenSSL::SSL::VERIFY_NONE request = Net::HTTP::Post.new(url) request["content-type"] = 'application/json' request["api-key"] = 'ABCDEFGHIJKL' request["cache-control"] = 'no-cache' request["postman-token"] = 'some value got generated' request.body = "{\"name\":\"Sample Name\",\"age\":\"20\"}" response = http.request(request) puts response.read_body I suspected is it some IP level issue is blocking hence tried used curl command to hit the api as below. I got proper response from the API. curl -X POST \ https://myapiurl.com/endpointname \ -H 'content-type: application/json' \ -H 'api-key: ABCDEGHIJKL' \ -d '{"name":"Sample Name","age":"20"}' I believe that the api-key is not getting passed to the request via the ruby code. Is my way of sending the api-key in headers via ruby code correct?
Microsoft Azure get access token API not working with cURL/Ruby but works with powershell
I am simply trying to get an access token from client id, client secret and tenant ID. Following powershell command works successfully Invoke-RestMethod -Uri https://login.microsoftonline.com/TENANT/oauth2/token?api-version=1.0 -Method Post -Body #{"grant_type" = "client_credentials"; "resource" = "https://management.core.windows.net/"; "client_id" = "CLIENTID"; "client_secret" = "SECRET" } But this curl doesn't work curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials&resource=https://management.core.windows.net/&client_id=CLIENTID&client_secret=SECRET" "https://login.microsoftonline.com/TENANT/oauth2/token?api-version=1.0" Neither this ruby script require 'json' require 'typhoeus' url = 'https://login.microsoftonline.com/TENANT/oauth2/token?api-version=1.0' params = "grant_type=client_credentials&resource=https://management.core.windows.net/&client_id=CLIENTID&client_secret=SECRET" HEADERS = { "Content-Type" => "application/x-www-form-urlencoded" } resp = Typhoeus::Request.post(url, body: params, headers: HEADERS) I am following this link. Any clues why neither of curl / ruby works ? Thanks in advance
I tried to reproduce your issue successfully, and I discovered that the issue was caused by curl without OpenSSL and Typhoeus request without setting ssl_verifypeer: false. So please follow this to check via curl --version and install openssl libraries on your environment. Here is my sample code. require "typhoeus" url = 'https://login.microsoftonline.com/<tanet-id>/oauth2/token?api-version=1.0' params = "grant_type=client_credentials&resource=https://management.core.windows.net/&client_id=<client-id>&client_secret=<client-key>" headers = { "Content-Type" => "application/x-www-form-urlencoded" } request = Typhoeus.post(url, body: params, headers: headers, ssl_verifypeer: false) puts request.code, request.body Hope it helps.
Add `Authorization Bearer` hash to Net::HTTP post request (Ruby)
How can I add Authorization Bearer to a POST request with Net::HTTP? I can only find help for "basic authentication" in the documentation. req.basic_auth 'user', 'pass' Source: https://docs.ruby-lang.org/en/2.0.0/Net/HTTP.html#class-Net::HTTP-label-Basic+Authentication I'm trying to replicate a curl that would look like: > curl 'http://localhost:8080/places' -d '{"_json":[{"uuid":"0514b...", > "name":"Athens"}]}' -X POST -H 'Content-Type: application/json' -H > 'Authorization: Bearer eyJ0eXAiO...' Currently I've gotten to: require 'net/http' require 'net/https' require 'uri' uri = URI('http://localhost:8080/places') res = Net::HTTP.post_form(uri, '_json' => [{'uuid': '0514b...', 'name':'Athens'}]) But I'm having trouble figuring out how to add the Authentication: Bearer... part. Does anyone have experience with this?
I dont think you can add custom headers with the post_form method. You can add it with post method. Try the code below: uri = URI("http://localhost:8080/places") params = [{'uuid': '0514b...', 'name':'Athens'}] headers = { 'Authorization'=>'Bearer foobar', 'Content-Type' =>'application/json', 'Accept'=>'application/json' } http = Net::HTTP.new(uri.host, uri.port) response = http.post(uri.path, params.to_json, headers)
Nodejs Ajax Call with XMLHttpRequest Header
I am trying to write a script for Hubot to make an AJAX call to Strawpoll.me. I have a cURL command that works exactly how I want but I am having trouble translating that into a Node.js function. curl --header "X-Requested-With: XMLHttpRequest" --request POST --data "options=1&options=2&options=3&options=4&options=5&title=Test&multi=false&permissive=false" http://strawpoll.me/api/v2/polls Here is what I currently have in my script. QS = require 'querystring' module.exports = (robot) -> robot.respond /strawpoll "(.*)"/i, (msg) -> options = msg.match[1].split('" "') data = QS.stringify({ title: "Strawpoll " + Math.floor(Math.random() * 10000), options: options, multi: false, permissive: true }) req = robot.http("http://strawpoll.me/api/v2/polls").headers({"X-Requested-With": "XMLHttpRequest"}).post(data) (err, res, body) -> if err msg.send "Encountered an error :( #{err}" return msg.reply(body) The script version is returning {"error":"Invalid request","code":40} I can't tell what I am doing wrong. Thanks for your help.
For POST requests, curl sets the Content-Type to application/x-www-form-urlencoded. Hubot uses Node's http client, which OTOH, doesn't use any defaults for the Content-Type header. Without an explicit Content-Type header, the resource at http://strawpoll.me/api/v2/polls is not able to discern the request body. You'll have to set the Content-Type header manually to mimic curl's request. robot.http('http://strawpoll.me/api/v2/polls') .headers({'X-Requested-With': 'XMLHttpRequest', 'Content-Type': 'application/x-www-form-urlencoded'}) .post(data)