Ruby split and parse batched HTTP Response (multipart/mixed) - ruby

I'm using the GMail API that allows me to get a batched response of multiple Gmail objects.
This comes back in the form of a multipart/mixed HTTP response with a set of separate HTTP responses separated by a boundary as defined in the header.
Each of the HTTP sub-Responses is a JSON format.
i.e.
result.response.response_headers = {...
"content-type"=>"multipart/mixed; boundary=batch_abcdefg"...
}
result.response.body = "----batch_abcdefg
<the response header>
{some JSON}
--batch_abcdefg
<another response header>
{some JSON}
--batch_abcdefg--"
Is there a library or an easy way to convert those responses from the string into a set of separate HTTP responses or JSON objects?

Thanks to Tholle above...
def parse_batch_response(response, json=true)
# Not the same delimiter in the response as we specify ourselves in the request,
# so we have to extract it.
# This should give us exactly what we need.
delimiter = response.split("\r\n")[0].strip
parts = response.split(delimiter)
# The first part will always be an empty string. Just remove it.
parts.shift
# The last part will be the "--". Just remove it.
parts.pop
if json
# collects the response body as json
results = parts.map{ |part| JSON.parse(part.match(/{.+}/m).to_s)}
else
# collates the separate responses as strings so you can do something with them
# e.g. you need the response codes
results = parts.map{ |part| part}
end
result
end

Related

Ruby return's HTTPVersionNotSupported object

I'm trying to make a get request to a service of mine with a valid URL string (if I put it into my browser, I get the expected response). However, when I run the following function:
def dispatch_uri(url)
uri = Addressable::URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request).to_s
response
end
The response variable holds a Net::HTTPVersionNotSupported, which has no body and isn't, of course, the expected response.
What am I doing wrong and how should I address this problem?
So, the answer is simpler than I thought.
Net::HTTP is both unable to work with an UTF-8 URL or Addressable::URI, however, Addressable gives us a fantastic tool to solve this problem: normalize.
Normalize converts your UTF=8 to a codified ASCII HTML compatible string, so a working code is:
def dispatch_uri(url)
uri = URI(Addressable::URI.parse(url).normalize.to_s)
response = Net::HTTP.get(uri)
response
end
This normalized string can be used to create a standard URI object and, thus, you are able to use a regular Net::HTTP request.

Using Sinatra to Parse JSON data from url

I'm using Sinatrarb to complete a task
I need to:
Parse the data of a JSON object from a url,
Single out one of attributes of the json data and store it as a variable
Run some arithmetic on the variable
Return the result as a new variable
then post this to a new url as a new json object.
I have seen bits and pieces of information all over including information on parsing JSON data in ruby and information on open-uri but I believe it would be very valuable having someone break this down step by step as most similar solutions given to this are either outdated or steeply complex.
Thanks in advance.
Here's a simple guide. I've done the same task recently.
Let's use this JSON (put it in a file called 'simple.json'):
{
"name": "obscurite",
"favorites": {
"icecream": [
"chocolate",
"pistachio"
],
"cars": [
"ferrari",
"porsche",
"lamborghini"
]
},
"location": "NYC",
"age": 100}
Parse the data of a JSON object from a url.
Step 1 is to add support for JSON parsing:
require 'json'
Step 2 is to load in the JSON data from our new .json file:
json_file = File.read('simple.json')
json_data = JSON.parse(json_file)
Single out one of attributes of the json data and store it as a variable
Our data is in the form of a Hash on the outside (curly braces with key:values). Some of the values are also hashes ('favorites' and 'cars'). The values of those inner hashes are lists (Arrays in Ruby). So what we have is a hash of hashes, where some hashes are arrays.
Let's pull out my location:
puts json_data['location'] # NYC
That was easy. It was just a top level key/value. Let's go deeper and pull out my favorite icecream(s):
puts json_data['favorites']['icecream'] # chocolate pistachio
Now only my second favorite car:
puts json_data['favorites']['cars'][1] # porsche
Run some arithmetic on the variable
Step 3. Let's get my age and cut it down by 50 years. Being 100 is tough!
new_age = json_data['age'] / 2
puts new_age
Return the result as a new variable
Step 4. Let's put the new age back into the json
json_data['age'] = new_age
puts json_data['age'] # 50
then post this to a new url as a new json object.
Step 5. Add the ability for your program to do an HTTP POST. Add this up at top:
require 'net/http'
and then you can post anywhere you want. I found a fake web service you could use, if you just want to make sure the request got there.
# use this guy's fake web service page as a test. handy!
uri = URI.parse("http://jsonplaceholder.typicode.com/posts")
header = {'Content-Type'=> 'text/json'}
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.request_uri, header)
request.body = json_data.to_json
response = http.request(request)
# Did we get something back?
puts response.body
On linux or mac you can open a localhost port and listen as a test:
nc -4 -k -l -v localhost 1234
To POST to this port change the uri to:
uri = URI.parse("http://localhost:1234")
Hope this helps. Let me know if you get stuck and I'll try to lend a hand. I'm not a ruby expert, but wanted to help a fellow explorer. Good luck.

Ruby: HTTP Put method

I am attempting to update the 'ip' parameter in a json object in an API.
I have the following case:
when "put"
uri = URI.parse("http://#{ip}:#{port}/api/v1/address_data/1.json")
jobj = Hash.new
jobj['ip'] = "1.1.1.1"
http = Net::HTTP.new(uri.hostname, uri.port)
response = http.send_request('PUT', '/api/v1/address_data/1.json', data = jobj.to_s)
end
This does not work, but this does:
curl -X PUT http://ip:port/api/v1/address_data/1.json -d "ip=1.1.1.1"
How do I more accurately translate the curl into a Put request in Ruby? I have tried several methods I've found through google searching, but none of them have had successful results.
A few things:
You're not sending JSON in the Ruby example, it's a string representation of a Ruby hash which isn't the same. You need the JSON module or similar.
In the Ruby code you're attempting to send a JSON object (which would look like {"ip":"1.1.1.1"} and in the curl example you're sending it in application/x-www-form-urlencoded format, so they're currently not equivalent.
Also I'd look at the type of data the server expects from your requests: both Ruby and curl send a request header of Content-Type: application/x-www-form-urlencoded by default, and you're expecting to send JSON. This is why the curl example works: the data format you're using and the header matches. Note the .json in the URL shouldn't really make any difference; the header takes precedence.
Your call to send_request has you picking out the data parameter as a Python-style keyword argument. Ruby doesn't do that: what you're actually doing there is assigning a local variable in-line with the call.
So try something like this:
require 'json' # put this at the top of the file
uri = URI.parse("http://#{ip}:#{port}/api/v1/address_data/1.json")
jobj = {"ip" => "1.1.1.1"}
http = Net::HTTP.new(uri.hostname, uri.port)
response = http.send_request('PUT', uri.path, JSON.dump(jobj),
{'Content-Type' => 'application/json'})
And just a friendly reminder, saying something "doesn't work" doesn't usually give enough information to people that might answer your question: try and remember to paste in error messages, stack traces, and things like that :)

how do I parse out this URL-encoded string with a JSON array using Ruby?

When I use a webhook with Mandrill and post to my Iron Worker, I get the following Raw (this is from RequestBin, as well) -- I didn't include the whole payload, just an example:
puts payload =>
mandrill_events=%5B%7B%22event%22%3A%22inbound%22%2C%22msg%22%3A%7B%22dkim%22%3A%7B%22signed%22%3Atrue%2C%22valid%22%3Atrue%7D%2C%22email%22%3A%22kaya%40hellokaya.com%22%2C%22from_email%22%3A%22example.sender%40mandrillapp.com%22%2C%22headers%22%3A%7B%22Content-Type%22%3A%22multipart%5C%2Falternative%3B+boundary%3D%5C%22_av-7r7zDhHxVEAo2yMWasfuFw%5C%22%22%2C%22Date%22%3A%22Fri%2C+10+May+2013+19%3A28%3A20+%2B0000%22%2C%22Dkim-Signature%22%3A%5B%22v%3D1%3B+a%3Drsa-
I tried to extract the value of the parameter mandrill_events using:
puts params = CGI::parse(#payload) =>
{"mandrill_events"=>["[{\"event\":\"inbound\",\"msg\":{\"dkim\":{\"signed\":true,\"valid\":true},\"email\":\"kaya#hellokaya.com\",\"from_email\":\"example.sender#mandrillapp.com\",\"headers\":{\"Content-Type\":\"multipart\\/alternative; boundary=\\\"_av-7r7zDhHxVEAo2yMWasfuFw\\\"\",\"Date\":\"Fri, 10 May 2013 19:28:20 +0000\",\"Dkim-Signature\":[\"v=1; a=rsa-sha1; c=relaxed\\/relaxed; s=mandrill; d=mail115.us4.mandrillapp.com; h=From:Sender:Subject:List-Unsubscribe:To:Message-Id:Date:MIME-Version:Content-Type; i=example.sender#mail115.us4.mandrillapp.com;
Then I am stuck. I want to extract the email value in the JSON.array.
I thought to try the following;
puts json_params = JSON.parse(params)
But I am now feeling there must be a better way....
How can I extract the elements from the JSON array in this URL-encoded string?

Check headers before downloading with Net::HTTP::Pipeline

I am trying to parse a list of image URL's and get some basic information before I actually commit to download.
Is the image there (solved with response.code?)
Do I have the image already (want to look at type and size?)
My script will check a large list every day (about 1300 rows) and each row has 30-40 image URLs. My #photo_urls variable allows me to keep track of what I have downloaded already. I would really like to be able to use that later as a hash (instead of an array in my example code) to interate through later and do the actual downloading.
Right now my problem (besides being a Ruby newbie) is that Net::HTTP::Pipeline only accepts an array of Net::HTTPRequest objects. The documentation for net-http-pipeline indicates that response objects will come back in the same order as the corresponding request objects that went in. The problem is that I have no way to correlate the request to the response other than that order. However, I don't know how to get relative ordinal position inside a block. I assume I could just have a counter variable but how would I access a hash by ordinal position?
Net::HTTP.start uri.host do |http|
# Init HTTP requests hash
requests = {}
photo_urls.each do |photo_url|
# make sure we don't process the same image again.
hashed = Digest::SHA1.hexdigest(photo_url)
next if #photo_urls.include? hashed
#photo_urls << hashed
# change user agent and store in hash
my_uri = URI.parse(photo_url)
request = Net::HTTP::Head.new(my_uri.path)
request.initialize_http_header({"User-Agent" => "My Downloader"})
requests[hashed] = request
end
# process requests (send array of values - ie. requests) in a pipeline.
http.pipeline requests.values do |response|
if response.code=="200"
# anyway to reference the hash here so I can decide whether
# I want to do anything later?
end
end
end
Finally, if there is an easier way of doing this, please feel free to offer any suggestions.
Thanks!
Make requests an array instead of a hash and pop off the requests as the responses come in:
Net::HTTP.start uri.host do |http|
# Init HTTP requests array
requests = []
photo_urls.each do |photo_url|
# make sure we don't process the same image again.
hashed = Digest::SHA1.hexdigest(photo_url)
next if #photo_urls.include? hashed
#photo_urls << hashed
# change user agent and store in hash
my_uri = URI.parse(photo_url)
request = Net::HTTP::Head.new(my_uri.path)
request.initialize_http_header({"User-Agent" => "My Downloader"})
requests << request
end
# process requests (send array of values - ie. requests) in a pipeline.
http.pipeline requests.dup do |response|
request = requests.shift
if response.code=="200"
# Do whatever checking with request
end
end
end

Resources