Issue while fetching data from nested json - ruby

I am trying to fetch data from a nested json. Not able to understand the issue over here. Please ignore the fields that I am passing to ChildArticle class. I can sort that out.
URL for JSON - http://api.nytimes.com/svc/mostpopular/v2/mostshared/all-sections/email/30.json?api-key=31fa4521f6572a0c05ad6822ae109b72:2:72729901
Below is my code:
url = 'http://api.nytimes.com'
#Define the HTTP object
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
#If the api being scraped uses https, then set use_ssl to true.
http.use_ssl = false
#Define the request_url
#Make a GET request to the given url
request = '/svc/mostpopular/v2/mostshared/all-sections/email/30.json?api-key=31fa4521f6572a0c05ad6822ae109b72:2:72729901'
response = http.send_request('GET', request)
#Parse the response body
forecast = JSON.parse(response.body)
forecast["results"]["result"].each do |item|
date = Date.parse(item["published_date"].to_s)
if (#start <= date) && (#end >= date)
article = News::ChildArticle.new(author: item["author"], title: item["title"], summary: item["abstract"],
images: item["images"],source: item["url"], date: item["published_date"],
guid: item["guid"], link: item["link"], section: item["section"],
item_type: item["item_type"], updated_date: item["updated_date"],
created_date: item["created_date"],
material_type_facet: item["material_type_facet"])
#articles.concat([article])
end
end
I get below error -
[]': no implicit conversion of String into Integer (TypeError) atforecast["results"]["result"].each do |item|`

Looks like forecast['results'] is simply an array, not a hash.
Take a look at this slightly modified script. Give it a run in your terminal, and check out its output.
require 'net/http'
require 'JSON'
url = 'http://api.nytimes.com'
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = false
request = '/svc/mostpopular/v2/mostshared/all-sections/email/30.json?api-key=31fa4521f6572a0c05ad6822ae109b72:2:72729901'
response = http.send_request('GET', request)
forecast = JSON.parse(response.body)
forecast["results"].each.with_index do |item, i|
puts "Item #{i}:"
puts '--'
item.each do |k, v|
puts "#{k}: #{v}"
end
puts '----'
end
Also, you may want to inspect the JSON structure of the API return from that URL. If you go to that URL, open your JavaScript console, and paste in
JSON.parse(document.body.textContent)
you can inspect the JSON structure very easily.
Another option would be downloading the response to a JSON file, and inspecting it in your editor. You'll need a JSON prettifier though.
File.open('response.json', 'w') do |f|
f.write(response.body)
end

Related

Ruby Net::HTTP passing headers through the creation of request

Maybe I'm just blind but many post about passing headers in Net::HTTP follows the lines of
require 'net/http'
uri = URI("http://www.ruby-lang.org")
req = Net::HTTP::Get.new(uri)
req['some_header'] = "some_val"
res = Net::HTTP.start(uri.hostname, uri.port) {|http|
http.request(req)
}
puts res.body
(From Ruby - Send GET request with headers metaphori's answer)
And from the Net::HTTP docs (https://docs.ruby-lang.org/en/2.0.0/Net/HTTP.html)
uri = URI('http://example.com/cached_response')
file = File.stat 'cached_response'
req = Net::HTTP::Get.new(uri)
req['If-Modified-Since'] = file.mtime.rfc2822
res = Net::HTTP.start(uri.hostname, uri.port) {|http|
http.request(req)
}
open 'cached_response', 'w' do |io|
io.write res.body
end if res.is_a?(Net::HTTPSuccess)
But what is the advantage of doing the above when you can pass the headers via the following way?
options = {
'headers' => {
'Content-Type' => 'application/json'
}
}
request = Net::HTTP::Get.new('http://www.stackoverflow.com/', options['headers'])
This allows you to parameterize the headers and can allow for multiple headers very easily.
My main question is, what is the advantage of passing the headers in the creation of Net::HTTP::Get vs passing them after the creation of Net::HTTP::Get
Net::HTTPHeader already goes ahead and assigns the headers in the function
def initialize_http_header(initheader)
#header = {}
return unless initheader
initheader.each do |key, value|
warn "net/http: duplicated HTTP header: #{key}", uplevel: 1 if key?(key) and $VERBOSE
if value.nil?
warn "net/http: nil HTTP header: #{key}", uplevel: 1 if $VERBOSE
else
value = value.strip # raise error for invalid byte sequences
if value.count("\r\n") > 0
raise ArgumentError, 'header field value cannot include CR/LF'
end
#header[key.downcase] = [value]
end
end
end
So doing
request['some_header'] = "some_val" almost seems like code duplication.
There is no advantage for setting headers one way or another, at least not that I can think of. It comes down to your own preference. In fact, if you take a look at what happens when you supply headers while initializing a new Net::Http::Get, you will find that internally, Ruby simply sets the headers onto a #headers variable:
https://github.com/ruby/ruby/blob/c5eb24349a4535948514fe765c3ddb0628d81004/lib/net/http/header.rb#L25
And if you set the headers using request[name] = value, you can see that Net::Http does the exact same thing, but in a different method:
https://github.com/ruby/ruby/blob/c5eb24349a4535948514fe765c3ddb0628d81004/lib/net/http/header.rb#L46
So the resulting object has the same configuration no matter which way you decide to pass the request headers.

TypeError problem: no implicit conversion in Sinatra + JSON.parse

I'm trying to set up a web hook, following this GitHub tutorial
require 'sinatra'
require 'json'
require 'net/http'
require 'pp'
set :port, 31415
# Descarga las diferencias hechas para un push
post '/' do
push = JSON.parse(request.body.read)
piezas = push["compare"].split("/")
api_url = "/repos/#{piezas[3]}/#{piezas[4]}/compare/#{piezas[6]}"
diff = Net::HTTP.get(URI("https://api.github.com#{api_url}"))
puts diff.class
pp(JSON.parse(diff))
end
diff.class prints:
String
And, as a matter of fact, the last sentence works correctly, printing via pp the structure. However, after printing, it yields the error
[2018-10-25 20:00:23] ERROR TypeError: no implicit conversion of Array into String
It's not referencing any line in the script, but would it be possible that the error would be in the first JSON.parse? Could it be that request.body.read would be an array?
Update I couldn't golf it down to any of the JSON.parse separately. Downloading the hook payload works OK, downloading the JSON from the GitHub API works without a glithc. Somehow it's using them together what does not work.
It's possible the library is treating the response like text. Try adding an Accept header. This worked for me:
request["Accept"] = "application/json"
example:
uri = URI.parse("https://api.github.com")
req = Net::HTTP::Get.new(URI("https://api.github.com/repos/JJ/microservices-broker/compare/d5d39c5db99d...bbbf695d1bf2"))
req["Accept"] = 'application/json'
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
response = http.request(req)
json = JSON.parse(response.body)
json['url']
# or
json = JSON.parse(response.body, symbolize_names: true)
json[:url]
(EDIT:) Also, Using Net::HTTP is really painful. Please checkout these libraries:
https://github.com/lostisland/faraday
https://github.com/octokit/octokit.rb

Cannot make HTTP Delete request with Ruby's net/http library

I've been trying to make an API call to my server to delete a user record help on a dev database. When I use Fiddler to call the URL with the DELETE operation I am able to immediately delete the user record. When I call that same URL, again with the DELETE operation, from my script below, I get this error:
{"Message":"The requested resource does not support http method 'DELETE'."}
I have changed the url in my script below. The url I am using is definitely correct. I suspect that there is a logical error in my code that I haven't caught. My script:
require 'net/http'
require 'json'
require 'pp'
require 'uri'
def deleteUserRole
# prepare request
url= "http://my.database.5002143.access" # dev
uri = URI.parse(url)
request = Net::HTTP::Delete.new(uri.path)
http = Net::HTTP.new(uri.host, uri.port)
# send the request
response = http.request(request)
puts "response: \n"
puts response.body
puts "response code: " + response.code + "\n \n"
# parse response
buffer= response.body
result = JSON.parse(buffer)
status= result["Success"]
if status == true
then puts "passed"
else puts "failed"
end
end
deleteUserRole
It turns out that I was typing in the wrong command. I needed to change this line:
request = Net::HTTP::Delete.new(uri.path)
to this line:
request = Net::HTTP::Delete.new(uri)
By typing uri.path I was excluding part of the URL from the API call. When I was debugging, I would type puts uri and that would show me the full URL, so I was certain the URL was right. The URL was right, but I was not including the full URL in my DELETE call.
if you miss the parameters to pass while requesting delete, it won't work
you can do like this
uri = URI.parse('http://localhost/test')
http = Net::HTTP.new(uri.host, uri.port)
attribute_url = '?'
attribute_url << body.map{|k,v| "#{k}=#{v}"}.join('&')
request = Net::HTTP::Delete.new(uri.request_uri+attribute_url)
response = http.request(request)
where body is a hashmap where you can define query params as a hashmap.. while sending request it can be joined in the url by the code above.
ex:body = { :resname => 'res', :bucket_name => 'bucket', :uploaded_by => 'upload' }

Getting data out of JSON

I keep hitting an error.
The path to my data in JSON is
["data"][i]["ip"]
where iis the index.
I tried
json_obj.each do |obj|
list << obj["data"[i]["ip"]
end
in order to store the values inside a list array. I keep getting some conversion error. Is there a better way to do this?
uri = URI.parse("SITE")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
req = Net::HTTP::Get.new(uri.request_uri)
req.basic_auth('USER', 'PASS')
res = http.request(req)
json_resp = JSON.parse(res.body)
list = []
json_resp.each do | obj |
print obj["data"][0]["fqdn"]
end
it should be json_obj["data"].each
json_obj is a hash (or hash like anyway) so you need to pass the key then you can access the array json_obj["data"] points to and use it's each method.
As #tadman says this error is from Array.each not liking a string index, and happens often to me...

Ruby + Net::HTTP: How do I send two XML documents in one POST request?

I have to send two XML documents in my request to the UPS API (here's my original question What is the root of this XML document? )
How would I do this?
def make_initial_request
uri = URI.parse(UPS_API['confirm_url'])
https = Net::HTTP.new(uri.host, uri.port)
https.use_ssl = true
headers = {'Content-Type' => 'text/xml'}
request = Net::HTTP::Post.new(uri.path, headers)
request.body = xml_for_initial_request #<-- how do i split this into two documents?
#request.body = second_xml_document #<-- i want something like that. could i just use << ?
begin
response = https.request(request)
rescue
return nil
end
puts "response: #{response.code} #{response.message}: #{response.body}"
return nil if response.body.include?("Error")
end
You should use MIME Multipart messages if the API support them (ruby gem).
Otherwise just try to concatenate files' contents request.body = "#{xml_for_initial_request}\n#{second_xml_document}"

Resources