Parsing HTTParty response - ruby

I'm using HTTParty to pull a list of a Facebook user's books but I'm having trouble parsing the response:
Facebook returns data this way:
{
"data": [
{
"name": "Title",
"category": "Book",
"id": "21192118877902",
"created_time": "2011-11-11T20:50:47+0000"
},
{
"name": "Title 2",
"category": "Book",
"id": "1886126860176",
"created_time": "2011-11-05T02:35:56+0000"
},
And HTTParty parses that into a ruby object. I've tried something like this (where ret is the response) ret.parsed_response and that returns the data array, but actually accessing the items inside returns a method not found error.
This is a sample of what HTTParty actually returns:
#<HTTParty::Response:0x7fd0d378c188 #parsed_response={"data"=>
[{"name"=>"Title", "category"=>"Book", "id"=>"21192111877902", "created_time"=>"2011-11-11T20:50:47+0000"},
{"name"=>"Title 2", "category"=>"Book", "id"=>"1886126860176", "created_time"=>"2011-11-05T02:35:56+0000"},
{"name"=>"Thought Patterns", "category"=>"Book", "id"=>"109129539157186", "created_time"=>"2011-10-27T00:00:16+0000"},

Do you have any code that is throwing an error? The parsed_response variable from the HTTParty response is a hash, not an array. It contains one key, "data" (the string, NOT the symbol). The value for the "data" key in the hash is an array of hashes, so you would iterate as such:
data = ret.parsed_response["data"]
data.each do |item|
puts item["name"]
puts item["category"]
puts item["id"]
# etc
end

Just an additional info - It's Not Always a default JSON response
HTTParty's result.response.body or result.response.parsed_response does not always have form of a Hash
It just depends generally on the headers which you are using in your request. For e.g., you need to specify Accept header with application/json value while hitting GitHub API, otherwise it simply returns as string.
Then you shall have to use JSON.parse(data) for same to convert the string response into Hash object.

Related

How can I remove dot from a nested object in logstash

We have a complex object with nested fields that the field names can be dynamic and contains dot.When I try to ingest data to elasticsearch it gives me the following error
Object mapping for [x] tried to parse field [x.y] as object, but found a concrete value
One record can have key/values like a.b.c:4 and in other record it can have a.b:3. We don't have control of the source of coming data so the only option can be changing the object in logstash. Here is an example of coming object:
{
"result": "https://www.yahoo.com",
"tags": {
"url": "https://www.yahoo.com",
"projectName": "monitor",
"host": "ttt",
"dd": 12345,
"vv": "kk"
},
"timestamp": 1586599441000,
"runId": 12345,
"performance": {
"x.y.z": 31307
},
"channel": "clientperf",
"asset": {
"a.b.c": 5,
"a.b":4
}
}
as you see values inside asset and performance has dot. The fields on the roots(like runId, performance and ...) are fine. How can I resolve this either with replacing the dot in logstash or anything that doesn't give me error. I'm aware of de_dot plugin but to use it we need to specifically tell what are the name of nested fields while we cannot enforce the naming for the coming records.I also know that we probably can achieve this with ruby plugin but I have zero knowledge of ruby. Any help can be appreciated.
Could use Hash#deep_transform_keys from ActiveSupport:
class Hash
def deep_transform_keys(&block)
result = {}
each do |key, value|
result[yield(key)] = value.is_a?(Hash) ? value.deep_transform_keys(&block) : value
end
result
end
end
puts hash.deep_transform_keys { |key| key.to_s.gsub(".", "" ) }

Questions with making calls to an API

I've been working on creating a CLI gem for a job board. I've been setting up my API class, but I have been struggling to get it to work correctly in terms of successful calls; I'm using HTTParty to parse. When I have been testing this, it keeps giving me a method error for "[]". I've gone over everything, made sure the syntax is correct but have hit a wall in figuring out what seems to break this. Here is the method I created to list all of the jobs on the specific board:
def all_jobs_call
url = "https://boards-api.greenhouse.io/v1/boards/flatironschoolcareers/jobs"
response = HTTParty.get(url)
response["absolute_url"]["location"]["metadata"]["id"]["title"].each do |job|
absolute_url = job["absolute_url"]
location = job["location"]
metadata = job["metadata"]
id = job["id"]
title = job["title"]
end
end
I would greatly appreciate any insight as to what I could be doing wrong or if I'm missing something glaring. Thanks!
The JSON response you get from https://boards-api.greenhouse.io/v1/boards/flatironschoolcareers/jobs looks like this:
{
"jobs": [
{
"absolute_url": "https://boards.greenhouse.io/flatironschoolcareers/jobs/4460392002",
"internal_job_id": 4375855002,
"location": {
"name": "New York, NY"
},
"metadata": [
{
"id": 4019377002,
"name": "Employment Type",
"value": "Full-time",
"value_type": "single_select"
},
...
HTTParty converts that response to Ruby objects. So just like in that JSON response, response has a top level "jobs" key which contains an array of jobs.
In order to get the 1st job you'd use:
response["jobs"][0]
#=> {"absolute_url"=>"https://boa...", "internal_job_id"=>4375855002, ...}
and to get it's absolute_url:
response["jobs"][0]["absolute_url"]
#=> "https://boards.greenhouse.io/flatironschoolcareers/jobs/4460392002"
And to traverse all jobs you call each on the array, i.e.:
response["jobs"].each do |job|
puts job["absolute_url"]
end
Output:
https://boards.greenhouse.io/flatironschoolcareers/jobs/4460392002
https://boards.greenhouse.io/flatironschoolcareers/jobs/4460383002
https://boards.greenhouse.io/flatironschoolcareers/jobs/4472889002
...

Parsing request JSON in Sinatra app

I am having some difficulty with parsing the JSON from a request to my Sinatra application:
response = JSON.pretty_generate(request.env)
reply = response["rack.request.form_hash"]
results in reply just returning:
rack.request.form_hash
as a string rather than just the relevant part of the response:
{...
"rack.request.form_hash": {
"token": "token",
"team_id": "team",
"team_domain": "teamname",
"service_id": "service",
"channel_id": "channel",
"channel_name": "testing-webhooks",
"timestamp": "1424480976.000910",
"user_id": "U029W1WF2",
"user_name": "myusername",
"text": "checkeverything",
"trigger_word": "checkeverything"
},
...}
which is within the JSON request object I'm trying to parse. When I use:
response["rack.request.form_hash"]["user_name"]
there is nothing returned. The following is returned in my log:
App 1662 stdout:
App 1640 stderr: JSON::ParserError - 746: unexpected token at 'No text specified':
So it looks like it's not iterating properly, or perhaps can't access it.
I've looked through other documentation and other posts, but found nothing that worked for me, but I am definitely overlooking something, but I'm not sure what.
What is the best way to parse this nested array in a request to Sinatra?
This should fix it :
res = JSON.parse( JSON.generate(request.env) )
res.class
# => Hash
res["rack.url_scheme"]
# => http
The reason is that the JSON.generate only generates JSON syntax for objects and arrays in a string. Then you need to parse the generated JSON string into a hash in Ruby with JSON.parse.

Fancytree complains Ajax request returned a string using Cherrypy

Hi I am very new to use cherrypy as backend with fanytree as front end.
here is my fanytree side of the code:
source: {
url : '/test_data'
},
on the cherrypy side, I implemented function called test_data
#cherrypy.expose
#cherrypy.tools.json_out()
def test_data(self, **kwargs):
cherrypy.response.headers["Content-Type"] = "application/json"
return '[ {"title":"abc", "folder": true, "key": "1", "children":[ {"title":"b","key":"2"}] }]'
So I see the request comes to cherrypy as
'GET /test_data?_=some number...
On browser I see my return object back but it failed on check:
if (typeof data === "string") {
$.error("Ajax request returned a string (did you get the JSON dataType wrong?).");
}
I read somewhere that you need content-type to be json but I already have. What am I missing?
CherryPy JSON output tool, cherrypy.tools.json_out, takes care of MIME and turning your data into a JSON string. So if you use it the method should look like:
#cherrypy.expose
#cherrypy.tools.json_out()
def test_data(self, **kwargs):
return [{
"title" : "abc",
"folder" : True,
"key" : 1,
"children" : [{"title": "b", "key": 2}]
}]
Otherwise if you want to do it yourself it'll be:
import json
#cherrypy.expose
def test_data(self, **kwargs):
cherrypy.response.headers["Content-Type"] = "application/json"
return json.dumps([{
"title" : "abc",
"folder" : True,
"key" : 1,
"children" : [{"title": "b", "key": 2}]
}])
Then make sure you've restarted CherryPy app, and look in web developer tools or FireBug network tab to verify response headers and content.
The content type is ok, but the string you are returning is not valid json (for example the keys must be enclosed in double quotes).
I would recommend to prepare your data as list of dicts and then use 'json.dumps()' to convert to JSON.
(Maybe the json_out tool does the same but I would guess that even then you should return a list of dicts instead of a string.)

Efficiently check that a JSON response contains a specific element within an array

Given the JSON response:
{
"tags": [
{
"id": 81499,
"name": "sign-in"
},
{
"id": 81500,
"name": "user"
},
{
"id": 81501,
"name": "authentication"
}
]
}
Using RSpec 2, I want to verify that this response contains the tag with the name authentication. Being a fairly new to Ruby, I figured there is a more efficient way than iterating the array and checking each value of name using include? or map/collect. I could simply user a regex to check for /authentication/i but that doesn't seem like the best approach either.
This is my spec so far:
it "allows filtering" do
response = #client.story(15404)
#response.tags.
end
So, if
t = JSON.parse '{ ... }'
Then this expression will either return nil, which is false, or it will return the thing it detected, which has a boolean evaluation of true.
t['tags'].detect { |e| e['name'] == 'authentication' }
This will raise NoMethodError if there is no tags key. I think that's handled just fine in a test, but you can arrange for that case to also show up as false (i.e., nil) with:
t['tags'].to_a.detect { |e| e['name'] == 'authentication' }

Resources