Difficulty processing json with ruby - ruby

Question

I have the following json...
{
"NumPages":"17",
"Page":"1",
"PageSize":"50",
"Total":"808",
"Start":"1",
"End":"50",
"FirstPageUri":"/v3/results?PAGE=1",
"LastPageUri":"/v3/results?PAGE=17",
"PreviousPageUri":"",
"NextPageUri":"/v3/results?PAGE=2",
"User":[
{
"RowNumber":"1",
"UserId":"86938",
"InternalId":"",
"CompletionPercentage":"100",
"DateTimeTaken":"2014-06-18T01:43:25Z",
"DateTimeLastUpdated":"2014-06-18T01:58:11Z",
"DateTimeCompleted":"2014-06-18T01:58:11Z",
"Account":{
"Id":"655",
"Name":"Technical Community College"
},
"FirstName":"Matthew",
"LastName":"Knice",
"EmailAddress":"knice#gmail.com",
"AssessmentResults":[
{
"Title":"Life Factors",
"Code":"LifeFactors",
"IsComplete":"1",
"AttemptNumber":"1",
"Percent":"58",
"Readiness":"fail",
"DateTimeCompleted":"2014-06-18T01:46:00Z"
},
{
"Title":"Learning Styles",
"Code":"LearnStyles",
"IsComplete":"0"
},
{
"Title":"Personal Attributes",
"Code":"PersonalAttributes",
"IsComplete":"1",
"AttemptNumber":"1",
"Percent":"52.08",
"Readiness":"fail",
"DateTimeCompleted":"2014-06-18T01:49:00Z"
},
{
"Title":"Technical Competency",
"Code":"TechComp",
"IsComplete":"1",
"AttemptNumber":"1",
"Percent":"100",
"Readiness":"pass",
"DateTimeCompleted":"2014-06-18T01:51:00Z"
},
{
"Title":"Technical Knowledge",
"Code":"TechKnowledge",
"IsComplete":"1",
"AttemptNumber":"1",
"Percent":"73.44",
"Readiness":"question",
"DateTimeCompleted":"2014-06-18T01:58:00Z"
},
{
"Title":"Reading Rate & Recall",
"Code":"Reading",
"IsComplete":"0"
},
{
"Title":"Typing Speed & Accuracy",
"Code":"Typing",
"IsComplete":"0"
}
]
},
{
"RowNumber":"2",
"UserId":"8654723",
"InternalId":"",
"CompletionPercentage":"100",
"DateTimeTaken":"2014-06-13T14:37:59Z",
"DateTimeLastUpdated":"2014-06-13T15:00:12Z",
"DateTimeCompleted":"2014-06-13T15:00:12Z",
"Account":{
"Id":"655",
"Name":"Technical Community College"
},
"FirstName":"Virginia",
"LastName":"Bustas",
"EmailAddress":"bigBusta#students.college.edu",
"AssessmentResults":[
{
...
I need to start processing where you see "User:" The stuff at the beginning (numpages, page, ect) I want to ignore. Here is the processing script I am working on...
require 'csv'
require 'json'
CSV.open("your_csv.csv", "w") do |csv| #open new file for write
JSON.parse(File.open("sample.json").read).each do |hash| #open json to parse
csv << hash.values
end
end
Right now this fails with the error:
convert.rb:6:in `block (2 levels) in <main>': undefined method `values' for ["NumPages", "17"]:Array (NoMethodError)
I have ran the json through a parser, and it seems to be valid. What is the best way to only process the "User" data?

Answer 1

You have to look at the structure of the JSON object being created. Here's a very small subset of your document being parsed, which makes it easier to see and understand:
require 'json'
foo = '{"NumPages":17,"User":[{"UserId":12345}]}'
bar = JSON[foo]
# => {"NumPages"=>17, "User"=>[{"UserId"=>12345}]}
bar['User'].first['UserId'] # => 12345
foo contains the JSON for a hash. bar contains the Ruby object created by the JSON parser after it reads foo.
User is the key pointing to an array of hashes. Because it's an array, you have to specify which of the hashes in the array you want to look at, which is what bar['User'].first does.
An alternate way to access that sub-hash is:
bar['User'][0]['UserId'] # => 12345
If there were multiple hashes inside the array, you could access them by using the appropriate index value. For example, if there are two hashes, and I want the second one:
foo = '{"NumPages":17,"User":[{"UserId":12345},{"UserId":12346}]}'
bar = JSON[foo]
# => {"NumPages"=>17, "User"=>[{"UserId"=>12345}, {"UserId"=>12346}]}
bar['User'].first['UserId'] # => 12345
bar['User'][0]['UserId'] # => 12345
bar['User'][1]['UserId'] # => 12346
I'm wondering if I am going down the wrong road with the JSON.parse(File.open("sample.json").read).each do |hash|?
Yes, you are. You need to understand what you're doing, and break your code into digestible pieces so they make sense to you. Consider this:
require 'csv'
require 'json'
json_object = JSON.parse(File.read("sample.json"))
CSV.open("your_csv.csv", "w") do |csv| #open new file for write
csv << %w[RowNumber UserID AccountID AccountName FirstName LastName EmailAddress]
json_object['User'].each do |user_hash|
puts 'RowNumber: %s' % user_hash['RowNumber']
puts 'UserID: %s' % user_hash['UserID']
account = user_hash['UserID']['Account']
puts 'Account->Id: %s' % account['Id']
puts 'Account->Name: %s' % account['Name']
puts 'FirstName: %s' % user_hash['FirstName']
puts 'LastName: %s' % user_hash['LastName']
puts 'EmailAddress: %s' % user_hash['EmailAddress']
csv << [
user_hash['RowNumber'],
user_hash['UserID'],
account['Id'],
account['Name'],
user_hash['FirstName'],
user_hash['LastName'],
user_hash['EmailAddress']
]
end
end
This reads the JSON file and parses it into a Ruby object immediately. There is no special magic or anything else that happens with the file, it's opened, read, closed, and its content is passed to the JSON parser and assigned to json_object.
Once parsed, the CSV file is opened and a header row is written. It could have been written as part of the open statement but this is clearer for explaining what's going on.
json_object is a hash, so to access the 'User' data you have to use a normal hash access json_object['User']. The value for the User key is an array of hashes, so those need to be iterated over, which is what json_object['User'].each does, passing the hash elements of that array into the block as user_hash.
Inside that block it's pretty much the same thing as access the value for 'User', each "element" is a key/value pair, except 'Account' which is an embedded hash.

Answer 2

Read the error message. each called on a hash is giving you a sequence of arrays with two members (the key and value together). There is no values method on an array. And in any case if what you have is a hash there seems little point cycling through it with each; if you want the "User" entry in the hash, why don't you ask for it up front?

Answer 3

Just for posterity and context this is the script I ended up using in its entity. I needed to pull from a url, and process the results and move them to a simple CSV. I needed to wite the student id, first name, last name, and the score from each of 4 assessments to the csv.
require 'csv'
require 'json'
require 'curb'
c = Curl::Easy.new('myURL/m/v3/results')
c.http_auth_types = :basic
c.username = 'myusername'
c.password = 'mypassword'
c.perform
json_object = JSON.parse(c.body_str)
CSV.open("your_csv.csv", "w") do |csv| #open new file for write
csv << %w[UserID FirstName LastName LifeFactors PersonalAttributes TechComp TechKnowledge]
json_object['User'].each do |user_hash|
csv << [
user_hash['UserId'],
user_hash['FirstName'],
user_hash['LastName'],
user_hash['AssessmentResults'][0]['Percent'],
user_hash['AssessmentResults'][2]['Percent'],
user_hash['AssessmentResults'][3]['Percent'],
user_hash['AssessmentResults'][4]['Percent']
]
end
end