Breaking out variably deeply nested hashes into separate hashes with Ruby - ruby

Question

I'm pretty new to Ruby, but I've done a ton of searches, research here on Stack, and experimentation.
I'm getting POST data that contains variable information which I am able to convert into a hash from XML.
My objectives are to:
Get and store the parentage key hierarchy.
I'm creating MongoDb records of what I get via these POSTs, and I need to record what keys I get storing any new ones I get that aren't already part of the collections keys.
Once I have the key hierarchy stored, I need to take the nested hash and break out each top level key and its children into another hash. These will end up as individual subdocuments in a MongoDb record.
A big obstacle is that I won't know the hierarchy structure or any of the key names up front, so I have to create an parser that doesn't really care what is in the hash, it just organizes the key structure, and breaks the hash up into separate hashes representing each 'top level' key contained in a hash.
I have a nested hash:
{"hashdata"=>
{"ComputersCount"=>
{"Total"=>1, "Licensed"=>1, "ByOS"=>{"OS"=>{"Windows 7 x64"=>1}}},
"ScansCount"=>
{"Total"=>8,
"Scheduled"=>8,
"Agent"=>0,
"ByScanningProfile"=>{"Profile"=>{"Missing Patches"=>8}}},
"RemediationsCount"=>{"Total"=>1, "ByType"=>{"Type"=>{"9"=>1}}},
"AgentsCount"=>{"Total"=>0},
"RelaysCount"=>{"Total"=>0},
"ScanResultsDatabase"=>{"Type"=>"MSAccess"}}}
In this example, ignoring the 'hashdata' key, the 'top level' parents are:
ComputersCount
ScansCount
RemediationsCount
RelaysCount
ScanResultsDatabase
So ideally, I would end up with a hash of each parent key and its children keys, and a separate hash for each of the top level parents.
EDIT: I'm not sure the best way to articulate the 'keys hash' but I know it needs to contain a sense of the hierarchy structure with regards to what level and parent a key in the structure might have.
For the separate hashes themselves it could be as simple as:
{"ComputersCount"=>{"Total"=>1, "Licensed"=>1, "ByOS"=>{"OS"=>{"Windows 7 x64"=>1}}}}
{"ScansCount"=>{"Total"=>8,"Scheduled"=>8,"Agent"=>0,"ByScanningProfile"=>{"Profile"=>{"Missing Patches"=>8}}}}
{"RemediationsCount"=>{"Total"=>1, "ByType"=>{"Type"=>{"9"=>1}}}}
{"AgentsCount"=>{"Total"=>0}}
{"RelaysCount"=>{"Total"=>0}}
{"ScanResultsDatabase"=>{"Type"=>"MSAccess"}}}
My ultimate goal is to take the key collections and the hash collections and store them in MongoDb, each sub hash is a sub-document, and the keys collection gives me a column name map for the collection so it can be queried against later.
I've come close to a solution using some recursive methods for example:
def recurse_hash(h,p=nil)
h.each_pair do |k,v|
case v
when String, Fixnum then
p "Key: #{k}, Value: #{v}"
when Hash then
h.find_all_values_for(v)
recurse_hash(v,k)
else raise ArgumentError "Unhandled type #{v.class}"
end
end
end
But so far, I've only been able to get close to what I'm after. Ultimately, I need to be prepared to get hashes with any level of nesting or value structures because the POST data is highly variable.
Any advice, guidance or other assistance here would be greatly appreciated - I realize I could very well be approaching this entire challenge incorrectly.

Answer 1

Looks like you want an array of hashes like the following:
array = hash["hashdata"].map { |k,v| { k => v } }
# => [{"ComputersCount"=>{"Total"=>1, "Licensed"=>1, "ByOS"=>{"OS"=>{"Windows 7 x64"=>1}}}}, ... ]
array.first
# => {"ComputersCount"=>{"Total"=>1, "Licensed"=>1, "ByOS"=>{"OS"=>{"Windows 7 x64"=>1}}}}
array.last
# => {"ScanResultsDatabase"=>{"Type"=>"MSAccess"}}

Answer 2

Here's my best guess at the "key structure hierarchy and parentage."
I gently suggest that it is overkill.
Instead, I think that all you really need to do is just store your hashdata directly as MongoDB documents.
Even if your POST data is highly variable,
in all likelyhood it will still be sufficiently well-formed that you can write your application without difficulty.
Here's a test that incorporates "key structure hierarcy and parentage",
but maybe more importantly just shows how trivial it is to store your hashdata directly as a MongoDB document.
The test is run twice to demonstrate new key discovery.
test.rb
require 'mongo'
require 'test/unit'
require 'pp'
def key_structure(h)
h.keys.sort.collect{|k| v = h[k]; v.is_a?(Hash) ? [k, key_structure(h[k])] : k}
end
class MyTest < Test::Unit::TestCase
def setup
#hash_data_coll = Mongo::MongoClient.new['test']['hash_data']
#hash_data_coll.remove
#keys_coll = Mongo::MongoClient.new['test']['keys']
end
test "extract cancer drugs" do
hash_data = {
"hashdata" =>
{"ComputersCount" =>
{"Total" => 1, "Licensed" => 1, "ByOS" => {"OS" => {"Windows 7 x64" => 1}}},
"ScansCount" =>
{"Total" => 8,
"Scheduled" => 8,
"Agent" => 0,
"ByScanningProfile" => {"Profile" => {"Missing Patches" => 8}}},
"RemediationsCount" => {"Total" => 1, "ByType" => {"Type" => {"9" => 1}}},
"AgentsCount" => {"Total" => 0},
"RelaysCount" => {"Total" => 0},
"ScanResultsDatabase" => {"Type" => "MSAccess"}}}
known_keys = #keys_coll.find.to_a.collect{|doc| doc['key']}.sort
puts "known keys: #{known_keys}"
hash_data_keys = hash_data['hashdata'].keys.sort
puts "hash data keys: #{hash_data_keys.inspect}"
new_keys = hash_data_keys - known_keys
puts "new keys: #{new_keys.inspect}"
#keys_coll.insert(new_keys.collect{|key| {key: key, structure: key_structure(hash_data['hashdata'][key]), timestamp: Time.now}}) unless new_keys.empty?
pp #keys_coll.find.to_a unless new_keys.empty?
#hash_data_coll.insert(hash_data['hashdata'])
assert_equal(1, #hash_data_coll.count)
pp #hash_data_coll.find.to_a
end
end
$ ruby test.rb
Loaded suite test
Started
known keys: []
hash data keys: ["AgentsCount", "ComputersCount", "RelaysCount", "RemediationsCount", "ScanResultsDatabase", "ScansCount"]
new keys: ["AgentsCount", "ComputersCount", "RelaysCount", "RemediationsCount", "ScanResultsDatabase", "ScansCount"]
[{"_id"=>BSON::ObjectId('535976177f11ba278d000001'),
"key"=>"AgentsCount",
"structure"=>["Total"],
"timestamp"=>2014-04-24 20:37:43 UTC},
{"_id"=>BSON::ObjectId('535976177f11ba278d000002'),
"key"=>"ComputersCount",
"structure"=>[["ByOS", [["OS", ["Windows 7 x64"]]]], "Licensed", "Total"],
"timestamp"=>2014-04-24 20:37:43 UTC},
{"_id"=>BSON::ObjectId('535976177f11ba278d000003'),
"key"=>"RelaysCount",
"structure"=>["Total"],
"timestamp"=>2014-04-24 20:37:43 UTC},
{"_id"=>BSON::ObjectId('535976177f11ba278d000004'),
"key"=>"RemediationsCount",
"structure"=>[["ByType", [["Type", ["9"]]]], "Total"],
"timestamp"=>2014-04-24 20:37:43 UTC},
{"_id"=>BSON::ObjectId('535976177f11ba278d000005'),
"key"=>"ScanResultsDatabase",
"structure"=>["Type"],
"timestamp"=>2014-04-24 20:37:43 UTC},
{"_id"=>BSON::ObjectId('535976177f11ba278d000006'),
"key"=>"ScansCount",
"structure"=>
["Agent",
["ByScanningProfile", [["Profile", ["Missing Patches"]]]],
"Scheduled",
"Total"],
"timestamp"=>2014-04-24 20:37:43 UTC}]
[{"_id"=>BSON::ObjectId('535976177f11ba278d000007'),
"ComputersCount"=>
{"Total"=>1, "Licensed"=>1, "ByOS"=>{"OS"=>{"Windows 7 x64"=>1}}},
"ScansCount"=>
{"Total"=>8,
"Scheduled"=>8,
"Agent"=>0,
"ByScanningProfile"=>{"Profile"=>{"Missing Patches"=>8}}},
"RemediationsCount"=>{"Total"=>1, "ByType"=>{"Type"=>{"9"=>1}}},
"AgentsCount"=>{"Total"=>0},
"RelaysCount"=>{"Total"=>0},
"ScanResultsDatabase"=>{"Type"=>"MSAccess"}}]
.
Finished in 0.028869 seconds.
1 tests, 1 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
34.64 tests/s, 34.64 assertions/s
$ ruby test.rb
Loaded suite test
Started
known keys: ["AgentsCount", "ComputersCount", "RelaysCount", "RemediationsCount", "ScanResultsDatabase", "ScansCount"]
hash data keys: ["AgentsCount", "ComputersCount", "RelaysCount", "RemediationsCount", "ScanResultsDatabase", "ScansCount"]
new keys: []
[{"_id"=>BSON::ObjectId('535976197f11ba278e000001'),
"ComputersCount"=>
{"Total"=>1, "Licensed"=>1, "ByOS"=>{"OS"=>{"Windows 7 x64"=>1}}},
"ScansCount"=>
{"Total"=>8,
"Scheduled"=>8,
"Agent"=>0,
"ByScanningProfile"=>{"Profile"=>{"Missing Patches"=>8}}},
"RemediationsCount"=>{"Total"=>1, "ByType"=>{"Type"=>{"9"=>1}}},
"AgentsCount"=>{"Total"=>0},
"RelaysCount"=>{"Total"=>0},
"ScanResultsDatabase"=>{"Type"=>"MSAccess"}}]
.
Finished in 0.015559 seconds.
1 tests, 1 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
64.27 tests/s, 64.27 assertions/s