Related
Writing a plugin for Jekyll, I am stucked at some ruby related code as I am not yet familiar on
how to do .select, .map and all their friends on arrays (except for simple one array cases)
So my problem now is, having this as an array of docs, with their related sha1sum codes:
[
[
#<Jekyll::Document _projects/1st-project.md collection=projects>,
"8f918a3d8263a957c206a9864d3507aaa5277a79"
],
[
#<Jekyll::Document _posts/2nd-project.markdown collection=posts>,
"ca81eda5d49100bdf23db16fe1c9d17040fd33f8"
],
[
#<Jekyll::Document _posts/3rd-project.markdown collection=posts>,
"37bf18464b00c9a808436853bc6d868aff218eb6"
],
...
...
]
And on the other side, the hash of linkage groups like so:
{
"linkage_groups"=>
[
{
"group_name"=>"test1",
"sha1_sums"=> [
"ca81eda5d49100bdf23db16fe1c9d17040fd33f8",
"37bf18464b00c9a808436853bc6d868aff218eb6"
]
},
{
"group_name"=>"test1",
"sha1_sums"=> [
"154c255d59e6063fc609ade2254dc1b09d1de8ab",
"3e9ef9f059786888e39df608d104263cf0fdae76"
]
}
]
}
How would I be able to cycle through these groups one at a time, and return for each group of sha1_sums docs from the above
array where sha1_sum of a document is present in a particular group.
Expected output would be an array of hashes for each of the groups, holding
docs which fulfill the condition, on their sha1_sum being present in a group:
Ex. 2nd and 3rd project fulfill the condition because their sha's are in a group named test1
[
{
"test1" => [#<Jekyll::Document _posts/2nd-project.markdown collection=posts>, #<Jekyll::Document _posts/3rd-project.markdown collection=posts]
},
{
"test2" => [..., ...]
},
{
"test3" => [..., ...]
},
...
]
As of reply from #Lukas Baliak -
Here is what I am getting in case of both hashes belonging to the same group:
{
"ca81eda5d49100bdf23db16fe1c9d17040fd33f8"=>"test1",
"b673be35ad73ab48da23b271ab0dda95ea07c905"=>"test1",
"154c255d59e6063fc609ade2254dc1b09d1de8ab"=>"test2",
"3e9ef9f059786888e39df608d104263cf0fdae76"=>"test2"
}
[
[
#<Jekyll::Document _projects/my-first-project.md collection=projects>,
"b673be35ad73ab48da23b271ab0dda95ea07c905"
],
[
#<Jekyll::Document _posts/2016-06-05-one-more-post.markdown collection=posts>,
"ca81eda5d49100bdf23db16fe1c9d17040fd33f8"
]
]
{
"test1"=> [#<Jekyll::Document _posts/2016-06-05-one-more-post.markdown collection=posts>, "ca81eda5d49100bdf23db16fe1c9d17040fd33f8"]
}
Only one document is listed, why? Where is b673be35ad73ab48da23b271ab0dda95ea07c905?
I preffer to use simple data structures, so i "migrate" groups to Hash
doc
doc = [
[
"#<Jekyll::Document _projects/my-first-project.md collection=projects>",
"8f918a3d8263a957c206a9864d3507aaa5277a79"
],
[
"#<Jekyll::Document _posts/2016-06-05-one-more-post.markdown collection=posts>",
"ca81eda5d49100bdf23db16fe1c9d17040fd33f8"
]
]
group_config
group_config = {
"linkage_groups" => [
{
"group_name" => "test1",
"sha1_sums" => [
"ca81eda5d49100bdf23db16fe1c9d17040fd33f8",
"b673be35ad73ab48da23b271ab0dda95ea07c905"
]
},
{
"group_name" => "test2",
"sha1_sums" => [
"154c255d59e6063fc609ade2254dc1b09d1de8ab",
"8f918a3d8263a957c206a9864d3507aaa5277a79"
]
}
]
}
Migrate to Hash
groups = group_config["linkage_groups"].each_with_object({}) do |h, exp|
h["sha1_sums"].each { |sha1| exp[sha1] = h["group_name"] }
end
export groups
p groups
# {
# "ca81eda5d49100bdf23db16fe1c9d17040fd33f8" => "test1",
# "b673be35ad73ab48da23b271ab0dda95ea07c905" => "test1",
# "154c255d59e6063fc609ade2254dc1b09d1de8ab" => "test2",
# "8f918a3d8263a957c206a9864d3507aaa5277a79" => "test2"
# }
And process to generate hash structure
export = doc.each_with_object({}) do |arr, exp|
exp[groups[arr[1]]] = arr
end
output
p export
# {
# "test2" => ["#<Jekyll::Document _projects/my-first-project.md collection=projects>", "8f918a3d8263a957c206a9864d3507aaa5277a79"],
# "test1" => ["#<Jekyll::Document _posts/2016-06-05-one-more-post.markdown collection=posts>", "ca81eda5d49100bdf23db16fe1c9d17040fd33f8"]
# }
EDIT:
If you need more then one use this modification
export = doc.each_with_object(Hash.new{|k, v| k[v] = []}) do |arr, exp|
exp[groups[arr[1]]] << arr
end
EDIT 2
Ok, if you need one sha1_hash in to more groups may be you can use this update.
groups = group_config["linkage_groups"].each_with_object(Hash.new { |k, v| k[v] = [] }) do |h, exp|
h["sha1_sums"].each { |sha1| exp[sha1] << h["group_name"] }
end
groups
p groups
# {
# "ca81eda5d49100bdf23db16fe1c9d17040fd33f8" => ["test1"],
# "b673be35ad73ab48da23b271ab0dda95ea07c905" => ["test1"],
# "8f918a3d8263a957c206a9864d3507aaa5277a79" => ["test1", "test2"],
# "154c255d59e6063fc609ade2254dc1b09d1de8ab" => ["test2"]
# }
process
export = doc.each_with_object(Hash.new { |k, v| k[v] = [] }) do |arr, exp|
groups[arr[1]].each { |group| exp[group] << arr }
end
output
p export
# {
# "test1" => [
# ["#<Jekyll::Document _projects/my-first-project.md collection=projects>", "8f918a3d8263a957c206a9864d3507aaa5277a79"],
# ["#<Jekyll::Document _posts/2016-06-05-one-more-post.markdown collection=posts>", "ca81eda5d49100bdf23db16fe1c9d17040fd33f8"]
# ],
# "test2" => [
# ["#<Jekyll::Document _projects/my-first-project.md collection=projects>", "8f918a3d8263a957c206a9864d3507aaa5277a79"],
# ["#<Jekyll::Document _posts/2016-06-0s-one-more-post.markdown collection=posts>", "154c255d59e6063fc609ade2254dc1b09d1de8ab"]
# ]
# }
I hope this will helps.
You are not very clear in your question. Sometimes it helps to be detailed about what output is expected for which input. I assume the array contains the Document/Sha1 pairs and that hash contains the linkage_groups.
The hash looks like this:
{"linkage_groups"=>[{"group_name"=>"test1", "sha1_sums"=>["ca81eda5d49100bdf23db16fe1c9d17040fd33f8", "b673be35ad73ab48da23b271ab0dda95ea07c905"]}, {"group_name"=>"test1", "sha1_sums"=>["154c255d59e6063fc609ade2254dc1b09d1de8ab", "3e9ef9f059786888e39df608d104263cf0fdae76"]}]}
And the array looks like this:
[["Document1", "8f918a3d8263a957c206a9864d3507aaa5277a79"], ["Document2", "ca81eda5d49100bdf23db16fe1c9d17040fd33f8"]]
I'd try something like this:
hash["linkage_groups"].each { |group| // for each linkage group
group["sha1_sums"].each { |sha1| // for each sha1 in group
array.each { |array_element| // for each array element (which itself is an array of document/sha1 pairs
if array_element.include?(sha1) then
puts "#{array_element[0]} found in #{group["group_name"]} with #{group["sha1"]}"
end
}
}
}
I leave it up to you to manage how to return the elements according to your needs.
The hash I'm working with has a hash for it's values which always contains an ID, name, and description. I am not interested in keeping the ID or name and just want to replace every hash value with its corresponding description.
Code
hsh['nested']['entries']['addr'] = hsh['nested']['entries']['addr']['description']
hsh['nested']['entries']['port'] = hsh['nested']['entries']['port']['description']
hsh['nested']['entries']['protocol'] = hsh['nested']['entries']['protocol']['description']
hsh['nested']['entries']['type'] = hsh['nested']['entries']['type']['description']
... (many more)
This works fine, but it is not very elegant--in reality, I have 20 entries/lines of code to get the job done.
Structure of the hash value (for hsh['nested']['entries']['addr'])
{ "id" => "27", "name" => "Instance", "description" => "**This is what I need.**" }
Taking the first line of code above as a sample, the end result would be the value of hsh['nested']['entries']['addr'] becomes **This is what I need.**
What is an elegant way to achieve this?
hsh = { 'nested'=>
{ 'entries'=>
{
'addr'=>{ "id" => "1", "description"=>"addr" },
'port'=>{ "id" => "2", "description"=>"port" },
'cats'=>{ "id" => "3", "description"=>"dogs" },
'type'=>{ "id" => "4", "description"=>"type" }
}
}
}
keys_to_replace = ["addr", "port", "type"]
hsh['nested']['entries'].tap { |h| keys_to_replace.each { |k| h[k]=h[k]["description"] }
#=> { "addr"=>"addr",
# "port"=>"port",
# "cats"=>{"id"=>"3", "description"=>"dogs"},
# "type"=>"type"
# }
hsh
#=> {"nested"=>
# { "entries"=>
# { "addr"=>"addr",
# "port"=>"port",
# "cats"=>{"id"=>"3", "description"=>"dogs"},
# "type"=>"type"
# }
# }
# }
sub_hash = hsh['nested']['entries']
categories = %w{addr port protocol type}
categories.each do |category|
sub_hash[category] = sub_hash[category]['description']
end
A GET to an API endpoint I'm working with returns json with an inconsistent order of contacts, either
{"contacts"=>[
{"id"=>$UUID_0, "name"=>nil, "email"=>$EMAIL_0, "phone"=>$PHONE_0, "type"=>"foo"},
{"id"=>$UUID_1, "name"=>nil, "email"=>$EMAIL_1, "phone"=>$PHONE_1, "type"=>"bar"}
]}
or
{"contacts"=>[
{"id"=>$UUID_1, "name"=>nil, "email"=>$EMAIL_1, "phone"=>$PHONE_1, "type"=>"bar"},
{"id"=>$UUID_0, "name"=>nil, "email"=>$EMAIL_0, "phone"=>$PHONE_0, "type"=>"foo"}
]}
The "type" values are the only static objects in these responses, so I'd like to map this so that the contact types are keys containing the other pairs:
{
"foo"=>{"id"=>$UUID_0, "name"=>$NAME_0, "email"=>$EMAIL_0, "phone"=>$PHONE_0},
"bar"=>{"id"=>$UUID_1, "name"=>$NAME_1, "email"=>$EMAIL_1, "phone"=>$PHONE_1}
}
A solution is not obvious to me.
If you use Ruby on Rails, or at least ActiveSupport, you can try index_by instead of group_by: it won't put the values into arrays.
hash['contacts'].index_by {|r| r['type']}
=>
{
"bar" => {
"id" => "asdf",
"name" => nil,
"email" => "EMAIL_1",
"phone" => "PHONE_1",
"type" => "bar"
},
"foo" => {
"id" => "asdf",
"name" => nil,
"email" => "EMAIL_0",
"phone" => "PHONE_0",
"type" => "foo"
}
}
Hash[data['contacts'].map { |c| [c['type'], c] }]
This can be done with Enumerable#reduce:
hash['contacts'].reduce({}) {|m,c| m[c['type']] = c;m}
How it works:
An empty hash is the starting point.
The block is called once for each element in the contacts list. The block receives the hash that we're building as m and the current contact as c.
In the block, assign c to the hash based on its type and return the hash so far.
Final result is the last return value of the block.
I have a collection in which one of the field is "type". I want to get some values of each type depending upon condition which is same for all the types. Like I want 2 documents for type A, 2 for type B like that.
How to do this in a single query? I am using Ruby Active Record.
Generally what you are describing is a relatively common question around the MongoDB community which we could describe as the "top n results problem". This is when given some input that is likely sorted in some way, how to get the top n results without relying on arbitrary index values in the data.
MongoDB has the $first operator which is available to the aggregation framework which deals with the "top 1" part of the problem, as this actually takes the "first" item found on a grouping boundary, such as your "type". But getting more than "one" result of course gets a little more involved. There are some JIRA issues on this about modifying other operators to deal with n results or "restrict" or "slice". Notably SERVER-6074. But the problem can be handled in a few ways.
Popular implementations of the rails Active Record pattern for MongoDB storage are Mongoid and Mongo Mapper, both allow access to the "native" mongodb collection functions via a .collection accessor. This is what you basically need to be able to use native methods such as .aggregate() which supports more functionality than general Active Record aggregation.
Here is an aggregation approach with mongoid, though the general code does not alter once you have access to the native collection object:
require "mongoid"
require "pp";
Mongoid.configure.connect_to("test");
class Item
include Mongoid::Document
store_in collection: "item"
field :type, type: String
field :pos, type: String
end
Item.collection.drop
Item.collection.insert( :type => "A", :pos => "First" )
Item.collection.insert( :type => "A", :pos => "Second" )
Item.collection.insert( :type => "A", :pos => "Third" )
Item.collection.insert( :type => "A", :pos => "Forth" )
Item.collection.insert( :type => "B", :pos => "First" )
Item.collection.insert( :type => "B", :pos => "Second" )
Item.collection.insert( :type => "B", :pos => "Third" )
Item.collection.insert( :type => "B", :pos => "Forth" )
res = Item.collection.aggregate([
{ "$group" => {
"_id" => "$type",
"docs" => {
"$push" => {
"pos" => "$pos", "type" => "$type"
}
},
"one" => {
"$first" => {
"pos" => "$pos", "type" => "$type"
}
}
}},
{ "$unwind" => "$docs" },
{ "$project" => {
"docs" => {
"pos" => "$docs.pos",
"type" => "$docs.type",
"seen" => {
"$eq" => [ "$one", "$docs" ]
},
},
"one" => 1
}},
{ "$match" => {
"docs.seen" => false
}},
{ "$group" => {
"_id" => "$_id",
"one" => { "$first" => "$one" },
"two" => {
"$first" => {
"pos" => "$docs.pos",
"type" => "$docs.type"
}
},
"splitter" => {
"$first" => {
"$literal" => ["one","two"]
}
}
}},
{ "$unwind" => "$splitter" },
{ "$project" => {
"_id" => 0,
"type" => {
"$cond" => [
{ "$eq" => [ "$splitter", "one" ] },
"$one.type",
"$two.type"
]
},
"pos" => {
"$cond" => [
{ "$eq" => [ "$splitter", "one" ] },
"$one.pos",
"$two.pos"
]
}
}}
])
pp res
The naming in the documents is actually not used by the code, and titles in the data shown for "First", "Second" etc, are really just there to illustrate that you are indeed getting the "top 2" documents from the listing as a result.
So the approach here is essentially to create a "stack" of the documents "grouped" by your key, such as "type". The very first thing here is to take the "first" document from that stack using the $first operator.
The subsequent steps match the "seen" elements from the stack and filter them, then you take the "next" document off of the stack again using the $first operator. The final steps in there are really justx to return the documents to the original form as found in the input, which is generally what is expected from such a query.
So the result is of course, just the top 2 documents for each type:
{ "type"=>"A", "pos"=>"First" }
{ "type"=>"A", "pos"=>"Second" }
{ "type"=>"B", "pos"=>"First" }
{ "type"=>"B", "pos"=>"Second" }
There was a longer discussion and version of this as well as other solutions in this recent answer:
Mongodb aggregation $group, restrict length of array
Essentially the same thing despite the title and that case was looking to match up to 10 top entries or greater. There is some pipeline generation code there as well for dealing with larger matches as well as some alternate approaches that may be considered depending on your data.
You will not be able to do this directly with only the type column and the constraint that it must be one query. However there is (as always) a way to accomplish this.
To find documents of different types, you would need to have some type of additional value that, on average distributed the types out according to how you want the data back.
db.users.insert({type: 'A', index: 1})
db.users.insert({type: 'B', index: 2})
db.users.insert({type: 'A', index: 3})
db.users.insert({type: 'B', index: 4})
db.users.insert({type: 'A', index: 5})
db.users.insert({type: 'B', index: 6})
Then when querying for items with db.users.find(index: {$gt: 2, $lt: 7}) you will have the right distribution of items.
Though I'm not sure this was what you were looking for
After parsing some JSON:
data = JSON.parse(data)['info']
puts data
I get:
[
{
"title"=>"CEO",
"name"=>"George",
"columns"=>[
{
"display_name"=> "Salary",
"value"=>"3.85",
}
, {
"display_name"=> "Bonus",
"value"=>"994.19",
}
, {
"display_name"=> "Increment",
"value"=>"8.15",
}
]
}
]
columns has nested data in itself.
I want to save the data in a database or CSV file.
title, name, value_Salary, value_Bonus, value_increment
But I'm not concerned about getting display_name, so just the values of first of columns, second of columns data, etc.
Ok I tried data.map after converting to hash & hash.flatten could find a way out.. .map{|x| x['columns']}
.map {|s| s["value"]}
tried to get the values atleast separately - but couldnt...
This is a simple problem, and resolves down to a couple nested map blocks.
Here's the data retrieved from JSON, plus an extra row to demonstrate how easy it is to handle a more complex JSON response:
data = [
{
"title" => "CEO",
"name" => "George",
"columns" => [
{
"display_name" => "Salary",
"value" => "3.85",
},
{
"display_name" => "Bonus",
"value" => "994.19",
},
{
"display_name" => "Increment",
"value" => "8.15",
}
]
},
{
"title" => "CIO",
"name" => "Fred",
"columns" => [
{
"display_name" => "Salary",
"value" => "3.84",
},
{
"display_name" => "Bonus",
"value" => "994.20",
},
{
"display_name" => "Increment",
"value" => "8.15",
}
]
}
]
Here's the code:
records = data.map { |record|
title, name = record.values_at('title', 'name')
values = record['columns'].map{ |column| column['value'] }
[title, name, *values]
}
Here's the resulting data structure, an array of arrays:
records
# => [["CEO", "George", "3.85", "994.19", "8.15"],
# ["CIO", "Fred", "3.84", "994.20", "8.15"]]
Saving it into a database or CSV is left for you to figure out, but Ruby's CSV class makes it trivial to write a file, and an ORM like Sequel makes it really easy to insert the data into a database.