Merging and updating hash keys in Redis with Ruby - ruby

I have a list the following hashes:
{
key_main1: {
k1: 1,
k2: 11,
k3: 33,
k4: 146,
k5: 12,
# etc
},
key_main2: {
k1: 1,
k2: 11,
k3: 33,
k4: 146,
k5: 12,
# etc
},
# etc
}
which is save in a redis as json:
redis_key1 = "redis_key1"
redis.set("redis_key", my_hash.to_json)
redis.get("redis_key") # =>
"{"key_main":{"k1":1,"k2":11,"k3":33,"k4":146,"k5":12}}"
The hashes have the same structure but can have different keys. On each iteration I want to update the existing keys by summing up the values of "k"s or/and insert the keys that don't exist:
So the second hash in the list looks like this
{
key_main1: {
k3: 44,
k4: 14,
k18: 99
},
key_main3: {
k2: 77
}
}
Then after the seconds iteration the result in Redis will look like the following:
{
key_main1: {
k1: 1,
k2: 11,
k3: 33,
k4: 160,
k5: 12,
k18: 99
},
key_main2: {
k1: 1,
k2: 11,
k3: 33,
k4: 146,
k5: 12
},
key_main3: {
k2: 77
}
}
What's the easiest and best way to do it? Do I have to parse (restore)json in each iteration in order to check if the keys exist and update or insert them?

The best way is to parse the JSON back into a hash, otherwise you will have a really hard time figuring out how to combine the hashes.
To combine them, the best and easiest is with Hash#merge:
h1 = { m1: { k1: 10, k2: 20, k3: 30 }, m2: { k1: 11, k2: 12 } }
h2 = { m1: { k1: 500, k2: 5, k4: 40 }, m3: { k2: 123 } }
pp h1.merge(h2) { |key, v1, v2|
v1.merge(v2) { |key, v1, v2| v1 + v2 }
}
=> { :m1 => { :k1 => 510,:k2 => 25, :k3 => 30, :k4 => 40 },
:m2 => { :k1 => 11, :k2 => 12},
:m3 => { :k2 => 123 } }
This code assumes the keys in h1 and h2 always contain a hash with integer keys.

Related

Inconsistant Results from Searchkick / Elasticsearch Aggregations

First off I will work on cleaning up the output from IRB (this was from production).
I was using some search kick aggregations in my Rails app. tested fine in development now in production I am seeing inconsistent results. I have data in fields that should clearly show up in my aggregations.
Here is my plain old Active Record Data - my BudgetItem model has a total:
irb(main):008:0> BudgetItem.where(budget_id: 3).order(:cbs_item_id).select(:id, :cbs_item_id, :total)
BudgetItem Load (1.4ms) SELECT "budget_items"."id", "budget_items"."cbs_item_id", "budget_items"."total" FROM "budget_items" WHERE "budget_items"."company_id" = 26 AND "budget_items"."budget_id" = $1 ORDER BY "budget_items"."cbs_item_id" ASC LIMIT $2 [["budget_id", 3], ["LIMIT", 11]]
=> #<ActiveRecord::Relation [#<BudgetItem id: 28, company_id: 26, cbs_item_id: 3, total: 0.1e5>, #<BudgetItem id: 29, company_id: 26, cbs_item_id: 12, total: 0.8e5>, #<BudgetItem id: 34, company_id: 26, cbs_item_id: 15, total: 0.1e5>, #<BudgetItem id: 41, company_id: 26, cbs_item_id: 16, total: 0.141e6>, #<BudgetItem id: 35, company_id: 26, cbs_item_id: 18, total: 0.1e5>, #<BudgetItem id: 33, company_id: 26, cbs_item_id: 18, total: 0.12e5>, #<BudgetItem id: 27, company_id: 26, cbs_item_id: 20, total: 0.2e5>, #<BudgetItem id: 6, company_id: 26, cbs_item_id: 23, total: 0.184e6>, #<BudgetItem id: 5, company_id: 26, cbs_item_id: 23, total: 0.2288e6>, #<BudgetItem id: 30, company_id: 26, cbs_item_id: 24, total: 0.45e5>, ...]>
The count on this is 41
the same search via Searchkick:
BudgetItem.search("*", where: {budget_id: 3}).count = 41
and even this:
irb(main):025:0> BudgetItem.search("*", where: {budget_id: 3, cbs_item_id: 16}).first.total
= 26 AND "budget_items"."id" = $1 [["id", 41]]
=> 0.141e6
Note cbs_item_id: 16, total: 0.141e6 (141000) - the value is clearly in the model.
Now I try and run aggregates on this:
irb(main):019:0> BudgetItem.search("*", body_options: { aggs: { cbs: { terms: { field: "cbs_item_id" }, aggs: { "total": { "sum": { "field": "total" } } } } } }, where: {budget_id: 3}).aggs
BudgetItem Search (5.4ms) pacific-canbriam-20191213_budget_items_production/_search {"query":{"bool":{"must":{"match_all":{}},"filter":[{"term":{"budget_id":{"value":3}}}]}},"timeout":"11s","_source":false,"size":10000,"aggs":{"cbs":{"terms":{"field":"cbs_item_id"},"aggs":{"total":{"sum":{"field":"total"}}}}}}
=> {"cbs"=>{"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>13, "buckets"=>[{"key"=>24, "doc_count"=>4, "total"=>{"value"=>90000.0}}, {"key"=>25, "doc_count"=>4, "total"=>{"value"=>114000.0}}, {"key"=>39, "doc_count"=>4, "total"=>{"value"=>107325.0}}, {"key"=>43, "doc_count"=>4, "total"=>{"value"=>209820.0}}, {"key"=>18, "doc_count"=>2, "total"=>{"value"=>22000.0}}, {"key"=>23, "doc_count"=>2, "total"=>{"value"=>412800.0}}, {"key"=>38, "doc_count"=>2, "total"=>{"value"=>13500.0}}, {"key"=>49, "doc_count"=>2, "total"=>{"value"=>161000.0}}, {"key"=>57, "doc_count"=>2, "total"=>{"value"=>20300.0}}, {"key"=>58, "doc_count"=>2, "total"=>{"value"=>32200.0}}]}}
The data is completely inconsistent and aggs missing. Note key 16 is missing. The crazy thing is I have another aggregation on another column and that one works absolutely fine. Am i missing something here? I have already tried settings: {number_of_shards: 1}.
To be more frustrating this works:
aggs: {
"grand_total": { "sum": { "field": "total" } },
}
The regular agg for total works in the same result.
Try setting size in the aggregation. Based on sum_other_doc_count in the response, 13 documents aren't being returned.

Processing hierarchical JSON data with d3.js

I'm trying to understand how to use more complicated data sets with d3. Below I will show the JSON data I would like to display;
{
"questions": ["Large choice of food", "Food quality", "Food freshness", "Taste of food", "Waiting time to recieve food", "Value for money"],
"places": ["US", "UK", "TK"],
"dates": ["Jan", "Feb", "Mar"],
"values": [
[
[24, 42, 72],
[29, 45, 79],
[34, 39, 84]
],
[
[33, 73, 41],
[21, 16, 45],
[43, 22, 17]
],
[
[75, 53, 78],
[55, 33, 22],
[94, 83, 99]
],
[
[63, 37, 11],
[47, 67, 62],
[33, 34, 35]
],
[
[43, 89, 78],
[99, 92, 87],
[41, 23, 71]
],
[
[92, 11, 45],
[100, 0, 50],
[40, 72, 62]
]
]
}
From here I would like to be able to select one question, then pair it with a place + date and then retrieve the value based on this.
I have tried to find resources online which could help educate me with how to access this kind of data in this way, but i've had no such luck. I have created a plnk to provide a set up for this.
http://plnkr.co/edit/Cdwm5RXoIdBNg0uxeeP6?p=preview
So the question is how can I retrieve this data in d3, and display it in the console in the order of question+[place + date][values based on this].
Any advice and links to good educational resources would be a big help for me at this stage,
Cheers
EDIT:
The above JSON format may be a little confusing, here is perhaps a more simplified version?
{
"dates": ["Jan", "Feb", "Mar"],
"questions": {
"Large choice of food": {
"US": [11, 15, 13],
"UK": [25, 24, 39],
"TK": [27, 23, 20]
},
"Food quality": {
"US": [11, 15, 13],
"UK": [25, 24, 40],
"TK": [27, 23, 20]
},
}
}
Here is something I put together but it isn't using D3, but vanilla JS. So I'm not sure if you want this but I'll throw it in anyway. Rather self explanatory: http://plnkr.co/edit/ckm45FB3RVhofGbUjdBl?p=preview
Main chunk of code :
var questionData = data.questions;
var monthData = data.dates;
var countryData = data.places;
//populate question drop down
var questionSelect = document.getElementById('questionSelect');
for(i=0;i<questionData.length;i++){
var option = document.createElement('option');
option.text = questionData[i];
option.value = questionData[i];
questionSelect.appendChild(option);
}
var submitButton = document.getElementById('submitButton');
submitButton.addEventListener('click', getData);
function getData(){
var thisQuestion = document.getElementById('questionSelect').value;
var thisCountry = document.getElementById('countrySelect').value;
var thisMonth = document.getElementById('monthSelect').value;
var questionIndex = questionData.indexOf(thisQuestion);
var countryIndex = countryData.indexOf(thisCountry);
var monthIndex = monthData.indexOf(thisMonth);
var getValue = data.values[questionIndex][monthIndex][countryIndex]; //this is the found value
console.log(getValue)
}

RethinkDB: How to get data from dictionary field and transform them to lists

I have objects that have a following structure:
[
{
'record': 1,
'tags': [1, 2],
'data': {
'1': 10,
'2': 15
}
},
...
{
'record': 1,
'tags': [3, 4, 5],
'data': {
'1': 100,
'2': 150
}
}
]
How can I get distinct lists of data: [10, ..., 100] and [15, ..., 150] to process each of them?
Thanks
If all data have same key, I proposed this map-reduce process.
.map(function(doc) {
return doc('data').keys().map(function(key) {
return [key, [doc('data')(key)]]
}).coerceTo('object')
})
.reduce(function(left, right) {
return left.keys().map(function(key) {
return [key, left(key).setUnion(right(key))]
})
.coerceTo('object')
})
With your data set, we have:
r.expr(
[
{
'record': 1,
'tags': [1, 2],
'data': {
'1': 10,
'2': 15
}
},
{
'record': 1,
'tags': [1, 2],
'data': {
'1': 19,
'2': 100
}
},
{
'record': 1,
'tags': [3, 4, 5],
'data': {
'1': 100,
'2': 150
}
}
]
)
.map(function(doc) {
return doc('data').keys().map(function(key) {
return [key, [doc('data')(key)]]
}).coerceTo('object')
})
.reduce(function(left, right) {
return left.keys().map(function(key) {
return [key, left(key).setUnion(right(key))]
})
.coerceTo('object')
})
Which produces:
{
"1": [10, 19, 100],
"2": [15, 100, 150]
}
You can call values on an object to get an array of values from it: https://www.rethinkdb.com/api/javascript/#values
If not all of the indexes are present as keys, you could write something like r.range(obj.keys().max()).map(function(i) { return obj(i.coerceTo('string')).default(nil); }).
If the data fields could be any string, something like this could work:
r.db('test')
.table('test')
.getAll(1, {index: 'record'})
.getField('data')
.concatMap(r.row.coerceTo('array'))
.group(r.row(0))
.concatMap([r.row(1)])
.ungroup()
Otherwise, mlucy's suggestion might be used to make it more efficient.
EDIT: on further experimentation, I could not get obj.keys() to be faster than group.

How to create a 3D matrix of values in VB.NET?

In 2D I write:
Dim matr2D(,) As Integer = { {10, 20, 30}, {11, 21, 31}, {12, 22, 32} }.
In 3D?
Found:
Dim matr3D(, ,) As Integer = { { { 1, 2, 3 }, { 4, 5, 6 } }, { { 7, 8, 9 }, { 10, 11, 12 } } }
https://msdn.microsoft.com/it-it/library/2yd9wwz4.aspx

Merge two hashes of hashes, without overlap

I have two hashes of hashes that look basically like:
sales = {
"2013-03-15": {sales: 5},
"2013-03-14": {sales: 3},
"2013-03-12": {sales: 8},
... }
and
views = {
"2013-03-15": {views: 30},
"2013-03-14": {views: 23},
"2013-03-13": {views: 35},
... }
How can I merge them into a single hash that looks like:
data = {
"2013-03-15": {views: 30, sales: 5},
"2013-03-14": {views: 23, sales: 3},
"2013-03-13": {views: 35, sales: 0}, # or just {views: 35}
... }
Basically, I need to keep the keys and all the data intact.
data = views.merge sales Seems to override all the views data, leaving me with basically just the sales hash.
Edit: I can also convert the sales or views hashes into simple hashes (not hashes of hashes), but I still don't know a good way to proceed.
This:
sales.merge(views) { |k, o, n| o.merge(n) }
Runnable example:
views = {
"2013-03-15" => {:views=> 30},
"2013-03-14" => {:views=> 23},
"2013-03-13" => {:views=> 35},
}
sales = {
"2013-03-15" => {:sales=> 5},
"2013-03-14" => {:sales=> 3},
"2013-03-12" => {:sales=> 8},
}
puts sales.merge(views) { |k, o, n| o.merge(n) }
=> {"2013-03-15"=>{:sales=>5, :views=>30}, "2013-03-14"=>{:sales=>3, :views=>23}, "2013-03-12"=>{:sales=>8}, "2013-03-13"=>{:views=>35}}

Resources