adding function to loops through - windows

I need to search in a big json nested collection which have unique IDs recursively. The collection contains key values or nested arrays which contains keys. Keys can be anywhere in the object. Keys can be number or string.
Please note: Key values are unique if they are not in array. If they are in array, the key duplicates per items in array. For example,
"WebData": {
WA1: 3, //not in array so unique
WA3: 2, so unique
WA3: "NEO",
WebGroup : [
{ Web1: 1, //duplicate Web1
Web2: 2
},
{ Web1: 2, //duplicate Web2
Web2: 2
}]
}
What I want:
I will pass an array of keys in different variations for example
Not in Arrays: I will pass key return either their values or sum for example:
function(["WA1",""WA3", "RAE1"],"notsum")
If I pass (not sum)
["WA1",""WA3", "RAE1"]
and the operation is not "sum", it should return an array of their values from the collection
[3,2,1]
If I pass the same but operation is sum)
function(["WA1",""WA3", "RAE1"],"sum")
["WA1",""WA3", "RAE1"]
it should return sum from the collection
return 6
If in Array: If the value to search are in the array means they duplicate, then it should return me sum or their individual values again For example
["WEB1","Web2"]
. It could either return me,
[7,1] //Again total of 3+4, 0+1 //see in example
or
[[3,4],[0,1]] //Because values are duplicate and in array, just collect them
I need to do in an elegant way:
Full example of JSON:
{
version: "1.0"
submission : "editing"
"WebData": {
WA1: 3,
WA3: 2,
WA3: "NEO",
WebGroup : [
{ Web1: 3,
Web2: 0
},
{ Web1: 4,
Web2: 1
}]
},
"NonWebData": {
NWA1: 3,
NWA2: "INP",
NWA3: 2,
},
"FormInputs": {
FM11: 3,
FM12: 1,
FM13: 2,
"RawData" : {
"RawOverview": {
"RAE1" : 1,
"RAE2" : 1,
},
"RawGroups":[
{
"name": "A1",
"id": "1",
"data":{
"AD1": 'period',
"AD2": 2,
"AD3": 2,
"transfers": [
{
"type": "in",
"TT1": 1,
"TT2": 2,
},
{
"type": "out",
"TT1": 1,
"TT2": 2,
}
]
}
},
{
"name": "A2",
"id": "2",
"data":{
"AD1": 'period',
"AD2": 2,
"AD3": 2,
"transfers": [
{
"type": "in",
"TT1": 1,
"TT2": 2,
},
{
"type": "out",
"TT1": 1,
"TT2": 2,
}
]
}
}
]
},
"Other":
{ O1: 1,
O2: 2,
O3: "hello"
},
"AddedBy": "name"
"AddedDate": "11/02/2019"
}
I am not able to write a function here, which can do this for me, my code is simply searching in this array, and I loop through to find it, which is I am sure not the correct way.
My code is not elegant, and I am using somehow repetitive functions. This is just one snippet, to find out the keys in one level. I want only 1 or 2 functions to do all this
function Search(paramKey, formDataArray) {
var varParams = [];
for (var key in formDataArray) {
if (formDataArray.hasOwnProperty(key)) {
var val = formDataArray[key];
for (var ikey in val) {
if (val.hasOwnProperty(ikey)) {
if (ikey == paramKey)
varParams.push(val[ikey]);
}
}
}
}
return varParams;
}
One more test case if in Array: to Return only single array of values, without adding. (Update - I achieved this through editing the code following part)
notsumsingle: function (target, key, value) {
if (target[key] === undefined) {
target[key] = value;
return;
}
target.push(value);
},
"groupData": [
{
"A1G1": 1,
"A1G2": 22,
"AIG3": 4,
"AIG4": "Rob"
},
{
"A1G1": 1,
"A1G2": 41,
"AIG3": 3,
"AIG4": "John"
},
{
"A1G1": 1,
"A1G2": 3,
"AIG3": 1,
"AIG4": "Andy"
}
],
perform(["AIG2",""AIG4"], "notsum")
It is returning me
[
[
22,
41,
3
]
],
[
[
"",
"Ron",
"Andy"
]
]
Instead, can I add one more variation "SingleArray" like "sum" and "notsum" and get the result as single Array.
[
22,
41,
3
]
[
"",
"Ron",
"Andy"
]
4th one, I asked, is it possible the function intelligent enough to pick up the sum of arrays or sum of individual fields automatically. for example, in your example, you have used "sum" and "total" to identify that.
console.log(perform(["WA1", "WA3", "RAE1"], "total")); // 6
console.log(perform(["Web1", "Web2"], "sum")); // [7, 1]
Can the function, just use "sum" and returns single or array based on if it finds array, return [7,1] if not return 6
5th : I found an issue in the code, if the json collection is added this way
perform(["RAE1"], "notsum") //[[1,1]]
perform(["RAE1"], "sum") //2
It returns [1, 1], or 2 although there is only one RAE1 defined and please note it is not an array [] so it should not be encoded into [[]] array, just the object key
"RawData" : {
"RawOverview": {
"RAE1" : 1,
"RAE2" : 1,
}

For making it easier, and to take the same interface for getting sums or not sums and a total, without any array, you could introduce another operation string total for getting the sum of all keys.
This approach takes an object for getting a function which either add an value to an array at the same index or stores the value at an specified index, which match the given keys array of the function.
For iterating the object, you could take the key/value pairs and iterate until no more object is found.
As result, you get an array, or the total sum of all items.
BTW, the keys of an object are case sensitive, for example 'WEB1' does not match 'Web1'.
function perform(keys, operation) {
function visit(object) {
Object
.entries(object)
.forEach(([k, v]) => {
if (k in indices) return fn(result, indices[k], v);
if (v && typeof v === 'object') visit(v);
});
}
var result = [],
indices = Object.assign({}, ...keys.map((k, i) => ({ [k]: i }))),
fn = {
notsum: function (target, key, value) {
if (target[key] === undefined) {
target[key] = value;
return;
}
if (!Array.isArray(target[key])) {
target[key] = [target[key]];
}
target[key].push(value);
},
sum: function (target, key, value) {
target[key] = (target[key] || 0) + value;
}
}[operation === 'total' ? 'sum' : operation];
visit(data);
return operation === 'total'
? result.reduce((a, b) => a + b)
: result;
}
var data = { version: "1.0", submission: "editing", WebData: { WA1: 3, WA3: 2, WAX: "NEO", WebGroup: [{ Web1: 3, Web2: 0 }, { Web1: 4, Web2: 1 }] }, NonWebData: { NWA1: 3, NWA2: "INP", NWA3: 2 }, FormInputs: { FM11: 3, FM12: 1, FM13: 2 }, RawData: { RawOverview: { RAE1: 1, RAE2: 1 }, RawGroups: [{ name: "A1", id: "1", data: { AD1: 'period', AD2: 2, AD3: 2, transfers: [{ type: "in", TT1: 1, TT2: 2 }, { type: "out", TT1: 1, TT2: 2 }] } }, { name: "A2", id: "2", data: { AD1: 'period', AD2: 2, AD3: 2, transfers: [{ type: "in", TT1: 1, TT2: 2 }, { type: "out", TT1: 1, TT2: 2 }] } }] }, Other: { O1: 1, O2: 2, O3: "hello" }, AddedBy: "name", AddedDate: "11/02/2019" };
console.log(perform(["WA1", "WA3", "RAE1"], "notsum")); // [3, 2, 1]
console.log(perform(["WA1", "WA3", "RAE1"], "total")); // 6
console.log(perform(["Web1", "Web2"], "sum")); // [7, 1]
console.log(perform(["Web1", "Web2"], "notsum")); // [[3, 4], [0, 1]]
.as-console-wrapper { max-height: 100% !important; top: 0; }

Related

improve leftJoin nested mapObject when equivalence do and do not match

I am using the following dataweave function, and it does works.
%dw 2.0
import * from dw::core::Arrays
output application/json
var mysqlInvoices = [
{
"id": 1,
"owner": "Joseph"
},
{
"id": 2,
"owner": "Maria"
}
]
var sapInvoices = [
{
"number": 3,
"issuedBy": "XYZ"
},
{
"number": 4,
"issuedBy": "ABC"
}
]
---
leftJoin(mysqlInvoices, sapInvoices, (m) -> m.id, (s) -> s.number) map (item, index) ->
(item.l mapObject (sItem, sKey) ->
(if ((sKey) as String == "id") "identifier"
else if ((sKey) as String == "owner") "ownerName"
else (sKey)): sItem)
++
(if (item.r != null)
item.r mapObject (sItem, sKey) ->
(sKey): sItem
else
sapInvoices[0] mapObject
(sItem, sKey) -> (sKey): "")
However, I am thinking if I can improve this function at two points:
change the key conditions:
I dont think that is the best practice to check every key match an if condition to change it:
(if ((sKey) as String == "id") "identifier"
else if ((sKey) as String == "owner") "ownerName"
else (sKey)): sItem
Use the original object to map it as an empty string when the leftJoin do not match keys:
sapInvoices[0] mapObject (sItem, sKey) ->
(sKey): ""
I am uncomfortable with these two points, and I believe that there are ways to improve this code, I just dont know how.
If there is a very different way of doing the same task, I also appreciate that kind of suggestion.
Based on George's answer, you can remove pluck and match and directly combine left and right table. See below:
%dw 2.0
import * from dw::core::Arrays
output application/json
var mysqlInvoices = [
{
"id": 1,
"owner": "Joseph"
},
{
"id": 2,
"owner": "Maria"
}
]
var sapInvoices = [
{
"number": 3,
"issuedBy": "XYZ"
},
{
"number": 4,
"issuedBy": "ABC"
}
]
var fs2rn = {
id: "identifier",
owner: "ownerName"
}
var rightEmpty= {number:"",issuedBy:""}
---
leftJoin(
// Do the field renaming at the very begining
mysqlInvoices map ($ mapObject {(fs2rn[$$] default $$): $}),
sapInvoices,
(m) -> m.identifier,
(s) -> s.number
) map (item) -> item.l ++ (item.r default rightEmpty)
Give the following a try, if anything the code seems a bit simpler:
%dw 2.0
import * from dw::core::Arrays
output application/json
var mysqlInvoices = [
{
"id": 1,
"owner": "Joseph"
},
{
"id": 2,
"owner": "Maria"
}
]
var sapInvoices = [
{
"number": 3,
"issuedBy": "XYZ"
},
{
"number": 4,
"issuedBy": "ABC"
}
]
var fs2rn = {
id: "identifier",
owner: "ownerName"
}
var rightEmpty= {number:"",issuedBy:""}
---
leftJoin(
// Do the field renaming at the very begining
mysqlInvoices map ($ mapObject {(fs2rn[$$] default $$): $}),
sapInvoices,
(m) -> m.identifier,
(s) -> s.number
)
// Iterate over the results
// Get just the values, and colapse the objects into a single object
map (
{($ pluck $)}
)
// Iterate over the results and use pattern-matching to
//
map (
$ match {
// Check if you have an id but not a number fields
// In which case add the rightEmpty object
case o if (o.identifier? and not (o.number?)) -> o ++ rightEmpty
// Or give the object because you now have both an id and a number
else o -> o
}
)
The features and functions I used are:
Dynamic Elements, documentation
pluck, documentation
Pattern-matching using the match operator, documentation
If I was to give you an advice, it would be to better indent your code. Nonetheless, pretty good job!

Apply delta values on nested fields

Suppose I have record like this:
{
id: 1,
statistics: {
stat1: 1,
global: {
stat2: 3
},
stat111: 99
}
}
I want to make update on record with object:
{
statistics: {
stat1: 8,
global: {
stat2: 6
},
stat4: 3
}
}
And it should be added to current record as delta. So, the result record should looks like this:
{
id: 1,
statistics: {
stat1: 9,
global: {
stat2: 9
},
stat4: 3,
stat111: 99
}
}
Is it possible to make this with one query?
Do you want something generic or something specific?
Specific is easy, this is the generic case:
const updateValExpr = r.expr(updateVal);
const updateStats = (stats, val) => val
.keys()
.map(key => r.branch(
stats.hasFields(key),
[key, stats(key).add(val(key))],
[key, val(key)]
))
.coerceTo('object')
r.table(...)
.update(stats =>
updateStats(stats.without('global'), updateValExpr.without('global'))
.merge({ global: updateStats(stats('global'), updateValExpr('global'))
)
There might be some bugs here sincce it's untested but the solution key point is the updateStats function, the fact that you can get all the keys with .keys() and that coerceTo('object') transforms this array: [['a',1],['b',2]] to this object: { a: 1, b: 2 },
Edit:
You can do it recursively, although with limited stack (since you can't send recursive stacks directly, they resolve when the query is actually built:
function updateStats(stats, val, stack = 10) {
return stack === 0
? {}
: val
.keys()
.map(key => r.branch(
stats.hasFields(key).not(),
[key, val(key)],
stats(key).typeOf().eq('OBJECT'),
[key, updateStats(stats(key), val(key), stack - 1)],
[key, stats(key).add(val(key))]
)).coerceTo('object')
}
r.table(...).update(row => updateStats(row, r(updateVal)).run(conn)
// test in admin panel
updateStats(r({
id: 1,
statistics: {
stat1: 1,
global: {
stat2: 3
},
stat111: 99
}
}), r({
statistics: {
stat1: 8,
global: {
stat2: 6
},
stat4: 3
}
}))

ElasticSearch comparative range results

Hi I would like to index objects that look like that
{
uuid: "123",
clauses: [{ order: 1, uuid: "345"},{ order: 2, uuid: "567"},{ order: 3, uuid: "789"}]
}
Is there a way to write a query that matches all the objects that contain
clauses with uuid: "345" and uuid: "789" but order of the second one is at most two bigger than first one?
So the above example would match but the next one wouldn't :
{
uuid: "999",
clauses: [{ order: 1, uuid: "345"},{ order: 2, uuid: "567"},{order: 3, uuid: "777"},{ order: 4, uuid: "789"}]
}
The reason is that order of "789" clause is 4 which is more than 2 bigger than "345" clause, which has order 1.
Any help is appreciated!
Thanks,
Michail
One way to achieve this involves using a script filter.
The script I'm using is the following:
def idxs = [];
for (int i = 0; i < doc['clauses.uuid'].values.size(); i++) {
if (matches.contains(doc['clauses.uuid'].values[i])){
idxs << i
}
};
def orders = idxs.collect{ doc['clauses.order'].values[it]};
return orders[1] - orders[0] <= 2
Basically, what I'm doing is first collection all indices of the clauses which contain a uuid in the matches array (i.e. 345 and 789).
Then, with the indices I got I gather all order values at those indices. And finally, I check that the second order minus the first order is not bigger than 2.
POST your_index/_search
{
"query": {
"bool": {
"filter": [
{
"term": {
"clauses.uuid": "345"
}
},
{
"term": {
"clauses.uuid": "789"
}
},
{
"script": {
"script": "def idxs = []; for (int i = 0; i < doc['clauses.uuid'].values.size(); i++) {if (matches.contains(doc['clauses.uuid'].values[i])){idxs << i}}; def orders = idxs.collect{doc['clauses.order'].values[it]}; return orders[1] - orders[0] <= 2",
"params": {
"matches": [
"345",
"789"
]
}
}
}
]
}
}
}
That will return only the first document and not the second.

Get specific object field with condition and make opertion on it

I have objects like this:
{
buildings: {
"1": {
"l": 0 ,
"r": 0 ,
"s": 0 ,
"type": "GoldMine" ,
"x": 2 ,
"y": 15
} ,
"10": {
"l": 0 ,
"r": 6 ,
"s": 2 ,
"type": "MagicMine" ,
"x": 26 ,
"y": 22
}
} ,
[...]
}
I want to get objects with buildings of type "GoldMine".
I tried something with map:
r.table("Characters").map(function(row) {
return row("planet")("buildings")
})
With keys() I can iterate it:
r.db("Unnyworld").table("Characters").map(function(row) {
return row("planet")("buildings").keys().map(function(key) {
return "need to get only buildings with type == GoldMine";
})
}).limit(2)
But it returns all buildings. I want to get only buildings with type == GoldMine and change field x.
Something like this may work:
r.table('Characters')
.concatMap(function(doc) {
return doc("planet")("buildings").keys().map(function(k) {
return {id: doc('id'), key: k, type: doc("planet")("buildings")(k)('type'), x: doc("planet")("buildings")(k)('x')}
})
})
.filter(function(building) {
return building('type').eq('GoldMine')
})
.forEach(function(doc) {
return r.table('Characters').get(doc('id'))
.update({
planet: {buildings: r.object(doc('key'), {x: 1111111})}
})
})
Basically create a flat array from building by using concatMap then filter it. With result data, we can iterator over it and update to value that we want.

Parsing JSON to highcharts - Data gathered using AJAX

My JSON looks like
[{"target": "sumSeries(integral(org.example.fib.hi.value),integral(org.example.fib.hi1.value))",
"datapoints":
[
[2, 1359214560],
[3, 1359215040],
[4, 1359215050],
[null, 1359215060],
[null, 1359215070],
[5, 1359215080],
[7, 1359215090],
[9, 1359215100],
[10, 1359215110],
[null, 1359215120],
[10, 1359215130],
[14, 1359215140],
[null, 1359215150]
]}
]
I am trying to grab this data from my localhost and have highcharts render a line graph.
I have something like :
$(function() {
$.getJSON('http://localhost/render?target=sumSeries(integral(org.example.fib.*.value))&from=-10minutes&format=json', function(data) {
// Create the chart
window.chart = new Highcharts.StockChart({
chart : {
renderTo : 'container'
},
rangeSelector : {
selected : 1
},
plotOptions: {
series: {
stacking: 'normal'
}
},
series : [{
name : 'AAPL',
data : data,
tooltip: {
yDecimals: 2
}
}]
});
});
});
How can I parse this data?
I assume that you use timestamps (as second parameter in JSON), so data should be parsing to display as datetime. If yes, you should revert your data parameters in JSON, and multiply timestamps by 1000 (javascript timestamps format). Example of parsing:
var tmpdata = [],
i = 0,
len = data[0].datapoints.length;
for(i=0;i<len;i++)
{
tmpdata[i] = [data[0].datapoints[i][1]*1000,data[0].datapoints[i][0]];
}
and then in series data:
series : [{
name : 'AAPL',
data : tmpdata,
tooltip: {
yDecimals: 2
}
}]

Resources