improve leftJoin nested mapObject when equivalence do and do not match - performance

I am using the following dataweave function, and it does works.
%dw 2.0
import * from dw::core::Arrays
output application/json
var mysqlInvoices = [
{
"id": 1,
"owner": "Joseph"
},
{
"id": 2,
"owner": "Maria"
}
]
var sapInvoices = [
{
"number": 3,
"issuedBy": "XYZ"
},
{
"number": 4,
"issuedBy": "ABC"
}
]
---
leftJoin(mysqlInvoices, sapInvoices, (m) -> m.id, (s) -> s.number) map (item, index) ->
(item.l mapObject (sItem, sKey) ->
(if ((sKey) as String == "id") "identifier"
else if ((sKey) as String == "owner") "ownerName"
else (sKey)): sItem)
++
(if (item.r != null)
item.r mapObject (sItem, sKey) ->
(sKey): sItem
else
sapInvoices[0] mapObject
(sItem, sKey) -> (sKey): "")
However, I am thinking if I can improve this function at two points:
change the key conditions:
I dont think that is the best practice to check every key match an if condition to change it:
(if ((sKey) as String == "id") "identifier"
else if ((sKey) as String == "owner") "ownerName"
else (sKey)): sItem
Use the original object to map it as an empty string when the leftJoin do not match keys:
sapInvoices[0] mapObject (sItem, sKey) ->
(sKey): ""
I am uncomfortable with these two points, and I believe that there are ways to improve this code, I just dont know how.
If there is a very different way of doing the same task, I also appreciate that kind of suggestion.

Based on George's answer, you can remove pluck and match and directly combine left and right table. See below:
%dw 2.0
import * from dw::core::Arrays
output application/json
var mysqlInvoices = [
{
"id": 1,
"owner": "Joseph"
},
{
"id": 2,
"owner": "Maria"
}
]
var sapInvoices = [
{
"number": 3,
"issuedBy": "XYZ"
},
{
"number": 4,
"issuedBy": "ABC"
}
]
var fs2rn = {
id: "identifier",
owner: "ownerName"
}
var rightEmpty= {number:"",issuedBy:""}
---
leftJoin(
// Do the field renaming at the very begining
mysqlInvoices map ($ mapObject {(fs2rn[$$] default $$): $}),
sapInvoices,
(m) -> m.identifier,
(s) -> s.number
) map (item) -> item.l ++ (item.r default rightEmpty)

Give the following a try, if anything the code seems a bit simpler:
%dw 2.0
import * from dw::core::Arrays
output application/json
var mysqlInvoices = [
{
"id": 1,
"owner": "Joseph"
},
{
"id": 2,
"owner": "Maria"
}
]
var sapInvoices = [
{
"number": 3,
"issuedBy": "XYZ"
},
{
"number": 4,
"issuedBy": "ABC"
}
]
var fs2rn = {
id: "identifier",
owner: "ownerName"
}
var rightEmpty= {number:"",issuedBy:""}
---
leftJoin(
// Do the field renaming at the very begining
mysqlInvoices map ($ mapObject {(fs2rn[$$] default $$): $}),
sapInvoices,
(m) -> m.identifier,
(s) -> s.number
)
// Iterate over the results
// Get just the values, and colapse the objects into a single object
map (
{($ pluck $)}
)
// Iterate over the results and use pattern-matching to
//
map (
$ match {
// Check if you have an id but not a number fields
// In which case add the rightEmpty object
case o if (o.identifier? and not (o.number?)) -> o ++ rightEmpty
// Or give the object because you now have both an id and a number
else o -> o
}
)
The features and functions I used are:
Dynamic Elements, documentation
pluck, documentation
Pattern-matching using the match operator, documentation
If I was to give you an advice, it would be to better indent your code. Nonetheless, pretty good job!

Related

How to use ConcatArrays in Spring Data MongoDB

I have the following JS code:
$concatArrays: [
"$some.path",
[
{
"foo": "baz",
"x": 5
}
]
]
How do I do the same with ArrayOperators.ConcatArrays?
Toy can use like following. You can pass the aggregation expresion or reference field name in concat(). So what you can do is, you can add the new array before concat() like
db.collection.aggregate([
{
$addFields: {
secondArray: [
{ "foo": "baz", "x": 5 }
]
}
},
{
"$project": {
combined: {
"$concatArrays": [ "$path", "$secondArray" ]
}
}
}
])
Working Mongo playground
ArrayOperators.ConcatArrays concatArrays = ArrayOperators.ConcatArrays
.arrayOf("path")
.concat("secondArray");
Assuming the array field is defined as follows in a document:
{
"_id": ObjectId("60bdc6b228acee4a5a6e1736"),
"some" : {
"path" : [
{
"foo" : "bar",
"x" : 99
}
]
},
}
The following Spring Data MongoDB code returns the result- an array with two elements. The array field "some.path" and the input array myArr are concatenated using the ArrayOperators.ConcatArrays API.
String json = "{ 'foo': 'baz', 'x': 5 }";
Document doc = Document.parse(json);
List<Document> myArr = Arrays.asList(doc);
MongoOperations mongoOps = new MongoTemplate(MongoClients.create(), "testDB");
Aggregation agg = newAggregation(
addFields()
.addFieldWithValue("myArr", myArr)
.build(),
addFields()
.addField("result").withValue(ConcatArrays.arrayOf("$some.path").concat("$myArr"))
.build()
);
AggregationResults<Document> results = mongoOps.aggregate(agg, "testColl", Document.class);

adding function to loops through

I need to search in a big json nested collection which have unique IDs recursively. The collection contains key values or nested arrays which contains keys. Keys can be anywhere in the object. Keys can be number or string.
Please note: Key values are unique if they are not in array. If they are in array, the key duplicates per items in array. For example,
"WebData": {
WA1: 3, //not in array so unique
WA3: 2, so unique
WA3: "NEO",
WebGroup : [
{ Web1: 1, //duplicate Web1
Web2: 2
},
{ Web1: 2, //duplicate Web2
Web2: 2
}]
}
What I want:
I will pass an array of keys in different variations for example
Not in Arrays: I will pass key return either their values or sum for example:
function(["WA1",""WA3", "RAE1"],"notsum")
If I pass (not sum)
["WA1",""WA3", "RAE1"]
and the operation is not "sum", it should return an array of their values from the collection
[3,2,1]
If I pass the same but operation is sum)
function(["WA1",""WA3", "RAE1"],"sum")
["WA1",""WA3", "RAE1"]
it should return sum from the collection
return 6
If in Array: If the value to search are in the array means they duplicate, then it should return me sum or their individual values again For example
["WEB1","Web2"]
. It could either return me,
[7,1] //Again total of 3+4, 0+1 //see in example
or
[[3,4],[0,1]] //Because values are duplicate and in array, just collect them
I need to do in an elegant way:
Full example of JSON:
{
version: "1.0"
submission : "editing"
"WebData": {
WA1: 3,
WA3: 2,
WA3: "NEO",
WebGroup : [
{ Web1: 3,
Web2: 0
},
{ Web1: 4,
Web2: 1
}]
},
"NonWebData": {
NWA1: 3,
NWA2: "INP",
NWA3: 2,
},
"FormInputs": {
FM11: 3,
FM12: 1,
FM13: 2,
"RawData" : {
"RawOverview": {
"RAE1" : 1,
"RAE2" : 1,
},
"RawGroups":[
{
"name": "A1",
"id": "1",
"data":{
"AD1": 'period',
"AD2": 2,
"AD3": 2,
"transfers": [
{
"type": "in",
"TT1": 1,
"TT2": 2,
},
{
"type": "out",
"TT1": 1,
"TT2": 2,
}
]
}
},
{
"name": "A2",
"id": "2",
"data":{
"AD1": 'period',
"AD2": 2,
"AD3": 2,
"transfers": [
{
"type": "in",
"TT1": 1,
"TT2": 2,
},
{
"type": "out",
"TT1": 1,
"TT2": 2,
}
]
}
}
]
},
"Other":
{ O1: 1,
O2: 2,
O3: "hello"
},
"AddedBy": "name"
"AddedDate": "11/02/2019"
}
I am not able to write a function here, which can do this for me, my code is simply searching in this array, and I loop through to find it, which is I am sure not the correct way.
My code is not elegant, and I am using somehow repetitive functions. This is just one snippet, to find out the keys in one level. I want only 1 or 2 functions to do all this
function Search(paramKey, formDataArray) {
var varParams = [];
for (var key in formDataArray) {
if (formDataArray.hasOwnProperty(key)) {
var val = formDataArray[key];
for (var ikey in val) {
if (val.hasOwnProperty(ikey)) {
if (ikey == paramKey)
varParams.push(val[ikey]);
}
}
}
}
return varParams;
}
One more test case if in Array: to Return only single array of values, without adding. (Update - I achieved this through editing the code following part)
notsumsingle: function (target, key, value) {
if (target[key] === undefined) {
target[key] = value;
return;
}
target.push(value);
},
"groupData": [
{
"A1G1": 1,
"A1G2": 22,
"AIG3": 4,
"AIG4": "Rob"
},
{
"A1G1": 1,
"A1G2": 41,
"AIG3": 3,
"AIG4": "John"
},
{
"A1G1": 1,
"A1G2": 3,
"AIG3": 1,
"AIG4": "Andy"
}
],
perform(["AIG2",""AIG4"], "notsum")
It is returning me
[
[
22,
41,
3
]
],
[
[
"",
"Ron",
"Andy"
]
]
Instead, can I add one more variation "SingleArray" like "sum" and "notsum" and get the result as single Array.
[
22,
41,
3
]
[
"",
"Ron",
"Andy"
]
4th one, I asked, is it possible the function intelligent enough to pick up the sum of arrays or sum of individual fields automatically. for example, in your example, you have used "sum" and "total" to identify that.
console.log(perform(["WA1", "WA3", "RAE1"], "total")); // 6
console.log(perform(["Web1", "Web2"], "sum")); // [7, 1]
Can the function, just use "sum" and returns single or array based on if it finds array, return [7,1] if not return 6
5th : I found an issue in the code, if the json collection is added this way
perform(["RAE1"], "notsum") //[[1,1]]
perform(["RAE1"], "sum") //2
It returns [1, 1], or 2 although there is only one RAE1 defined and please note it is not an array [] so it should not be encoded into [[]] array, just the object key
"RawData" : {
"RawOverview": {
"RAE1" : 1,
"RAE2" : 1,
}
For making it easier, and to take the same interface for getting sums or not sums and a total, without any array, you could introduce another operation string total for getting the sum of all keys.
This approach takes an object for getting a function which either add an value to an array at the same index or stores the value at an specified index, which match the given keys array of the function.
For iterating the object, you could take the key/value pairs and iterate until no more object is found.
As result, you get an array, or the total sum of all items.
BTW, the keys of an object are case sensitive, for example 'WEB1' does not match 'Web1'.
function perform(keys, operation) {
function visit(object) {
Object
.entries(object)
.forEach(([k, v]) => {
if (k in indices) return fn(result, indices[k], v);
if (v && typeof v === 'object') visit(v);
});
}
var result = [],
indices = Object.assign({}, ...keys.map((k, i) => ({ [k]: i }))),
fn = {
notsum: function (target, key, value) {
if (target[key] === undefined) {
target[key] = value;
return;
}
if (!Array.isArray(target[key])) {
target[key] = [target[key]];
}
target[key].push(value);
},
sum: function (target, key, value) {
target[key] = (target[key] || 0) + value;
}
}[operation === 'total' ? 'sum' : operation];
visit(data);
return operation === 'total'
? result.reduce((a, b) => a + b)
: result;
}
var data = { version: "1.0", submission: "editing", WebData: { WA1: 3, WA3: 2, WAX: "NEO", WebGroup: [{ Web1: 3, Web2: 0 }, { Web1: 4, Web2: 1 }] }, NonWebData: { NWA1: 3, NWA2: "INP", NWA3: 2 }, FormInputs: { FM11: 3, FM12: 1, FM13: 2 }, RawData: { RawOverview: { RAE1: 1, RAE2: 1 }, RawGroups: [{ name: "A1", id: "1", data: { AD1: 'period', AD2: 2, AD3: 2, transfers: [{ type: "in", TT1: 1, TT2: 2 }, { type: "out", TT1: 1, TT2: 2 }] } }, { name: "A2", id: "2", data: { AD1: 'period', AD2: 2, AD3: 2, transfers: [{ type: "in", TT1: 1, TT2: 2 }, { type: "out", TT1: 1, TT2: 2 }] } }] }, Other: { O1: 1, O2: 2, O3: "hello" }, AddedBy: "name", AddedDate: "11/02/2019" };
console.log(perform(["WA1", "WA3", "RAE1"], "notsum")); // [3, 2, 1]
console.log(perform(["WA1", "WA3", "RAE1"], "total")); // 6
console.log(perform(["Web1", "Web2"], "sum")); // [7, 1]
console.log(perform(["Web1", "Web2"], "notsum")); // [[3, 4], [0, 1]]
.as-console-wrapper { max-height: 100% !important; top: 0; }

loopback REST API filter by nested data

I would like to filter from REST API by nested data. For example this object:
[
{
"name": "Handmade Soft Fish",
"tags": "Rubber, Rubber, Salad",
"categories": [
{
"name": "women",
"id": 2,
"parent_id": 0,
"permalink": "/women"
},
{
"name": "kids",
"id": 3,
"parent_id": 0,
"permalink": "/kids"
}
]
},
{
"name": "Tasty Rubber Soap",
"tags": "Granite, Granite, Chair",
"categories": [
{
"name": "kids",
"id": 3,
"parent_id": 0,
"permalink": "/kids"
}
]
}
]
is comming by GET /api/products?filter[include]=categories
and i would like to get only products which has category name "women". How do this?
LoopBack does not support filters based on related models.
This is a limitation that we have never had bandwidth to solve, unfortunately :(
For more details, see the discussion and linked issues here:
Filter on level 2 properties: https://github.com/strongloop/loopback/issues/517
Filter by properties of related models (use SQL JOIN in queries): https://github.com/strongloop/loopback/issues/683
Maybe you want to get this data by the Category REST API. For example:
GET /api/categories?filter[include]=products&filter[where][name]=woman
The result will be a category object with all products related. To this, will be necessary declare this relation on the models.
Try like this.It has worked for me.
const filter = {
where: {
'categories.name': {
inq: ['women']**strong text**
}
}
};
Pass this filter to request as path parameters and the request would be like bellow
GET /api/categoriesfilter=%7B%22where%22:%7B%categories.name%22:%7B%22inq%22:%5B%women%22%5D%7D%7D%7D
Can you share how it looks like without filter[include]=categorie, please ?
[edit]
after a few questions in comment, I'd build a remote method : in common/models/myModel.js (inside the function) :
function getItems(filter, categorieIds = []) {
return new Promise((resolve, reject) => {
let newInclude;
if (filter.hasOwnProperty(include)){
if (Array.isArray(filter.include)) {
newInclude = [].concat(filter.include, "categories")
}else{
if (filter.include.length > 0) {
newInclude = [].concat(filter.include, "categories");
}else{
newInclude = "categories";
}
}
}else{
newInclude = "categories";
}
myModel.find(Object.assign({}, filter, {include: newInclude}))
.then(data => {
if (data.length <= 0) return resolve(data);
if (categoriesIds.length <= 0) return resolve(data);
// there goes your specific filter on categories
const tmp = data.filter(
item => item.categories.findIndex(
categorie => categorieIds.indexOf(categorie.id) > -1
) > -1
);
return resolve(tmp);
})
}
}
myModel.remoteMethod('getItems', {
accepts: [{
arg: "filter",
type: "object",
required: true
}, {
arg: "categorieIds",
type: "array",
required: true
}],
returns: {arg: 'getItems', type: 'array'}
});
I hope it answers your question...

mongoose sort by a function

I have a schema that looks like this
var user = new Schema({
preference1: { // preference is a number between 1 - 10
type: Number
},
preference2: { // preference is a number between 1 - 10
type: Number
}
})
how do I find the documents and sort by a function on the preferences fields? Say fn is something like this
fn = Math.abs(preference1 - 3) + preference2 ^ 2
I kind of find a temporary solution. It works but isn't really what I was looking for... the code is really messy and apparently you cannot take arbitrary fn for sorting..
for example, say fn = (a+3) * (b+5)
Media.aggregate()
.project({
"type": 1,
"status": 1,
"newField1": { "$add": [ "$type", 3 ] },
"newField2": { "$add": [ 5, "$status" ] },
})
.project({
"newField3": { "$multiply": [ "$newField1", "$newField2" ] },
})
.sort("newField3")
.exec()

Rethinkdb insert query results into a table

I'm trying to insert the results of a query from one table into another table. However, when I attempt to run the query I am receiving an error.
{
"deleted": 0 ,
"errors": 1 ,
"first_error": "Expected type OBJECT but found ARRAY." ,
"inserted": 0 ,
"replaced": 0 ,
"skipped": 0 ,
"unchanged": 0
}
Here is the the insert and query:
r.db('test').table('destination').insert(
r.db('test').table('source').map(function(doc) {
var result = doc('result');
return result('section_list').concatMap(function(section) {
return section('section_content').map(function(item) {
return {
"code": item("code"),
"name": item("name"),
"foo": result("foo"),
"bar": result("bar"),
"baz": section("baz"),
"average": item("average"),
"lowerBound": item("from"),
"upperBound": item("to")
};
});
});
});
);
Is there a special syntax for this, or do I have to retrieve the results and then run a separate insert?
The problem is that your inner query is returning a stream of arrays. You can't insert arrays into a table (only objects), so the query fails. If you change the outermost map into a concatMap it should work.
The problem here was that the result was a sequence of an array of objects. i.e
[ [ { a:1, b:2 }, { a:1, b:2 } ], [ { a:2, b:3 } ] ]
Therefore, I had to change the outer map call to a concatMap call. The query then becomes:
r.db('test').table('destination').insert(
r.db('test').table('source').concatMap(function(doc) {
var result = doc('result');
return result('section_list').concatMap(function(section) {
return section('section_content').map(function(item) {
return {
"code": item("code"),
"name": item("name"),
"foo": result("foo"),
"bar": result("bar"),
"baz": section("baz"),
"average": item("average"),
"lowerBound": item("from"),
"upperBound": item("to")
};
)});
});
});
}
Thanks goes to #AtnNn on the #rethinkdb freenode for pointing me in the right direction.

Resources