couchdb - problems grouping collections - view

I'm having problems trying to get my head around getting a collection of types along with number of times a skill is found in that doc type.
There are a number of document types that have a list of skills.
{
"skills": "Windows, Network Admin, Linux",
"type": "Experience"
},
{
"skills": "Windows, Erlang, Linux",
"type": "Experience"
},
{
"skills": "Ruby, Rails, Erlang",
"type": "Project"
}
I'm trying to get the number of times a skill is found in an document type.
The end result should look something like this:
{
'type': Experience,
'skills': [
{'skill': 'Erlang', 'count': 1},
{'skill': 'Linux', 'count': 2},
{'skill': 'Network Admin', 'count': 1},
{'skill': 'Rails', 'count': 0},
{'skill': 'Ruby', 'count': 0},
{'skill': 'Windows', 'count': 2}
]
},
{
'type': Project,
'skills': [
{'skill': 'Erlang', 'count': 1},
{'skill': 'Linux', 'count': 0},
{'skill': 'Network Admin', 'count': 0},
{'skill': 'Rails', 'count': 1},
{'skill': 'Ruby', 'count': 1},
{'skill': 'Windows', 'count': 0}
]
}
What would be the best way to do this?

You should store the skill list in your docs as a real list.
{
"skills": ["Windows", "Network Admin", Linux"],
"type": "Experience"
}
From there, your map function becomes:
function(doc) {
for(var skill in doc.skills) {
emit([doc.type, skill], 1);
}
}
And the reduce function just sums the result, so use "_sum". You can easily choose to show only Experience or Project with start and end keys.

Here are map and reduce functions (view named skill_split) and a list function that will produce output close to what you asked for using a URL like this http://127.0.0.1:5984/myskills/_design/myskills/_list/transform/skill_split?group=true:
{
"Experience": {"Erlang":1, "Linux":2, "Network Admin":1, "Windows":2},
"Project":{"Erlang":1, "Rails":1, "Ruby":1}
}
I can tweak the code to make the output exactly what you asked for, but the list function would be a bit longer.
map:
function(doc) {
if (doc.type && doc.skills) {
doc.skills.split(', ').forEach(function(skill) {
emit([doc.type, skill], 1);
});
}
}
reduce:
function(keys, values, rereduce) {
return sum(values)
}
or use "reduce": "_sum" for short
list (named transform):
function(head, req) {
var results = {};
var row;
while (row = getRow()) {
var skill_map;
if (results.hasOwnProperty(row.key[0])) {
skill_map = results[row.key[0]];
} else {
skill_map = {};
results[row.key[0]] = skill_map;
}
if (skill_map.hasOwnProperty(row.key[1])) {
skill_map[row.key[1]] = skill_map[row.key[1]] + row.value;
} else {
skill_map[row.key[1]] = row.value;
}
}
send(JSON.stringify(results));
}
If you don't have access to the JSON object (you do inside a CouchApp), you might have to jump through some hoops to get access to it.

Related

Converting mongoshell projection to Spring MongoTemplate projection

Given following mongoshell query - which works fine
[
{
'$match': {
'id': '1'
}
}, {
'$graphLookup': {
'from': 'pages',
'startWith': '$cID',
'connectFromField': 'parent',
'connectToField': 'cID',
'as': 'result',
'depthField': 'level'
}
}, {
'$unwind': {
'path': '$result',
'preserveNullAndEmptyArrays': true
}
}, {
'$sort': {
'result.level': 1
}
}, {
'$group': {
'_id': '$id',
'result': {
'$push': '$result'
}
}
}, {
'$project': {
'result': {
'id': 1,
'cID': 1,
'level': 1
}
}
}
]
How do I achieve the projection using the Spring MongoTemplate and the its given DSL?
Given the source and the AggregationTests provided by the github repo, I could be
project().and("result").nested(Fields.fields("id","cID","level")
which just returns an empty result list.
I didn't come up with a solution to make it work with the MongoTemplate directly. As I'm already using the MongoRepository given by Spring Data, creating an aggregation query is quite straightforward.
public interface Repository extends MongoRepository<Object, String> {
#Aggregation(pipeline = {
"{'$match': {'cID': '?0'}}",
"{'$graphLookup': {'from': 'pages', 'startWith': '$cID', 'connectFromField': 'cID', 'connectToField': 'parent', 'as': 'result', 'depthField': 'level'}}",
"{'$unwind': {'path': '$result', 'preserveNullAndEmptyArrays': true}}",
"{'$sort': {'result.level': -1}}",
"{'$group': {'_id': '$id', 'result': {'$push': '$result'}}}",
"{'$project': {'cID_list': '$result.cID', 'result': {'level': 1, 'cID': 1, 'id': 1}}}"
})
AggregationResults<Map<?, List<String>>> getChildrencIDs(final String cID);
}

adding function to loops through

I need to search in a big json nested collection which have unique IDs recursively. The collection contains key values or nested arrays which contains keys. Keys can be anywhere in the object. Keys can be number or string.
Please note: Key values are unique if they are not in array. If they are in array, the key duplicates per items in array. For example,
"WebData": {
WA1: 3, //not in array so unique
WA3: 2, so unique
WA3: "NEO",
WebGroup : [
{ Web1: 1, //duplicate Web1
Web2: 2
},
{ Web1: 2, //duplicate Web2
Web2: 2
}]
}
What I want:
I will pass an array of keys in different variations for example
Not in Arrays: I will pass key return either their values or sum for example:
function(["WA1",""WA3", "RAE1"],"notsum")
If I pass (not sum)
["WA1",""WA3", "RAE1"]
and the operation is not "sum", it should return an array of their values from the collection
[3,2,1]
If I pass the same but operation is sum)
function(["WA1",""WA3", "RAE1"],"sum")
["WA1",""WA3", "RAE1"]
it should return sum from the collection
return 6
If in Array: If the value to search are in the array means they duplicate, then it should return me sum or their individual values again For example
["WEB1","Web2"]
. It could either return me,
[7,1] //Again total of 3+4, 0+1 //see in example
or
[[3,4],[0,1]] //Because values are duplicate and in array, just collect them
I need to do in an elegant way:
Full example of JSON:
{
version: "1.0"
submission : "editing"
"WebData": {
WA1: 3,
WA3: 2,
WA3: "NEO",
WebGroup : [
{ Web1: 3,
Web2: 0
},
{ Web1: 4,
Web2: 1
}]
},
"NonWebData": {
NWA1: 3,
NWA2: "INP",
NWA3: 2,
},
"FormInputs": {
FM11: 3,
FM12: 1,
FM13: 2,
"RawData" : {
"RawOverview": {
"RAE1" : 1,
"RAE2" : 1,
},
"RawGroups":[
{
"name": "A1",
"id": "1",
"data":{
"AD1": 'period',
"AD2": 2,
"AD3": 2,
"transfers": [
{
"type": "in",
"TT1": 1,
"TT2": 2,
},
{
"type": "out",
"TT1": 1,
"TT2": 2,
}
]
}
},
{
"name": "A2",
"id": "2",
"data":{
"AD1": 'period',
"AD2": 2,
"AD3": 2,
"transfers": [
{
"type": "in",
"TT1": 1,
"TT2": 2,
},
{
"type": "out",
"TT1": 1,
"TT2": 2,
}
]
}
}
]
},
"Other":
{ O1: 1,
O2: 2,
O3: "hello"
},
"AddedBy": "name"
"AddedDate": "11/02/2019"
}
I am not able to write a function here, which can do this for me, my code is simply searching in this array, and I loop through to find it, which is I am sure not the correct way.
My code is not elegant, and I am using somehow repetitive functions. This is just one snippet, to find out the keys in one level. I want only 1 or 2 functions to do all this
function Search(paramKey, formDataArray) {
var varParams = [];
for (var key in formDataArray) {
if (formDataArray.hasOwnProperty(key)) {
var val = formDataArray[key];
for (var ikey in val) {
if (val.hasOwnProperty(ikey)) {
if (ikey == paramKey)
varParams.push(val[ikey]);
}
}
}
}
return varParams;
}
One more test case if in Array: to Return only single array of values, without adding. (Update - I achieved this through editing the code following part)
notsumsingle: function (target, key, value) {
if (target[key] === undefined) {
target[key] = value;
return;
}
target.push(value);
},
"groupData": [
{
"A1G1": 1,
"A1G2": 22,
"AIG3": 4,
"AIG4": "Rob"
},
{
"A1G1": 1,
"A1G2": 41,
"AIG3": 3,
"AIG4": "John"
},
{
"A1G1": 1,
"A1G2": 3,
"AIG3": 1,
"AIG4": "Andy"
}
],
perform(["AIG2",""AIG4"], "notsum")
It is returning me
[
[
22,
41,
3
]
],
[
[
"",
"Ron",
"Andy"
]
]
Instead, can I add one more variation "SingleArray" like "sum" and "notsum" and get the result as single Array.
[
22,
41,
3
]
[
"",
"Ron",
"Andy"
]
4th one, I asked, is it possible the function intelligent enough to pick up the sum of arrays or sum of individual fields automatically. for example, in your example, you have used "sum" and "total" to identify that.
console.log(perform(["WA1", "WA3", "RAE1"], "total")); // 6
console.log(perform(["Web1", "Web2"], "sum")); // [7, 1]
Can the function, just use "sum" and returns single or array based on if it finds array, return [7,1] if not return 6
5th : I found an issue in the code, if the json collection is added this way
perform(["RAE1"], "notsum") //[[1,1]]
perform(["RAE1"], "sum") //2
It returns [1, 1], or 2 although there is only one RAE1 defined and please note it is not an array [] so it should not be encoded into [[]] array, just the object key
"RawData" : {
"RawOverview": {
"RAE1" : 1,
"RAE2" : 1,
}
For making it easier, and to take the same interface for getting sums or not sums and a total, without any array, you could introduce another operation string total for getting the sum of all keys.
This approach takes an object for getting a function which either add an value to an array at the same index or stores the value at an specified index, which match the given keys array of the function.
For iterating the object, you could take the key/value pairs and iterate until no more object is found.
As result, you get an array, or the total sum of all items.
BTW, the keys of an object are case sensitive, for example 'WEB1' does not match 'Web1'.
function perform(keys, operation) {
function visit(object) {
Object
.entries(object)
.forEach(([k, v]) => {
if (k in indices) return fn(result, indices[k], v);
if (v && typeof v === 'object') visit(v);
});
}
var result = [],
indices = Object.assign({}, ...keys.map((k, i) => ({ [k]: i }))),
fn = {
notsum: function (target, key, value) {
if (target[key] === undefined) {
target[key] = value;
return;
}
if (!Array.isArray(target[key])) {
target[key] = [target[key]];
}
target[key].push(value);
},
sum: function (target, key, value) {
target[key] = (target[key] || 0) + value;
}
}[operation === 'total' ? 'sum' : operation];
visit(data);
return operation === 'total'
? result.reduce((a, b) => a + b)
: result;
}
var data = { version: "1.0", submission: "editing", WebData: { WA1: 3, WA3: 2, WAX: "NEO", WebGroup: [{ Web1: 3, Web2: 0 }, { Web1: 4, Web2: 1 }] }, NonWebData: { NWA1: 3, NWA2: "INP", NWA3: 2 }, FormInputs: { FM11: 3, FM12: 1, FM13: 2 }, RawData: { RawOverview: { RAE1: 1, RAE2: 1 }, RawGroups: [{ name: "A1", id: "1", data: { AD1: 'period', AD2: 2, AD3: 2, transfers: [{ type: "in", TT1: 1, TT2: 2 }, { type: "out", TT1: 1, TT2: 2 }] } }, { name: "A2", id: "2", data: { AD1: 'period', AD2: 2, AD3: 2, transfers: [{ type: "in", TT1: 1, TT2: 2 }, { type: "out", TT1: 1, TT2: 2 }] } }] }, Other: { O1: 1, O2: 2, O3: "hello" }, AddedBy: "name", AddedDate: "11/02/2019" };
console.log(perform(["WA1", "WA3", "RAE1"], "notsum")); // [3, 2, 1]
console.log(perform(["WA1", "WA3", "RAE1"], "total")); // 6
console.log(perform(["Web1", "Web2"], "sum")); // [7, 1]
console.log(perform(["Web1", "Web2"], "notsum")); // [[3, 4], [0, 1]]
.as-console-wrapper { max-height: 100% !important; top: 0; }

loopback REST API filter by nested data

I would like to filter from REST API by nested data. For example this object:
[
{
"name": "Handmade Soft Fish",
"tags": "Rubber, Rubber, Salad",
"categories": [
{
"name": "women",
"id": 2,
"parent_id": 0,
"permalink": "/women"
},
{
"name": "kids",
"id": 3,
"parent_id": 0,
"permalink": "/kids"
}
]
},
{
"name": "Tasty Rubber Soap",
"tags": "Granite, Granite, Chair",
"categories": [
{
"name": "kids",
"id": 3,
"parent_id": 0,
"permalink": "/kids"
}
]
}
]
is comming by GET /api/products?filter[include]=categories
and i would like to get only products which has category name "women". How do this?
LoopBack does not support filters based on related models.
This is a limitation that we have never had bandwidth to solve, unfortunately :(
For more details, see the discussion and linked issues here:
Filter on level 2 properties: https://github.com/strongloop/loopback/issues/517
Filter by properties of related models (use SQL JOIN in queries): https://github.com/strongloop/loopback/issues/683
Maybe you want to get this data by the Category REST API. For example:
GET /api/categories?filter[include]=products&filter[where][name]=woman
The result will be a category object with all products related. To this, will be necessary declare this relation on the models.
Try like this.It has worked for me.
const filter = {
where: {
'categories.name': {
inq: ['women']**strong text**
}
}
};
Pass this filter to request as path parameters and the request would be like bellow
GET /api/categoriesfilter=%7B%22where%22:%7B%categories.name%22:%7B%22inq%22:%5B%women%22%5D%7D%7D%7D
Can you share how it looks like without filter[include]=categorie, please ?
[edit]
after a few questions in comment, I'd build a remote method : in common/models/myModel.js (inside the function) :
function getItems(filter, categorieIds = []) {
return new Promise((resolve, reject) => {
let newInclude;
if (filter.hasOwnProperty(include)){
if (Array.isArray(filter.include)) {
newInclude = [].concat(filter.include, "categories")
}else{
if (filter.include.length > 0) {
newInclude = [].concat(filter.include, "categories");
}else{
newInclude = "categories";
}
}
}else{
newInclude = "categories";
}
myModel.find(Object.assign({}, filter, {include: newInclude}))
.then(data => {
if (data.length <= 0) return resolve(data);
if (categoriesIds.length <= 0) return resolve(data);
// there goes your specific filter on categories
const tmp = data.filter(
item => item.categories.findIndex(
categorie => categorieIds.indexOf(categorie.id) > -1
) > -1
);
return resolve(tmp);
})
}
}
myModel.remoteMethod('getItems', {
accepts: [{
arg: "filter",
type: "object",
required: true
}, {
arg: "categorieIds",
type: "array",
required: true
}],
returns: {arg: 'getItems', type: 'array'}
});
I hope it answers your question...

Lodash - filter by string 1,2

I have string
"1,2" // which is from the db.field
I am trying to filter with lodash and some thing like the following worked
_.filter(jsonArray, function(res) { return (res.id == 1 || res.id == 2); });
Please assume jsonArray as follows:
[
{ 'id': '1', 'age': 60 },
{ 'id': '2', 'age': 70 },
{ 'id': '3', 'age': 22 },
{ 'id': '4', 'age': 33 }
];
here the problem is I need to split the sting 1,2 and apply,
But note that 1,2 is not always 1,2 - it could be 1,2,3 and this string is dynamic from db.field.
Now I am searching if there is any way I can just use the string say like
-.filter(jsonArray, function(res){ return res.id <is equal to one of the value in 1,2,3,4 >})
I think its obvious the split this string into array and ... but I am not sure of doing it, help it out please.
You first need to split db.field to make it as array of ids that can be evaluated easily when matching items. Next, use the filter() that you've already constructed to check if such items match the ids using includes.
var ids = db.field.split(',').map(Number);
var result = _.filter(jsonArray, function(res) {
return _.includes(ids, res.id);
});
var db = { field: '1,2' };
var jsonArray = [
{ 'id': 1, 'age': 60 },
{ 'id': 2, 'age': 70 },
{ 'id': 3, 'age': 22 },
{ 'id': 4, 'age': 33 }
];
var ids = db.field.split(',').map(Number);
var result = _.filter(jsonArray, function(res) {
return _.includes(ids, res.id);
});
document.write('<pre>' + JSON.stringify(result, 0, 4) + '</pre>');
<script src="https://cdn.jsdelivr.net/lodash/4.12.0/lodash.min.js"></script>

Parsing JSON to highcharts - Data gathered using AJAX

My JSON looks like
[{"target": "sumSeries(integral(org.example.fib.hi.value),integral(org.example.fib.hi1.value))",
"datapoints":
[
[2, 1359214560],
[3, 1359215040],
[4, 1359215050],
[null, 1359215060],
[null, 1359215070],
[5, 1359215080],
[7, 1359215090],
[9, 1359215100],
[10, 1359215110],
[null, 1359215120],
[10, 1359215130],
[14, 1359215140],
[null, 1359215150]
]}
]
I am trying to grab this data from my localhost and have highcharts render a line graph.
I have something like :
$(function() {
$.getJSON('http://localhost/render?target=sumSeries(integral(org.example.fib.*.value))&from=-10minutes&format=json', function(data) {
// Create the chart
window.chart = new Highcharts.StockChart({
chart : {
renderTo : 'container'
},
rangeSelector : {
selected : 1
},
plotOptions: {
series: {
stacking: 'normal'
}
},
series : [{
name : 'AAPL',
data : data,
tooltip: {
yDecimals: 2
}
}]
});
});
});
How can I parse this data?
I assume that you use timestamps (as second parameter in JSON), so data should be parsing to display as datetime. If yes, you should revert your data parameters in JSON, and multiply timestamps by 1000 (javascript timestamps format). Example of parsing:
var tmpdata = [],
i = 0,
len = data[0].datapoints.length;
for(i=0;i<len;i++)
{
tmpdata[i] = [data[0].datapoints[i][1]*1000,data[0].datapoints[i][0]];
}
and then in series data:
series : [{
name : 'AAPL',
data : tmpdata,
tooltip: {
yDecimals: 2
}
}]

Resources