I have a schema that looks like this
var user = new Schema({
preference1: { // preference is a number between 1 - 10
type: Number
},
preference2: { // preference is a number between 1 - 10
type: Number
}
})
how do I find the documents and sort by a function on the preferences fields? Say fn is something like this
fn = Math.abs(preference1 - 3) + preference2 ^ 2
I kind of find a temporary solution. It works but isn't really what I was looking for... the code is really messy and apparently you cannot take arbitrary fn for sorting..
for example, say fn = (a+3) * (b+5)
Media.aggregate()
.project({
"type": 1,
"status": 1,
"newField1": { "$add": [ "$type", 3 ] },
"newField2": { "$add": [ 5, "$status" ] },
})
.project({
"newField3": { "$multiply": [ "$newField1", "$newField2" ] },
})
.sort("newField3")
.exec()
Related
Here's the logic I am trying to accomplish:
I am using Elasticsearch to display top selling Products and randomly inserting newly created products in the results using function_score query DSL.
The issue I am facing is that I am using random_score fn for newly created products and the query does inserts new products up till page 2 or 3 but then rest all the other newly created products pushed towards the end of search results.
Here's the logic written for function_score:
function_score: {
query: query,
functions: [
{
filter: [
{ terms: { product_type: 'sponsored') } },
{ range: { live_at: { gte: 'CURRENT_DATE - 1.MONTH' } } }
],
random_score: {
seed: Time.current.to_i / (60 * 10), # new seed every 10 minutes
field: '_seq_no'
},
weight: 0.975
},
{
filter: { range: { live_at: { lt: 'CURRENT_DATE - 1.MONTH' } } },
linear: {
weighted_sales_rate: {
decay: 0.9,
origin: 0.5520974289580515,
scale: 0.5520974289580515
}
},
weight: 1
}
],
score_mode: 'sum',
boost_mode: 'replace'
}
And then I am sorting based on {"_score" => { "order" => "desc" } }
Let's say there are 100 sponsored products created in last 1 month. Then the above Elasticsearch query displays 8-10 random products (3 to 4 per page) as I scroll through 2 or 3 pages but then all other 90-92 products are displayed in last few pages of the result. - This is because the score calculated by random_score for 90-92 products is coming lower than the score calculated by linear
decay function.
Kindly suggest how can I modify this query so that I continue to see newly created Products as I navigate through pages and can prevent pushing new records towards the end of results.
[UPDATE]
I tried adding gauss decay function to this query (so that I can somehow modify the score of the products appearing towards the end of result) like below:
{
filter: [
{ terms: { product_type: 'sponsored' } },
{ range: { live_at: { gte: 'CURRENT_DATE - 1.MONTH' } } },
{ range: { "_score" => { lt: 0.9 } } }
],
gauss: {
views_per_age_and_sales: {
origin: 1563.77,
scale: 1563.77,
decay: 0.95
}
},
weight: 0.95
}
But this too is not working.
Links I have referred to:
https://intellipaat.com/community/12391/how-to-get-3-random-search-results-in-elasticserch-query
Query to get random n items from top 100 items in Elastic Search
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/query-dsl-function-score-query.html
I am not sure if this is the best solution, but I was able to accomplish this with wrapping up the original query with script_score query + I have added a new ElasticSearch indexing called sort_by_views_per_year. Here's how the solution looks:
Link I referred to: https://github.com/elastic/elasticsearch/issues/7783
attribute(:sort_by_views_per_year) do
object.live_age&.positive? ? object.views_per_year.to_f / object.live_age : 0.0
end
Then while querying ElasticSearch:
def search
#...preparation of query...#
query = original_query(query)
query = rearrange_low_scoring_docs(query)
sort = apply_sort opts[:sort]
Product.search(query: query, sort: sort)
end
I have not changed anything in original_query (i.e. using random_score to products <= 1.month.ago and then use linear decay function).
def rearrange_low_scoring_docs query
{
function_score: {
query: query,
functions: [
{
script_score: {
script: "if (_score.doubleValue() < 0.9) {return 0.9;} else {return _score;}"
}
}
],
#score_mode: 'sum',
boost_mode: 'replace'
}
}
end
Then finally my sorting looks like this:
def apply_sort
[
{ '_score' => { 'order' => 'desc' } },
{ 'sort_by_views_per_year' => { 'order' => 'desc' } }
]
end
It would be way too helpful if ElasticSearch random_score query DSL starts supporting something like: max_doc_to_include and min_score attributes. So that I can use it like:
{
filter: [
{ terms: { product_type: 'sponsored' } },
{ range: { live_at: { gte: 'CURRENT_DATE - 1.MONTH' } } }
],
random_score: {
seed: 123456, # new seed every 10 minutes
field: '_seq_no',
max_doc_to_include: 10,
min_score: 0.9
},
weight: 0.975
},
I am using the following dataweave function, and it does works.
%dw 2.0
import * from dw::core::Arrays
output application/json
var mysqlInvoices = [
{
"id": 1,
"owner": "Joseph"
},
{
"id": 2,
"owner": "Maria"
}
]
var sapInvoices = [
{
"number": 3,
"issuedBy": "XYZ"
},
{
"number": 4,
"issuedBy": "ABC"
}
]
---
leftJoin(mysqlInvoices, sapInvoices, (m) -> m.id, (s) -> s.number) map (item, index) ->
(item.l mapObject (sItem, sKey) ->
(if ((sKey) as String == "id") "identifier"
else if ((sKey) as String == "owner") "ownerName"
else (sKey)): sItem)
++
(if (item.r != null)
item.r mapObject (sItem, sKey) ->
(sKey): sItem
else
sapInvoices[0] mapObject
(sItem, sKey) -> (sKey): "")
However, I am thinking if I can improve this function at two points:
change the key conditions:
I dont think that is the best practice to check every key match an if condition to change it:
(if ((sKey) as String == "id") "identifier"
else if ((sKey) as String == "owner") "ownerName"
else (sKey)): sItem
Use the original object to map it as an empty string when the leftJoin do not match keys:
sapInvoices[0] mapObject (sItem, sKey) ->
(sKey): ""
I am uncomfortable with these two points, and I believe that there are ways to improve this code, I just dont know how.
If there is a very different way of doing the same task, I also appreciate that kind of suggestion.
Based on George's answer, you can remove pluck and match and directly combine left and right table. See below:
%dw 2.0
import * from dw::core::Arrays
output application/json
var mysqlInvoices = [
{
"id": 1,
"owner": "Joseph"
},
{
"id": 2,
"owner": "Maria"
}
]
var sapInvoices = [
{
"number": 3,
"issuedBy": "XYZ"
},
{
"number": 4,
"issuedBy": "ABC"
}
]
var fs2rn = {
id: "identifier",
owner: "ownerName"
}
var rightEmpty= {number:"",issuedBy:""}
---
leftJoin(
// Do the field renaming at the very begining
mysqlInvoices map ($ mapObject {(fs2rn[$$] default $$): $}),
sapInvoices,
(m) -> m.identifier,
(s) -> s.number
) map (item) -> item.l ++ (item.r default rightEmpty)
Give the following a try, if anything the code seems a bit simpler:
%dw 2.0
import * from dw::core::Arrays
output application/json
var mysqlInvoices = [
{
"id": 1,
"owner": "Joseph"
},
{
"id": 2,
"owner": "Maria"
}
]
var sapInvoices = [
{
"number": 3,
"issuedBy": "XYZ"
},
{
"number": 4,
"issuedBy": "ABC"
}
]
var fs2rn = {
id: "identifier",
owner: "ownerName"
}
var rightEmpty= {number:"",issuedBy:""}
---
leftJoin(
// Do the field renaming at the very begining
mysqlInvoices map ($ mapObject {(fs2rn[$$] default $$): $}),
sapInvoices,
(m) -> m.identifier,
(s) -> s.number
)
// Iterate over the results
// Get just the values, and colapse the objects into a single object
map (
{($ pluck $)}
)
// Iterate over the results and use pattern-matching to
//
map (
$ match {
// Check if you have an id but not a number fields
// In which case add the rightEmpty object
case o if (o.identifier? and not (o.number?)) -> o ++ rightEmpty
// Or give the object because you now have both an id and a number
else o -> o
}
)
The features and functions I used are:
Dynamic Elements, documentation
pluck, documentation
Pattern-matching using the match operator, documentation
If I was to give you an advice, it would be to better indent your code. Nonetheless, pretty good job!
I would like to filter from REST API by nested data. For example this object:
[
{
"name": "Handmade Soft Fish",
"tags": "Rubber, Rubber, Salad",
"categories": [
{
"name": "women",
"id": 2,
"parent_id": 0,
"permalink": "/women"
},
{
"name": "kids",
"id": 3,
"parent_id": 0,
"permalink": "/kids"
}
]
},
{
"name": "Tasty Rubber Soap",
"tags": "Granite, Granite, Chair",
"categories": [
{
"name": "kids",
"id": 3,
"parent_id": 0,
"permalink": "/kids"
}
]
}
]
is comming by GET /api/products?filter[include]=categories
and i would like to get only products which has category name "women". How do this?
LoopBack does not support filters based on related models.
This is a limitation that we have never had bandwidth to solve, unfortunately :(
For more details, see the discussion and linked issues here:
Filter on level 2 properties: https://github.com/strongloop/loopback/issues/517
Filter by properties of related models (use SQL JOIN in queries): https://github.com/strongloop/loopback/issues/683
Maybe you want to get this data by the Category REST API. For example:
GET /api/categories?filter[include]=products&filter[where][name]=woman
The result will be a category object with all products related. To this, will be necessary declare this relation on the models.
Try like this.It has worked for me.
const filter = {
where: {
'categories.name': {
inq: ['women']**strong text**
}
}
};
Pass this filter to request as path parameters and the request would be like bellow
GET /api/categoriesfilter=%7B%22where%22:%7B%categories.name%22:%7B%22inq%22:%5B%women%22%5D%7D%7D%7D
Can you share how it looks like without filter[include]=categorie, please ?
[edit]
after a few questions in comment, I'd build a remote method : in common/models/myModel.js (inside the function) :
function getItems(filter, categorieIds = []) {
return new Promise((resolve, reject) => {
let newInclude;
if (filter.hasOwnProperty(include)){
if (Array.isArray(filter.include)) {
newInclude = [].concat(filter.include, "categories")
}else{
if (filter.include.length > 0) {
newInclude = [].concat(filter.include, "categories");
}else{
newInclude = "categories";
}
}
}else{
newInclude = "categories";
}
myModel.find(Object.assign({}, filter, {include: newInclude}))
.then(data => {
if (data.length <= 0) return resolve(data);
if (categoriesIds.length <= 0) return resolve(data);
// there goes your specific filter on categories
const tmp = data.filter(
item => item.categories.findIndex(
categorie => categorieIds.indexOf(categorie.id) > -1
) > -1
);
return resolve(tmp);
})
}
}
myModel.remoteMethod('getItems', {
accepts: [{
arg: "filter",
type: "object",
required: true
}, {
arg: "categorieIds",
type: "array",
required: true
}],
returns: {arg: 'getItems', type: 'array'}
});
I hope it answers your question...
I have objects like this:
{
buildings: {
"1": {
"l": 0 ,
"r": 0 ,
"s": 0 ,
"type": "GoldMine" ,
"x": 2 ,
"y": 15
} ,
"10": {
"l": 0 ,
"r": 6 ,
"s": 2 ,
"type": "MagicMine" ,
"x": 26 ,
"y": 22
}
} ,
[...]
}
I want to get objects with buildings of type "GoldMine".
I tried something with map:
r.table("Characters").map(function(row) {
return row("planet")("buildings")
})
With keys() I can iterate it:
r.db("Unnyworld").table("Characters").map(function(row) {
return row("planet")("buildings").keys().map(function(key) {
return "need to get only buildings with type == GoldMine";
})
}).limit(2)
But it returns all buildings. I want to get only buildings with type == GoldMine and change field x.
Something like this may work:
r.table('Characters')
.concatMap(function(doc) {
return doc("planet")("buildings").keys().map(function(k) {
return {id: doc('id'), key: k, type: doc("planet")("buildings")(k)('type'), x: doc("planet")("buildings")(k)('x')}
})
})
.filter(function(building) {
return building('type').eq('GoldMine')
})
.forEach(function(doc) {
return r.table('Characters').get(doc('id'))
.update({
planet: {buildings: r.object(doc('key'), {x: 1111111})}
})
})
Basically create a flat array from building by using concatMap then filter it. With result data, we can iterator over it and update to value that we want.
I'm trying to insert the results of a query from one table into another table. However, when I attempt to run the query I am receiving an error.
{
"deleted": 0 ,
"errors": 1 ,
"first_error": "Expected type OBJECT but found ARRAY." ,
"inserted": 0 ,
"replaced": 0 ,
"skipped": 0 ,
"unchanged": 0
}
Here is the the insert and query:
r.db('test').table('destination').insert(
r.db('test').table('source').map(function(doc) {
var result = doc('result');
return result('section_list').concatMap(function(section) {
return section('section_content').map(function(item) {
return {
"code": item("code"),
"name": item("name"),
"foo": result("foo"),
"bar": result("bar"),
"baz": section("baz"),
"average": item("average"),
"lowerBound": item("from"),
"upperBound": item("to")
};
});
});
});
);
Is there a special syntax for this, or do I have to retrieve the results and then run a separate insert?
The problem is that your inner query is returning a stream of arrays. You can't insert arrays into a table (only objects), so the query fails. If you change the outermost map into a concatMap it should work.
The problem here was that the result was a sequence of an array of objects. i.e
[ [ { a:1, b:2 }, { a:1, b:2 } ], [ { a:2, b:3 } ] ]
Therefore, I had to change the outer map call to a concatMap call. The query then becomes:
r.db('test').table('destination').insert(
r.db('test').table('source').concatMap(function(doc) {
var result = doc('result');
return result('section_list').concatMap(function(section) {
return section('section_content').map(function(item) {
return {
"code": item("code"),
"name": item("name"),
"foo": result("foo"),
"bar": result("bar"),
"baz": section("baz"),
"average": item("average"),
"lowerBound": item("from"),
"upperBound": item("to")
};
)});
});
});
}
Thanks goes to #AtnNn on the #rethinkdb freenode for pointing me in the right direction.