How to get the total page number for pagination - spring

I try to make my pageable data for weeks everything works but I would like to know the total number of pagination pages here are the data of my mongonDB.
{
"content": [
{
"mame": "iPhone X",
"eategory": "High-Tech",
"productId": "aDPXF7Xq",
"details": {
"vating": "00",
"stocks": "20",
"price": "800000",
"tags": [
{
"tagsl": "Apple",
"tags2": "aull",
"tags3": null
}
],
"brand": "Apple",
"description": "Iphone xX",
"picture": [
{
"picturel": "photol",
"picture2": null,
"picture3": null,
"picture4": null,
"picture5": null
}
],
"thumbnails": [
{
"thumbnaill": "thumbails 1",
"thumbnail2": null,
"thumbnail3": null,
"thumbnail4": null,
"thumbnail5": null
}
]
}
},
{
"mame": "iPhone X",
"eategory": "High-Tech",
"productId": "cjjVqOBk",
"details": {
"rating": "10",
"stocks": "20",
"price": "800000",
"tags": [
{
"tagsl": "Apple",
"tags2": "aull",
"tags3": null
}
],
"brand": "Apple",
"description": "Iphone xX",
"picture": [
{
"picturel": "photol",
"picture2": null,
"picture3": null,
"picture4": null,
"picture5": null
}
],
"thumbnails": [
{
"thumbnaill": "thumbails 1",
"thumbnail2": null,
"thumbnail3": null,
"thumbnail4": null,
"thumbnail5": null
}
]
}
},
{
"mame": "iPhone X",
"eategory": "High-Tech",
"productId": "LhKiRGr6",
"details": {
"vating": "10",
"stocks": "20",
"price": "800000",
"tags": [
{
"tagsl": "Apple",
"tags2": "aull",
"tags3": null
}
],
"brand": "Apple",
"description": "Iphone xX",
"picture": [
{
"picturel": "photol",
"picture2": null,
"picture3": null,
"picture4": null,
"picture5": null
}
],
"thumbnails": [
{
"thumbnaill": "thumbails 1",
"thumbnail2": null,
"thumbnail3": null,
"thumbnail4": null,
"thumbnail5": null
}
]
}
},
{
"mame": "iPhone X",
"eategory": "High-Tech",
"productId": "dgCvi8NJ",
"details": {
"vating": "10",
"stocks": "20",
"price": "800000",
"tags": [
{
"tagsl": "Apple",
"tags2": "aull",
"tags3": null
}
],
"brand": "Apple",
"description": "Iphone xX",
"picture": [
{
"picturel": "photol",
"picture2": null,
"picture3": null,
"picture4": null,
"picture5": null
}
],
"thumbnails": [
{
"thumbnaill": "thumbails 1",
"thumbnail2": null,
"thumbnail3": null,
"thumbnail4": null,
"thumbnail5": null
}
]
}
}
],
"pageable": {
"sort": {
"sorted": false,
"unsorted": true,
"empty": true
},
"offset": 0,
"pageNumber": 0,
"pageSize": 20,
"paged": true,
"unpaged": false
},
"totalPages": 1,
"totalElements": 4,
"last": true,
"size": 20,
"sort": {
"sorted": false,
"unsorted": true,
"empty": true
},
"number": 0,
"numberOfElements": 4,
"first": true,
"empty": false
}
Now here is the data paging code
#GetMapping("/engine/search")
public PageImpl<Products> engineSearch(#RequestParam("p") String query, #RequestParam(value = "page", defaultValue = "0") int page, Pageable pageable) {
Criteria criteria = new Criteria();
final Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match( criteria.orOperator(
Criteria.where( "name" ).regex( query, "i" ),
Criteria.where( "details.brand" ).regex( query, "i" ),
Criteria.where( "details.tags.tags1" ).regex( query, "i" ),
Criteria.where( "details.tags.tags2" ).regex( query, "i" ),
Criteria.where( "details.tags.tags3" ).regex( query, "i" )
)),
Aggregation.skip(page * 4),
Aggregation.limit(4)
);
List<Products> filter = mongoTemplate.aggregate(aggregation, "Product", Products.class).getMappedResults();
return new PageImpl<Products>(filter, pageable, filter.size());
}

Mongodb 3.4 has introduced $facet aggregation which processes multiple aggregation pipelines within a single stage on the same set of input documents.
Using $facet and $group you can find documents with $limit and can get total count.
You can use below aggregation in mongodb 3.6
db.collection.aggregate([
{ "$facet": {
"totalData": [
{ "$match": { }},
{ "$skip": 10 },
{ "$limit": 10 }
],
"totalCount": [
{ "$count": "count" }
]
}}
])

Related

Change entry in json file that matches the condition in CloudFront Update Distribution

We have a scenario where we need to replace the TargetOriginID of CacheBehaviour in the distribution of a json file. We need to replace the Existing TargetOriginID with New Values. I have tried with below jq command but not getting any closer
for targetoriginID in $(jq '(.CacheBehaviors.Items[].TargetOriginId)' distconfig.json);
do
echo "#######fetch the new value from change behaviour json file"#######
NewValue=$(jq -r "map(select(.targetoriginid == ""$targetoriginID""))[].targetorigindr" changebehaviour.json)
echo "#########replace value in dist config json file with new value from change behaviour###########"
jq -r '(.CacheBehaviors.Items[].TargetOriginId | select(. == "$targetoriginID")) = "$NewValue"' distconfig.json > "tmp" && mv "tmp" distconfig.json
{
"CachedMethods": {
"Quantity": 3,
"Items": [
"HEAD",
"GET",
"OPTIONS"
]
}
},
"SmoothStreaming": false,
"Compress": false,
"LambdaFunctionAssociations": {
"Quantity": 0
},
"FunctionAssociations": {
"Quantity": 0
},
"FieldLevelEncryptionId": "",
"ForwardedValues": {
"QueryString": true,
"Cookies": {
"Forward": "none"
},
"Headers": {
"Quantity": 9,
"Items": [
"Authorization",
"Origin",
"access-control-allow-credentials",
"expires",
"access-control-max-age",
"access-control-allow-headers",
"cache-control",
"access-control-allow-methods",
"pragma"
]
},
"QueryStringCacheKeys": {
"Quantity": 1,
"Items": [
"*"
]
}
},
"MinTTL": 0,
"DefaultTTL": 86400,
"MaxTTL": 31536000
},
"CacheBehaviors": {
"Quantity": 2,
"Items": [
{
"PathPattern": "jkl/*",
"TargetOriginId": "nkl/Prod",
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"TrustedKeyGroups": {
"Enabled": false,
"Quantity": 0
},
"ViewerProtocolPolicy": "redirect-to-https",
"AllowedMethods": {
"Quantity": 7,
"Items": [
"HEAD",
"DELETE",
"POST",
"GET",
"OPTIONS",
"PUT",
"PATCH"
],
"CachedMethods": {
"Quantity": 3,
"Items": [
"HEAD",
"GET",
"OPTIONS"
]
}
},
"SmoothStreaming": false,
"Compress": false,
"LambdaFunctionAssociations": {
"Quantity": 0
},
"FunctionAssociations": {
"Quantity": 0
},
"FieldLevelEncryptionId": "",
"ForwardedValues": {
"QueryString": true,
"Cookies": {
"Forward": "all"
},
"Headers": {
"Quantity": 9,
"Items": [
"Authorization",
"Origin",
"access-control-allow-credentials",
"access-control-max-age",
"access-control-allow-headers",
"cache-control",
"access-control-allow-methods",
"expirers",
"pragma"
]
},
"QueryStringCacheKeys": {
"Quantity": 1,
"Items": [
"*"
]
}
},
"MinTTL": 0,
"DefaultTTL": 86400,
"MaxTTL": 31536000
},
{
"PathPattern": "fgh/*",
"TargetOriginId":"xyz/Prod",
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"TrustedKeyGroups": {
"Enabled": false,
"Quantity": 0
},
"ViewerProtocolPolicy": "redirect-to-https",
"AllowedMethods": {
"Quantity": 7,
"Items": [
"HEAD",
"DELETE",
"POST",
"GET",
"OPTIONS",
"PUT",
"PATCH"
],
"CachedMethods": {
"Quantity": 3,
"Items": [
"HEAD",
"GET",
"OPTIONS"
]
}
},
"SmoothStreaming": false,
"Compress": false,
"LambdaFunctionAssociations": {
"Quantity": 0
},
"FunctionAssociations": {
"Quantity": 0
},
"FieldLevelEncryptionId": "",
"ForwardedValues": {
"QueryString": true,
"Cookies": {
"Forward": "none"
},
"Headers": {
"Quantity": 10,
"Items": [
"access-control-allow-origin",
"authorization",
"Origin",
"access-control-allow-credentials",
"access-control-max-age",
"access-control-allow-headers",
"cache-control",
"access-control-allow-methods",
"expirers",
"pragma"
]
},
"QueryStringCacheKeys": {
"Quantity": 1,
"Items": [
"*"
]
}
},
"MinTTL": 0,
"DefaultTTL": 0,
"MaxTTL": 0
}
]
}
Looking for a solution to make bulk change in CacheBehaviour for all TargetOriginID's
Providing a second answer with some assumption. Let's start with a minimal example input:
{
"CacheBehaviors": {
"Quantity": 2,
"Items": [
{
"PathPattern": "jkl/*",
"TargetOriginId": [
"nkl",
"something else"
]
},
{
"PathPattern": "fgh/*",
"TargetOriginId": [
"xyz",
"abc"
]
}
]
},
"IsIPV6Enabled": true
}
Assuming you want to define a mapping "{old1: new1, old2: new2, old3: new3, …}", the following program could work and is easily extendable:
(.CacheBehaviors.Items[].TargetOriginId[]) |= ({
nkl: "wkl",
xyz: "123",
"something else": "is changed too",
"not found": "never replaced"
}[.] // .)
And if you like to have your mapping at the top/start of your program, bind it to a variable:
{
nkl: "wkl",
xyz: "123",
"something else": "is changed too",
"not found": "rever replaced"
} as $mapping
| (.CacheBehaviors.Items[].TargetOriginId[]) |= ($mapping[.] // .)
Output after running through above's program:
{
"CacheBehaviors": {
"Quantity": 2,
"Items": [
{
"PathPattern": "jkl/*",
"TargetOriginId": [
"wkl",
"is changed too"
]
},
{
"PathPattern": "fgh/*",
"TargetOriginId": [
"123",
"abc"
]
}
]
},
"IsIPV6Enabled": true
}
Is that what you are after? This chooses the new value by key from an object/dictionary/map and falls back to the current value if no mapping was found.
… |= ({ old: "new" }[.] // .)
The mapping itself could be provided as an argument to jq from a separate file:
jq --slurpfile mapping old_to_new_ids.json \
'(.CacheBehaviors.Items[].TargetOriginId[]) |= ($mapping[0][.] // .)'
With the contents of file old_to_new_ids.json simply being:
{
"nkl": "wkl",
"xyz": "123",
"something else": "is changed too",
"not found": "never replaced"
}
As of this writing, the posted JSON is invalid, so it's not clear exactly what is needed, but the following is illustrative:
jq --arg newvalue xyz '
(.. | objects | select(has("TargetOriginId")) | .TargetOriginId) |= $newvalue
'
Fixed (i.e. made valid) and reduced your JSON input down to the necessary minimum (the M in MRE), that is 50 lines.
{
"CacheBehaviors": {
"Quantity": 2,
"Items": [
{
"PathPattern": "jkl/*",
"TargetOriginId": [
"nkl"
],
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"TrustedKeyGroups": {
"Enabled": false,
"Quantity": 0
},
"ViewerProtocolPolicy": "redirect-to-https"
},
{
"PathPattern": "fgh/*",
"TargetOriginId": [
"xyz"
],
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"TrustedKeyGroups": {
"Enabled": false,
"Quantity": 0
},
"ViewerProtocolPolicy": "redirect-to-https"
}
]
},
"CustomErrorResponses": {
"Quantity": 0
},
"Comment": "vvvv",
"Logging": {
"Enabled": true,
"IncludeCookies": false,
"Bucket": "abc.s3.amazonaws.com",
"Prefix": "std"
},
"WebACLId": "",
"HttpVersion": "http2",
"IsIPV6Enabled": true
}
You probably want to run a jq program similar to the following:
(.CacheBehaviors.Items[].TargetOriginId | select(. as $id | "nkl" | IN($id[]))) = ["wkl"]
It selects all TargetOriginIds which contain the value "nkl" and changes the list to only contain "wkl".
Output:
{
"CacheBehaviors": {
"Quantity": 2,
"Items": [
{
"PathPattern": "jkl/*",
"TargetOriginId": [
"wkl"
],
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"TrustedKeyGroups": {
"Enabled": false,
"Quantity": 0
},
"ViewerProtocolPolicy": "redirect-to-https"
},
{
"PathPattern": "fgh/*",
"TargetOriginId": [
"xyz"
],
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"TrustedKeyGroups": {
"Enabled": false,
"Quantity": 0
},
"ViewerProtocolPolicy": "redirect-to-https"
}
]
},
"CustomErrorResponses": {
"Quantity": 0
},
"Comment": "vvvv",
"Logging": {
"Enabled": true,
"IncludeCookies": false,
"Bucket": "abc.s3.amazonaws.com",
"Prefix": "std"
},
"WebACLId": "",
"HttpVersion": "http2",
"IsIPV6Enabled": true
}
From the question it is unclear if "TargetOriginId": ["nkl", "xyz"] should become ["wkl"] or ["wkl", "xyz"]. Or whether ["nkl", "xyz"] is a match because it contains different values too. Maybe you only want to select those where the full TargetOriginId array matches?
If you only want to match items with single-valued target origin ids, the program becomes a bit simpler:
(.CacheBehaviors.Items[].TargetOriginId | select(. == ["nkl"])) = ["wkl"]
If your target origin ids are always an array and you want to change say "[abc,nkl,xyz]" to "[abc,REPLACED,xyz]", then select only those array elements and assign them the new value.
New input (as I understood it, the Q is quite vague on this):
{
"CacheBehaviors": {
"Quantity": 2,
"Items": [
{
"PathPattern": "jkl/*",
"TargetOriginId": [
"nkl",
"something else"
]
},
{
"PathPattern": "fgh/*",
"TargetOriginId": [
"xyz",
"abc"
]
}
]
},
"IsIPV6Enabled": true
}
jq:
(.CacheBehaviors.Items[].TargetOriginId[] | select(. == "nkl")) = "wkl"
Output:
{
"CacheBehaviors": {
"Quantity": 2,
"Items": [
{
"PathPattern": "jkl/*",
"TargetOriginId": [
"wkl",
"something else"
]
},
{
"PathPattern": "fgh/*",
"TargetOriginId": [
"xyz",
"abc"
]
}
]
},
"IsIPV6Enabled": true
}

How to filter an array of objects nested array

Im implementing a new feature into my project for when a user begins to search for a tag input, which is a value stored inside of a nested array. With the code that I have currently written, if the whole tag keyword has been typed out, we get back the corresponding user data. Im trying to get this work on every keystroke. I tried to use 'match()' but no luck. Any tips are greatly appreciated.
.... I figured it out
const filterDataByTag = (value) => {
let tagFilter;
let valueLowercase = value.toLowerCase();
if (initialData.data) {
tagFilter = initialData.data.filter((item) => {
item.tag.map((tag) => tag.toLowerCase());
return item.tag.toString().match(valueLowercase);
});
setFilterData(tagFilter);
}
};
[
{
"city": "Fushë-Muhurr",
"company": "Yadel",
"email": "iorton0#imdb.com",
"firstName": "Ingaberg",
"grades": [
"78",
"100",
"92",
"86",
"89",
"88",
"91",
"87"
],
"id": "1",
"lastName": "Orton",
"pic": "https://storage.googleapis.com/hatchways-app.appspot.com/assessments/data/frontend/images/voluptasdictablanditiis.jpg",
"skill": "Oracle",
"tag": ["hello", "test"]
},
{
"city": "Sanghan",
"company": "Avamm",
"email": "cboards1#weibo.com",
"firstName": "Clarke",
"grades": [
"75",
"89",
"95",
"93",
"99",
"82",
"89",
"76"
],
"id": "2",
"lastName": "Boards",
"pic": "https://storage.googleapis.com/hatchways-app.appspot.com/assessments/data/frontend/images/voluptasautreprehenderit.jpg",
"skill": "Sports",
"tag": ["test"]
},
{
"city": "Kugesi",
"company": "Skalith",
"email": "lromanet2#wired.com",
"firstName": "Laurens",
"grades": [
"88",
"90",
"79",
"82",
"81",
"99",
"94",
"73"
],
"id": "3",
"lastName": "Romanet",
"pic": "https://storage.googleapis.com/hatchways-app.appspot.com/assessments/data/frontend/images/aspernaturnonsapiente.jpg",
"skill": "Employee Handbooks",
"tag": []
},
......
const filterDataByTag = (value) => {
let tagFilter;
let valueLowercase = value.toLowerCase();
if (initialData.data) {
tagFilter = initialData.data.filter((item) => {
item.tag.map((tag) => tag.toLowerCase());
return item.tag.forEach((tag) => tag.match(valueLowercase));
});
console.log(tagFilter);
}
};

Weighted Average value over documents in elastic search

I need to calculate the weighted average value using the elastic search, I can't change the structure of the documents. If we assume that there are 2 indexed documents. The first document
const doc1 = {
"id": "1",
"userId: "2",
"scores" : [
{
"name": "score1",
"value": 93.0
},
{
"name": "score2",
"value": 90.0
},
{
"name": "score3",
"value": 76.0
}
],
"metadata": {
"weight": 130
}
}
Second document
const doc2 = {
"id": "2",
"userId: "2",
"scores" : [
{
"name": "score1",
"value": 80.0
},
{
"name": "score2",
"value": 70.0
},
{
"name": "score3",
"value": 88.0
}
],
"metadata": {
"weight": 50
}
}
Calculations should be done by the following formula:
score1Avg = (doc1.scores['score1'].value * doc1.metadata.weight +
doc2.scores['score1'].value * doc2.metadata.weight)/(doc1.weight+doc2.weight)
score2Avg = (doc1.scores['score2'].value * doc1.metadata.weight +
doc2.scores['score2'].value * doc2.metadata.weight)/(doc1.weight+doc2.weight)
score3Avg = (doc1.scores['score3'].value * doc1.metadata.weight +
doc2.scores['score3'].value * doc2.metadata.weight)/(doc1.weight+doc2.weight)
I tried something with nested type for mapping scores, but I can't access the parent document field metadata.weight. How this should be approached, should I use nested type mapping or this can be done in some other way without that?
Edit: I ended up storing scores element as separated documents. Instead of doc1, now I have the following documents.
{
"id": "1",
"userId: "2",
"score": {
"name": "score1",
"value": 93.0
},
"metadata": {
"weight": 130
}
}
{
"id": "1",
"userId: "2",
"score": {
"name": "score2",
"value": 90.0
},
"metadata": {
"weight": 130
}
}
{
"id": "1",
"userId: "2",
"score": {
"name": "score3",
"value": 76.0
},
"metadata": {
"weight": 130
}
}
And the query is:
GET /scores/_search
{
"size": 0,
"aggs": {
"group_by_score_and_user": {
"composite": {
"sources": [
{
"scoreName": {
"terms": {
"field": "score.name.keyword"
}
}
},{
"userId": {
"terms": {
"field": "userId.keyword"
}
}
}
]
},
"aggs": {
"avg": {
"weighted_avg": {
"value":{ "field": "score.value" },
"weight":{ "field": "metadata.weight" }
}
}
}
}
}
}
Btw, the query with the script approach against 5k documents takes 120 ms on average compared to this which takes about 35-40 ms over 100k documents.
Edited to fit the requirement in the comment, like I said before this is not an optimal solution at all, the usage of scripts + params._source + my subpar java will cause this to be very slow or unusable with a lot of docs.
Still I learned a lot
Mapping:
{
"mappings": {
"properties": {
"id": {
"type": "keyword"
},
"userId": {
"type": "keyword"
},
"scores": {
"properties": {
"name": {
"type": "keyword"
},
"value": {
"type": "float"
}
}
},
"metadata": {
"properties": {
"weight": {
"type": "float"
}
}
}
}
}
}
Docs:
POST ron_test/_doc/1
{
"id": "1",
"userId": "2",
"scores" : [
{
"name": "score1",
"value": 93.0
},
{
"name": "score2",
"value": 90.0
},
{
"name": "score3",
"value": 76.0
}
],
"metadata": {
"weight": 130
}
}
POST ron_test/_doc/2
{
"id": "2",
"userId": "2",
"scores" : [
{
"name": "score1",
"value": 80.0
},
{
"name": "score2",
"value": 70.0
},
{
"name": "score3",
"value": 88.0
}
],
"metadata": {
"weight": 50
}
}
POST ron_test/_doc/3
{
"id": "2",
"userId": "2",
"scores" : [
{
"name": "score1",
"value": 80.0
},
{
"name": "score2",
"value": 70.0
},
{
"name": "score9",
"value": 88.0
}
],
"metadata": {
"weight": 12
}
}
POST ron_test/_doc/4
{
"id": "2",
"userId": "2",
"scores" : [
{
"name": "score9",
"value": 50.0
}
],
"metadata": {
"weight": 17
}
}
Query
GET ron_test/_search
{
"size": 0,
"aggs": {
"weigthed_avg": {
"scripted_metric": {
"init_script": """
state.name_to_sum = new HashMap();
state.name_to_weight = new HashMap();
""",
"map_script": """
for (score in params._source['scores']){
def name = score['name'];
def value = score['value'];
def weight = doc['metadata.weight'].value;
if (state.name_to_sum.containsKey(name)){
state.name_to_sum[name] += value * weight;
}
else {
state.name_to_sum[name] = value * weight;
}
if (state.name_to_weight.containsKey(name)){
state.name_to_weight[name] += weight;
}
else {
state.name_to_weight[name] = weight;
}
}
""",
"combine_script": "return [state.name_to_sum, state.name_to_weight]",
"reduce_script": """
def total_score_per_name = new HashMap();
def total_weigth_per_name = new HashMap();
for (state in states){
total_score_per_name = Stream.concat(total_score_per_name.entrySet().stream(), state[0].entrySet().stream())
.collect(Collectors.groupingBy(Map.Entry::getKey,
Collectors.summingDouble(Map.Entry::getValue)));
total_weigth_per_name = Stream.concat(total_weigth_per_name.entrySet().stream(), state[1].entrySet().stream())
.collect(Collectors.groupingBy(Map.Entry::getKey,
Collectors.summingDouble(Map.Entry::getValue)));
}
def results = new HashMap();
total_score_per_name.forEach((name, score) -> results[name] = score / total_weigth_per_name[name]);
return results;
"""
}
}
}
}
Results
{
"took" : 258,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"weigthed_avg" : {
"value" : {
"score9" : 65.72413793103448,
"score2" : 83.54166666666667,
"score3" : 79.33333333333333,
"score1" : 88.80208333333333
}
}
}
}
More info on scripted metrics
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html
Btw, the way I would choose to simplify this is to insert the metadata.weight value inside every nested score

ElasticSearch - Combine filters & Composite Query to get unique fields combinations

Well.. I am quite "newb" regarding ES so regarding aggregation... there is no words in the dictionary to describe my level regarding it :p
Today I am facing an issue where I am trying to create a query that should execute something similar to a SQL DISTINCT, but among filters. I have this document given (of course, an abstraction of the real situation):
{
"id": "1",
"createdAt": 1626783747,
"updatedAt": 1626783747,
"isAvailable": true,
"kind": "document",
"classification": {
"id": 1,
"name": "a_name_for_id_1"
},
"structure": {
"material": "cartoon",
"thickness": 5
},
"shared": true,
"objective": "stackoverflow"
}
As all the data of the above document can vary, I however have some values that can be redundant, such as classification.id, kind, structure.material.
So, in order to fullfit my requirements, I would like to "group by" these 3 fields in order to have a unique combination of each. If we go deeper, with the following data, I should get the following possibilities:
[{
"id": "1",
"createdAt": 1626783747,
"updatedAt": 1626783747,
"isAvailable": true,
"kind": "document",
"classification": {
"id": 1,
"name": "a_name_for_id_1"
},
"structure": {
"material": "cartoon",
"thickness": 5
},
"shared": true,
"objective": "stackoverflow"
},
{
"id": "2",
"createdAt": 1626783747,
"updatedAt": 1626783747,
"isAvailable": true,
"kind": "document",
"classification": {
"id": 2,
"name": "a_name_for_id_2"
},
"structure": {
"material": "iron",
"thickness": 3
},
"shared": true,
"objective": "linkedin"
},
{
"id": "3",
"createdAt": 1626783747,
"updatedAt": 1626783747,
"isAvailable": false,
"kind": "document",
"classification": {
"id": 2,
"name": "a_name_for_id_2"
},
"structure": {
"material": "paper",
"thickness": 1
},
"shared": false,
"objective": "tiktok"
},
{
"id": "4",
"createdAt": 1626783747,
"updatedAt": 1626783747,
"isAvailable": true,
"kind": "document",
"classification": {
"id": 3,
"name": "a_name_for_id_3"
},
"structure": {
"material": "cartoon",
"thickness": 5
},
"shared": false,
"objective": "snapchat"
},
{
"id": "5",
"createdAt": 1626783747,
"updatedAt": 1626783747,
"isAvailable": true,
"kind": "document",
"classification": {
"id": 3,
"name": "a_name_for_id_3"
},
"structure": {
"material": "paper",
"thickness": 1
},
"shared": true,
"objective": "twitter"
},
{
"id": "6",
"createdAt": 1626783747,
"updatedAt": 1626783747,
"isAvailable": false,
"kind": "document",
"classification": {
"id": 3,
"name": "a_name_for_id_3"
},
"structure": {
"material": "iron",
"thickness": 3
},
"shared": true,
"objective": "facebook"
}
]
based on the above, I should get the following results in the "buckets":
document 1 cartoon
document 2 iron
document 2 paper
document 3 cartoon
document 3 paper
document 3 iron
Of course, for the sake of this example (and to make it easier, I yet don't have any duplicates)
However, on top of that, I need some "pre-filters" as I only want:
Documents that are available isAvailable=true
Documents'structure's thickness should range between 2 and 4 included: 2 >= structure.thickness >= 4
Document's that are shared shared=true
I should so then get only the following combinations compared to the first set of results:
document 1 cartoon -> not a valid result, thickness > 4
document 2 iron
document 2 paper -> not a valid result, isAvailable != true
document 3 cartoon -> not a valid result, thickness > 4
document 3 cartoon -> not a valid result, thickness < 2
document 3 iron -> not a valid result, isAvailable != true
If you're still reading, well.. thanks! xD
So, as you can see, I need all the possible combination of this field regarding the static pattern kind <> classification_id <> structure_material that are matching the filters regarding isAvailable, thickness, shared.
Regarding the output, the hits doesn't matter to me as I don't need the documents but only the combination kind <> classification_id <> structure_material :)
Thanks for any help :)
Max
You can got with Cardinatily aggregations with your existing filters.Please check this url and let me know if you have any queries.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html
Thanks to a colleague, I could finally get it working as expected!
QUERY
GET index-latest/_search
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"term": {
"isAvailable": true
}
},
{
"range": {
"structure.thickness": {
"gte": 2,
"lte": 4
}
}
},
{
"term": {
"shared": true
}
}
]
}
},
"aggs": {
"my_agg_example": {
"composite": {
"size": 10,
"sources": [
{
"kind": {
"terms": {
"field": "kind.keyword",
"order": "asc"
}
}
},
{
"classification_id": {
"terms": {
"field": "classification.id",
"order": "asc"
}
}
},
{
"structure_material": {
"terms": {
"field": "structure.material.keyword",
"order": "asc"
}
}
}
]
}
}
}
}
The given result is then:
{
"took": 11,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": null,
"hits": []
},
"aggregations": {
"my_agg_example": {
"after_key": {
"kind": "document",
"classification_id": 2,
"structure_material": "iron"
},
"buckets": [
{
"key": {
"kind": "document",
"classification_id": 2,
"structure_material": "iron"
},
"doc_count": 1
}
]
}
}
}
So, as we can see, we get the following bucket:
{
"key": {
"kind": "document",
"classification_id": 2,
"structure_material": "iron"
},
"doc_count": 1
}
Note: Be careful regarding the type of your field.. putting .keyword on classification.id was resulting to no results in the buckets... .keyword should be use only on types such as string (as far as I understood, correct me if I am wrong)
As expected, we have the following result (compared to the initial question):
document 2 iron
Note: Be careful, the order of the elements within the aggs.<name>.composite.sources does play a role in the returned results.
Thanks!

How to get value in nested field using Kibana scripted field?

I'm new to Elastic Stack.
Here, I'm trying to get the value for "pressure" and then convert it to numeric value(string⇒numeric) using Kibana scripted field.
I tried scripted field, but it didn't work for me.
Any idea? I really appreciate your support in advance.
One of my data records is as below.
{
"_index": "production",
"_type": "_doc",
"_id": "4570df7a0d4ec1b0e624e868a5861a0f1a9a7f6c35fdsssafafsa734fb152f4bed",
"_version": 1,
"_score": null,
"_source": {
"factorycode": "AM-NY",
"productcode": "STR",
"lastupdatetime": "2020-05-28T04:16:17.590Z",
"#timestamp": "2020-05-28T04:14:48.000Z",
"massproduction": {
"errorcode": null,
"equipment": "P17470110",
"operatorldap": null,
"machinetime": null,
"quantity": "1",
"externalfilename": null,
"errorcomment": null,
"datas": {
"data": [
{
"value": "45.4",
"id": "001",
"name": "pressure"
},
{
"value": "0.45",
"id": "002",
"name": "current"
}
]
},
"ladderver": null,
"eid": null,
"setupid": null,
"model": "0",
"identificationtagid": null,
"workid": "GD606546sf0B002020040800198",
"reuse": {
"num": "0"
},
"registrydate": "2020-05-28T13:14:48",
"product": "GD604564550B00",
"line": "STRS001",
"judge": "1",
"cycletime": null,
"processcode": "OP335",
"registryutcdate": "2020-04-28T04:14:48",
"name": "massproduction"
}
},
"fields": {
"massproduction.registrydate": [
"2020-05-28T13:14:48.000Z"
],
"#timestamp": [
"2020-05-28T04:14:48.000Z"
],
"lastupdatetime": [
"2020-05-28T04:16:17.590Z"
],
"registrydate": [
"2020-05-28T13:14:48.000Z"
],
"massproduction.registryutcdate": [
"2020-05-28T04:14:48.000Z"
],
"registryutcdate": [
"2020-05-28T04:14:48.000Z"
]
},
"sort": [
158806546548000
]
}
This is my "painless" scripted field in Kibana.
for(item in params._source.massproduction.datas.data)
{
if(item.name=='pressure'){
return item.value;
}
}
return 0;
You can use Float.parseFloat(value) to convert string to float
if(params._source.massproduction!=null && params._source.massproduction.datas!=null &&params._source.massproduction.datas.data.size()>0)
{
def data = params._source.massproduction.datas.data;
if(data instanceof ArrayList)
{
for(item in data)
{
if(item.name=='pressure')
{
return Float.parseFloat(item.value);
}
}
}else
{
if(data.name=='pressure')
{
return Float.parseFloat(data.value);
}
}
}
return 0;

Resources