In Elasticsearch, how do I perform nested sub-aggregations? - elasticsearch

Question

In Kibana, I create my index as follows:
PUT cars
{
"mappings":{
"_doc":{
"properties":{
"metadata":{
"type":"nested",
"properties":{
"str_value":{
"type":"keyword"
}
}
}
}
}
}
}
I then insert three records:
POST /cars/_doc/1
{
"metadata": [
{
"key": "model",
"str_value": "Ford"
},
{
"key": "price",
"int_value": 1000
}
]
}
PUT /cars/_doc/2
{
"metadata": [
{
"key": "model",
"str_value": "Ford"
},
{
"key": "price",
"int_value": 2000
}
]
}
PUT /cars/_doc/3
{
"metadata": [
{
"key": "model",
"str_value": "Holden"
},
{
"key": "price",
"int_value": 2500
}
]
}
The schema is a bit unconventional, but I've designed the index this way to avoid mapping explosion:
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
What I'd like to be able to do is to get all my car models, and the sum of prices for those models ie Ford $3000, and Holden $2500. So far I have:
GET /cars/_search
{
"aggs":{
"metadata":{
"nested":{
"path":"metadata"
},
"aggs":{
"model_filter":{
"filter":{
"term":{
"metadata.key":"model"
}
},
"aggs":{
"model_counter":{
"terms":{
"field":"metadata.str_value",
"size":1000
}
}
}
}
}
}
}
}
This gets me part of the way there, because it returns car models and document counts:
"aggregations": {
"metadata": {
"doc_count": 6,
"model_filter": {
"doc_count": 3,
"model_counter": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "Ford",
"doc_count": 2
},
{
"key": "Holden",
"doc_count": 1
}
]
}
}
}
}
How can I modify my query to add a sub-aggregation which will show the sum of prices ie 3000 for Ford (sum of two documents) and 2500 for Holden (sum of one document)

Answer 1

Below query should help you for what you are looking for.
I've simply added on to your solution for that. I've made use of Reverse Nested Aggregation and then applied Sum Aggregation post again using Nested Aggregation.
So your query hierarchy is as below:
Nested Aggregation
- Terms Aggregation
- Reverse Nested Aggregation to back to parent doc
- Nested Aggregation to enter into nested price document
- Sum Aggregation to calculate all the prices
Aggregation Query:
POST <your_index_name>/_search
{
"size":0,
"aggs":{
"metadata":{
"nested":{
"path":"metadata"
},
"aggs":{
"model_filter":{
"filter":{
"term":{
"metadata.key":"model"
}
},
"aggs":{
"model_counter":{
"terms":{
"field":"metadata.str_value",
"size":1000
},
"aggs":{
"reverseNestedAgg":{
"reverse_nested":{},
"aggs":{
"metadata":{
"nested":{
"path":"metadata"
},
"aggs":{
"sum":{
"sum":{
"field":"metadata.int_value"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Note that I've added "size": 0 so as to only return aggregation query. You can modify it according to your requirements.
Aggregation Solution:
{
"took" : 7,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"metadata" : {
"doc_count" : 6,
"model_filter" : {
"doc_count" : 3,
"model_counter" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "Ford",
"doc_count" : 2,
"reverseNestedAgg" : {
"doc_count" : 2,
"metadata" : {
"doc_count" : 4,
"sum" : {
"value" : 3000.0
}
}
}
},
{
"key" : "Holden",
"doc_count" : 1,
"reverseNestedAgg" : {
"doc_count" : 1,
"metadata" : {
"doc_count" : 2,
"sum" : {
"value" : 2500.0
}
}
}
}
]
}
}
}
}
}
Note that I've tested the above query in ES version 7.
Important Note:
If your document ends up in the below format, then the above query wouldn't work.
POST /cars/_doc/1
{
"metadata": [
{
"key": "model",
"str_value": "Ford"
},
{
"key": "price",
"int_value": 1000
},
{
"key": "something else",
"int_value": 1000
}
]
}
// There are three nested documents with two documents having int_value field
I see you mentioned that you'd want to avoid mapping explosion and for that matter your schema is the way it is. However if the above scenario occurs, in that case you may want to take a step back and redesign your model or have your service layer handle this aggregation scenario.
Hope this helps!