Elasticsearch group, filter and aggregation in the same request - elasticsearch

Question

I have a use case to filter and group by two separate fields in Elasticsearch, while aggregating values in rest of the fields. Looking for help formulating a query for Elasticsearch. Here is an example. Each document has a locationId, category, productId, visitCount, purchaseCount and sampleCount. I want to view sums of visitCount, purchaseCount and sampleCount for each category within each location. Note that productId is unique across all entries. I have tried reading up Elasticsearch documentation but could not find a good source to learn how I can do grouping, filtering and aggregation all together. Please note that this is for a website use case where we show this data in a table with pages. Due to the amount of locations and categories, it is likely that there will be several groups that will go beyond 1 page. Please help formulate a query for Elasticsearch.
Sample documents:
[{
"locationId": 12345,
"category": "Food",
"productId": "JKHNG98",
"visitCount": 10,
"purchaseCount": 9,
"sampleCount": 7
}, {
"locationId": 12345,
"category": "Food",
"productId": "HJUSY68",
"visitCount": 1,
"purchaseCount": 15,
"sampleCount": 7
}, {
"locationId": 12345,
"category": "Entertainment",
"productId": "KGUJKHG78",
"visitCount": 20,
"purchaseCount": 15,
"sampleCount": 10
}, {
"locationId": 12345,
"category": "Entertainment",
"productId": "67912HYK",
"visitCount": 5,
"purchaseCount": 15,
"sampleCount": 10
}, {
"locationId": 54321,
"category": "Food",
"productId": "9823HYKN",
"visitCount": 15,
"purchaseCount": 12,
"sampleCount": 5
}, {
"locationId": 54321,
"category": "Food",
"productId": "KJHKJSAHD22",
"visitCount": 55,
"purchaseCount": 12,
"sampleCount": 5
}, {
"locationId": 54321,
"category": "Entertainment",
"productId": "SDJFHSF788",
"visitCount": 45,
"purchaseCount": 44,
"sampleCount": 23
}, {
"locationId": 54321,
"category": "Entertainment",
"productId": "2131286JH",
"visitCount": 80,
"purchaseCount": 44,
"sampleCount": 23
}]
Input can be multiple location IDs but always just 1 category.
Expected result for filter input category "Food":
[{
"locationId": 12345,
"category": "Food",
"sumOfVisitCount": 11,
"sumOfPurchaseCount": 24,
"sumOfSampleCount": 14
},{
"locationId": 54321,
"category": "Food",
"sumOfVisitCount": 15,
"sumOfPurchaseCount": 12,
"sumOfSampleCount": 5
}]
Expected result for filter input of location "12345":
[{
"locationId": 12345,
"category": "Food",
"sumOfVisitCount": 11,
"sumOfPurchaseCount": 24,
"sumOfSampleCount": 14
}, {
"locationId": 12345,
"category": "Entertainment"
"sumOfVisitCount": 25,
"sumOfPurchaseCount": 30,
"sumOfSampleCount": 20
}]
Expected result for filter input of location "12345" and category "Food":
[{
"locationId": 12345,
"category": "Food",
"sumOfVisitCount": 11,
"sumOfPurchaseCount": 24,
"sumOfSampleCount": 14
}]

Answer 1

You need to use a combination of query and aggregation to achieve your required result. Below query uses :
Search Query for filter input category "Food"
Query part uses term query to match the documents that have category equal to Food
Terms aggregation is used, to create buckets based on locationId, and in order to find the sum of sampleCount, viewCount and purchaseCount, sum aggregation is used
POST idxtest/_search
{
"size": 0,
"query": {
"term": {
"category.keyword": "Food"
}
},
"aggs": {
"NAME": {
"terms": {
"field": "locationId"
},
"aggs": {
"sumOfPurchaseCount": {
"sum": {
"field": "purchaseCount"
}
},
"sumOfVisitCount": {
"sum": {
"field": "visitCount"
}
},
"sumOfSampleCount": {
"sum": {
"field": "sampleCount"
}
}
}
}
}
}
Search Result:
"aggregations" : {
"NAME" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : 12345,
"doc_count" : 2,
"sumOfSampleCount" : {
"value" : 14.0
},
"sumOfVisitCount" : {
"value" : 11.0
},
"sumOfPurchaseCount" : {
"value" : 24.0
}
},
{
"key" : 54321,
"doc_count" : 2,
"sumOfSampleCount" : {
"value" : 10.0
},
"sumOfVisitCount" : {
"value" : 70.0
},
"sumOfPurchaseCount" : {
"value" : 24.0
}
}
]
}
Search Query for filter input of location "12345" and category "Food"
POST idxtest/_search
{
"size": 0,
"query": {
"bool": {
"must": [
{
"term": {
"category.keyword": "Food"
}
},
{
"term": {
"locationId": 12345
}
}
]
}
},
"aggs": {
"NAME": {
"terms": {
"field": "locationId"
},
"aggs": {
"sumOfPurchaseCount": {
"sum": {
"field": "purchaseCount"
}
},
"sumOfVisitCount": {
"sum": {
"field": "visitCount"
}
},
"sumOfSampleCount": {
"sum": {
"field": "sampleCount"
}
}
}
}
}
}
Search Result :
"aggregations" : {
"NAME" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : 12345,
"doc_count" : 2,
"sumOfSampleCount" : {
"value" : 14.0
},
"sumOfVisitCount" : {
"value" : 11.0
},
"sumOfPurchaseCount" : {
"value" : 24.0
}
}
]
}
}