how to sort and limit aggregations in ElasticSearch - elasticsearch

Question

For example I have the following records with the columns as:(Country,City,Date,Income)
USA SF 2015-08 50
USA SF 2015-05 30
USA SF 2015-01 20
USA NY 2015-05 70
USA NY 2015-02 10
U.K LD 2015-05 90
My sql as: select country,city,max(date) as maxDate,sum(income) as sumIncome from testTable group by country,city order by maxDate desc,sumIncome desc limit 3.
So the result should be:
USA SF 2015-08 100
U.K LD 2015-05 90
USA NY 2015-05 80
I wrote the ES aggregates as following, but it's wrong:
"aggs":{"sub1": {"terms":{"field":"contry"},
"aggs":{"sub2":{"terms":{"field":"city",
"order":[{"submax":"DESC"},{"subsum":"DESC"}]},
"aggs":{"submax":{"max":{"field":"date"}},"subsum":{"sum":{"field":"income"}}}}}}}
By my above script, it got the wrong result as following:
USA SF 2015-08 100
USA NY 2015-05 80
U.K LD 2015-05 90

Answer 1

You, actually, have two options, now that I understood the requirement.
Option 1
Use a script to "concatenate" country field and city field. Using the regular aggregations per field to do what you want is not possible in Elasticsearch.
Instead you need to do something like this:
GET /test/test/_search?search_type=count
{
"aggs": {
"sub1": {
"terms": {
"script": "doc['country'].value + ' ' + doc['city'].value",
"size": 3,
"order": [
{
"submax": "DESC"
},
{
"subsum": "DESC"
}
]
},
"aggs": {
"submax": {
"max": {
"field": "date"
}
},
"subsum": {
"sum": {
"field": "income"
}
}
}
}
}
}
With curl:
curl -XPOST "http://localhost:9200/livebox/type1/_search?search_type=count" -d'
{
"aggs": {
"sub1": {
"terms": {
"script": "doc[\"boxname\"].value + \" \" + doc[\"app\"].value",
"size": 3,
"order": [
{
"submax": "DESC"
},
{
"subsum": "DESC"
}
]
},
"aggs": {
"submax": {
"max": {
"field": "date"
}
},
"subsum": {
"sum": {
"field": "count"
}
}
}
}
}
}'
And the result of the aggregation will generate terms that are of the following form: country + + city.
"buckets": [
{
"key": "usa sf",
"doc_count": 3,
"subsum": {
"value": 100
},
"submax": {
"value": 1438387200000,
"value_as_string": "2015-08"
}
},
{
"key": "uk ld",
"doc_count": 1,
"subsum": {
"value": 90
},
"submax": {
"value": 1430438400000,
"value_as_string": "2015-05"
}
},
{
"key": "usa ny",
"doc_count": 2,
"subsum": {
"value": 80
},
"submax": {
"value": 1430438400000,
"value_as_string": "2015-05"
}
}
]
Option 2
Use _source transformation that will build a new field at indexing time, which will "move" the performance impact of running a script at aggregation time.
The mapping of the index, as it needs some changes, whatever you have now:
PUT /test
{
"mappings": {
"test": {
"transform": {
"script": "ctx._source['country_and_city'] = ctx._source['country'] + ' ' + ctx._source['city']"
},
"properties": {
"country": {
"type": "string"
},
"city": {
"type": "string"
},
"income": {
"type": "integer"
},
"date": {
"type": "date",
"format": "yyyy-MM"
},
"country_and_city": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
The query:
GET /test/test/_search?search_type=count
{
"aggs": {
"sub1": {
"terms": {
"field": "country_and_city",
"order": [
{
"submax": "DESC"
},
{
"subsum": "DESC"
}
]
},
"aggs": {
"submax": {
"max": {
"field": "date"
}
},
"subsum": {
"sum": {
"field": "income"
}
}
}
}
}
}
And the result:
"buckets": [
{
"key": "usa sf",
"doc_count": 3,
"subsum": {
"value": 100
},
"submax": {
"value": 1438387200000,
"value_as_string": "2015-08"
}
},
{
"key": "uk ld",
"doc_count": 1,
"subsum": {
"value": 90
},
"submax": {
"value": 1430438400000,
"value_as_string": "2015-05"
}
},
{
"key": "usa ny",
"doc_count": 2,
"subsum": {
"value": 80
},
"submax": {
"value": 1430438400000,
"value_as_string": "2015-05"
}
}
]