Understanding Elasticsearch aggregations - elasticsearch

Question

My scenario is the following:
I have people, who can have regular or one-time income. I would like to sum the regular income of every people, who are not deleted and was born within a date range. The query part just works well, but when I start to put together the aggregation part of the Elastic query, I got the wrong figures and can't understand, what do I do wrong.
This is how I've created the mapping for my data type:
curl -X PUT -i http://localhost:9200/people --data '{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"person" : {
"properties" : {
"birthDate" : {
"type" : "date",
"format" : "strict_date_optional_time||epoch_millis"
},
"company" : {
"type" : "string"
},
"deleted" : {
"type" : "boolean"
},
"income" : {
"type": "nested",
"properties" : {
"income_type" : {
"type" : "string"
},
"value" : {
"type" : "double"
}
}
},
"name" : {
"type" : "string"
}
}
}
}
}
}'
This is the data:
curl -X PUT -H 'Content-Type: application/json' -i http://localhost:9200
/people/person/1 --data '{
"deleted":false,
"birthDate":"1980-10-10",
"name":"John Smith",
"company": "IBM",
"income": [{"income_type":"regular","value":55.5}]
}'
curl -X PUT -H 'Content-Type: application/json' -i http://localhost:9200/people/person/2 --data '{
"deleted":true,
"birthDate":"1960-10-10",
"name":"Mary Legend",
"company": "ASUS",
"income": [{"income_type":"one-time","value":10},{"income_type":"regular","value":55}]
}'
curl -X PUT -H 'Content-Type: application/json' -i http://localhost:9200/people/person/3 --data '{
"deleted":false,
"birthDate":"2000-10-10",
"name":"F. King Elastic",
"income": [{"income_type":"one-time","value":1},{"income_type":"regular","value":5}]
}'
curl -X PUT -H 'Content-Type: application/json' -i http://localhost:9200/people/person/4 --data '{
"deleted":false,
"birthDate":"1989-10-10",
"name":"Prison Lesley",
"income": [{"income_type":"regular","value":120.7},{"income_type":"one-time","value":99.3}]
}'
curl -X PUT -H 'Content-Type: application/json' -i http://localhost:9200/people/person/5 --data '{
"deleted":false,
"birthDate":"1983-10-10",
"name":"Prison Lesley JR.",
"income": [{"income_type":"one-time","value":99.3}]
}'
curl -X PUT -H 'Content-Type: application/json' -i http://localhost:9200/people/person/6 --data '{
"deleted":true,
"birthDate":"1986-10-10",
"name":"Hono Lulu",
"income": [{"income_type":"regular","value":11.3}]
}'
This is a query, which filters for undeleted people, who have at least one regular income, and was born between the given dates. The below query still works as expected (two persons were fulfilling the criteria):
curl -X POST -H 'Content-Type: application/json' -i 'http://localhost:9200/people/person/_search?pretty=true' --data '{
"size": 100,
"filter": {
"bool": {
"must": [
{
"match": {
"deleted": false
}
},
{
"range": {
"birthDate": {
"gte": "1980-01-01",
"lte": "1990-12-31"
}
}
},
{
"nested": {
"path": "income",
"query": {
"bool": {
"must": [
{
"match": {
"income.income_type": "regular"
}
}
]
}
}
}
}
]
}
}
}'
But when I add the aggregation section, everything goes wrong, and I do not understand, why :(
curl -X POST -H 'Content-Type: application/json' -i 'http://localhost:9200/people/person/_search?pretty=true' --data '{
"size": 100,
"filter": {
"bool": {
"must": [
{
"match": {
"deleted": false
}
},
{
"range": {
"birthDate": {
"gte": "1980-01-01",
"lte": "1990-12-31"
}
}
},
{
"nested": {
"path": "income",
"query": {
"bool": {
"must": [
{
"match": {
"income.income_type": "regular"
}
}
]
}
}
}
}
]
}
},
"aggs": {
"incomes": {
"nested": {
"path": "income"
},
"aggs": {
"income_type": {
"filter": {
"bool": {
"must": [
{
"match": {
"income.income_type": "regular"
}
},
{
"match": {
"deleted": false
}
}
]
}
},
"aggs": {
"totalIncome": {
"sum": {
"field": "income.value"
}
}
}
}
}
}
}
}'
The result is this:
...
"aggregations": {
"incomes": {
"doc_count": 9,
"income_type": {
"doc_count": 0,
"totalIncome": {
"value": 0.0
}
}
}
}
}
I was expecting the doc_count to be 2, and the totalIncome should be 176.2 (120.7 + 55.5)
Does anyone have an idea, what do I do wrong?

Answer 1

Great start! You don't need the filter on the deleted field in your aggregation since your query is already filtering out all deleted documents. Try this:
"aggs": {
"incomes": {
"nested": {
"path": "income"
},
"aggs": {
"income_type": {
"filter": {
"match": {
"income.income_type": "regular"
}
},
"aggs": {
"totalIncome": {
"sum": {
"field": "income.value"
}
}
}
}
}
}
}