Why is my query ignoring my filter aggregation? - elasticsearch

Question

Preface
I have 4 days experience of Elasticsearch 1.7.2.
Setup
I have a collection of documents, each document is a User. The User has a number of Answers which is linked through UserAnswers. Which gives a document reference of user_answers.answer[]. Where the answers array is an array of objects.
The user_answers.answer[].correct is a boolean field which tells me if the answer given by the user is correct or not.
Objective
I would like to list the users and also display the total number of correct and incorrect answers they have.
Approach
So far I have tried a number of different approaches and the one I'll include here is as close as I've got so far in 1.5 days of trying.
Use a terms aggregation to create a bucket for each User by username.
Filter each bucket to leave only correct or incorrect answers.
Count the number of filtered answers.
Query
{
"size": 0,
"filter": {
"bool": {
"must_not": {
// Remove users who already have this award
"term": {"awards_users.award_id": 2}
}
}
},
"aggs": {
"users": {
"terms": {"field": "username"},
"aggs": {
"correct": {
"filter": {
"term": {"user_answers.answer.correct": true}
},
"aggs": {
"count": {
"value_count": {
"field": "user_answers.answer.id"
}
}
}
},
// Same for incorrect, but inverted correct value
}
}
}
}
Sample response
{
"key": "neon1024",
"doc_count": 1,
"correct": {
"doc_count": 1,
"count": {
"value": 7 // Expected 1 correct & 6 incorrect
}
}
},
This is the record which I am testing against, and I am expecting that 1 is returned instead of 7. There are 7 answers in total, 6 incorrect and 1 correct. This I have verified in my document index.
The problem
For some reason the actual filter seems to be being ignored, and leaving all possible related answers in the bucket. Hence the aggregation is seeing them all, rather than showing the expected value.
Question
How can I use an aggregation to segregate my counts based on the value of the related answers values?
Thanks for reading my long question!

Answer 1

As suggested, you probably have your answers mapped as object, while you should be using nested type.
Using nested type, elasticsearch will store your answers as individual documents linked to the root one and will let you do expected aggregations on them. You'll have to use nested type aggregation in your query to achieve that.
So I'd say it would be best to map your document like this:
PUT /test
{
"mappings" : {
"your_type" : {
"properties" : {
"username" : {
"type" : "string",
"index" : "not_analyzed"
},
"user_answers" : {
"type" : "nested",
"properties" : {
"id" : {
"type" : "integer"
},
"answer" : {
"type" : "string"
},
"correct" : {
"type" : "boolean"
}
}
}
}
}
}
}
Test document:
PUT /test/your_type/1
{
"username": "neon1024",
"user_answers": [
{
"id": 1,
"answer": "answer1",
"correct": true
},
{
"id": 2,
"answer": "answer2",
"correct": true
},
{
"id": 3,
"answer": "answer3",
"correct": false
}
]
}
Query:
POST /test/_search?search_type=count
{
"aggs": {
"users": {
"terms": {
"field": "username"
},
"aggs": {
"DiveIn": {
"nested": {
"path": "user_answers"
},
"aggs": {
"CorrectVsIncorrect": {
"terms": {
"field": "user_answers.correct",
"size": 2
}
}
}
}
}
}
}
}
And Final result:
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0,
"hits": []
},
"aggregations": {
"users": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "neon1024",
"doc_count": 1,
"DiveIn": {
"doc_count": 3,
"CorrectVsIncorrect": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "T",
"doc_count": 2
},
{
"key": "F",
"doc_count": 1
}
]
}
}
}
]
}
}
}
Where "key": "T" represents correct answers and "doc_count": 2 represents amount of them.