Term aggregation on filtered array items - elasticsearch

Question

I want to aggregate on term that are inside an array but I am only interested in some of the array item. I made up a simplified example. Basically I want to aggregate on Type.string if Type.field is valid.
POST so/question
{
"Type": [
[
{
"field": "invalid",
"string": "A"
}
],
[
{
"field": "valid",
"string": "B"
}
]
]
}
GET /so/_search
{
"size": 0,
"aggs": {
"xxx": {
"filter": {
"term": {
"Type.field": "valid"
}
},
"aggs": {
"yyy": {
"terms": {
"field": "Type.string.keyword",
"min_doc_count": 0
}
}
}
}
}
}
The agregation result has 2 keys whereas I only need the "B" key.
"aggregations": {
"xxx": {
"doc_count": 1,
"yyy": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "A",
"doc_count": 1
},
{
"key": "B",
"doc_count": 1
}
]
}
}
}
Is there a way to aggregate on array items which match the filter?
Unfortunately I can't change the data format which would be the obvious solution.

Answer 1

Unless, the documents are of Nested Type, I don't think its possible with simple array types because of the way Elasticsearch Flattens the objects and stores them.
Querying anything on these flattened objects will give you completely unexpected results.
Now I've come up with the below query, making use of Terms Aggregation using Script works perfectly fine for the document you've mentioned in the question
POST so/_search
{
"size": 0,
"aggs": {
"xxx": {
"filter": {
"term": {
"Type.field": "valid"
}
},
"aggs": {
"yyy": {
"terms": {
"script": {
"source": """
int size = doc['Type.string.keyword'].values.length;
for(int i=0; i<size; i++){
String myString = doc['Type.string.keyword'][i];
if(myString.equals("B") && doc['Type.field.keyword'][i].equals("valid")){
return myString;
}
}""",
"lang": "painless"
}
}
}
}
}
}
}
However if you ingest the below document, you see that the aggregation response would be completely different. That is because, array types doesn't store each Type.field value and Type.string value in an ith location in their respective arrays.
POST so/question/2
{
"Type": [
[
{
"field": "valid",
"string": "A"
}
],
[
{
"field": "invalid",
"string": "B"
}
]
]
}
Notice even the below simple Bool query wouldn't work as expected and ends up displaying both the documents.
POST so/_search
{
"query": {
"bool": {
"must": [
{ "match": { "Type.field.keyword": "valid" }},
{ "match": { "Type.string.keyword": "B" }}
]
}
}
}
Hope it helps!