Elasticsearch: find documents with distinct values and then aggregate over them - elasticsearch

Question

My index has a log-like structure: I insert a version of a document whenever an event occurs. For example, here are documents in the index:
{ "key": "a", subkey: 0 }
{ "key": "a", subkey: 0 }
{ "key": "a", subkey: 1 }
{ "key": "a", subkey: 1 }
{ "key": "b", subkey: 0 }
{ "key": "b", subkey: 0 }
{ "key": "b", subkey: 1 }
{ "key": "b", subkey: 1 }
I'm trying to construct a query in ElasticSearch which is basically equivalent to the following SQL query:
SELECT COUNT(*), key, subkey
FROM (SELECT DISTINCT key, subkey FROM t)
The answer to this query would obviously be
(1, a, 0)
(1, a, 1)
(1, b, 0)
(1, b, 1)
How would I replicate this query in Elasticsearch? I came up with the following:
GET test_index/test_type/_search?search_type=count
{
"aggregations": {
"count_aggr": {
"terms": {
"field": "concatenated_key"
},
"aggs": {
"sample_doc": {
"top_hits": {
"size": 1
}
}
}
}
}
}
concatenated_key is a concatenation of key and subkey. This query would create a bucket for each (key, subkey) combination and return a sample document from each bucket. However, I don't know how can I aggregate over the fields of _source.
Would appreciate any ideas. Thanks!

Answer 1

If you don't have the possibility to re-index the documents and to add your own concatenated key field, this is a way of doing it:
GET /my_index/my_type/_search?search_type=count
{
"aggs": {
"key_agg": {
"terms": {
"field": "key",
"size": 10
},
"aggs": {
"sub_key_agg": {
"terms": {
"field": "subkey",
"size": 10
}
}
}
}
}
}
It will give you something like this:
"buckets": [
{
"key": "a",
"doc_count": 4,
"sub_key_agg": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": 0,
"doc_count": 2
},
{
"key": 1,
"doc_count": 2
}
]
}
},
{
"key": "b",
"doc_count": 4,
"sub_key_agg": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": 0,
"doc_count": 2
},
{
"key": 1,
"doc_count": 2
}
]
}
}
]
where you have the key - "key": "a" - and then each combination with this key and the number of docs that match key=a and subkey=0 or key=a and subkey=1:
"buckets": [
{
"key": 0,
"doc_count": 2
},
{
"key": 1,
"doc_count": 2
}
]
Same goes for the other key.