How to define a bucket aggregation where buckets are defined by arbitrary filters on a field (GROUP BY CASE equivalent) - elasticsearch

Question

ElasticSearch enables us to filter a set of documents by regex on any given field, and also to group the resulting documents by the terms in a given (same or different field, using "bucket aggregations". For example, on an index that contains a "Url" field and a "UserAgent" field (some kind of web server log), the following will return the top document counts for terms found in the UserAgent field.
{
query: { filtered: { filter: { regexp: { Url : ".*interestingpage.*" } } } },
size: 0,
aggs: { myaggregation: { terms: { field: "UserAgent" } } }
}
What I'd like to do is use the power of the regexp filter (which operates on the whole field, not just terms within a field) to manually define my aggregation buckets, so that I can relatively reliably split my documents/counts/hits by "user agent type" data, rather than the arbitrary terms parsed by elastic search in the field.
Basically, I am looking for the equivalent of a CASE statement in a GROUP BY, in SQL terms. The SQL query that would express my intent would be something like:
SELECT Bucket, Count(*)
FROM (
SELECT CASE
WHEN UserAgent LIKE '%android%' OR UserAgent LIKE '%ipad%' OR UserAgent LIKE '%iphone%' OR UserAgent LIKE '%mobile%' THEN 'Mobile'
WHEN UserAgent LIKE '%msie 7.0%' then 'IE7'
WHEN UserAgent LIKE '%msie 8.0%' then 'IE8'
WHEN UserAgent LIKE '%firefox%' then 'FireFox'
ELSE 'OTHER'
END Bucket
FROM pagedata
WHERE Url LIKE '%interestingpage%'
) Buckets
GROUP BY Bucket
Can this be done in an ElasticSearch query?

Answer 1

This is an interesting use-case.
Here's a more Elasticsearch-way solution.
The idea is to do all this regex matching at indexing time and the search time to be fast (scripts during search time, if there are many documents, are not performing well and will take time). Let me explain:
define a sub-field for your main field, in which the manipulation of terms is customized
this manipulation will be performed so that the only terms that will be kept in the index will be the ones you defined: FireFox, IE8, IE7, Mobile. Each document can have more than one of these fields. Meaning a text like msie 7.0 sucks and ipad rules will generate only two terms: IE7 and Mobile.
All this is made possible by the keep token filter.
there should be another list of token filters that will actually perform the replacement. This will be possible by using the pattern_replace token filter.
because you have two words that should be replaced (msie 7.0 for example), you need a way to capture these two words (msie and 7.0) one beside the other. This will be possible using the shingle token filter.
Let me put all this together and provide the complete solution:
PUT /test
{
"settings": {
"analysis": {
"analyzer": {
"my_pattern_replace_analyzer": {
"tokenizer": "whitespace",
"filter": [
"filter_shingle",
"my_pattern_replace1",
"my_pattern_replace2",
"my_pattern_replace3",
"my_pattern_replace4",
"words_to_be_kept"
]
}
},
"filter": {
"filter_shingle": {
"type": "shingle",
"max_shingle_size": 10,
"min_shingle_size": 2,
"output_unigrams": true
},
"my_pattern_replace1": {
"type": "pattern_replace",
"pattern": "android|ipad|iphone|mobile",
"replacement": "Mobile"
},
"my_pattern_replace2": {
"type": "pattern_replace",
"pattern": "msie 7.0",
"replacement": "IE7"
},
"my_pattern_replace3": {
"type": "pattern_replace",
"pattern": "msie 8.0",
"replacement": "IE8"
},
"my_pattern_replace4": {
"type": "pattern_replace",
"pattern": "firefox",
"replacement": "FireFox"
},
"words_to_be_kept": {
"type": "keep",
"keep_words": [
"FireFox", "IE8", "IE7", "Mobile"
]
}
}
}
},
"mappings": {
"test": {
"properties": {
"UserAgent": {
"type": "string",
"fields": {
"custom": {
"analyzer": "my_pattern_replace_analyzer",
"type": "string"
}
}
}
}
}
}
}
Test data:
POST /test/test/_bulk
{"index":{"_id":1}}
{"UserAgent": "android OS is the best firefox"}
{"index":{"_id":2}}
{"UserAgent": "firefox is my favourite browser"}
{"index":{"_id":3}}
{"UserAgent": "msie 7.0 sucks and ipad rules"}
Query:
GET /test/test/_search?search_type=count
{
"aggs": {
"myaggregation": {
"terms": {
"field": "UserAgent.custom",
"size": 10
}
}
}
}
Results:
"hits": {
"total": 3,
"max_score": 0,
"hits": []
},
"aggregations": {
"myaggregation": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "FireFox",
"doc_count": 2
},
{
"key": "Mobile",
"doc_count": 2
},
{
"key": "IE7",
"doc_count": 1
}
]
}
}

Answer 2

You could use a terms aggregation with a scripted field:
{
query: { filtered: { filter: { regexp: { Url : ".*interestingpage.*" } } } },
size: 0,
aggs: {
myaggregation: {
terms: {
script: "doc['UserAgent'] =~ /.*android.*/ || doc['UserAgent'] =~ /.*ipad.*/ || doc['UserAgent'] =~ /.*iphone.*/ || doc['UserAgent'] =~ /.*mobile.*/ ? 'Mobile' : doc['UserAgent'] =~ /.*msie 7.0.*/ ? 'IE7' : '...you got the idea by now...'"
}
}
}
}
But beware of the performance hit!