Calculate the change percent of a sliding time window in Elasticsearch - elasticsearch

Question

Elasticsearch newbie here. I have a series of log messages like these
{
"#timestamp": "whatever",
"type": "toBeMonitored",
"success": true
}
I was tasked to react on a change of -30% of the total amount of successful messages compared to yesterday's same interval. So if I do the check at 8 AM today, I should compare today's total count from midnight to 8 AM to yesterday's same interval.
I tried creating a date histogram aggregation but I would like to have the diff percentage as a query result and not do the math on the development side.
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"term": {
"type": "toBeMonitored"
}
},
{
"term": {
"status": true
}
},
{
"range": {
"#timestamp": {
"gte": "now-1d/d",
"lte": "now/h"
}
}
}
]
}
},
"aggs": {
"histo": {
"date_histogram": {
"field": "#timestamp",
"fixed_interval": "1h"
}
}
}
}
Any idea on how this might be accomplished?

Answer 1

You can leverage the derivative pipeline aggregation to achieve exactly what you expect:
POST /sales/_search
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"term": {
"type": "toBeMonitored"
}
},
{
"term": {
"status": true
}
},
{
"range": {
"#timestamp": {
"gte": "now-1d/d",
"lte": "now/h"
}
}
}
]
}
},
"aggs": {
"histo": {
"date_histogram": {
"field": "#timestamp",
"fixed_interval": "1h"
},
"aggs": {
"successDiff": {
"derivative": {
"buckets_path": "_count"
}
}
}
}
}
}
In each bucket you're going to get the difference between the document count in the previous bucket vs the current bucket.

Answer 2

Ended up dropping the date_histogram aggregation and using the date_range one. It's much easier to work with, even though it does not return the difference compared to yesterday's same time period. I did that in code.
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"term": {
"type": "toBeMonitored"
}
},
{
"term": {
"status": true
}
},
{
"range": {
"#timestamp": {
"gte": "now-1d/d",
"lte": "now/h"
}
}
}
]
}
},
"aggs": {
"ranged_documents": {
"date_range": {
"field": "#timestamp",
"ranges": [
{
"key": "yesterday",
"from": "now-1d/d",
"to": "now-24h/h"
},
{
"key": "today",
"from": "now/d",
"to": "now/h"
}
],
"keyed": true
}
}
}
}
This query would yield a result similar to the one below
{
"_shards": {
"total": 42,
"failed": 0,
"successful": 42,
"skipped": 0
},
"hits": {
"hits": [],
"total": {
"value": 10000,
"relation": "gte"
},
"max_score": null
},
"took": 134,
"timed_out": false,
"aggregations": {
"ranged_documents": {
"buckets": {
"yesterday": {
"from_as_string": "2020-10-12T00:00:00.000Z",
"doc_count": 268300,
"to_as_string": "2020-10-12T12:00:00.000Z",
"from": 1602460800000,
"to": 1602504000000
},
"today": {
"from_as_string": "2020-10-13T00:00:00.000Z",
"doc_count": 251768,
"to_as_string": "2020-10-13T12:00:00.000Z",
"from": 1602547200000,
"to": 1602590400000
}
}
}
}
}