How to determine ETL health status from time series logs using Elasticsearch? - elasticsearch

Question

TL;DR: what is the Elasticsearch equivalent to this Postgres query?
SELECT latest_pipeline_logs.* FROM (
SELECT pipeline_logs.*,
rank() OVER (
PARTITION BY pipeline_name
ORDER BY updated_at DESC
)
FROM pipeline_logs
) latest_pipeline_logs WHERE RANK = 1
I have hundreds of ETL pipelines with logs that are dumped into Elasticsearch. They are each executed independently at different intervals. I would like to derive a simple health status for each of my ETL pipelines using Elasticsearch aggregations.
Every pipeline logs its state on execution. My current thought process is to determine the health of each pipeline based on the two most important states that occur: succeeded and failed.
I know I can make an aggregation query and group by each pipeline with a sub-aggregation for statuses. For example, something along the lines of this:
{
...
"aggs": {
"pipelines": {
"field": "pipeline_name"
},
"aggs": {
"states": {
"terms": {
"field": "pipeline_state"
}
}
}
}
}
The problem with the above example is I could get several states because of the time series data-set, such as this:
{
"key": "some-pipeline-name",
"buckets": [
{
"key": "succeeded",
"doc_count": 123
},
{
"key": "failed",
"doc_count": 567
}
]
}
I could theoretically filter the results based on the date the pipeline is executed, but because some pipelines run every other month or so, I don't think this is an option.
The end state is to drive a simple dashboard using an Elasticsearch result set that looks something like this:
[
{
"key": "some-pipeline-name",
"latest-status": "succeeded"
},
{
"key": "some-other-pipeline",
"latest-status": "failed"
}
]
One thing to note is in this use case the historical data is not important. The dashboard will simply convey the latest state for each pipeline.
How would you go about achieving this with Elasticsearch?

Answer 1

If you're only interested in the latest status for each pipeline you could use top_hits as a sub-aggregation and then sort on time
{
"size": 0,
"aggs": {
"pipeline": {
"terms": {
"field": "pipeline_name",
"size": 1000
},
"aggs": {
"top_hits_status": {
"top_hits": {
"size": 1,
"sort": [
{
"timestamp": {
"order": "desc"
}
}
],
"_source": {
"includes": [
"pipeline_state"
]
}
}
}
}
}
}
}