Nested query in Elasticsearch? - elasticsearch

Question

My team owns several dashboard and are considering the possibility to move to Elasticsearch in order to consolidate the software stacks. One type of common charts we expose is like "What's the pending workflow by the end of each day?". Here are some example data:
day workflow_id version status
20151101 1 1 In Progress
20151101 2 1 In Progress
20151102 1 2 In Progress
20151102 3 1 In Progress
20151102 4 1 In Progress
20151102 2 2 Completed
20151103 1 3 Completed
20151103 3 2 In Progress
20151104 3 3 Completed
20151105 4 2 Completed
Every time when something changed in the workflow, a new record is inserted, which might or might not change the status. The record with the max(version) is the most recent data for the workflow_id.
The goal is to have a chart to show what's the total number of 'In Progress' and 'Completed' workflows at the end of each day. This should only consider the record that has the largest version number until the day. This can be done in SQL with nested queries:
with
snapshot_dates as
(select distinct day from workflow),
snapshot as
(select d.day, w.workflow_id, max(w.version) as max_version
from snapshot_dates d, workflow w
where d.day >= w.day
group by d.day, w.workflow_id
order by d.day, w.workflow_id)
select s.day, w.status, count(1)
from workflow w join snapshot s on w.workflow_id=s.workflow_id and w.version = s.max_version
group by s.day, w.status
order by s.day, w.status;
Here is expected output from the query:
day,status,count
20151101,In Progress,2
20151102,Completed,1
20151102,In Progress,3
20151103,Completed,2
20151103,In Progress,2
20151104,Completed,3
20151104,In Progress,1
20151105,Completed,4
I am still new to Elasticsearch and wonder if Elasticsearch can do a similar query without using application side logic by properly define the mapping and query. More generally, what is the best practice to solve such problem using Elasticsearch?

Answer 1

I tried to find the solution using bucket selector aggregation, but I was stuck at one point. I discussed the same in elasticsearch forum. Following is what suggested by Christian Dahlqvist.
In addition to this you also index the record into a workflow-centric
index with a unique identifier, e.g. workflow id, as the document id.
If several updates come in for the same workflow, each will result in
an update and the latest state will be preserved. Running aggregations
across this index to find the current or latest state will be
considerably more efficient and scalable as you only have a single
record per workflow and do not need to filter out documents based on
relationships to other documents.
So as per this suggestion, you should use Workflow Id as document id while indexing. And whenever there is an update for that workflow, you update new version and date using workflow id. Let's say the index name is workflow and index type is workflow_status. So mapping of this workflow_status type will be as follow:
{
"workflow_status": {
"properties": {
"date": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"status": {
"type": "string",
"index": "not_analyzed"
},
"version": {
"type": "long"
},
"workFlowId": {
"type": "long"
}
}
}
}
Keep adding/updating the documents to this index type keeping workFlowId as document id.
Now for showing a chart day wise, you may need to create another index type, let's say per_day_workflow with following mapping:
{
"per_day_workflow": {
"properties": {
"date": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"in_progress": {
"type": "long"
},
"completed": {
"type": "long"
}
}
}
}
This index will hold data for each day. So you need to create a job which will run at the end of day and fetch total "In Progress" & "Completed" workflow from workflow_status index type using following aggregation search:
POST http://localhost:9200/workflow/workflow_status/_search?search_type=count
{
"aggs": {
"per_status": {
"terms": {
"field": "status"
}
}
}
}
Response will look as follow (I ran for date 2015-11-02 on your sample data):
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 4,
"max_score": 0,
"hits": []
},
"aggregations": {
"per_status": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "In Progress",
"doc_count": 3
},
{
"key": "Completed",
"doc_count": 1
}
]
}
}
}
From this response you need to extract In Progress and Completed count and add them to per_day_workflow index type with today's' date.
Now when you need per day data for your graph, then you can fetch easily from this per_day_workflow index type.