Visualization in Elasticsearch using customized query - elasticsearch

Question

Here's the situation I have, suppose my index document looks like this :
{
"user" : 1
"started" : "2021-06-05"
"finished" : -1
"status": "ONGOING"
}
{
"user" : 2
"started" : "2021-06-05"
"finished" : "2021-06-06"
"status": "DONE"
}
Like this I have 100 docs indexed. The ongoing documents have -1 as the finished time and completed once have a valid timestamp. I want to visualize a graph that can give me the number of ongoing applications with the "started" field in the X-axis.
In the date histogram, I'm only able to get the filtered ongoing processes for that specific interval. But I want the count for the ongoing application to be counted for every interval until the document is updated with the finish time.
Is there anyway I can visualize this in Kibana? Even an elastic search query that can give me this output will do.

Answer 1

This is really similar to a problem I had and have now solved. I spent ages trying to create a query that does this to no avail, but luckily this can be achieved using Vega's transforms.
If you want to bin it evenly, not using start time as your x-values Here is the posted solution (look for my answer). The one thing I would add is; for the documents where you have "-1" as the finished time, if you do a formula transform you can round these to the end bin times.
However, if you still want to stick to the "started"/"finished" field being the points of summation/evaluation this is also possible. I'll give you a quick rundown on how to do this...
Method:
First thing you need to do is create two copies of your data with a common field referring to the "timestamp". The first dataset will have the "started" value assigned to the field "timestamp" (started dataset) and the second will have "finished" (finished dateset). You can achieve this using the formula transform.
You will then need to create a column in each dataset named "operation" referring to what that that data entry does - add a user or remove a user. For the finished dataset you want to assign a column of '-1's and the started dataset '1's. Again using formula transforms.
Then join these datasets back up. You now need to order by "timestamp" and cumulatively sum the "operation" column up. This can be achieved using the window transform.
This should give you the data needed to plot it. Arguably this is much more accurate than binning, but if your data set is large it can yield quite messy results - binning in this case is much cleaner.
Good luck, there is obviously a lot to fill in but a working example would of taken me quite a while to draw up - plus where is the fun in copying.