Let's say I have a log of events of ad displays and ad clicks stored via Logstash in Elasticsearch and displayed in Kibana 4. I would like to calculate a simple metric like CTR (Click-Through-Rate) of the events , which is :
CTR = #clicks/#displays.
First of all, does anyone know if it's possible to do in Elasticsearch + Kibana? I don't see a possibility to do it in Kibana.
I was thinking about doing it in raw Elasticsearch by scripted aggregation. But I don't know how to define such in Kibana.
Any ideas on how to do it would be very welcome! Comments explaining that it's impossible to do it would be also valuable.
Kibana 4 includes support for Elasticsearch scripting. You can go to Settings > Indices (pick your pattern) > Scripted Fields and add a new scripted field that computes your CTR. Take a look at "Scripted Fields" at elastic blog for more info.
I had to do something similar for a customer, but couldn't manage to get it done using scripted fields, because I had to sum UP all the values on each aggregation. I ended up developing a custom visualization (Plugin). This was for Kibana 4.1 and 4.5.1.
Let me know if this is what you are looking for, or if you want to give it a go yourself I can give you some pointers.
Related
I created an index from a Storm topology to ElasticSearch (ES). The index map is basically:
index: btc-block
miner: text
reward: double
datetime: date
From those documents I would like to create a histogram of the richest miner, on a daily scale.
I am wondering if I should aggregate first in storm and just use ES and Kibana to store, query and then display the data or if ES and Kibana can handle such requests.
I have been looking at the Transforms, in the index management section, that allows to create new indices from queries and aggregations in continuous modes but I can't succeed to get to the expected result.
Any help will be appreciated.
Sometimes we need to ask a question to find the answer...
I kept looking at the documentation and eventually I could solve the issue by using a sibling pipeline aggregation, in the visualization. In my case, a max bucket aggregation of the sum of reward on Y-axis.
In that case get like 6 records/hour so I guess it's ok to let Kibana and ES work. What if I got lot more data? Would it not be wiser to aggregate in Storm?
In Kibana (ElasticSearch v6.8) I'm storing documents containing a date field and a LaunchTime field, and I have a scripted field uptime as their difference (in seconds):
(doc['date'].value.millis - doc['LaunchTime'].value.millis) / 1000 / 60
I'm trying to create a monitor (under alerting) on the max value of this field of the index, but the field 'Uptime' doesn't show up in the list of fields I can do a max query on. Its type is number and in visualisations I can do max/min etc. displays of this field.
Is this a limitation of Kibana alerting - that I can't use a scripted field? Or is there some way I can make it available to use?
I'm afraid it is a limitation of kibana's scripted fields. See this post about the same subject referring to the scripted field official documentation. I believe that the watcher are handled by ES itself while the scripted field are handled by kibana (they can be used in discovery and visualisations because kibana is handlind those too)
But have no fear! you already have the script for the calculation and you could just add it into logstash to add the field to you actual documents when you index them, which would enable you to use it for watchers AND would probably optimize the load at runtime, since the val is only calculated one, when you ingest it. Then you could run an update by query with a the script and add the field in you existing documents.
If you don't use logstash, you could look into ES's ingestion pipelines, but it's a rather advanced subject and i'm not sure if it was implemented in 5.x.
I am using Kibana 6.2.2 to visualize DNS traffic, and I want to filter out data with more than 4 answers, like this. However, I can't seem to filter the count of it. I tried using Terms count on the visualization tab, but I can't figure out how to group the answer section per query.
I am trying to visualize time series data stored in elastic search using grafana.
I have the legend setup to show 2 decimal places but it does not reflect in the UI.
The decimal places show up for other dashboard panels with a tsdb datasource. So this issue is specific to using grafana with elasticsearch. Is there any other configuration setup I am missing here which will help me achieve this?
Just found out that elastic search does not allow displaying values without some sort of aggregation and in my case aggregation is resulting in values getting rounded.
There was a related request which seemed to not get much traction in kibana.
https://github.com/elastic/kibana/issues/3572
In short not feasible as of [2.x] elastic search.
How do i plot in kibana the data that is returned from an aggregate query in elastic search.
I have tried the existing methods but could not achieve this.
Kibana does not currently have a way to display output of an aggregate request according to the bug tracker on github. It looks like maybe it's coming in 4.0.