In Kibana (ElasticSearch v6.8) I'm storing documents containing a date field and a LaunchTime field, and I have a scripted field uptime as their difference (in seconds):
(doc['date'].value.millis - doc['LaunchTime'].value.millis) / 1000 / 60
I'm trying to create a monitor (under alerting) on the max value of this field of the index, but the field 'Uptime' doesn't show up in the list of fields I can do a max query on. Its type is number and in visualisations I can do max/min etc. displays of this field.
Is this a limitation of Kibana alerting - that I can't use a scripted field? Or is there some way I can make it available to use?
I'm afraid it is a limitation of kibana's scripted fields. See this post about the same subject referring to the scripted field official documentation. I believe that the watcher are handled by ES itself while the scripted field are handled by kibana (they can be used in discovery and visualisations because kibana is handlind those too)
But have no fear! you already have the script for the calculation and you could just add it into logstash to add the field to you actual documents when you index them, which would enable you to use it for watchers AND would probably optimize the load at runtime, since the val is only calculated one, when you ingest it. Then you could run an update by query with a the script and add the field in you existing documents.
If you don't use logstash, you could look into ES's ingestion pipelines, but it's a rather advanced subject and i'm not sure if it was implemented in 5.x.
Related
I am using Elastic Cloud v 7.5.2. I am trying to transform the index, where i want the term count to be aggregated. In Kibana UI, Define Pivot does not have provision to take terms aggregation. How to achieve it? Is the version didn't support or we can achieve the same using Transform API?
we have a field eventType which will have values like task-started, task-completed, task-inprogress. Each document will have an jobId and each job can have multiple tasks. I need to transform the index to a new index in such a way where task-started, task-completed and task-inprogress will be separate field and it will have value count aggregated to it.
Our ultimate goal, in Kibana we need to show additional columns which will have percentage and ratio of these task fields.
Kibana's UI allows the user to create a scripted field which is stored as part of the index (screenshot below). How can that be done programatically? In particular, using either the NEST client or the Elasticsearch low level client.
Kibana UI for the Indice with the Scripted Fields tab highlighted
Note that I am not asking how to create add an expression/script field as part of a query, I'm specifically looking for how to add it as part of the Index when the mapping is created so that queries can reference it without having to explicitly include it.
Kibana dashboards are stored in the .kibana index. To export dashboards, you can query the Kibana index as you would any other index. For example, curl -XGET http://localhost:9200/.kibana/_search?type=dashboard&pretty would show the JSON for your dashboards. You could export the template, add the scripted field to the JSON, and then POST it again. Since Kibana uses a standard Elasticsearch index, the normal Elasticsearch API would apply to modifying Kibana dashboards. This may provide a little more clarification.
At the time of writing, current version 5.2 does not have an official way to do this.
This is how I do it:
Get index fields: GET /.kibana/index-pattern/YOUR_INDEX
Add your scripted field to _source.fields (as string, notice scaped quotation marks)
"fields":"[...,{\"name\":\"test\",\"type\":\"number\",\"count\":0,\"scripted\":true,\"script\":\"doc['area_id'].value\",\"lang\":\"painless\",\"indexed\":false,\"analyzed\":false,\"doc_values\":false,\"searchable\":true,\"aggregatable\":true}]"
Post back _source json to /.kibana/index-pattern/YOUR_INDEX
{
"title":"YOUR_INDEX",
"timeFieldName":"time",
"fields":"[...,{\"name\":\"test\",...}]"
}
I have imported weblogs into Elasticsearch via Logstash. This has completed successfully.
I have a field in the log file (clientip) that is always populated and another field that is sometimes populated (trueclientip). I want to aggregate based on the coalescing of the two; e.g. if trueclientip is not empty then use that otherwise use clientip.
How can I do this with the Visualisation in Kibana? Do I need to generate a scripted field or is there another approach?
Thanks.
Define a scripted field that should have this formula: doc['trueclientip'].value ? doc['trueclientip'].value : doc['clientip'].value and use this in your aggregations.
But, there is a downside to this scripted fields functionality AND the ip type: it seems what you get back from the script is the number itself (which is logic because the scripted fields in Kibana 4 only use Lucene expressions as a language), not the string representation. IPs internally are actually long numbers in Lucene.
For example, 127.0.0.1 is represented internally as 2130706433. And this is what you will see in Visualize.
Is not ideal, indeed, and it would be good to have a more advanced scripting language in scripted fields, but a github issue already exists.
I'm using java API to index data into ElasticSearch and generate graphs in Kibana.
I have a field named "Event_TS" which holds values of type long (time at which event was created in milliseconds). I could generate Date Histograms using it.
(I'm getting JSON document from a separate method.)
But, when I finally reindexed the whole data, Kibana is not identifying "Event_TS" as time-based anymore and hence I can't generate Date Histograms. How do I resolve this?
Let's say I have a log of events of ad displays and ad clicks stored via Logstash in Elasticsearch and displayed in Kibana 4. I would like to calculate a simple metric like CTR (Click-Through-Rate) of the events , which is :
CTR = #clicks/#displays.
First of all, does anyone know if it's possible to do in Elasticsearch + Kibana? I don't see a possibility to do it in Kibana.
I was thinking about doing it in raw Elasticsearch by scripted aggregation. But I don't know how to define such in Kibana.
Any ideas on how to do it would be very welcome! Comments explaining that it's impossible to do it would be also valuable.
Kibana 4 includes support for Elasticsearch scripting. You can go to Settings > Indices (pick your pattern) > Scripted Fields and add a new scripted field that computes your CTR. Take a look at "Scripted Fields" at elastic blog for more info.
I had to do something similar for a customer, but couldn't manage to get it done using scripted fields, because I had to sum UP all the values on each aggregation. I ended up developing a custom visualization (Plugin). This was for Kibana 4.1 and 4.5.1.
Let me know if this is what you are looking for, or if you want to give it a go yourself I can give you some pointers.