I have imported weblogs into Elasticsearch via Logstash. This has completed successfully.
I have a field in the log file (clientip) that is always populated and another field that is sometimes populated (trueclientip). I want to aggregate based on the coalescing of the two; e.g. if trueclientip is not empty then use that otherwise use clientip.
How can I do this with the Visualisation in Kibana? Do I need to generate a scripted field or is there another approach?
Thanks.
Define a scripted field that should have this formula: doc['trueclientip'].value ? doc['trueclientip'].value : doc['clientip'].value and use this in your aggregations.
But, there is a downside to this scripted fields functionality AND the ip type: it seems what you get back from the script is the number itself (which is logic because the scripted fields in Kibana 4 only use Lucene expressions as a language), not the string representation. IPs internally are actually long numbers in Lucene.
For example, 127.0.0.1 is represented internally as 2130706433. And this is what you will see in Visualize.
Is not ideal, indeed, and it would be good to have a more advanced scripting language in scripted fields, but a github issue already exists.
Related
I'm working in a Go application that makes queries to an elasticsearch index (version 7.17). I have a requirement to build a filter/query that compares 2 long fields in that index (A and B) and returns documents where A < B within the same document. I was trying to determine the best way to do this, and a lot of the info I had found while searching pointed me to script queries. The script queries are pretty straight forward, but I've read they can be resource intensive.
One idea I had to avoid a script query is adding a boolean field mapping called ~ "a_less_than_b". Our Go code performs some transformations on structs before they're sent to elastic as documents, so it would be trivial for me to set that field based on the corresponding A and B attributes on the Go struct.
Is there some simpler way to implement the comparison on the elasticsearch side instead of a script query, or would setting that dedicated boolean field be a cleaner, more performant solution?
In Kibana (ElasticSearch v6.8) I'm storing documents containing a date field and a LaunchTime field, and I have a scripted field uptime as their difference (in seconds):
(doc['date'].value.millis - doc['LaunchTime'].value.millis) / 1000 / 60
I'm trying to create a monitor (under alerting) on the max value of this field of the index, but the field 'Uptime' doesn't show up in the list of fields I can do a max query on. Its type is number and in visualisations I can do max/min etc. displays of this field.
Is this a limitation of Kibana alerting - that I can't use a scripted field? Or is there some way I can make it available to use?
I'm afraid it is a limitation of kibana's scripted fields. See this post about the same subject referring to the scripted field official documentation. I believe that the watcher are handled by ES itself while the scripted field are handled by kibana (they can be used in discovery and visualisations because kibana is handlind those too)
But have no fear! you already have the script for the calculation and you could just add it into logstash to add the field to you actual documents when you index them, which would enable you to use it for watchers AND would probably optimize the load at runtime, since the val is only calculated one, when you ingest it. Then you could run an update by query with a the script and add the field in you existing documents.
If you don't use logstash, you could look into ES's ingestion pipelines, but it's a rather advanced subject and i'm not sure if it was implemented in 5.x.
I have Kibana and ES. I have many indexes. I am using message field in ElasticSearch. My goal is to mask all IP addresses, which I already do using Logstash.
Now, given the fact there are many different indexes, and also different log types, I would like to run either Kibana or ES query for any occurence of IP. Just in case, that I missed any of them. Also, I would like to do it for email format as well.
Question is, how can I run IP/email regex search on ElasticSearch or Kibana?
Message field is string type, and is indexed.
I have found what I was looking for. In my case this approach is valid, since I do not care about performance. This was just a test to make sure I don't 'leak' information.
ElasticSearch regex query.
I've been given a set up where I have a program creating live data and posting them into Elasticsearch.
I am trying to visualise this data in Kibana, but I'm coming across many problems such as numbers for a field being of type string instead of integers or there being certain missing fields.
But mainly for now certain fields being integer instead of string would be useful. How do I go about this? Is it possible?
I have no access to source code of the system creating the live events data.
Thanks in advance.
Update: I should also mention additionally that for now I am restricted to Elasticsearch version 2.4
If your data is coming straight into Elasticsearch, your options are limited.
The best option is to have the program that is creating the data send valid, properly formatted data.
If that's not an option, you can set your Elasticsearch mapping to force the field to be numeric. This will have the side-effect of dropping all documents where this field is not numeric.
There is also the elasticsearch injest node, which allows for some (logstash-like) transformations of the data. Converting the type is one such allowed "processor".
Kibana 4 has a new feature to add scripted fields and write custom scripts. I wish to write a script that uses aggregations. Its easy to do simple arithmetic operations in scripted scripts but for doing aggregations I am puzzled. I am a new comer to Kibana and elasticsearch, I am looking for a sample script for beginning..
Scripted fields in Kibana are powered by lucene expressions, which only support numeric operations right now. Support for things like string manipulation and date parsing will probably be added at some point, but I doubt scripts will even support executing aggregations.
Scripted fields are primarily for converting a number before using it, or creating a synthetic field which is the combination of two or more other fields. Down the road they may even support things like extracting the day of the week from a date, or the portion of a string that matches a regular expression.