In Elastic search we store events, I have built data table which aggregates based on event types. I have filter which checks for event.keyword : "job-completed". I am getting the count as 1 or 0 but i want to display as completed / in-progress.
How to achieve this in Kibana ?
The best and more effective way to do this is to add another field with and to complete it at the ingest time.
It the best solution regarding on performance. But it can lead to an heavy work.
You can also use a scripted field to do this without touching your data.
Do to stack management > kibana > index pattern and select your index.
Select scripted field tab and fill in form.
Name : your_field
language: painless
type: string
format: string
script:
if(doc['event.keyword'].value=='job-completed'){
return "completed";
}else {
return "in progress";
}
I got to few information on your real data to be able to give you a working code, so you'll have to modify it to fit your needs.
Then refresh you visualization and you can use your new field
Related
Currently, I'm importing data into Elastic through logstash, at this time by reading csv files.
Now let's say I have two numeric fields in the csv, age, and weight.
I would need to add a 3rd field on the fly, by making a math on the age, the weight and another external data ( or function result ); and I need that 3rd field to be created when importing the data.
There is any way to do this?
What could be the best practice?
In all Logstash filter sections, you can add fields via add_field, but that's typically static data.
Math calculations need a separate plugin
As mentioned there, the ruby filter plugin would probably be your best option. Here is an example snippet for your pipeline
filter {
# add calculated field, for example BMI, from height and weight
ruby {
code => "event['data']['bmi'] = event['data']['weight'].to_i / (event['data']['height'].to_i)"
}
}
Alternatively, in Kibana, there are Scripted fields meant to be visualized, but cannot be queried
I'm currently trying to create a graph on Grafana to monitor the status of my servers, however, I can't seem to find a way to use the value of a field as the value to be displayed on the graph. (Datasource is ElasticSearch)
The following "document" is going to be sent to GrayLog (which saves to Elastic) every 1 minute for an array of regions.
{
"region_key": "some_key",
"region_name": "Some Name",
"region_count": 1610
}
By using the following settings, I can get Grafana to display the count of messages it received for each region, however, I want to display the number on the region_count field instead.
Result:
How can I accomplish this? is this even possible using Elastic as the datasource?
1) Make sure that your document includes a timestamp in ElasticSearch.
2) In the Query box, provide the Lucene query which narrows down the documents to only those related to this metric
3) In the Metric line, press "Count" and change that to one which takes a specific field: for example, "Average"
4) Next to the "Average" box will appear "select field", which is a dropdown of the available fields. If you see unexpected fieldnames here, it's probably because your Lucene query isn't specific enough. (Kibana can be useful for getting this query right)
I'm using elasticsearch and kibana for storing my logs.
Now what I want is to extract a number from a field and store it a new field.
So for instance, having this:
accountExist execution time: 1046 ms
I would like to extract the number (1046) and see it in a new field in kibana.
Is it possible? how?
Thanks for the help
You'll need to do this before/during indexing.
Within Elasticsearch, you can get what you need during indexing:
Define a new analyzer using the Pattern Analyzer to wrap a regular expression (for your purposes, to capture consecutive digits in the string - good answer on this topic).
Create your new numeric field in the mapping to hold the extracted times.
Use copy_to to copy the log message from the input field to the new numeric field from (2) where the new analyzer will parse it.
The Analyze API can be helpful for testing purposes.
While not performant, if you must avoid reindexing, you could use scripted fields in kibana.
Introduction here: https://www.elastic.co/blog/using-painless-kibana-scripted-fields
enable painless regex support by putting the following in your elasticsearch.yaml:
script.painless.regex.enabled: true
restart elasticsearch
create a new scripted field in Kibana through Management -> Index Patterns -> Scripted Fields
select painless as the language and number as the type
create the actual script, for example:
def logMsg = params['_source']['log_message'];
if(logMsg == null) {
return -10000;
}
def m = /.*accountExist execution time: ([0-9]+) ms.*$/.matcher(params['_source']['log_message']);
if ( m.matches() ) {
return Integer.parseInt(m.group(1))
} else {
return -10000
}
you must reload the website completely for the new fields to be executed, simply re-doing a search on an open discover site will not pick up the new fields. (This almost made me quit trying to get this working -.-)
use the script in discover or visualizations
While I do understand, that it is not performant to script fields for millions of log entries, my usecase is a very specific log entry, that is logged 10 times a day in total and I only use the resulting fields to create a visualization or in analysis where I reduce the candidates through regular queries in advance.
Would be interesing if it is possible to have those fields only be calculated in situations where you need them (or they make sense & are computable to begin with; i.e. to make the "return -1000" unnecessary). Currently they will be applied and show up for every log entry.
You can generate scripted fields inside of queries like this: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-script-fields.html but that seems a bit too much of burried under the hood, to maintain easily :/
Using Kibana, is it possible to display one row of data which is a summary of other rows?
This is our requirement:
Given an entry in an index with the following structure:
string requestId
boolean raisedException
boolean requiredExternalLookup
We want to create a tabular output with the following structure
requestId numberRaisedException numberNoException numberRequiredLookup
So, if there were three rows (or entries) in the index for the same request id, two where an exception was raised, the output may look like this:
requestId numberRaisedException numberNoException numberRequiredLookup
REQUEST_123 2 1 3
Presumably the correct Kibana visualization widget to represent this would be a Data Table. But how in Kibana would one create a row like the above which is a summary of several rows, somewhat akin to a sql GROUP BY clause. Is it at all possible?
You can probably do this with 'scripted_fields', but the status of the 'scripted_fields' feature in kibana isn't clear. I think it was recently blocked in kibana due to security issues - Leaving this open is dangerous since you can do anything.
If you have access to your elasticsearch cluster then you might be able to create the field on your elasticsearch index.
You can read about it here : http://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-script-fields.html
I'm looking to search for a particular JSON document in a bucket and I don't know its document ID, all I know is the value of one of the sub-keys. I've looked through the API documentation but still confused when it comes to my particular use case:
In mongo I can do a dynamic query like:
bucket.get({ "name" : "some-arbritrary-name-here" })
With couchbase I'm under the impression that you need to create an index (for example on the name property) and use startKey / endKey but this feels wrong - could you still end up with multiple documents being returned? Would be nice to be able to pass a parameter to the view that an exact match could be performed on. Also how would we handle multi-dimensional searches? i.e. name and category.
I'd like to do as much of the filtering as possible on the couchbase instance and ideally narrow it down to one record rather than having to filter when it comes back to the App Tier. Something like passing a dynamic value to the mapping function and only emitting documents that match.
I know you can use LINQ with couchbase to filter but if I've read the docs correctly this filtering is still done client-side but at least if we could narrow down the returned dataset to a sensible subset, client-side filtering wouldn't be such a big deal.
Cheers
So you are correct on one point, you need to create a view (an index indeed) to be able to query on on the content of the JSON document.
So in you case you have to create a view with this kind of code:
function (doc, meta) {
if (doc.type == "youtype") { // just a good practice to type the doc
emit(doc.name);
}
}
So this will create a index - distributed on all the nodes of your cluster - that you can now use in your application. You can point to a specific value using the "key" parameter