So I'm using grafana to display some data in a table. So in my grafana table I have a column named 'request_types'. Its value is represented by an array such as
["all"], ["Music", "Film"], ...etc
What I'm trying to do is create value mappings in grafana to map the array to specific values. For example
["all"] -> "All"
["Artist", "Film"] -> Artist, Film
How can I achieve this in grafana? Is value mapping the only option I have?
So in order for me to achieve this I had to create a custom data source plugin. Basically my data source plugin makes an http request to retrieve Elasticsearch data. From there I can write some typescript to manipulate my data.
Related
I am new to apache NiFi.
I am trying to put data into elasticsearch using nifi.
I want to specify an index name by combining a specific string and the value converted from a timestamp field into date format.
I was able to create the desired shape with the expression below, but failed to create the index name with the value of the timestamp field of the content.
${now():format('yyyy-MM-dd')}
example json data
{
"timestamp" :1625579799000,
"data1": "abcd",
"date2": 12345
}
I would like to get the following result:
index : "myindex-2021.07.06"
What should I do? please tell me how
I know that if you use the PutElasticSearch Processor, you can provide it with a specific index name to use. And as long as the index name meets the proper ElasticSearch format for naming a new index, if the enable auto index creation in ElasticSearch is turned on, then when sent, Elastic will create the new index with the desired name. This has worked for me. Double check the Elastic Naming Rules that can be found here or https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-indexing.html
I am using Kibana scripted fields using painless to populate a URL like this:
Url Template : https://dummy_url?branch={{value}}&id=abc
Script:
if (!doc['branch_name'].empty) {
return (doc['branch_name'].value)
}
When I access this scripted field for my index pattern in Discover, the value gets populated correctly and I can access the URL but when I am accessing the same scripted field in "data table" Visualize, the URL is missing the value populated by {{value}}.
Already tried using {{rawValue}} and doc['branch_name.keyword'] but none worked.
Can you please help on how I can populate the scripted field correctly on data table visualize?
I am using Kibana Version, 5.4.1.
The doc has values like this:
branch_name
develop
master
release/f1
release/f2
I’m trying to tag my data according to a lookup table.
The lookup table has these fields:
• Key- represent the field name in the data I want to tag.
In the real data the field is a subfield of “Headers” field..
An example for the “Key” field:
“Server. (* is a wildcard)
• Value- represent the wanted value of the mentioned field above.
The value in the lookup table is only a part of a string in the real data value.
An example for the “Value” field:
“Avtech”.
• Vendor- the value I want to add to the real data if a combination of field- value is found in an document.
An example for combination in the real data:
“Headers.Server : Linux/2.x UPnP/1.0 Avtech/1.0”
A match with that document in the look up table will be:
Key= Server (with wildcard on both sides).
Value= Avtech(with wildcard on both sides)
Vendor= Avtech
So baisically I’ll need to add a field to that document with the value- “ Avtech”.
the subfields in “Headers” are dynamic fields that changes from document to document.
of a match is not found I’ll need to add to the tag field with value- “Unknown”.
I’ve tried to use the enrich processor , use the lookup table as the source data , the match field will be ”Value” and the enrich field will be “Vendor”.
In the enrich processor I didn’t know how to call to the field since it’s dynamic and I wanted to search if the value is anywhere in the “Headers” subfields.
Also, I don’t think that there will be a match between the “Value” in the lookup table and the value of the Headers subfield, since “Value” field in the lookup table is a substring with wildcards on both sides.
I can use some help to accomplish what I’m trying to do.. and how to search with wildcards inside an enrich processor.
or if you have other idea besides the enrich processor- such as parent- child and lookup terms mechanism.
Thanks!
Adi.
There are two ways to accomplish this:
Using the combination of Logstash & Elasticsearch
Using the only the Elastichsearch Ingest node
Constriant: You need to know the position of the Vendor term occuring in the Header field.
Approach 1
If so then you can use the GROK filter to extract the term. And based on the term found, do a lookup to get the corresponding value.
Reference
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-jdbc_static.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-jdbc_streaming.html
Approach 2
Create an index consisting of KV pairs. In the ingest node, create a pipeline which consists of Grok processor and then Enrich it. The Grok would work the same way mentioned in the Approach 1. And you seem to have already got the Enrich part working.
Reference
https://www.elastic.co/guide/en/elasticsearch/reference/current/grok-processor.html
If you are able to isolate the sub field within the Header where the Term of interest is present then it would make things easier for you.
Currently, I'm importing data into Elastic through logstash, at this time by reading csv files.
Now let's say I have two numeric fields in the csv, age, and weight.
I would need to add a 3rd field on the fly, by making a math on the age, the weight and another external data ( or function result ); and I need that 3rd field to be created when importing the data.
There is any way to do this?
What could be the best practice?
In all Logstash filter sections, you can add fields via add_field, but that's typically static data.
Math calculations need a separate plugin
As mentioned there, the ruby filter plugin would probably be your best option. Here is an example snippet for your pipeline
filter {
# add calculated field, for example BMI, from height and weight
ruby {
code => "event['data']['bmi'] = event['data']['weight'].to_i / (event['data']['height'].to_i)"
}
}
Alternatively, in Kibana, there are Scripted fields meant to be visualized, but cannot be queried
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"4eb9f840-3969-11e8-ae19-552e148747c3\",\"filter\":[],\"query\":{\"language\":\"lucene\",\"query\":\"\"}}"
}
The above mentioned snippet is a exported JSON of a Kibana visualization. Without exporting this json is there a direct way to get this
\"index\":\"4eb9f840-3969-11e8-ae19-552e148747c3\ index id.
And if i am not wrong this is supposed to be the index id as its common across visualization with same index.
So, you can retrieve all index patterns using this query
GET .kibana/_search?q=type:index-pattern&size=100
Additionally, you can retrieve a specific set of index pattern given its name using
GET .kibana/_search?q=type:index-pattern%20AND%20index-pattern.title:indexname
Similarly, regarding visualizations, you can retrieve one by name using
GET .kibana/_search?q=type:visualization%20AND%20visualization.title:vizname