I was posting a question about the automatic (ie : scripted automation) creation of visualizations in Kibana. BUT I found the solution while writing it ;) Let me explain :
I was kind of blocked because I can't find nothing on the web about it. And the documentation of Elasticsearch doesn't explain that.
In a few words, the global projet I'm trying to do is to write a script to automatically create a new dashboard according to a specific user (for that, it is the search that changes).
More precisely, as a dashboard is made of visualizations, we need to create the visualizations in the first place. I was blocked at this step, creating the visualization directly via a CURL -XPOST request to ElasticSearch.
Because in order to do the final dashboard, we need to :
Write the search(es)
Create the visualization(s)
Create the dashboard made of visualizations
Schematically :
Input : Username --> Myscript --> Dashboard of the user
To make things clear, a visualization is only a JSON document in the path (in my case, but surely in yours too) :
'http://localhost:9200/.kibana/visualization/*'
In ElasticSearch.
So it's simple, add a new visualization as you add a new document through the ElasticSearch API.
To do that, and to know what the visualization you want to create looks like, you can create it using Kibana web interface; and once you validate and it's added in ElasticSearch, you see the ElasticSearch document.
Let's say we created a Pie Chart using Kibana named "Test1" : we request it in ElasticSearch to see the document.
curl -XGET 'http://localhost:9200/.kibana/visualization/Test1'
You should have this kind of result :
{"_index":".kibana","_type":"visualization","_id":"Test1","_version":1,"found":true,"_source":{"title":"Test1","visState":"{\"aggs\":[{\"id\":\"1\",\"params\":{},\"schema\":\"metric\",\"type\":\"count\"},{\"id\":\"2\",\"params\":{\"field\":\"type.raw\",\"order\":\"desc\",\"orderBy\":\"1\",\"size\":10},\"schema\":\"segment\",\"type\":\"terms\"}],\"listeners\":{},\"params\":{\"addLegend\":true,\"addTooltip\":true,\"isDonut\":false,\"shareYAxis\":true},\"title\":\"New Visualization\",\"type\":\"pie\"}","uiStateJSON":"{}","description":"","version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"YOUR KIBANA DISCOVER SEARCH HERE\"}},\"filter\":[]}"}}}
So, then, we want to create a similar document into ElasticSearch directly, without using Kibana, as we would need it in a script. The query looks like this :
curl -XPOST 'http://localhost:9200/.kibana/visualization/Test2' '{"title":"Test2","visState":"{\"aggs\":[{\"id\":\"1\",\"params\":{},\"schema\":\"metric\",\"type\":\"count\"},{\"id\":\"2\",\"params\":{\"field\":\"type.raw\",\"order\":\"desc\",\"orderBy\":\"1\",\"size\":10},\"schema\":\"segment\",\"type\":\"terms\"}],\"listeners\":{},\"params\":{\"addLegend\":true,\"addTooltip\":true,\"isDonut\":false,\"shareYAxis\":true},\"title\":\"New Visualization\",\"type\":\"pie\"}","uiStateJSON":"{}","description":"","version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"YOUR KIBANA DISCOVER SEARCH HERE\"}},\"filter\":[]}"}}'
That's it ! You can check the Pie Chart has been well created in Kibana ;)
Have a good day, I hope this has been useful.
Tony
Related
I am trying to Use my Elastic search server installed in my local machine to use Postman .i.e., With the help of Postman I want to Post Data and retrieve it with a get operation but unable to do it as I am getting error unknown key [High] for create index
So please help me with the same.
If you want to add a document to your index,
your url should look something like this ( for document ID 1 ) :
PUT http://localhost:9200/test/_doc/1
A good place to start :
https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index.html
For indexing document in the index
PUT http://localhost:9200/my_index/_doc/1
Retrieving indexed document
GET http://localhost:9200/my_index/_doc/1
Introduction:
Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease.
Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack. Do anything from tracking query load to understanding the way requests flow through your apps.
Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite “stash.” .
Elasticsearch exposes itself through rest API so in this case you don't have to use logstash as we are directly adding data to elastic search
How to add it directly
you can create an index and type using :
{{url}}/index/type
where index is like a table and type is like just a unique data type that we will be storing to the index. Eg {{url}/movielist/movie
https://praveendavidmathew.medium.com/visualization-using-kibana-and-elastic-search-d04b388a3032
Goal: I want to create a dashboard which shows user requests made to my website. For this, I created a filter in my java web-app and started capturing user requests and storing them in an ES index. The document is in the form of:
{
'user': 'user1',
'url': 'domain.com/page1',
'hitcount': 12
}
So, now I have an index which contains the information as to how many times a user requested which URLs.
Now, I want to create visualizations to show usage trends per user.
Question:
Which visualizations should be used for this use-case?
If I need to show the change in user-trends over time, how should I save the data? For e.g. is there a visualization where I could show, that a user has stopped/reduced requesting a page and now accesses a different page more frequently.
Any direction will be helpful.
Note: I understand, this could be done with grafana + prometheus, but I wish to do this with elastic stack.
I’d recommend logging user requests to a log file and have filebeat read and index them into ES. It is better to send non aggregated data into ES and then let ES aggregate it to create required visualizations
I am starting to use ES and Kibana, so apologies in advance if this question doesn't make sense!
I'd like to be able to see in Kibana a list of my current indices, similar to what you get with:
curl 'localhost:9200/_cat/indices?v'
I was expecting to be able to see in Kibana functionality partly like a DB client where you can connect to a DB server and see all the databases, then drill down in each of them to see tables and content. I'd love to have that kind of workflow in Kibana.
The closest I can find is in Management -> Index Pattern, but it'll display a list of all fields, which is too much information and I can't see any column in the table that points to which index each field belongs.
As I said I'm just starting so it might be I'm not looking in the right place!
I don't think we have any option to see the hierarchy like you see in traditional DB application.
If you are looking for something in kibana which can give you information similar to curl 'localhost:9200/_cat/indices?v', then you can go to "Monitoring-> Indices" which will list out all the index with there stat's(document count, data size, index rate etc).
If you don't have the x-pack installed then you have to use the "Discover" tab where you can see the list of all the index from the drop down, also the _type and all the available fields in the index.
Kibana's UI allows the user to create a scripted field which is stored as part of the index (screenshot below). How can that be done programatically? In particular, using either the NEST client or the Elasticsearch low level client.
Kibana UI for the Indice with the Scripted Fields tab highlighted
Note that I am not asking how to create add an expression/script field as part of a query, I'm specifically looking for how to add it as part of the Index when the mapping is created so that queries can reference it without having to explicitly include it.
Kibana dashboards are stored in the .kibana index. To export dashboards, you can query the Kibana index as you would any other index. For example, curl -XGET http://localhost:9200/.kibana/_search?type=dashboard&pretty would show the JSON for your dashboards. You could export the template, add the scripted field to the JSON, and then POST it again. Since Kibana uses a standard Elasticsearch index, the normal Elasticsearch API would apply to modifying Kibana dashboards. This may provide a little more clarification.
At the time of writing, current version 5.2 does not have an official way to do this.
This is how I do it:
Get index fields: GET /.kibana/index-pattern/YOUR_INDEX
Add your scripted field to _source.fields (as string, notice scaped quotation marks)
"fields":"[...,{\"name\":\"test\",\"type\":\"number\",\"count\":0,\"scripted\":true,\"script\":\"doc['area_id'].value\",\"lang\":\"painless\",\"indexed\":false,\"analyzed\":false,\"doc_values\":false,\"searchable\":true,\"aggregatable\":true}]"
Post back _source json to /.kibana/index-pattern/YOUR_INDEX
{
"title":"YOUR_INDEX",
"timeFieldName":"time",
"fields":"[...,{\"name\":\"test\",...}]"
}
I want to automate the creation of a set visualizations for new kibana/elasticsearch installations.
So I need to know if I can automate this, independent the programming language.
There are no APIs yet in Kibana to manage the searches, visualizations and dashboards. Some feature requests have been suggested (here and here) but they are still being discussed.
However, since Kibana visualizations are stored in the .kibana index with the visualization mapping type, you can definitely GET them, learn how they are built, modify them and PUT them again.
For a visualization named "Top consumers by country", you can get the visualization spec using
curl -XGET http://localhost:9200/.kibana/visualization/Top-consumers-by-country
You'll get a document containing the title of your visualization, another field called visState containing the specification of your visualization (obvisouly different for each visualization) and finally a field named kibanaSavedObjectMeta which contains the Elasticsearch query and index details.
You can also view/edit/export the same data in Settings > Objects > Visualizations