I am new for developing the ELK. So, I have an idea to do with the Elasticsearch data with some dashboard design using Kibana. I have tried we are able to create dashboards like pie, graph etc. However, they are all based on the count and average related. So, I want data like table with whole data. But, a way of the form the query based on that it will generate table. I would like to know whether it is possible in Kibana?
Related
Goal: I want to create a dashboard which shows user requests made to my website. For this, I created a filter in my java web-app and started capturing user requests and storing them in an ES index. The document is in the form of:
{
'user': 'user1',
'url': 'domain.com/page1',
'hitcount': 12
}
So, now I have an index which contains the information as to how many times a user requested which URLs.
Now, I want to create visualizations to show usage trends per user.
Question:
Which visualizations should be used for this use-case?
If I need to show the change in user-trends over time, how should I save the data? For e.g. is there a visualization where I could show, that a user has stopped/reduced requesting a page and now accesses a different page more frequently.
Any direction will be helpful.
Note: I understand, this could be done with grafana + prometheus, but I wish to do this with elastic stack.
I’d recommend logging user requests to a log file and have filebeat read and index them into ES. It is better to send non aggregated data into ES and then let ES aggregate it to create required visualizations
I would like to know how kibana dashboard can be made dynamic? I mean , each time data is posted into the es, i want the visualisation charts in kibana to reflect the new data.
I have been working on ways to import Google Analytics raw data without having to use a premium account .So far this is the nearest link to what I want to do
How to extract data from Google Analytics and build a data warehouse (webhouse) from it?
I want to load that data into elastic search and display using kibana .What is the best ETL approach for this ? Has anyone tried to display GA data using ELK stack ?
You should do it in two times
First, get the info, a very very useful site is https://developers.google.com/webmaster-tools/v3/how-tos/search_analytics but you have first to have a google wembaster tool account and create oauth credential on https://console.developers.google.com/apis
Then once you have your data, find a way to import them in elasticsearch, I'm still looking for the best way to do so, maybe transform the result table into csv and then using https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html
Have a look at this:
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html
You can use this to poll an endpoint, in this case GA, and load the response data into Elasticsearch. You may want to filter the response with the Split and / or Mutate plugins as well.
I have done this same setup.
Extracted data from Google Analytics with 7 Dimensions and 6 Metrics, out of which 2 Dimensions were primary key (Timestamp and ID). This was done using R.
Did some transformations on the data using linux awk and sed commands.
Loaded the data into Apache Hive with the row column formatting, created like total 9 tables.
Joined all the 9 tables in Hive using Hive Join queries, with 2 primary keys.
Used elasticsearch-hadoop connector to load the final resulting table to elasticsearch. Had to do a little data transformations to match Hive and Elasticsearch data types.
Used Kibana to visualize the data in Elasticsearch.
Now I am planning to avoid all the manual steps and somehow automate all the steps above.
I want to automate the creation of a set visualizations for new kibana/elasticsearch installations.
So I need to know if I can automate this, independent the programming language.
There are no APIs yet in Kibana to manage the searches, visualizations and dashboards. Some feature requests have been suggested (here and here) but they are still being discussed.
However, since Kibana visualizations are stored in the .kibana index with the visualization mapping type, you can definitely GET them, learn how they are built, modify them and PUT them again.
For a visualization named "Top consumers by country", you can get the visualization spec using
curl -XGET http://localhost:9200/.kibana/visualization/Top-consumers-by-country
You'll get a document containing the title of your visualization, another field called visState containing the specification of your visualization (obvisouly different for each visualization) and finally a field named kibanaSavedObjectMeta which contains the Elasticsearch query and index details.
You can also view/edit/export the same data in Settings > Objects > Visualizations
I have some requirement where I need to create different visualization for different users which will differ very slightly on the query param. So, I am considering to create a script which will enable me to do this.Have anyone done this on Kibana 4. Some pointers on how to create visualization using query would be of great help.
I would also like to create Dashboards on the fly but that can wait till I get this one sorted out.
If you want to go ahead with Java plugin (as mentioned in comments), here are the steps:
Create different visualizations with different X-axis parameters. Visualizations are basically json strings so you can write a java code which changes the value of x aggregation based on the mapping that you have. Now each chart will have different ids.
While you are creating a custom dashboard based on the user, check the mapping between user and the visualization and use the following command to add the visualization:
client.prepareIndex(,"visualization",).setSource().execute();