I'm new to Kibana and Elastic Search and i have run into this problem:
My ES contains (among other stuff) also data containing the current value of one custom performance counter and i would like my dashboard to show this value, e.g., as a big number - therefore i tried to use the Metric visualization, but i have no idea on how to show only the last value. Any help would be highly appreciated. Thanks.
We had a similar issue for our use case. We found two ways to handle it:
If the data is periodically generated then you can use the Kibana feature of showing data of recent n days to see the latest data.
In our case, the above option was not possible so we went with a hack where we have a property in our documents called "IsLatest" so we apply a filter "IsLatest":true in all our charts where we need latest info. We have written our code which feeds data to ElasticSearch in such a way that it updates the older data and sets it's "IsLatest" to false.
Hope it helps
Related
I have an ElasticSearch instance and it does one type of search - it takes a few parameters and returns the companies in its index that match the parameters given.
I'd like to be able to pull some stats that essentially says "This company has been returned from search queries X number of times in the past week".
Does ElasticSearch store metadata that will allow to pull this kind of info from it? If this kind of data isn't stored in ES out of the box, is there a way to enable it?
Elasticsearch (not ElasticSearch ;) ) does not do this natively, no. you can build something using the slow log, where you set the timing to 0 to get it to log everything, but that then logs everything which may not be useful/too noisy
things like https://www.elastic.co/enterprise-search, built on top of Elasticsearch, do provide this sort of insight
Consider the following use case:
I want the information from one particular log line to be indexed into Elasticsearch, as a document X.
I want the information from some log line further down the log file to be indexed into the same document X (not overriding the original, just adding more data).
The first part, I can obviously achieve with filebeat.
For the second, does anyone have any idea about how to approach it? Could I still use filebeat + some pipeline on an ingest node for example?
Clearly, I can use the ES API to update the said document, but I was looking for some solution that doesn't require changes to my application - rather, it is all possible to achieve using the log files.
Thanks in advance!
No, this is not something that Beats were intended to accomplish. Enrichment like you describe is one of the things that Logstash can help with.
Logstash has an Elasticsearch input that would allow you to retrieve data from ES and use it in the pipeline for enrichment. And the Elasticsearch output supports upsert operations (update if exists, insert new if not). Using both those features you can enrich and update documents as new data comes in.
You might want to consider ingesting the log lines as is to Elasticearch. Then using Logstash, build a separate index that is entity specific and driven based on data from the logs.
Let's say we have 1 million documents indexed in elasticsearch with name,id,skills,etc like a form a resume.
If I search for kartheek in the search box it will retrieve some results right and I found kartheek resume and I have clicked and view the resume.
Once I will view the resume I need to view the similar profiles based on the viewed resume.
Is it possible in elasticsearch
1.I learned about More Like This Query but we need to give input paramter i.e 'LIKE'.
2.I have seen this link
http://www.datasciencecentral.com/profiles/blogs/document-similarity-analysis-using-elasticsearch-and-python
Please any can give me ideas regarding the above mentioned request.
Thanks
Kartheek Gummaluri
To go from 1.x to 2.x you need to do what is known as a full cluster restart upgrade, you wont lose data in this case, here are the relevant steps laid out well from the docs:
Elastic Search Upgrade Docs
I have some requirement where I need to create different visualization for different users which will differ very slightly on the query param. So, I am considering to create a script which will enable me to do this.Have anyone done this on Kibana 4. Some pointers on how to create visualization using query would be of great help.
I would also like to create Dashboards on the fly but that can wait till I get this one sorted out.
If you want to go ahead with Java plugin (as mentioned in comments), here are the steps:
Create different visualizations with different X-axis parameters. Visualizations are basically json strings so you can write a java code which changes the value of x aggregation based on the mapping that you have. Now each chart will have different ids.
While you are creating a custom dashboard based on the user, check the mapping between user and the visualization and use the following command to add the visualization:
client.prepareIndex(,"visualization",).setSource().execute();
Our application exposes various metrics through an MBean, these are pulled by logstash and saved to ElasticSearch. Typical metrics are "number_of_connected_users", connection status for external dependencies etc. I'd like to visualize this, but I can't see that Kibana supports displaying only the last query result, discarding the older ones.
I figure I need to either 1) Limit a query to return only one result and then visualize it, or 2) Have the query return multiple results and selecting the last one in a visualization. Anyone done this?
[ Deleted old answer as your question makes more sense today! ]
Sadly, kibana doesn't support the idea of a "latest" record.
Here's their issue for it.