I am running into a weird issue, I have a lot of data in ES and Kibana displays it perfectly. I have a couple of visualization which include bar-chart & pie-chart, the problem is when i click on a bar-graph of a particular object i get only a limited result but when i search for the object on the "Discover" tab i can see all the result.
The dashboard basically consists of searches which were saved earlier, so why is the behaviour occurring ?
Can someone help me on this or point me why/how is this happening ?
Visualizations are mostly aggregations under the hood and the latter are usually limited to the top 10 results.
On the other hand, the Discover tab shows the actual records/documents stored in ES, which is why you can see many more results, usually capped at 500 by default.
Related
I am using Elastic search to show a paginated list of products in a grid view in a mobile app. Now the user can scroll through the list and click on any product to view the details.
Now the detail view also supports scrolling through the products via swipe left and right. So for the detail view, I want to fetch paginated results from elastic search starting from a particular product.
For now I am calculating the index of the product in list view and then doing the math to fetch that particular page and scroll to the index.
Is there a better way to do this?
You can use the scroll api to get paginated results for your search query.
Alternatively, you can use the search_after api which seems to perform better for large number of results but it is available only for 7.x+ elastic versions.
I'm not totally certain — I haven't tested it myself — but I think the suggestion of using the search after API is the way to go.
You need to use something like the point in time feature, which is what the search after uses. Without it, you have no guarantee that the data in the database aren't changing. If the data change, then your search result may change. If that changes, then what comes "next" also changes, and that may no longer correspond to what you want.
E.g., if you currently have 10 search results and your item of interest is at index 5, if someone adds a document that moves your point of interest to index 6, then naïvely asking for the next item would return the same thing!
The point in time feature creates a snapshot of the database at a moment in time, so you don't have to worry about new or modified documents messing things up.
As an aside, using the point in time feature at scale is probably (again, making educated guesses here) not a very good idea. Elasticsearch has to keep a mini-snapshot of the whole database (!) every time you call that for the duration.
You're probably better off limiting the number of items people can page through to something large but manageable, and then reloading a new page if someone gets to the end. If you pull 500 products initially and someone gets to the end (which seems unlikely to me a priori), you could re-issue the search paged forward 500 items, deduplicate at the boundary and no one will be the wiser.
I have a problem with Kibana: Dashboard and visualizations don't show any results!
As you can see in this screenshot, in the discover tab I get some results! Which means data exists in my index "as-*". right? But I used a trick in order to display this data :
1) I changed the range to "Today" ==> it shows no result found !
2) I clicked "New" button ==> then i get my data displayed!
Is there an other way "more proper" to get data displayed?
Then in the my dashboard (or visualization) I can't get any results!! even if the range is the same as discover tab!
I restarted Kibana ==> no changes!
I deleted as-* then I created it => no changes!
I'm using Curator to create daily index and logstash to index the data into ES!
I'm stuck here ! I ll be glad if you can help me figure it out !
Thank you very much!
I am fairly new to elasticsearch and Kibana, but here are three mistakes that I made in the past:
Are you using the correct index? Make sure that the index you have chosen for the logs that are displayed on the visualise page and for the actual visualisations is the same.
Correct timeperiod: does the time period you have chosen contain the data you are looking for? Or did you happen to have zero logs during that time?
Correct filters and aggregations: when you were making the visualisations, did they show any results? Or were they empty from the beginning? Maybe one of your filters or aggregations is wrong and it's excluding the results you're expecting to see.
Not sure if this is any help, hope you've sold the problem by now :)
If you can see information in "discover", it means that kibana has connected to the database, and that the database has information. You shouldn't have to click the "New" button to see information in the discover view. I believe the "New" button in the "discover" page is used to create a new search.
Maybe try zooming on the time period of the data on the "discover" page, or
try checking the systems logs to see if logstash is successfully pushing information to Elasticsearch.
With Kibana, dashboards are made up of visualisations, and visulations are made up of searches.
The "No results found" on the dashboard page, shown in your second page is due to the visualisation having no results. I guess you imported visulations into Kibana.
I hope that helps.
We've got an ELK setup. All logfiles are collected successfully and pushed into elasticsearch. Then we're using kibana 4 to visualize the logs. But there is only WARN as 'level' shown. Within the elasticsearch index there are many more and all other 'level's. But these are not shown. All other fields in the index, where multiple values are available are shown, except 'level'.
Is this a bug or is there something wrong in the default configuration?
It was a bug within logstash. It corrupted some log-events so only these wich have the loglevel WARN were processed correctly. So only these events are shown.
Thanks for your help. ;)
I am not entirely sure what you tried and how did you end up not getting all level in Kibana. Because you've mentioned:
Then we're using kibana 4 to visualize the logs.But there is only WARN as 'level' shown.
This does not make it clear what kind/type of (out of the box) visualization (graph, chart, data table, etc) is being used and what aggregation(s) are implemented.
Also, When you say
Within the elasticsearch index there are many more and all other
'level's. But these are not shown.
I'd assume that the everything is implemented/configured correctly at the logstash end (the grok filter, and the pattern if you're using one)
I recently did something similar (parse and visualize tomcat logs) and dealt with the same 'level' issue. I discovered that in Kibana while using a (sub)aggregation in any visualization, when you choose Significant Terms -> fieldname it might neglect the value that is very less compared to other values of that field. Ex if occurrence frequency of WARN >>> ERROR in the level field for the given timeframe, Kibana will show only WARN irrespective of the size that you put in. This is what it is meant for - showing only significant terms.
To overcome this, use Terms -> Field -> fieldname. Set Order to Top/Bottom
and most importantly, Size to 0.
You'll see every value for that field irrespective of its occurrence frequency. Hope this helps.
PS: Choosing bottom gets you slight relief from that default GREEN color in the Kibana visualizations.
It sounds like you're on the Discover tab, and are commenting on the values shown in the facets on the left side.
These values are generated from the documents in the current table (default is the most recent 500 rows returned by your query), not from the results of the entire query.
So, if the most recent messages are all WARN, that's all that will show in the facet on the left side.
To confirm this theory, exclude the WARN messages and see if your other levels appear.
I'm new to Kibana and Elastic Search and i have run into this problem:
My ES contains (among other stuff) also data containing the current value of one custom performance counter and i would like my dashboard to show this value, e.g., as a big number - therefore i tried to use the Metric visualization, but i have no idea on how to show only the last value. Any help would be highly appreciated. Thanks.
We had a similar issue for our use case. We found two ways to handle it:
If the data is periodically generated then you can use the Kibana feature of showing data of recent n days to see the latest data.
In our case, the above option was not possible so we went with a hack where we have a property in our documents called "IsLatest" so we apply a filter "IsLatest":true in all our charts where we need latest info. We have written our code which feeds data to ElasticSearch in such a way that it updates the older data and sets it's "IsLatest" to false.
Hope it helps
Specifically, I'm using Elasticsearch to do pagination, but this question could apply to any database.
Elasticsearch provides methods to paginate search results with handy from and to parameters.
So I run a query get me the most recent data from result 1 to 10
This works great.
The user clicks "next page" and the query is:
get me the most recent data from result 11 to 20
The problem is that in the time between the two queries, 2 new records have been added to the backing database, which means the paginated results will overlap (the last 2 from the first page show up as first two on the second page).
What's the best solution to avoid this? Right now, I'm adding a filter to the query that tell it to only include results later than the last result of the previous query. But it just seems hackish.
A filter is not a bad option, if you're already indexing a relevant timestamp. You have to track that timestamp on the client side in order to correctly prepare your queries. You also have to know when to get rid of it. But those aren't insurmountable problems.
The Scroll API is a solid option for this, because it effectively snapshots in time on the Elasticsearch side. The intent of the Scroll API is to provide a stable search query for deep pagination, which has to deal with the exact issue of change that you're experiencing.
You begin a Scrolling Search by supplying your query and the scroll parameter, for which Elasticsearch returns a scroll_id. You then make requests to /_search/scroll supplying that ID, each of which return a page of results and a new scroll_id for the next request.
(Note that you don't want the scan search type here. That's used to extract documents en masse, and does not apply any sorting.)
Compared to filtering, you do still have to track a value: the scroll_id for your next page of results. Whether that's easier than tracking a timestamp depends on your app.
There are other potential downsides to consider. Elasticsearch persists the context for your search on a single node within the cluster. Conceivably these could accumulate in your cluster, depending on how heavily you rely on scrolling search. You'll want to test the performance implications there. And if I recall correctly, scrolling searches also do not persist through a node failure or restart.
The ES documentation for the Scroll API provides good details on all of the above.
Bottom line: filtering by timestamp is actually not a bad choice. The Scroll API is another valid option, designed for a similar use case, but is not without its drawbacks.
Realise this is a bit old but with ElasticSearch 6.3 there's now the search_after feature for the request body which allows for cursor type paging:
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-after.html
It is very similar to the scroll API but unlike it, the search_after parameter is stateless, it is always resolved against the latest version of the searcher.
You need to use scan API for this. Scan and scroll API let's you do point in time search and pagination.
Scan API -