I have a field in elastic search loaded that has information in it as:
message: Requesting 30 containers
message: Requesting 40 containers
.
.
.
message: Requesting 50 containers
I want to get a total of all containers used in the job. (30+40+50=120, in this case).
Is it more efficient to extract these values in a field in logstash and then use aggregation queries in elasticsearch or given the message above everything is possible in elasticsearch?
Also, if I write a aggregation query in Dev Tools of Kibana, then is it possible to store the result to be used for visualization?
It is better and is the solution to extract the number in logstash and then use it in aggregations
No , You cant use a string in sum aggregation , Everything is never possible
You dont need you write aggregation query in dev tools if you are using kibana , in kibana you can do aggregations without writing queries
Related
I have an ElasticSearch instance and it does one type of search - it takes a few parameters and returns the companies in its index that match the parameters given.
I'd like to be able to pull some stats that essentially says "This company has been returned from search queries X number of times in the past week".
Does ElasticSearch store metadata that will allow to pull this kind of info from it? If this kind of data isn't stored in ES out of the box, is there a way to enable it?
Elasticsearch (not ElasticSearch ;) ) does not do this natively, no. you can build something using the slow log, where you set the timing to 0 to get it to log everything, but that then logs everything which may not be useful/too noisy
things like https://www.elastic.co/enterprise-search, built on top of Elasticsearch, do provide this sort of insight
I want to log all the queries made to Elasticsearch along with their response bodies in kibana.
Is there a way to do that?
I came to know a way to set. t he slowlogs threshold to 0 and log all the queries i slowlogs and then use filebeat to push those queries to kibana.
Is there any other way to do that
As far as I know, this is not available atleast in basic and free version and even if you set search slowlog threshold to 0ms it will just log the search query and other metadata of search query but wouldn't log the search query response.
It would be better to do this in your application which generated the search query and parse the response, then using filebeat you can send the application logs to Elasticsearch.
I am pretty new to elasticsearch and I want to create statistics and kibana dashboards on queries sent to elasticsearch index , what is the best approach to do so ? Any advice or recommendations will be highly appreciated?
The idea is to analyze all queries sent to the index and do some performance optimisation in the future when the userbase increase ...
I am planning for the moment to store the logs in different index , but parsing seems to be kind of complex activity ...
Ideally I need to have:
-Counting of user queries
-Counting of queries that returned no results
-Logging of all search terms
-Sorting of queries, and queries that returned no results, by most frequently contained search term
-A view of top queries, including the search term not found results for and the exact query
-A view of top queries returning no results, including the search term not found results for and the exact query
Thanks
There is no OOTB functionality available in Elasticsearch for search analysis. But there are some workaround you can do for same and get information what you are asking.
First option, you can enable slow log in Elasticsearch by executing below command and it will log each and every request to coming to Elasticsearch.
PUT /my-index-000001/_settings
{
"index.search.slowlog.threshold.query.info": "0s",
"index.search.slowlog.threshold.fetch.info": "0s"
}
Second option, You can log all the query the application layer or intermediate level using which application and elasticsearch talking to each other.
Once you have logs, You can configured Logstash / Filebeat / Fleet to read log and transform and index to Elasticsearch. Logstash provide differnt kind of filter which you can use and easily transofrm your plain text logs to strcture logs (grok filter).
Is it possible to see which are the most popular searched phrases/words within a particular index in elasticsearch.
Can this be set up in kibana at all.
You can do that by using Search Slow log - https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html
You can set the slow log setting dynamically too. Once this is set you should see the logs in index_search_slowlog.log. Ingest these logs back to elasticsearch and visualize in kibana. You can create the dashboard from this data.
We use these slow logs to monitor slow queries, popular queries etc.
When I am trying to pull the rawdocuments in a table from elasticsearch using Grafana, It does not show me the all the documents which are available there in elasticsearch index .No matter how many docs are in my elasticsearch index , it shows <=1000 docs only .
I guess when Grafana is firing the query for getting the docs . It is fixing the document size 1000 in query , and not using scan and scroll .
Is there some way possible, from where I can increase the size of documents which are getting retrieved from elasticsearch .
Can I write lucene query in query box and get all the records ? if yes what kind of query I need to specify in Grafana lucene query box , any example?
why do you want to scroll through more than 1000 docs? Is there not a filter / query you can specify to limit the list so you can find one you want?