I am new to Elastic Search. Is there any way to get all the search results for a search keyword? Elastic Search is limited to 10 or else we can set the size but we need to get the size??
Yes, the default number of search results is 10.
You need to set the size parameter on the query.
I don't think you an say "all results", though, there must always be a size limit.
If you use the JAVA API you can simple get the total hit number from the SearchResponse
SearchRequestBuilder srb = ..
SearchResponse sr = srb.execute().actionGet();
long totalHits = sr.getHits().getTotalHits();
You can do this in couple of steps using some code
Fix a size say 1000 and get all 1000 records.
Identify from hits.total whether size is smaller than 1000. (if small then you got all the records :) )
Otherwise use from and size to provide 1001 in from and total as size from previous query to get full result.
Related
For some reason I cannot set index.max_result_window from https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#dynamic-index-settings 1
I have tried different options like
PUT /articles/_settings { "max_result_window" : 25000 }
and I can see that it was set when I run
GET /articles/_settings
but still, when I run any search like match_all I get only 10000 results back.
I'm aware about all memory complications associated with it but I have pretty small index less than 25K documents so I would like to enable it
Any suggestions what am I doing wrong?
max_result_window will only set the maximum number of hits you can request from a given query, it doesn't change the default number of hits (which is 10000)
So in your case, you now need to specify ?size=25000 in your query (or "size": 25000 in your JSON query) if you want to get all the possible results back.
Or you can go 10 by 10 (with size=10) until you get to from=24990&size=10.
Is there a reason that a generic search on a index only returns 10 items?
If so how can I set the limit higher?
http://localhost:9200/journal/_search
10 is just the default number of search hits that will get returned. Take a look here https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html
There you find a lot of parameters to append to your URI Search. You need to set the size parameter like so:
http://localhost:9200/journal/_search?size=100
So when querying ElasticSearch, I know you can constrain the size with the "size" parameter. By default, it's 10,000. I was wondering how to know what's the max (if it has been changed from 10,000)?
I have tried "/index/_settings" in hopes of finding the max_window_size, but couldn't find anything. I'm not necessarily sure if that's because it doesn't have a limit at all, or if I am doing something wrong.
So to rephrase my question: I basically want to know how to find the max size when trying to query "size: xx" to an elastic search server. If the size is 10,000/the default, then I want to know where I can find this number.
Any tips or guidance?
If the value isn't specified on the index itself (in _settings where you were looking), then it is 10000. You can change this setting only on the index itself as far as I know. To automatically apply it to new indices you can use an index template.
It appears to be an oversight by the devs to me, if you use rolling indices by date for example then there is no single index for you to query modifications to the value from (sure you could guess one). I think you just have to make sure to match your query code assumptions to your index template. In my opinion there should be a way to just ask for max results possible without needing to know that value beforehand.
You are correct in that elastic search default max query size is 10000. The way to get more is to use the "scroll" api:
https://www.elastic.co/guide/en/elasticsearch/reference/7.3/search-request-body.html#request-body-search-scroll
This essentially uses pagination to split your result into user defined segments and allows you to "scroll" to the next one using a "Scroll_id" that's returned from the initial query.
For a given date range in the query and with a search_after parameter I am able to successfully extract the relevant results. How do I figure out if I am at the end of the search results for the given date range and I dont have to continue querying with the search_after parameter.
There is a pretty cool "trick" that does not involve any additional queries or knowledge of the total number of results:
Say you have a page size of 20. Instead of asking elasticsearch for 20 results, ask it for 21.
If you got 21 results back, only use the first 20 of them. But you now know that the next query will have at least one more result (If you use the sort values of the 20th result for the search_after parameter, not the 21st!).
If you get 20 results or fewer, there will be no additional results.
This github issue gives some more details into why elasticsearch does not have this feature out of the box: https://github.com/elastic/elasticsearch/issues/22364
You can either keep querying until it starts returning zero results, or it does return the total, so you could keep a track of how many you've already retrieved and stop searching once you've met the total. (I do a combination of both)
I am having an issue where i want to reduce the number of results from Elastic search to 1,000 no matter how many matching results are there matching, but this should not affect the ranking and scoring.
I was trying terminate_after, but that seems to just tell the elastic search to just get the top N results without considering the scores. Correct me if am wrong.
Any help on this?
EDIT:
I am already using pagination. So, using Size in From/Size will only affect the size of current page. But i want to limit the size of total results to 1,000 and then pagination on that.
How about using From/Size in order to return the requirement number of results:
GET /_search
{
"from" : 0, "size" : 1000,
"query" : {
//your query
}
}
You can just specify the size as an parameter.
GET /_search?size=1000
{
"query" : {
//your query
}
}
I know this question aged a little since it was asked, but i stumbled over this and i am surprised no one could give the correct answer.
Elasticsearch indices have an index module called max_result_window. You can find it in the documentation under dynamic index settings.
index.max_result_window
The maximum value of from + size for searches to this index. Defaults to 10000. Search requests take heap memory and time proportional to from + size and this limits that memory. See Scroll or Search After for a more efficient alternative to raising this.
So basically instead of limiting from or size (or a combination of those), you set max_result_window to 1000 and ES will only return a maximum of 1000 hits per request.
If you are using an index definition in a separate JSON file to create your index, you can set this value there under yourindexname.settings.index.max_result_window.
I hope this helps the folks still looking for a solution to this problem!
did you try with
terminate_after
The maximum number of documents to collect for each shard, upon reaching which the query execution will terminate early. If set, the response will have a boolean field terminated_early to indicate whether the query execution has actually terminated_early. Defaults to no terminate_after.