How to use SearchAfter API correctly? - elasticsearch

Question

I recently work with elasticsearch and I have such a question. I have a million documents in index and I wanna to get more than 10_000. For this I can use scroll API or SearchAfter API. I understood how does scroll api works but I have some problem with SearchAfter.
Here is my SearchSourceBuilder method:
public SearchRequest buildRequest(SearchDistanceParameters args) {
final SearchSourceBuilder searchSourceBuilder = prepareSearchSourceBuilder(args);
final SearchRequest searchRequest = new SearchRequest();
return searchRequest.source(searchSourceBuilder);
}
private SearchSourceBuilder prepareSearchSourceBuilder(SearchDistanceParameters searchDistanceParameters) {
final FieldSortBuilder fieldSortBuilder = new FieldSortBuilder("_id").order(SortOrder.ASC);
final SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
final GeoDistanceQueryBuilder geoDistanceQueryBuilder = geoDistanceQuery(GeoLocationModelFieldName.LOCATION.name().toLowerCase());
geoDistanceQueryBuilder.point(searchDistanceParameters.getLatitude(), searchDistanceParameters.getLongitude());
geoDistanceQueryBuilder.distance(searchDistanceParameters.getDistance(), DistanceUnit.KILOMETERS);
searchSourceBuilder.query(geoDistanceQueryBuilder);
searchSourceBuilder.sort(fieldSortBuilder);
searchSourceBuilder.searchAfter();
return searchSourceBuilder;
}
Here I do sort before searchAfter() as mention in SearchAfter API doc.
Here I am sending my request to ElasticSearch:
public SearchResponse sendRequestToElastic(SearchDistanceParameters args) throws IOException {
SearchRequest searchRequest = searchByDistanceRequestBuilder.buildRequest(args);
return elasticDao.search(searchRequest, RequestOptions.DEFAULT); // standard RestHighLevelClient.search method inside elasticDao.
}
And finally I am trying to get my objects from SearchResponse:
public List<GeoPointsFromElasticSearchResponse> searchByDistance(SearchDistanceParameters searchDistanceParameters) {
final SearchResponse searchResponse = searchRepository.searchByDistance(searchDistanceParameters);
return getGeoPointsFromElasticSearchResponses(searchResponse);
}
private List<GeoPointsFromElasticSearchResponse> getGeoPointsFromElasticSearchResponses(SearchResponse searchResponse) {
SearchHit[] hits = searchResponse.getHits().getHits();
return Arrays.stream(hits)
.map(hit -> {
final GeoPointsFromElasticSearchResponse geoPointsFromElasticSearchResponse = new GeoPointsFromElasticSearchResponse();
final Map<String, Object> sourceMap = hit.getSourceAsMap();
final Map map = (Map) sourceMap.get(GeoLocationModelFieldName.LOCATION.name().toLowerCase());
geoPointsFromElasticSearchResponse.setLatitude((Double) map.get("lat"));
geoPointsFromElasticSearchResponse.setLongitude((Double) map.get("lon"));
log.info("Sorted hits: {}", hit.getSortValues());
return geoPointsFromElasticSearchResponse;
}).collect(Collectors.toList());
}
But I've got only 10_000 objects. It seems I am doing something wrong in last part. What am I doing wrong? How to use SearchAfter api in java correctly?

Answer 1

Well the search API wont return all the documents in one request, the behaviour is similar as pagination.
You have to pass argument to search after:
https://www.elastic.co/guide/en/elasticsearch/reference/6.7/search-request-search-after.html
According to the constructor : searchSourceBuilder.searchAfter(new Object[]{sortAfterValue});
The value you want to set is the one returned by the first search request (hits => getAt(lastIndex) => getSortValues())