Elasticsearch get matching documents after specific document id - elasticsearch

Question

When I search for documents I took the first 10 and give them to the view, if the user scrolls to the end of the list the next 10 elements should be displayed.
I know the last document id of the displayed documents, now I have to get the next 10. Basically I would perform the exact same search with an offset of 10 but it would be much better to be able to search with the same query, putting the document id of the last retrieved document to it and retrieve the matching documents after the document with that id.
Is that possible with elasticsearch?
=== UPDATE
I want to point out my issue a bit more, because it seems it is not clear enough as it is described right now. Sorry for that.
The case:
You have a kind of feed, the feed will grow every second. If a user goes to the feed he gets the most recent 10 entries, if he scrolls down he wants to get the next 10 entries.
Because the feed is growing every second, a usual offset / limit (from / size in elasticsearch) can't solve this problem, you would display already displayed entries or completely newer entries, depending on the time between first request (first 10 entries) and the request for the next entries.
The request to get the next 10 elements AFTER the already displayed entries gives the backend the id of the last entry which got displayed. The backend knows to ignore all entries before this specific one.
At the moment I'm handling this in code, I request the list with all matching entries from Elasticsearch and iterate it, this way I can do everything I want (no surprise) and extract the needed chunk of entires.
My question is: Is there is a build in solution for this issue in elasticsearch. Because solving the problem on my way is not the fastest.

Answer 1

It's an old topic, but it feels that Search After API, which is available since elasticsearch 5.0, does exactly what is needed. Provide an id of your last doc and it's timestamp, for example:
GET twitter/tweet/_search
{
"size": 10,
"query": {
"match": {
"title": "elasticsearch"
}
},
"search_after": [
1463538857,
"tweet#654323"
],
"sort": [
{
"date": "asc"
},
{
"_uid": "desc"
}
]
}
Source: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-after.html

Answer 2

You just have to create your query DSL and a pagination system with
{ "size": 10, "from" : YOUR_OFFSET }

Answer 3

If I understood your question correctly then you can use ES scrolls for such thing.
This is an example on how would one do that in java, note that it uses SearchType.SCAN
SearchRequestBuilder requestBuilder = ....getClient().prepareSearch(indices)
/**
* Set up scroll and go from there.....
* To do that need to change search type to <code>SearchType.SCAN</code>
* and set up scroll it self
* Once search type and scroll are set and search is executed, whoever
* handles the result will need to check and poll the scroll
*/
requestBuilder.setSearchType(SearchType.SCAN);
requestBuilder.setScroll(new TimeValue(SCROLL_WINDOW_IN_MILLISECONDS)); // this is in MILLISECONDS
requestBuilder.setSize(10); // this is how many hits per shrad per scroll will be returned
SearchResponse response = requestBuilder.execute().actionGet();
while (true) {
results = client.prepareSearchScroll(results.getScrollId()).setScroll(new TimeValue(60000)).execute().actionGet();
if (results.getHits().getHits().length == 0) {
break;
}
// do what you need to do w/ scroll result here
}
So every time inside of while loop you would grab 10 consequtive results until you would get all your results

Answer 4

I know this is old, but I encountered the same dilemma and I'd rather think out loud.
In that feed, you seem to care about less and less relevant documents with every request. I'm not saying a timestamp/comment count etc on purpose, in terms of ES you talk about a score that can be calculated by many factors, and what you want is to continue to search down that scoring road.
The solution that came to my mind was: If you also care about more relevant documents (like Facebook shows you on the top "X new stories available") you can first search from the beginning until you reach the first document you encountered (the one that was previously most relevant), and by adding the count of documents before to the count of documents you already displayed in the feed you can determine an estimated offset (you might get a few duplicates in race conditions, just drop them).
So what you actually need to do is search the top until you reach the first document, and then search the estimated bottom and drop everything more relevant than the last document.
This is all assuming the bulks of feeds never change, if document Y was between X and Z, it will stay there forever.
If the score is constant (unlikely as this means it will always rise for the feed to keep changing), you can also filter by everything below the score of the last document.