Elastic Search, from-start doesn't work correctly - elasticsearch

Question

I am going to fetch some records between two dates using Elastic Search query.
First, I check the number of records between two dates to know whether it is greater than 10000 or not. If it is, i try to fetch them 10000 by 10000.
//get count
var result_count = client.Count<TelegramMessageStructure>(s => s
.AllTypes()
.AllIndices()
.Query(q => q
.DateRange(r => r
.Field(f => f.messageDate)
.GreaterThanOrEquals("2018-06-03 00:00:00.000")
.LessThan("2018-06-03 00:59:00.000")
)
)
);
long count = result_count.Count; //count = 27000
it returns 27000. So I want to fetch them 10000 by 10000. I use this query to do that:
int MaxMessageCountPerQuery=10000;
for (int i = 0; i < count; i += MaxMessageCountPerQuery)
{
client = new ElasticClient(connectionSettings);
// No change whether the client is renewed or not
var result = client.Search<TelegramMessageStructure>(s => s
.AllTypes()
.AllIndices()
.MatchAll()
.From(i)
.Size(MaxMessageCountPerQuery)
.Sort(ss => ss.Ascending(p => p.id))
.Query(q => q
.DateRange(r => r
.Field(f => f.messageDate)
.GreaterThanOrEquals("2018-06-03 00:00:00.000")
.LessThan("2018-06-03 00:59:00.000")
)
)
);
//when i=0, result.documents contains 10000 records otherwise it has 0
}
In The first round, when i=0, result.documents contains 10000 records otherwise it contains 0 records.
What is wrong with this?

Answer 1

Based on this link:
scroll in elastic net-api
Your codes should contains below steps:
1- Search with all parameters that you need plus .Scroll("5m") (I assume from(0) and size(10000) is set too and save response in result variable)
2- Now you have first 10000 records (in result.Documents)
3- For receive more records, you should use ScrollId param to get more results. (Each call of bellow code give you next 10000 records)
var result_new = client.Scroll<TelegramMessageStructure>("10m", result.ScrollId);
In fact, your codes should be like this:
int MaxMessageCountPerQuery=10000;
client = new ElasticClient(connectionSettings);
// No change whether the client is renewed or not
var result = client.Search<TelegramMessageStructure>(s => s
.AllTypes()
.AllIndices()
.MatchAll()
.From(i)
.Size(MaxMessageCountPerQuery)
.Sort(ss => ss.Ascending(p => p.id))
.Query(q => q
.DateRange(r => r
.Field(f => f.messageDate)
.GreaterThanOrEquals("2018-06-03 00:00:00.000")
.LessThan("2018-06-03 00:59:00.000")
)
)
.Scroll("5m") // Add this parameter
);
// TODO some code:
// save and use result.Documents
for (int i = 0; i < result.Total; i += MaxMessageCountPerQuery)
{
var result_new = client.Scroll<TelegramMessageStructure>("10m", result.ScrollId); // Add this line to loop , Each loop you can get next 10000 record.
// TODO some code:
// save and use result_new.Documents
}

Answer 2

Elasticsearch has a default index.max_result_window = 10000 and it's well explained at
https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html
To understand why deep paging is problematic, let’s imagine that we
are searching within a single index with five primary shards. When we
request the first page of results (results 1 to 10), each shard
produces its own top 10 results and returns them to the coordinating
node, which then sorts all 50 results in order to select the overall
top 10.
Now imagine that we ask for page 1,000—results 10,001 to 10,010.
Everything works in the same way except that each shard has to produce
its top 10,010 results. The coordinating node then sorts through all
50,050 results and discards 50,040 of them!
You can see that, in a distributed system, the cost of sorting results
grows exponentially the deeper we page. There is a good reason that
web search engines don’t return more than 1,000 results for any query.