How to stream many documents from Elasticsearch index (Scroll, Sliced Scroll) - elasticsearch

Question

I am looking for a way to stream all (~ 10^6+) documents via .NET Nest Client.
I want to boost performance by using parallel async requests. (e.g ActionBlock, Task.WhenAll())
old fashioned without boosting:
var objects = new List<object>();
var searchResponse = await elasticClient.SearchAsync<object>(
new SearchRequest<object>("myIndex")
{
Size = 7000,
Query = new BoolQuery
{
//...
},
// why here and in scroll itself?
Scroll = "2s",
Sort = new List<ISort>
{
//..
}
});
while (searchResponse.Documents.Any())
{
objects.AddRange(searchResponse.Documents);
searchResponse = await elasticClient.ScrollAsync<object>("2s", searchResponse.ScrollId).ConfigureAwait(false);
}
return objects;
then a try using parallel sliced scroll
var result = new ConcurrentBag<object>();
var tasks = Enumerable.Range(0, 4).Select(
id => new SearchRequest<object>("myIndex")
{
// hast to be lower than 1024?
Size = 1000,
Query = new BoolQuery
{
//...
},
// why here and in scroll itself?
Scroll = "2s",
Sort = new List<ISort>
{
//..
}
}).Select(
async searchRequest =>
{
var searchResponse = await elasticClient.SearchAsync<object>(searchRequest).ConfigureAwait(false);
while (searchResponse.Documents.Any())
{
searchResponse.Documents.Each(result.Add);
searchResponse = await elasticClient.ScrollAsync<object>("2s", searchResponse.ScrollId).ConfigureAwait(false);
}
// good idea right?
//await elasticClient.ClearScrollAsync(x => x.ScrollId(searchResponse.ScrollId)).ConfigureAwait(false);
});
await Task.WhenAll(tasks).PreserveAllExceptions().ConfigureAwait(false);
return result.ToList();
But this only gives me a fraction of the actual available documents.
More over slices scroll is limited to 1024 documents per slice.
I was not able to increase this value to 7000:
{
"myIndex_template": {
"settings": {
"index": {
"number_of_shards": "1",
"number_of_replicas": "0",
"max_slices_per_scroll": "10000"
}
}
}
}