Elasticsearch .NET only allows me to bulk upload 80 times - elasticsearch

Question

I am using Elasticsearch.NET (5.6) on ASP.NET API (.NET 4.6) on Windows, and try to publish to elasticsearch hosted on AWS (I have tried both 5.1.1 and 6, both same behaviour).
I have the following code which bulk index the documents to Elasticsearch. Image calling the below code block many times:
var node = new System.Uri(restEndPoint);
var settings = new ConnectionSettings(node);
var lowlevelClient = new ElasticLowLevelClient(settings);
var index = indexStart + indexSuffix;
var items = new List<object>(list.Count() * 2);
foreach (var conn in list)
{
items.Add(new { index = new { _index = index, _type = "doc", _id = getId(conn) } });
items.Add(conn);
}
try
{
var indexResponse = lowlevelClient.Bulk<Stream>(items);
if (indexResponse.HttpStatusCode != 200)
{
throw new Exception(indexResponse.DebugInformation);
}
return indexResponse.HttpStatusCode;
}
catch (Exception ex)
{
ExceptionManager.LogException(ex, "Cannot publish to ES");
return null;
}
It runs fine, can publish documents to Elasticsearch, but it only can run 80 times, after 80 times, it will always get exception:
# OriginalException: System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData) in C:\Users\russ\source\elasticsearch-net-5.x\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 148
The most interesting part is that: I have tried to change the bulk size to be 200 or 30, and it turned out to be 16000 and 2400, meaning both ends up at 80 times. (Each document size is very similar)
Any ideas? Thanks

Answer 1

There is a connection limit (Also refer to comments from #RussCam under the question). So the real issue is that the Stream in the response holding the connections.
So the fix is either indexResponse.Body.Dispose (I haven't tried this one) or use VoidResponse: reportClient.BulkAsync<VoidResponse>(items); which does not require the response stream. I've tried the second and it works.