PushStreamContent stream does not flush under load - asp.net-web-api

Question

I am using PushStreamContent to keep a persistent connection to each client. Pushing short heartbeat messages to each client stream every 20 seconds works great with 100 clients, but at about 200 clients, the client first starts receiving it a few seconds delayed, then it doesn't show up at all.
My controller code is
// Based loosely on https://aspnetwebstack.codeplex.com/discussions/359056
// and http://blogs.msdn.com/b/henrikn/archive/2012/04/23/using-cookies-with-asp-net-web-api.aspx
public class LiveController : ApiController
{
public HttpResponseMessage Get(HttpRequestMessage request)
{
if (_timer == null)
{
// 20 second timer
_timer = new Timer(TimerCallback, this, 20000, 20000);
}
// Get '?clientid=xxx'
HttpResponseMessage response = request.CreateResponse();
var kvp = request.GetQueryNameValuePairs().Where(q => q.Key.ToLower() == "clientid").FirstOrDefault();
string clientId = kvp.Value;
HttpContext.Current.Response.ClientDisconnectedToken.Register(
delegate(object obj)
{
// Client has cleanly disconnected
var disconnectedClientId = (string)obj;
CloseStreamFor(disconnectedClientId);
}
, clientId);
response.Content = new PushStreamContent(
delegate(Stream stream, HttpContent content, TransportContext context)
{
SaveStreamFor(clientId, stream);
}
, "text/event-stream");
return response;
}
private static void CloseStreamFor(string clientId)
{
Stream oldStream;
_streams.TryRemove(clientId, out oldStream);
if (oldStream != null)
oldStream.Close();
}
private static void SaveStreamFor(string clientId, Stream stream)
{
_streams.TryAdd(clientId, stream);
}
private static void TimerCallback(object obj)
{
DateTime start = DateTime.Now;
// Disable timer
_timer.Change(Timeout.Infinite, Timeout.Infinite);
// Every 20 seconds, send a heartbeat to each client
var recipients = _streams.ToArray();
foreach (var kvp in recipients)
{
string clientId = kvp.Key;
var stream = kvp.Value;
try
{
// ***
// Adding this Trace statement and running in debugger caused
// heartbeats to be reliably flushed!
// ***
Trace.WriteLine(string.Format("** {0}: Timercallback: {1}", DateTime.Now.ToString("G"), clientId));
WriteHeartBeat(stream);
}
catch (Exception ex)
{
CloseStreamFor(clientId);
}
}
// Trace... (this trace statement had no effect)
_timer.Change(20000, 20000); // re-enable timer
}
private static void WriteHeartBeat(Stream stream)
{
WriteStream(stream, "event:heartbeat\ndata:-\n\n");
}
private static void WriteStream(Stream stream, string data)
{
byte[] arr = Encoding.ASCII.GetBytes(data);
stream.Write(arr, 0, arr.Length);
stream.Flush();
}
private static readonly ConcurrentDictionary<string, Stream> _streams = new ConcurrentDictionary<string, Stream>();
private static Timer _timer;
}
Could there be some ASP.NET or IIS setting that affects this? I am running on Windows Server 2008 R2.
UPDATE:
Heartbeats are reliably sent if 1) the Trace.WriteLine statement is added, 2) Visual Studio 2013 debugger is attached and debugging and capturing the Trace.WriteLines).
Both of these are necessary; if the Trace.WriteLine is removed, running under the debugger has no effect. And if the Trace.WriteLine is there but the program is not running under the debugger (instead SysInternals' DbgView is showing the trace messages), the heartbeats are unreliable.
UPDATE 2:
Two support incidents with Microsoft later, here are the conclusions:
1) The delays with 200 clients were resolved by using a business class Internet connection instead of a Home connection
2) whether the debugger is attached or not really doesn't make any difference;
3) The following two additions to web.config are required to ensure heartbeats are sent timely, and failed heartbeats due to client disconnecting "uncleanly" (e.g. by unplugging computer rather than normal closing of program which cleanly issues TCP RST) trigger a timely ClientDisconnected callback as well:
<httpRuntime executionTimeout="5" />
<serverRuntime appConcurrentRequestLimit="50000" uploadReadAheadSize="1" frequentHitThreshold="2147483647" />