WCF hosted in IIS peformance issue during load test - performance

Question

Summary
A WCF service hosted in IIS has only 1 method that takes 1 second to complete (Thread.Sleep in sample code). When 5 'clients' hammer the server with requests average response times is 1 second, 10 clients - around 2 seconds and with 20 clients the performance drops below the floor. I have tried all kind of settings and nothing helps.
I think the problem is in WCF (MS tried to make it fail-safe and put a lot of safeguards from developers): it tries to use as few threads as possible and thus the performance suffer.
Detailed description
Code
I have a very simple WCF service hosted in IIS:
using System.IO;
using System.ServiceModel;
using System.ServiceModel.Web;
using System.Threading;
namespace WCFPerf
{
[ServiceContract]
public interface IService1
{
[OperationContract]
[WebInvoke(Method = "POST",
ResponseFormat = WebMessageFormat.Xml,
RequestFormat = WebMessageFormat.Xml,
BodyStyle = WebMessageBodyStyle.Bare,
UriTemplate = "Test")]
Stream DoWork(Stream s);
}
public class Service1 : IService1
{
public Stream DoWork(Stream s)
{
Thread.Sleep(1000); // simulate work
return s;
}
}
}
The configuration file:
<configuration>
<system.web>
<compilation debug="true" targetFramework="4.0" />
<httpRuntime />
</system.web>
<system.serviceModel>
<services>
<service behaviorConfiguration="ServiceBehavior" name="WCFPerf.Service1">
<endpoint address="" behaviorConfiguration="web" binding="customBinding" bindingConfiguration="" contract="WCFPerf.IService1" />
<host>
<baseAddresses>
<add baseAddress="http://localhost:57676/Service1.svc" />
</baseAddresses>
</host>
</service>
</services>
<bindings>
<customBinding>
<binding closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00">
<webMessageEncoding />
<httpTransport manualAddressing="true" />
</binding>
</customBinding>
</bindings>
<behaviors>
<endpointBehaviors>
<behavior name="web">
<webHttp />
</behavior>
</endpointBehaviors>
<serviceBehaviors>
<behavior name="ServiceBehavior">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="true" />
<serviceThrottling maxConcurrentCalls="200" maxConcurrentSessions="200" maxConcurrentInstances="200" />
</behavior>
</serviceBehaviors>
</behaviors>
</system.serviceModel>
<system.net>
<connectionManagement>
<add address="*" maxconnection="2000" />
</connectionManagement>
</system.net>
</configuration>
The test suit:
void Main()
{
var numberOfThreads = new[] {5, 10, 20, 30};
var table = new List<ResultInfo>();
foreach (var threadNumber in numberOfThreads)
{
var tasks = new List<Task<List<double>>>();
for (var i = 0; i < threadNumber; i++)
{
tasks.Add(Task<List<double>>.Factory.StartNew(() =>
{
var results = new List<double>();
for (var j = 0; j < 5; j++)
{
results.Add(SendRequest());
}
return results;
}));
}
Task.WaitAll(tasks.ToArray());
var allResults = tasks.SelectMany(t => t.Result);
table.Add(new ResultInfo{ Threads = threadNumber, Avg = allResults.Average().ToString("F2"), Min = allResults.Min().ToString("F2"), Max = allResults.Max().ToString("F2") });
}
table.Dump();
}
public double SendRequest()
{
var stopwatch = new Stopwatch();
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(#"http://localhost:8081/");
stopwatch.Start();
var t = client.PostAsync("WCFPerf/Service1.svc/Test", new StringContent("123")).Result;
stopwatch.Stop();
return stopwatch.Elapsed.TotalSeconds;
}
}
public class ResultInfo
{
public int Threads {get;set;}
public string Avg {get;set;}
public string Min {get;set;}
public string Max {get;set;}
}
The test results are very inpredictable, but always there is a bad number:
This run I changed execution time from 1 second to 5 seconds just to demonstrate that the diff (2x times, 3x times, etc.) is relative, not absolute. Also notice that 'warm up' helps, but just a little bit.
Comments
During all my testing (and I did about 100 runs of different kind) the maximum amount of threads reported by Windows Task Manager in w3wp process was 54. I am using Core i3 CPU with 2 physical cores and 4 logical cores, 12 Gb RAM, running under Windows 8.1 and .Net 4.5. Memory print of the process always grows, but very slowly, the highest value was around 110 Mb.
dotTrace reports that about 7% of time is spent in my code and everything else - in system code.
I have tried doing the following:
In C:\Windows\Microsoft.NET\Framework\v4.0.30319\Aspnet.config:
<system.web>
<applicationPool maxConcurrentRequestsPerCPU="5000" maxConcurrentThreadsPerCPU="0" requestQueueLimit="5000"/>
</system.web>
In C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config
<system.net>
<connectionManagement>
<add address = "*" maxconnection = "400" />
</connectionManagement>
</system.net>
<system.web>
<processModel autoConfig="false" maxWorkerThreads="80" maxIoThreads="80" />
<httpRuntime minFreeThreads="10" minLocalRequestFreeThreads="10" />
Links: Web Settings Schema, Element, applicationPool maxConcurrentRequestsPerCPU, Element, Thread Throttling in IIS-hosted WCF, processModel Element, httpRuntime Element, ASP Registry Entries (IIS 6.0)
Ideas
My rationale (how it is supposed to work): while we have enough memory for every IIS/WCF creates a new thread for every request. For example when we have 40 'clients' connecting to the server - there will be about 45 threads in w3wp process (40 working threads and a few others for good measure). Because all my 'clients' wait for response before sending next request, there might be a little difference in response times (like 20-30%, not 2x or even 10x I see now).
The problems (as I see them):
There are not enough threads (only 54 threads for 70 clients)
There is some kind of queue, because already at 20 'clients' the average response time is at least twice longer than the actual logic
There is a non deterministic behavior, because results differ greatly from run to run

Answer 1

I have found an answer in kb aritcle number 2538826, the sample code also can be found in this blog post (with some additional information like how to add perf counters and write load test).
This solution will help you only if you have the described behavior (check provided links for details): number of threads you use (Thread count perf counter) does not scale as fast as your requests. The picture from KB article:
So the fix is to use some other way to create threads... and what is a better way than yo use a ThreadPool?
You need these two classes:
public class WorkerThreadPoolSynchronizer : SynchronizationContext
{
public override void Post(SendOrPostCallback d, object state)
{
// WCF almost always uses Post
ThreadPool.QueueUserWorkItem(new WaitCallback(d), state);
}
public override void Send(SendOrPostCallback d, object state)
{
// Only the peer channel in WCF uses Send
d(state);
}
}
and
[AttributeUsage(AttributeTargets.Class)]
public class WorkerThreadPoolBehaviorAttribute : Attribute, IContractBehavior
{
private static WorkerThreadPoolSynchronizer synchronizer = new WorkerThreadPoolSynchronizer();
void IContractBehavior.AddBindingParameters(
ContractDescription contractDescription,
ServiceEndpoint endpoint,
BindingParameterCollection bindingParameters)
{
}
void IContractBehavior.ApplyClientBehavior(
ContractDescription contractDescription,
ServiceEndpoint endpoint,
ClientRuntime clientRuntime)
{
}
void IContractBehavior.ApplyDispatchBehavior(
ContractDescription contractDescription,
ServiceEndpoint endpoint,
DispatchRuntime dispatchRuntime)
{
dispatchRuntime.SynchronizationContext = synchronizer;
}
void IContractBehavior.Validate(
ContractDescription contractDescription,
ServiceEndpoint endpoint)
{
}
}
And then apply the attribute to your service:
[WorkerThreadPoolBehavior] // this is what changed
public class Service1 : IService1
{
public Stream DoWork(Stream s)
{
Thread.Sleep(1000); // simulate work
return s;
}
}