Visual Studio Load Testing framework vs Console App - strange results - visual-studio

Question

I've been using the Visual Studio Load Testing framework to load test a web service.
If I keep my test simple and use a constant load pattern of 1 user from my local machine, I am able to generate 'x' requests per second.
Alternatively, if I use console app that runs the same test, making synchronous calls to the web service, the console app is generating twice the load that I get using the Visual Studio Load Testing framework.
The same is true if I try to scale my load tests to use multiple test agents (8 cores) - the VS framework does not generate near the amount of load as a console app running multiple instances.
These are the two different unit tests I am using to generate load:
//Unit test used for load testing
[TestMethod]
public void HappyReturnCase_Test()
{
HttpWebRequest req = WebRequest.Create("http://myurl") as HttpWebRequest;
req.Method = "GET";
req.GetResponse().Close();
}
//Console app version
private static void Main(string[] args)
{
for (int i = 0; i < 200000; i++)
{
HttpWebRequest req = WebRequest.Create("http://myurl") as HttpWebRequest;
req.Method = "GET";
req.GetResponse().Close();
}
}
Can any explain to me why I might be seeing this kind of behaviour?
Thanks in advance.
Kevin

Answer 1

There are a few things that comes into mind when dealing with load testing in Visual Studio.
What kind of data capture are you using?
Is there instrumentation for Code Coverage?
You also need to keep in mind that in Visual Studio you're tunning the code within a test framework that does way more than simply call your code. It analyse the result, check for exceptions, logs all the data to and fro the code being called, generate reports with the captured data...
And all this stuff that comes out of the box, and that we think is "free" does have its toll on performance of the said test.
While the number of request per seconds, as you put, is lower in VS than its app counterpart, you also needs to weigh in all the other stuff the testing framework is doing for you.

Answer 2

With the console application, there's no throttling of the requests--you're just going full-bore from the client. With the VS Load Tests, there are other factors that limit the number of requests (like the total number of iterations).
For example, if you have test iterations enabled, you'll spread them throughout the duration of the load test. Generally, that will bring your test frequency down. If you have 100 test iterations set, and you're running your test for an hour, and each test takes 30 seconds, you'll run 20 fewer tests because of it (evenly spread throughout the hour).
There is also a callback model going on here. The load tests support a load test plugin model and a request plugin model, so the unit test will yield to the load test runner which may be swapping out to a new virtual user; even if the test is set for 1 virtual user, it may not be the same virtual user throughout the test. You'll be reporting and logging, plus you may be starting up a new application host "container" for your unit test, and a few other activities. Even if that's not the case, you're not spending all your time in the context of the unit test.
Even inside the unit test, there are other methods running, like ClassInitialize, TestInitialize, setting timers, etc. Plus, there is a thread pool being used, even if only for one user. See http://blogs.msdn.com/b/billbar/archive/2007/10/12/features-and-behavior-of-load-tests-containing-unit-tests-in-vsts-2008.aspx for some more information on how unit tests are run by the load test runner. Even if you data bound that unit test to run 100 rows of data, it probably wouldn't run as quickly as the loop that you'd written, but it's got the benefit of easily configuring extra work and running multiple unit tests together.
You may want to take a read through of the performance testing quick reference guide at http://vsptqrg.codeplex.com/.
Now, setting the constant load to 1 user doesn't take advantage of any of the benefits of the load rig--you've taken on the overhead of the thread pool without running multiple users. You'd expect to start seeing benefits if you increase the number of users and let the VS load test manage that context switching for you. Another benefit is creating a test mix that you can easily alter, plus collecting the perfmon statistics, applying threshold rules, etc. You're not really doing any of those in the console app.