Visual Studio 2008 has a test type called a Load Test. It looks like I can just drop a collection of other tests into it, give it a test profile (say, 50 users constantly hitting the program), a running time, and let it run.
What isn't clear to me, is how to actually setup the tests it consumes. I need to launch a instance of the program to be tested (technically, a HTTP server; but not a web project or anything - just a console app) with some command line parameters, and have it remain up while the actual tests are running.
Basically, how would I setup a Load Test (or any test) in Visual Studio 2008 that would allow me to startup a project, then run a number of tests against it - in a configuration of my choosing as in Load Test.
The best way of doing it is to create a LoadTestPlugin to do what you want at various points in the load test.
The events exposed are:
LoadTestStarting
LoadTestFinished
LoadTestWarmupComplete
TestStarting
TestFinished
ThresholdExceeded
HeartBeat
LoadTestAborted
Related
I am trying to get code coverage metrics for an ASP.NET REST service (that uses a global.asax file) running in IIS. I have followed the following basic steps:
Set environment variable using "VSPerfClrEnv /globaltraceon" (then reboot computer) (I have also tried /globalsampleon)
Instrument the DLLs for code coverage using “vsinstr –coverage ” and I do this for the 5 DLLs I am interested in
Start the profiler using "VSPerfCmd /start:coverage /output:cc.coverage /CS /user:Everyone"
Start the service in IIS 10 1703, use Task Manager to note the PID for w3wp.exe
Attach the profiler to the service using "VSPerfCmd /attach:"
Run tests from Visual Studio '17 Enterprise Test Explorer against the service
Use "VSPerfCmd /detach"
Use "VSPerfCmd /shutdown"
But then when I open the cc.coverage file that was created, only one or two of the DLLs (it's not consistent) have coverage results, and the tests absolutely would have exercised code in all 5 DLLs. Am I doing something wrong or missing a step? Thank you!
According to https://blogs.msdn.microsoft.com/tfssetup/2015/08/13/steps-to-check-the-code-coverage-of-a-web-application-via-command-line/, you should issue a iisreset /STOP command before vsperfcmd /shutdown, after vsperfcmd /detach. Not sure if this is the root cause of your problem or not, but it could be considering that it seems like a buffer flush is not occurring faithfully in your scenario. The buffer flush for each module is required in order to get the coverage data out of the session - if all the things aren't properly shutdown, then this can cause buffers to fail to flush their data.
Is there an elegant way to setup StartUp and Cleanup methods to run for a Load Test in Visual Studio ?
I have a Controller - Agents system for load testing and I need to find a way to run a method before the Test runs and after it finishes but only in the controller.
Plugins can be written for load tests. Basically the plugin provides code to be run during the initialisation of the test run. That code can attach event handlers for the following and hence the test can be manipulated in many ways. (List of events copied from here.)
LoadTestStarting, LoadTestFinished, LoadTestWarmupComplete, TestStarting, TestFinishedTestSelected, ThresholdExceeded, HeartBeat, and LoadTestAborted.
See this Microsoft page (and the pages it links to) for more details on writing load test plugins.
Visual Studio tests also use a ".testsettings" file to specify several aspects of the test. This file allows set-up and clean-up scripts to be specified. These can be seen in the image below "Configure the test controller and roles for remote data collection and execution" and in the "Test Settings: Setup and Cleanup Scripts" parts of this Microsoft page.
Not part of the question, but added for completeness. Plugins can also be written for the web tests. Two forms are provided, web test request plugins and http://msdn.microsoft.com/en-us/library/ms243191(v=vs.110).aspx plugins.
See also this Stackoverflow question VS2010 Load Testing: How can I perform custom action that is run once prior to each load test
I have an ordered test that contains 4 web tests. The problem is that Visual Studio seems to have a problem loading the test results for an individual test.
I included a screenshot to supplement for the 1000 words. :)
As you may see the 3rd test, called intuitively Webservice03, failed. The screenshot above was taken after I double-clicked on the corresponding line to open the detailed information about that test run. The panels with requests and other useful information are empty.
The issue is quite annoying, because the test is part of a bigger test suite that runs periodically, so when it fails, I have to manually disable the setup and cleanup scripts from the test configuration, run them manually, then run each of the tests manually, instead of just comfortably opening the test results.
Did anyone experienced this problem?
Thanks.
Version details:
Microsoft Visual Studio 2010 Ultimate
Version 10.0.40219.1 SP1Rel
Microsoft .NET Framework
Version 4.0.30319 SP1Rel
I had the same problem and tryed to clarify it via Microsoft forum:
Web test details are not displayed when ran from an ordered test
According to the answer you can not have detailed test result for a web test run as a part of an ordered test.
Here is an alternative to Ordered Tests:
Add the web tests to a load test
In the test mix choose 'Test mix based on sequential test order'. This will achieve the same thing as an Ordered Test.
In the load test run settings, make sure 'Save Log on Test Failure' is set to True:
Set a Constant Load Pattern of 1 user
At the end of the "load" test you will be able to see the number of failures and on the Tables view you can click on the error count which takes you to a dialog that lets you browse the individual web test failures.
Is there an alternate way to test VS 2010 extension projects? Right now I have to spin a new instance of VS 2010 and wait. This takes couple of minutes and would love to know if there's another way of doing it quickly.
Depending on the project type you used to create your package (empty VSIX container or one of the extension projects from the SDK like Editor Margin, Text Adornment, ViewPort Adornment, etc) It will ask you if you want to create unit/integration tests.
The integration tests provide all the methods simulate the shell and allow you to test without actually spinning up the UI. Of course this depends if you need to do any visual verification, but if you're testing package load, service calls, pretty much anything else it's great.
Let me know how it goes.
is there a way? do I have to wait for building every time I start the test? I want to build from visual studio not from test
thanks
Any time your code changes and you run your test it is going to do a build... so technically you can run your test over and over again and they will only build the first time, but once you run your test why would you run them again without making a code change?
Couple of things that I use that make your test run faster are:
Check the box for "Only build startup projects and dependencies on Run", located Options->Projects and Solution->Build and Run.
Learn the short cut keys
a. "Ctrl+R, T" Runs test in current context, so if your cursor is inside a test method it will only run that test, but when you do it inside of a non test class it will run all of your test.
b. "Crtl+R, Ctrl+T" Debug test same except debug.
c. Others can be found here, those are 2008 if you need to reference others you can find them via google.
Make sure your test are not calling the database or other time intensive resources, use mocking and stubbing.
Run only small sets of test, ie if I am working in a service class I run only the service class test.
Edit: Reading your question again if you want to build and not from a test you can just go to the menu and click Build->Build Solution or press F6. Also it would be helpful if you indicated which version of visual studio you are using because 2010 is different in the sense that you have to click refresh. Either way are you able to clarify?
This is an old question, but I keep seeing people ask it and the issue is still true in VS2017, and it's also true of other test frameworks (Xunit, etc) run from within VS.
I don't know how to make VS stop building all the time. But I do know how to circumvent the compile - run your tests from a console runner, not from within VS. If you're using ReSharper, it has one.
If you aren't using ReSharper, for MSTest, you can start here. https://msdn.microsoft.com/en-us/library/ms182489.aspx
If you aren't using ReSharper, for XUnit, you can start here. https://xunit.github.io/docs/getting-started-desktop.html#add-xunit-runner-ref
Any changes to source code cause compilation, because in order to run tests VS needs up to date DLL with tests.
If you have already compiled project then you can run test multiple times without compilation.
PS: I run MSTest using TestDriven.NET as for me it is faster.