All -- I am using Visual studio 2013 ultimate edition to run my load tests. I have more than one scenario configured on my load test. Each of the scenario run a different set of unit tests. Is is there a way to selectively run the scenarios ? In other words if there are more than one scenario configured, can I just run one of them and not run the rest ?
If the answer is 'NO', any ideas/suggestions would be very helpful.
Thanks in advance.
Two ways I have found:
One way is to set the user load to a constant zero in the unwanted scenarios.
Another way is to set the "Delay start time" of the unwanted scenarios to longer that the total test duration. This option only makes sense when "User test iterations" is false.
Related
We have a legacy code base with some unit tests still coupled by global state (static variables etc.). To find them, I need to know the exact execution order the tests ran when I ran them via VS test explorer.
Is there a way to log execution order in VS test explorer?
I know that vstest.console does output/log the execution order but then I need to narrow down the subset of tests which is very inconvenient with vstest. In VS test runner, i can just run subsets.
I also know that there are other tools (like resharper test runner) but this is also not an option.
Not out of the box that I am aware of, but I guess you can find ways to log it, like have a global counter that you increment in a before method.
But, since the order of the tests are non deterministic by design I am not sure how valuable that information is.
If you know what test are coupled then run those as Ordered Tests or use a different test traits to control how you execute those test. The best option is of course to break their external dependencies.
I remember running a set of test cases and generating a report (.html) using Selenium and also TestNG. I would like to know how to do this for a set of .webtest files that I recorded using Visual Studio 2013. I can run them individually but how about one by one automatically and getting a final result that says which tests failed.
Thank you.
Create a load test in Visual studio, add all the wanted tests, set a constant user load of 1. Then all the tests will run one after another. The load tests results will include the pass/fail counts. Setting a higher use load may be useful, depending on your application, and the whole test will run faster.
The "Details" page will show all the executed tests on a time-graph, a tick box allows the failed tests to be shown in red.
If you set each web test to stop on the first error then the error page in the results should name precisely the tests that failed, and the reason.
The "Summary" page will give a summary of the results, it may be copied (use right-click context menu) into Word document and then edited as needed.
Which test is executed and when is chosen according to the "Test mix model" by Visual Studio, some of these models are based on random numbers. With a user load of one it may be best to select the "Test mix based on sequential test order". The "Test mix percentage based on the number of tests started" should also work but I can imagine occasionally getting a test run where one test is executed twice and another not executed. If more than one virtual user is used then the "Test mix based on sequential test order" is probably the wrong choice. See here for more on test mix models.
I need to generate some delays to make a load test more realistic. In Visual Studio I have found 2 methods that delay execution of a transaction in a loadtest:
Convert to code
or
Set "Think Time" to first request in the transaction
What would be the best practice here. I am not completely sure that Visual Studio executes these test in sequencional manner, so 2nd option might return false results.
On the other hand, I would loose the ability to change settings from Visual Studios UI, if I generate code from a recorded test. And that increases time spent editing the test quite a bit.
Is there a better way than any of my methods?
I believe that the second way is more safier.
Just have in mind that the Think Time is aplied to each Test Run not to each contained Test. So, if you have a Load Test which contains more than one Tests in the Test Mix the Think Time will apllied at the end of the test run (when all included tests were completed).
Also, if you have more than one virtual user, since it's a load test, Think Time is applied to each user seperately. If, for example, you have 100 user then they will concurently start the tests but they may not finish them at the same time. Each user will wait according to the Think Time before starting the second test without waiting for the others to complete their first test.
I have an ordered test that contains 4 web tests. The problem is that Visual Studio seems to have a problem loading the test results for an individual test.
I included a screenshot to supplement for the 1000 words. :)
As you may see the 3rd test, called intuitively Webservice03, failed. The screenshot above was taken after I double-clicked on the corresponding line to open the detailed information about that test run. The panels with requests and other useful information are empty.
The issue is quite annoying, because the test is part of a bigger test suite that runs periodically, so when it fails, I have to manually disable the setup and cleanup scripts from the test configuration, run them manually, then run each of the tests manually, instead of just comfortably opening the test results.
Did anyone experienced this problem?
Thanks.
Version details:
Microsoft Visual Studio 2010 Ultimate
Version 10.0.40219.1 SP1Rel
Microsoft .NET Framework
Version 4.0.30319 SP1Rel
I had the same problem and tryed to clarify it via Microsoft forum:
Web test details are not displayed when ran from an ordered test
According to the answer you can not have detailed test result for a web test run as a part of an ordered test.
Here is an alternative to Ordered Tests:
Add the web tests to a load test
In the test mix choose 'Test mix based on sequential test order'. This will achieve the same thing as an Ordered Test.
In the load test run settings, make sure 'Save Log on Test Failure' is set to True:
Set a Constant Load Pattern of 1 user
At the end of the "load" test you will be able to see the number of failures and on the Tables view you can click on the error count which takes you to a dialog that lets you browse the individual web test failures.
is there a way? do I have to wait for building every time I start the test? I want to build from visual studio not from test
thanks
Any time your code changes and you run your test it is going to do a build... so technically you can run your test over and over again and they will only build the first time, but once you run your test why would you run them again without making a code change?
Couple of things that I use that make your test run faster are:
Check the box for "Only build startup projects and dependencies on Run", located Options->Projects and Solution->Build and Run.
Learn the short cut keys
a. "Ctrl+R, T" Runs test in current context, so if your cursor is inside a test method it will only run that test, but when you do it inside of a non test class it will run all of your test.
b. "Crtl+R, Ctrl+T" Debug test same except debug.
c. Others can be found here, those are 2008 if you need to reference others you can find them via google.
Make sure your test are not calling the database or other time intensive resources, use mocking and stubbing.
Run only small sets of test, ie if I am working in a service class I run only the service class test.
Edit: Reading your question again if you want to build and not from a test you can just go to the menu and click Build->Build Solution or press F6. Also it would be helpful if you indicated which version of visual studio you are using because 2010 is different in the sense that you have to click refresh. Either way are you able to clarify?
This is an old question, but I keep seeing people ask it and the issue is still true in VS2017, and it's also true of other test frameworks (Xunit, etc) run from within VS.
I don't know how to make VS stop building all the time. But I do know how to circumvent the compile - run your tests from a console runner, not from within VS. If you're using ReSharper, it has one.
If you aren't using ReSharper, for MSTest, you can start here. https://msdn.microsoft.com/en-us/library/ms182489.aspx
If you aren't using ReSharper, for XUnit, you can start here. https://xunit.github.io/docs/getting-started-desktop.html#add-xunit-runner-ref
Any changes to source code cause compilation, because in order to run tests VS needs up to date DLL with tests.
If you have already compiled project then you can run test multiple times without compilation.
PS: I run MSTest using TestDriven.NET as for me it is faster.