The MSTest allows to run tests in specify assemblies to test through /testcontainer or metalist through /testmetadata; But is there a way to automatically run all tests in a solution, whithout specifing any name of dlls/meta files? Thanks.
It's not what you want to hear, but the answer is no, by default you are limited to the /testcontainer or /testmetadata options for specifying which tests to run.
You could easily run MSTest multiple times for each DLL in your project which contains tests. You could even run a process that analyzes your DLLs for tests and build up a list of DLLs which contain tests and then run a script that calls mstest for each of those DLLs.
Related
I have a suite of XUnit Tests in a .NET solution that is built in a TeamCity CI environment. The tests in question generate html templates. In our development environment, they write the results to the Visual Studio console using ITestOutputHelper. In Team City, the output is making the build artifacts way too big, and I want to find a way of suppressing the output as it is not needed in the build.
I imagine this will be achieved using an XUnit console runner command line option (but open to suggestions). I have found this list of command line options fsharp/FAKE#759, but it doesn't include one to suppress output. Is there any other (configuration-based) way of achieving what I want? (There are a lot of test classes and I don't want to have to refactor them all).
We have a TFS build definition that kicks off NUnit tests tagged with the 'Regression' test category. This uses the NUnit console runners annotation of
where cat = 'Regression'
However, we have multiple different environments where some tests will fail in one environment, they will pass in the other. We have not made much use of the Playlist feature, because I can not find a way to target a playlist when running remotely on TFS. Does anyone know how this can be done? Thanks!
Unfortunately, there is no way of specifying this/playlist in the TFS Build Definition for now. A related uservoice:
Allow test playlists to be used by TFS build servers
https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/3853614-allow-test-playlists-to-be-used-by-tfs-build-serve
As a workaround, you could use .orderedtest instead of .playlist.
Ordered tests can be created and edited in VS2013 and later. The format is otherwise similar to .playlist but it contains links to test GUIDs so its more complicated to modify programmatically.
TFS is able to run orderedtest in build pipeline, how to achieve this you could refer below links:
TFS - order of automata tests to execute
How to use Vnext build: ordered tests, distribute test, collect results
I'm new to load testing in Visual Studio/MSTest, and I created a new Load Test recently to validate some high-traffic scenarios for a WCF service. I want to add this to the tests project for the service, but I don't want the test to be executed whenever I "Run All Tests in Solution" nor as part of our Continuous Integration build-verification process because a) it takes 5 minutes to run, and b) the service call that it is testing generates many thousands of email messages. Basically, I'd like to do the equivalent of adding the [Ignore] attribute to a unit test so that the load test is only executed when I explicitly choose to run it.
This MSDN Article ("How to: Disable and Enable Tests") suggests that the only to disable the test is to use Test Lists (.vsmdi files), but I don't have much experience with them, they seem like a hassle to manage, I don't want to have to modify our CI Build Definition, and this blog post says that Test Lists are deprecated in VS2012. Any other ideas?
Edit: I accepted Mauricio's answer, which was to put the load tests into a separate project and maintain separate solutions, one with the load tests and one without. This enables you to run the (faster-running) unit tests during development and also include the (slower-running) load tests during build verification without using test lists.
This should not be an issue for your CI Build Definition. Why?
To run unit tests as part of your build process you need to configure the build definition to point to a test container (usually a .dll file containint your test classes and methods). Load tests do not work this way, they are defined within .loadtest files (which are just xml files) that are consumed by the MSTest engine.
If you do not make any further changes to your CI Build definition the load test will be ignored.
If you want to run the test as part of a build, then you need to configure the build definition to use the .loadtest file.
Stay away from testlists. Like you said, they are being deprecated in VS11.
Edit: The simplest way to avoid running the load test as part of Visual Studio "Run All" tests is to create a different solution for your load tests.
Why don't you want to use Test Lists. I think is the best way to do that. Create different Test Lists for each test type (unit test, load test...) and then in your MSTest command run the Test List(s) you want:
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests (only UnitTests)
MSTest \testmetadata:testlists.vsmdi \testlist:LoadTests (only LoadTests)
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests \testlist:LoadTests (UnitTests & LoadTests)
I have two unit test projects in my VS 2010 solution. Each project has a Data directory with input data needed for the unit tests.
I annotated the test classes that need the data with
[DeploymentItem("Data")]
When I run tests individually, the run fine. However, when I run (or debug) all tests in the solution, I find that only one of the two Data directories is copied to TestResults\MyTestDir-YYYY-MM-DD HH_mm_SS\Out, causing unit tests that rely on the other data directory to fail.
Interestingly if I then use the Test Results window to re-run checked (as in failed) tests, the tests still fail (they do not recognize that the correct Data directory's files are missing). If I then navigate directly to a failed test and select Run Tests in Current Context, the test run gets the correct Data directory and the test succeeds.
My question: How can I cause DeploymentItems from two separate test projects all to be copied to the Out directory where the tests are executed?
For reference, I have reviewed these questions without noting a solution
Problems with DeploymentItem attribute
Do MSTest deployment items only work when present in the project test settings file?
I found that giving each "Data" directory a unique name (e.g. "TestProjectAData") resolved the issue. There seems to be some sort of bug when multiple directories in different paths all have the same relative path to different test projects (i.e. if every test project has a subdirectory called "Data").
We use Visual Studio 2010 Ultimate with tests written in MSTest. Both our unit tests and integration tests* are written in MSTest.
**By our definition, an integration test is an MSTest TestMethod that takes a while to run and/or calls out to external components such as a database or web services.*
I'm looking for a way of easily filtering out the integration tests so that only unit tests run without all the integration tests running too.
My ideas so far:
Mark integration tests with the [Ignore] attribute. This works but is a real pain when you do want to run the integration tests.
Assign a [TestCategory] attribute to the different test types. This allows them to be run separately but only through the Test View panel. You can't use CTRL+R, A (Run All Tests in Solution) or other similar shortcuts/hotkeys.
The integration tests are in a separate project, is there something that could be done to stop them running at the project level? As long as it's easy to toggle.
Write the integration tests in a different test framework, e.g. NUnit. This would keep them totally separate from a tooling point of view.
Does anyone have any other suggestions? Are there any plug-ins that can help with this?
I recommend having a different project (or projects) for integration tests because depending on your runner the only real way to run or not run tests across all runners is to include or not include a test class library.
But, I also recommend, if you're using MSTest, to use the TestCategoryAttribute to tag non-unit tests. You can then "filter" tests to be run in Test View with MSTest.
Runners like Resharper and apparently TestDriven.net (http://bit.ly/tmtYC2) allow you to then filter-out those tests from general unit-test executions.
If your unit test project is in a separate namespace, you could use the keyboard shortcut CTRL+R, T to run all tests in the current context (i.e. namespace MyApp.Tests.Unit). To do this place the cursor just after the opening curly brace in the namespace clause of any unit test class.
I have a suggestion but you won't like it.
Abandon MSTest entirely, while other unit test frameworks have been evolving MSTest as almost stopped in time. Yes, it has a major benefit of integrating directly with VS, but if I'm not mistaken that will change in VS 2011 which will provide native support for custom unit test runners integration.
(Note: The stopped in time part may be not true because I confess not paying to much attention to MSTest since I used it sparingly with VS 2008)
I use NUnit and separate my unit tests from the integration tests by using a different class library project. Then I automate the running of the tests using Gallio command line runner allowing me to configure separate scripts for running unit and integration tests.
Finally, personal opinions aside, I'm not sure but the TestDriven.net plugin may have support for running tests with a specific category only, so you could check that.