We want to parallelize our unittests, however the nunit-console.exe cannot do this out of the box. And the NUnit.MultiCore project on codeplex.com seems to fail loading our custom log4net appenders - probably AppDomain error in the framework.
However, Resharpers own test runner, runs nunit tests parallel pr. assembly and my question is whether is it possible to use Resharpers unit test runner, JetBrains.ReSharper.TaskRunner.exe, as a CLI tool and run this on our build server?
Related
I am currently looking for ways on how to test my .net core unit/integration test projects inside a docker container.
I already know about dotnet test and I would know how to create the correct Dockerfile to get this working, but I'm looking for a way to do this through VS2019 directly, instead of doing it via cli or vs code.
I'd like to do this for potentially debugging unit tests, as well as for making use of the nice VS unit test session overview.
Essentially, I would have to teach the embedded unit test start / debug buttons to trigger the dotnet test, and potentially attaching the debugger.
Is there any way to do this?
I am trying to get parallel tests going with NUnit and C# in visual studio using the NUnit3Test adapter and .runsettings file. It's basically this sample from Souce Labs I am unable to get to run in parallel with the NUnit3Test adpter, instead of using the console.
According to the NUnit documentation (https://github.com/nunit/docs/wiki/Tips-And-Tricks)is should be sufficient to have a .runsettings files in the project and configure number of workers there.
I tried with this, but test are still running sequentially.
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<NUnit>
<NumberOfTestWorkers>4</NumberOfTestWorkers>
</NUnit>
</RunSettings>
Any thoughts on what is wrong?
The "tips and tricks" entry is info for the user of the adapter. It tells you what you might need to do to configure parallelism of your tests under the adapter in addition to the general steps that are involved in the nunit framework itself and assumes you have read all the docs regarding parallel execution.
By default, the nunit framework runs nothing in parallel. The ParallelizableAttribute is available to tell it to run a particular test method or fixture in parallel. It can even be used at the assembly level if you are quite sure that all your tests are able to run in parallel.
That last point deserves repeating with emphasis! When you apply [Parallelizable] to a test, you are telling the framework that this particular test is capable of running in parallel - that it will not interfere with any other tests. NUnit assumes you know what you are talking about. It will run that test on a separate thread along with other parallelizable tests even if that causes problems due to the way the tests are written.
For that reason, you should pick a few tests first and mark them as parallelizable. The main thing that makes a test impossible to run in parallel is that it shares non-readonly state with other tests.
The info you provide in the .runsettings file is completely optional. NUnit will use a default value for the number of worker threads if you don't specify it. The main thing is the use of the attribute.
I created a build definition that runs automated tests using MTM build environments and test suites. I recently created a Visual Studio Load Test, which can be added to a test suite just like any test method marked with the [TestMethod] attribute. However, when I run the build, I get no errors and it appears the aggregate tests don't run. Is there a way to make this work?
I found this article: https://blogs.msdn.microsoft.com/testingspot/2013/01/22/how-to-automatically-run-a-load-test-as-part-of-a-build/ which describes a way to do it, but I can't find a build template that matches what he describes, and it appears this only allows you to run a single load test.
Also, when you configure a test controller, there is an option to configure it for load testing, but to do this, you must unregister it from the Team Project Collection. If this is done, it appears the controller can no longer be used in an environments to run project automated tests. This defeats the purpose of what I want to do and makes it seem that Load Tests and Team Projects are mutually exclusive. Is this the case? If so, this is a big oversight. Load tests are the kind of thing you would like to run automatically. Thanks for the help.
You are unfortunately right. A test controller used for load testing cannot be used for other automated test execution 'at the same time'. In your scenario I would recommend that you setup a different test controller and agent for load testing and you would be able to queue it as a part of your build to achieve what you are looking for.
There is no special build process template for this case.
I'm new to load testing in Visual Studio/MSTest, and I created a new Load Test recently to validate some high-traffic scenarios for a WCF service. I want to add this to the tests project for the service, but I don't want the test to be executed whenever I "Run All Tests in Solution" nor as part of our Continuous Integration build-verification process because a) it takes 5 minutes to run, and b) the service call that it is testing generates many thousands of email messages. Basically, I'd like to do the equivalent of adding the [Ignore] attribute to a unit test so that the load test is only executed when I explicitly choose to run it.
This MSDN Article ("How to: Disable and Enable Tests") suggests that the only to disable the test is to use Test Lists (.vsmdi files), but I don't have much experience with them, they seem like a hassle to manage, I don't want to have to modify our CI Build Definition, and this blog post says that Test Lists are deprecated in VS2012. Any other ideas?
Edit: I accepted Mauricio's answer, which was to put the load tests into a separate project and maintain separate solutions, one with the load tests and one without. This enables you to run the (faster-running) unit tests during development and also include the (slower-running) load tests during build verification without using test lists.
This should not be an issue for your CI Build Definition. Why?
To run unit tests as part of your build process you need to configure the build definition to point to a test container (usually a .dll file containint your test classes and methods). Load tests do not work this way, they are defined within .loadtest files (which are just xml files) that are consumed by the MSTest engine.
If you do not make any further changes to your CI Build definition the load test will be ignored.
If you want to run the test as part of a build, then you need to configure the build definition to use the .loadtest file.
Stay away from testlists. Like you said, they are being deprecated in VS11.
Edit: The simplest way to avoid running the load test as part of Visual Studio "Run All" tests is to create a different solution for your load tests.
Why don't you want to use Test Lists. I think is the best way to do that. Create different Test Lists for each test type (unit test, load test...) and then in your MSTest command run the Test List(s) you want:
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests (only UnitTests)
MSTest \testmetadata:testlists.vsmdi \testlist:LoadTests (only LoadTests)
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests \testlist:LoadTests (UnitTests & LoadTests)
We use Visual Studio 2010 Ultimate with tests written in MSTest. Both our unit tests and integration tests* are written in MSTest.
**By our definition, an integration test is an MSTest TestMethod that takes a while to run and/or calls out to external components such as a database or web services.*
I'm looking for a way of easily filtering out the integration tests so that only unit tests run without all the integration tests running too.
My ideas so far:
Mark integration tests with the [Ignore] attribute. This works but is a real pain when you do want to run the integration tests.
Assign a [TestCategory] attribute to the different test types. This allows them to be run separately but only through the Test View panel. You can't use CTRL+R, A (Run All Tests in Solution) or other similar shortcuts/hotkeys.
The integration tests are in a separate project, is there something that could be done to stop them running at the project level? As long as it's easy to toggle.
Write the integration tests in a different test framework, e.g. NUnit. This would keep them totally separate from a tooling point of view.
Does anyone have any other suggestions? Are there any plug-ins that can help with this?
I recommend having a different project (or projects) for integration tests because depending on your runner the only real way to run or not run tests across all runners is to include or not include a test class library.
But, I also recommend, if you're using MSTest, to use the TestCategoryAttribute to tag non-unit tests. You can then "filter" tests to be run in Test View with MSTest.
Runners like Resharper and apparently TestDriven.net (http://bit.ly/tmtYC2) allow you to then filter-out those tests from general unit-test executions.
If your unit test project is in a separate namespace, you could use the keyboard shortcut CTRL+R, T to run all tests in the current context (i.e. namespace MyApp.Tests.Unit). To do this place the cursor just after the opening curly brace in the namespace clause of any unit test class.
I have a suggestion but you won't like it.
Abandon MSTest entirely, while other unit test frameworks have been evolving MSTest as almost stopped in time. Yes, it has a major benefit of integrating directly with VS, but if I'm not mistaken that will change in VS 2011 which will provide native support for custom unit test runners integration.
(Note: The stopped in time part may be not true because I confess not paying to much attention to MSTest since I used it sparingly with VS 2008)
I use NUnit and separate my unit tests from the integration tests by using a different class library project. Then I automate the running of the tests using Gallio command line runner allowing me to configure separate scripts for running unit and integration tests.
Finally, personal opinions aside, I'm not sure but the TestDriven.net plugin may have support for running tests with a specific category only, so you could check that.