I've got a Visual Studio Solution File conatining multiple projects. About half of them are Unit Test Projects.
Visual Studio Test Explorer reliably picks up all Test Projects and allows me to run all of them at once.
We also got a CI that should run these Tests. Is there a way to use the Test Explorer functionality from the command line? Currently I have to pass a list of dll's and executables to the vstest.console.exe Tool.
I searched google for solutions but only found solutions that pass dll files to vstest.console.exe. I would expect a possibility to run the tests from a solution also from the command line similar to how dotnet test allows me to do this.
Unfortunately I cannot use dotnet test to reach my goal, as dotnet does not yet support native projects (which are also conatined in my solution).
Related
I recently installed XUnit and the XUnit VS runner (i.e. xunit.runner.visualstudio) on Visual Studio following the documentation. The installation process uses NuGet and on completion I can successfully see all tests in the MS Test Explorer.
However, if I start a new solution and reference all the required XUnit dll's, i.e., I want to use XUnit without using NuGet to install it, the tests never show up in MS Test Explorer (no problems seeing them in the ReSharper test explorer).
I think this has something to do with the XUnit test adapter that Visual Studio uses to find XUnit tests. It appears that NuGet somehow instructs Visual Studio to create the VisualStudioTestExplorerExtensions folder on each build putting the necessary XUnit test adapter dll's (e.g.xunit.runner.visualstudio.testadapter.dll) in that folder. The MS Test Explorer then uses this test adapter to find XUnit test (see SO Question).
When I attempt to use XUnit without NuGet, the VisualStudioTestExplorerExtensions folder never gets created on build. Adding the folder manually with all XUnit dll's in it also doesn't work. It's almost as if i need to instruct Visual Studio to look in that folder, but I can't see any obvious way of doing that.
How do I use/install XUnit without using NuGet? Or, how do I get VS to look in the `VisualStudioTestExplorerExtensions' when looking for tests?
If you want to install a test runner manually, copy all the necessary dlls to:
C:\Program Files (x86)\Microsoft Visual Studio {version}\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions
Replacing {version} with the appropriate number.
The drawback here is that this is machine wide (maybe desirable in your case), and that you can't quite as easily replicate this setup on a build server. You also will probably have to restart VS whenever you make changes to these files.
I migrated my web performance tests project's build to TFS 2013 build process template for auto nuget restore by following this document.
My solution has .webtest files which are the containers for web performance tests. But VsTest.console.exe doesn't recognize the .webtest tests. I looked at the discoverers and found that none of the ones installed support the .webtest extension.
I tried to find test adapters for running .webtests using vstest.console.exe but found nothing.
This document says the following for vstest.console.exe:
You can run automated unit and coded UI tests from the command line.
And this for mstest.exe:
You can run automated Web performance and load tests from the command line either locally or by using a test controller or test agents.
Why doesn't vstest.console.exe support .webtests ? Wasn't it designed as a replacement to mstest.exe for running any kind of tests using adapters ?
Also, if it means that I need to write a custom adapter that can read and run tests with .webtest extension, what does it take for me to do that ?
vstest.console can run webtests using the following:
"c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" SmokeTest.orderedtest /Settings:..\..\..\local.testsettings"
You need to create an orderedtest and put your webtest in it.
vstest.console can run .webtest Webtests. Enter the path to the exe in quotation marks and then the path to the webtest:
"c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" WebTest1.webtest
Looks like vstest.console.exe cannot run load tests and web tests. My only option is to use MsTest.exe through TFS 2013 by customizing the build process template to suit my needs.
Edit: I'm wrong. You can run web tests using vstest.console.exe.
I've setup my build as below using the build definition. I'm using XUnit and locally my tests are discovered and run. I've tested the glob **\*spec*.dll and it finds all my test dlls and the build log shows that those dlls are in fact built.
However in the build log I get
Run VS Test Runner
No test found. Make sure that installed test
discoverers & executors, platform & framework
version settings are appropriate and try again.
Which seems to suggest it is trying to use the MSTest test runner instead of the XUnit test runner. How do I tell the build for visual studio online to use the XUnit test runner and discoverer?
This might be out of date now, but this is how I have it setup and working - downvote and let me know if it's wrong and I'll delete this. I got it from a blog post/MSDN page, but I can't locate it any more.
First you need to create a TFVC Team Project (doesn't matter if you don't use it again).
Into $/MyTFVC/BuildProcessTemplate/CustomActivities/
Checking the following files from xunit.net:
Now in VS, click the BUILD, Manage Build Controllers... option. Select the "Hosted Build Controller (Hosted)" and click "Properties...".
Enter the path where you checked in the DLL's into the "Version control path to custom assemblies" field:
You should be good to go.
I have Nunit unit test which i need to run as the part of my MS build.. I know that running all the test will slow up the build So, i need to run only the impacted test is there any way to find that out..
AFAIK running only impacted tests is not possible yet for NUnit tests. But this is possible for MSTests in Visual Studio 2010 Ultimate/Test Professional:
Recommending Tests to Run That are Affected by Code Changes
You can use Visual Studio Ultimate or Visual Studio Test Professional
2010 to help you determine which tests might have to be run, based on
coding changes that were made to the application you are testing. To
be able to use this functionality, you have to use Team Foundation
Build to build your application and use Microsoft Visual Studio 2010
for version control for your source code
Anyway you can use MSBuild Community NUnit Task to run tests from a set of the assemblies. You can do this as dependency target of standard AfterBuild target by specifying DependsOnTargets attribute.
<NUnit Assemblies="..."
IncludeCategory="..."
ExcludeCategory="..."
ToolPath="$(NUnitDllsPath)"
ProjectConfiguration="$(Configuration)"
OutputXmlFile="$(NUnitOutputPath)\UnitTests.xml"
ContinueOnError="true">
To know the impacted tests you need to track the test case code coverage. Only this way you can examine what test is impacted by the changes you are checking in. I don't know of any tool that does what you want besides Microsofts Team Foundation Server.
Running your tests as a part of the build can be done trough the Build Events properties of your project. You can execute the command line tool for NUnit.
But as PVitt already pointed out, I don't know if NUnit can work with Test Impact analysis.
I realise that there are many older questions addressing the general question of NUnit v MSTest for versions of Visual Studio up to 2008 (such as this one).
Microsoft have a history of getting things right in their 3rd version. For MSTest, that is VS2010.
Have they done so with MSTest? Would you use it in a new project in preference to NUnit?
My specific concerns:
speed
running tests within CruiseControl.NET (either commandline or MSBuild task)
code coverage reports from CC.NET
can you run MSTest tests in debug mode
(We use ReSharper, so test-runners are not an issue for us. We have used NUnit for the last few years. We do not have TFS.)
List item speed is same, but MsTest may be a bit slower because it creates folder for test run every time
MSBuid and CC.Net is big pain. You can't run MSTest on computer without VS on it (not 100 sure about 2010, but with 2008 it is so)
not sure, sorry
yes you can, from visual studio
My recommendation is following: if NUnit satisfies you - use it, forget about MSTest
To correct some old information on the thread;
It IS possible to run 64 bit tests in 2010
From VS2008 forward it is not neccesary to have MSTEST create directories anc opy the binaries in, just disable deployment, in 2010 thats the default but you have to set it in 2008
2010 MSTEST is faster but as its a generalised test framework that also runs load/web/UI tests there are compromises in the design that will lead to it being slower. Jamie Cansdale appears to have managed to get perf increases with the lastest releases of TestDriven.net's support for MSTEST
I've mainly used NUnit, some xUnit and some MSTest. They seem functionality equivalent, but I don't like the MSTest test runner. It runs in visual studio so it either crowds the screen or is on another monitor getting in the way everytime I tab to visual studio. (I run NUnit on another monitor, but it doesn't cover everything on that monitor everytime I focus visual studio). It takes too many clicks to find out what test failed and why.
NUnit can run in the background until a test fails, at which point it shows you information about the breaking test. This seems like the ideal for keeping red/green/refactor going smoothly.
Nope. Same issues regarding appdomains and assembly resolving still exist. I would avoid unless you want the new goodness for other functional testing or integration with Team System.
I don't know much about CruseControl.net, but you can debug tests. We currently don't use TFS either, and the MSTest is working for us.
If you think you'll ever run your tests in 64 bit mode, use NUnit. MsTest is only x86.
One major difference between the two is that MSTest makes a copy of the current DLLs every time it runs a test. If you're doing TDD and running your tests frequently, this can eat up a lot of hard drive space.
If you're using MSTest, you can change this setting in Tools > Options > Test Tools > Test Execution. "Limit number of old Test Results to" is set to 25 by default in Visual Studio 2010. I usually change it to 1.
MSUnit runs your test cases under conditions that are different from the actual execution environment. Specifically, the deployed files differ from those that are deployed when you run your actual project. Nethertheless, there is the [DeploymentItem]-Attribute to specify which files shall be deployed by MSUnit. So if your application depends on any external
files, such as
database files
database configuration file
application configuration file
...
then MSUnit is not the right choice, because the MSUnit tests never cover what your file system is going to look like in execution environment. The Visual Studio Project File settings for deploying files (Copy always, Content, etc.) are ignored by the MSUnit runner. So those settings cannot be tested.