How do I tell MSTEST to run all test projects in a Solution? - tdd

Question

I need to know how to tell MSTEST to run all test projects in a solution file. This needs to be done from the command line. Right now I have to pass it a specific project file, I'm trying to get it to run from a SOLUTION file.
I'm hoping this is possible, because in Visual Studio, hitting Ctrl+R, A, runs ALL tests in the currently opened solution.
The way I've interpretted the help files, you have to pass in each DLL specifically.
I want to run this from the command line from my CruiseControl.NET server, so I can write other utilities to make this happen. If there is a wierd way of getting this to happen through some OTHER method, let me know.
How do I tell MSTEST to run all test projects for a solution?
<exec>
<!--MSTEST seems to want me to specify the projects to test -->
<!--I should be able to tell it a SOLUTION to test!-->
<executable>mstest.exe</executable>
<baseDirectory>C:\projects\mysolution\</baseDirectory>
<buildArgs>/testcontainer:testproject1\bin\release\TestProject1.dll
/runconfig:localtestrun.Testrunconfig
/resultsfile:C:\Results\testproject1.results.trx</buildArgs>
<buildTimeoutSeconds>600</buildTimeoutSeconds>
</exec>

Answer 1

To elaborate on VladV's answer and make things a bit more concrete, following the suggested naming convention running your tests can be easily be automated with MSBuild. The following snippet from the msbuild file of my current project does exactly what you asked.
<Target Name="GetTestAssemblies">
<CreateItem
Include="$(WorkingDir)\unittest\**\bin\$(Configuration)\**\*Test*.dll"
AdditionalMetadata="TestContainerPrefix=/testcontainer:">
<Output
TaskParameter="Include"
ItemName="TestAssemblies"/>
</CreateItem>
</Target>
<!-- Unit Test -->
<Target Name="Test" DependsOnTargets="GetTestAssemblies">
<Message Text="Normal Test"/>
<Exec
WorkingDirectory="$(WorkingDir)\unittest"
Command="MsTest.exe #(TestAssemblies->'%(TestContainerPrefix)%(FullPath)',' ') /noisolation /resultsfile:$(MSTestResultsFile)"/>
<Message Text="Normal Test Done"/>
</Target>
Furthermore integrating MsBuild with CruiseControl is a piece of cake.
Edit
Here's how you can 'call' msbuild from your ccnet.config.
First if you do not already use MSBuild for your build automation add the following xml around the snippet presented earlier:
<Project DefaultTargets="Build"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
..... <insert snippet here> .....
</Project>
Save this in e.g. RunTests.proj next to your solution in your source tree. Now you can modify the bit of ccnet.config above to the following:
<msbuild>
<executable>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe</executable>
<workingDirectory>C:\projects\mysolution\</workingDirectory>
<baseDirectory>C:\projects\mysolution\</baseDirectory>
<projectFile>RunTests.proj</projectFile>
<targets>Test</targets>
<timeout>600</timeout>
<logger>C:\Program Files\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll</logger>
</msbuild>

Answer 2

This is an old thread, but I have been struggling with the same issue and I realized that you can really just run MSTest on every dll in the whole solution and it doesn't really cause any problems. MSTest is looking for methods in the assemblies marked with the [TestMethod] attribute, and assemblies that aren't "test" assemblies just won't have any methods decorated with that attribute. So you get a "No tests to execute." message back and no harm done.
So for example in NAnt you can do this:
<target name="default">
<foreach item="File" property="filename">
<in>
<items>
<include name="**\bin\Release\*.dll" />
</items>
</in>
<do>
<echo message="${filename}" />
<exec program="C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe">
<arg value="/testcontainer: ${filename}" />
<arg value="/nologo" />
</exec>
</do>
</foreach>
</target>
and it will run all the test methods in every dll in every bin\Release folder in the solution. Those which are not test dlls will return a "No tests to execute." and those that have tests will have the tests run. The only part I haven't figured out yet is that (in NAnt) execution stops the first time a command returns a non-zero value. So if any unit tests fail it doesn't keep going to execute any tests in subsequent assemblies. That is not great, but if all the tests pass, then they will all run.

Answer 3

I just resolve this problem recently. Here is my proposal: Use testmetadata + testlist option of mstest
First you should create a testlist in testmetadata file(vsmdi)
the commandline should be mstest /testmetadata:....vsmdi /testlist:<name>
Then use ccnet config to run mstest

Answer 4

i know this thread is quite Old, but its still high on Google so i thought i might help one or two.
Anyway, since there is no satisfactory solution for this.
I've written an msbuild task for this.
details can be found here:
http://imistaken.blogspot.com/2010/08/running-all-tests-in-solution.html

Answer 5

You could enforce some convention on the naming and location of test projects, then you could run MSTest on, say, all *Test.dll below the location of your solution.
As far as I know, there is no way to tell a test project from a 'normal' DLL project based soleley on a solution file. So, an alternative could be to analyze the project files and/or .vsmdi files to find the test projects, but that could be rather tricky.

Answer 6

I don't know directly but this is where VSMDI [fx:spits in a corner] can help. In your solution add all the tests to the VSMDI. And then pass the VSMDI to mstest using /testmetadata.
However I would suggest that you follow the conventions above. And use a naming convention and dump that out from the SLN file using say a for loop in the command script

Answer 7

I would just write a target that calls it the way you want, then whip up a batch file that calls the target that contains all the DLL's to be tested.
Unless you're adding test projects all the time, you'll very rarely need to modify it.

Answer 8

Why not just have msbuild output all the test assemblies to a folder.
Try setting OutputPath,OutputDir,OutDir properties in msbuild to accomplish this.
then have mstest execute against all assemblies in that folder.