I am using TFS 2010 and Visual Studio 2010. I have a build definition which points to my solution. The build runs overnight
I have a set of test case files(*.tst) and i would like my current build to include these as part of the build and to execute them overnight. The test case files are in source control
I read that i have to use MsTest.exe but unsure how to get started?
Can anyone point me please how i can get started on running the test case files as part of the build? Any examples please?
Thanks in advance,
There are several ways you can have your test cases run. One way is to add an invoke process or Exec Command to your build project file or workflow.
Place the call to mstest in the AfterBuild target or workflow step. The other way would be to separate out the build and test cases into different builds.
Here are a couple of links to get you started:
How to: Configure and Run Scheduled Tests After Building Your Application
Example of MSBuild with MSTest
*Based on your comments here is an update of something you can do to get more information about the error or to continue if you encounter an error. You do not need to check for the error code if you just want to stop on any error but if you do want to check the error code then you would do something like this:
<Target Name="AfterBuild">
<Message Text="Running tests..." />
<PropertyGroup Label="TestSuccessOrNot">
<TestSuccessOrNot>5</TestSuccessOrNot>
</PropertyGroup>
<!-- Run MSTest exe-->
<Exec Command="cd ." IgnoreExitCode="False" ContinueOnError="ErrorAndContinue">
<Output TaskParameter="ExitCode" PropertyName="TestSuccessOrNot" />
</Exec>
<Message Text="ExitCode = $(TestSuccessOrNot)" />
<Error Condition="'$(TestSuccessOrNot)' != '0'" Text="Unit tests fail!" /> </Target>
Related
I created a custom Task that executes after the build operation.
<Target Name="AfterBuild" />
<Target Name="MyTarget"
AfterTargets="AfterBuild">
<MyTask ... />
</Target>
QUESTION: Is it possible to execute the task, if the build operation was triggered, but did not perform, because there are no changes in the project / no need to build again?
In other words: I want to execute the task always at the end of the build process, even if the project was not built again.
UPDATE: Using AfterTargets="Build" or setting the property <DisableFastUpToDateCheck>true</DisableFastUpToDateCheck> does not help.
After triggering the Build process a second time, I only get the Output: Build: 0 succeeded, 0 failed, 1 up-to-date, 0 skipped
Is it possible to execute the task, if the build operation was triggered, but did not perform, because there are no changes in the project / no need to build again?
If I understand you correctly, you can define this property in your project file:
<PropertyGroup>
<DisableFastUpToDateCheck>true</DisableFastUpToDateCheck>
</PropertyGroup>
Note: This method seems that Visual Studio is bypassing normal up-to-date checks of MSBuild and using some sort of custom check that is faster, but has a side effect of breaking customized build targets.
Update:
Not sure the reason why this method not work on your project, let me make the answer more detail:
Define the property in your project file:
Add the custom MSBuild task with some messages info.
Build the project, check the output(log file verbosity is Normal).
Build the project again, check the output again.
If I use AfterTargets="Build" instead of AfterBuild, the message is written to the Output window every time I build the solution (.NET Core Console App with a .NET Standard Class Libary).
<Target Name="MyAfterBuild" AfterTargets="Build">
<Message Importance="High" Text="Hello World!" />
</Target>
I have a target to execute after publication in my csproj:
<Target Name="CustomPostPublishActions" AfterTargets="MSDeployPublish">
<Message Text="After publish" Importance="high" />
<Exec Command="git branch -f $(PublishProfile)" />
</Target>
This indeed get called when I publish from the command line like so:
msbuild /p:DeployOnBuild=true /p:PublishProfile=Staging /p:Password=*** /p:AllowUntrustedCertificate=true
It does not get hit when publishing from Visual Studio, however. Neither do I see the message, nor does git do any moving of the branch pointer.
Any ideas what I could try?
EDIT:
For those who care, the problem here was that PublishProfile was not set while publishing from VS. The property that works for both VS and command line is PublishProfileName.
Turn on "detailed" build debugging in VS (Tools|Options|Projects and Solutions|Build and Run)
Also set your MSBuild verbosity at the commandline as well.
/v:detailed
Then compare the output from each to find the differences.
FYI - I couldn't reproduce your situation but this should help you debug further.
I am new to MSBuild and busy automating tests of Visual Studio solutions.
I previously worked with the command line of Devenv, which provides a convenient /Runexit mode of operation. From the manual:
/Runexit (devenv.exe)
Compiles and runs the specified solution, minimizes the IDE when the solution is run,
and closes the IDE after the solution has finished running.
This is exactly the functionality that I need. I am now migrating to MSBuild. I have discovered that the project files in the Solution can be directly used for building, as the default target is precisely Build.
What can I do to handle a different target, that will have the same effect as /Runexit ? Can you help me through the maze ?
This is the most basic Target which runs a projects' output file:
<Target Name="RunTarget">
<Exec Command="$(TargetPath)" />
</Target>
For c++ unittests I use something like this; it's a property sheet so it's easy to add to any project without needing to manually modify it. It automatcially runs the output after the build so there is no need to specify an extra target and it works the same for VS and from the command line. Moreover in VS you'll get unittest errors from frameworks like Unittest++ or Catch displayed right away in the error list, so you can doubleclick them. Also the UnitTestExtraPath property can be set elsewhere just in case (e.g. on a buildserver we always want to keep the PATH clean but sometimes we do need to modify it to run built exes).
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ImportGroup Label="PropertySheets" />
<PropertyGroup Label="UserMacros" />
<PropertyGroup />
<ItemDefinitionGroup />
<ItemGroup />
<!--Used to be AfterTargets="AfterBuild", but that is unusable since a failing test marks the build as unsuccessful,
but in a way that VS will always try to build again. As a consequence debugging in VS is impossible since
VS will build the project before starting the debugger but building fails time and time again.-->
<Target Name="RunUnitTests" AfterTargets="FinalizeBuildStatus">
<Exec Condition="$(UnitTestExtraPath)!=''" Command="(set PATH="%PATH%";$(UnitTestExtraPath)) & $(TargetPath)" />
<Exec Condition="$(UnitTestExtraPath)==''" Command="$(TargetPath)" />
</Target>
</Project>
My company already has a Team Foundation Server as a Continuous Integration platform. However, what I am looking to setup is a build configuration that a developer can run on their own development machine.
Let's say I have a Visual Studio solution that contains a .NET C# Class Library project (I'll call this the Library Project). It also contains another project containing the Unit Testing classes for Library Project (I'll call this the Testing Project).
I have the normal Debug and Release build configurations for each project and at the solution level. For both of these configurations, I have set it to only build the Library Project (so Testing Project does not get built).
What I would like to do is set up 2 new build configurations called Debug With Testing and Release With Testing. They will each be the same as the Debug and Release, respectively but I need them to have the following extra features:
Builds the Testing Project.
Run all test cases in the Testing Project.
Run Code Analysis on Library Project.
Generate report for testing and code analysis.
Save report in a specific location.
Doing item 1 is easy. However, I can't figure out how to do items 2 to 5. Can anyone point me in the right direction?
Any help will be greatly appreciated. TIA
You will need to write custom MS build code, I already do some similar task as the following:
Get the latest change from TFS
Build the solution including all projects
Deploy the Main Database locally
Deploy the Test Database locally which hold the test data used in the
data driven test
Run the sanity test or BVT (Build Verification Test) which has
belong to category 1 (Test the integration between DB and code)
Check-in the pending change
And hear the code of this tasks
<Target Name="GetLatestFromTFS2010" AfterTargets="build" >
<Message Importance="high" Text ="start GetLatest for the project "></Message>
<Exec Command='"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\TF.exe" get $/AutoDBand/AutomateDatabaseAndTest/AutomateDatabaseAndTest /recursive /login:YourUsername,YourPassword' ContinueOnError='false'/>
</Target>
<!--===========Deploy Database============-->
<Target Name="DeployDatabase" AfterTargets="GetLatestFromTFS2010" Condition="'$(Configuration)' == 'DebugForCheck-in'">
<Message Importance="high" Text="-------------------------------- Deploying Database according to the connection string -------------------------------- " />
<Message Importance="high" Text=" "/>
<MSBuild Projects="..\DB\DB.dbproj" Targets="Build;Deploy" />
</Target>
<!--============Run the Test==================-->
<Target Name="UnitTests" AfterTargets="DeployDatabase" Condition="'$(Configuration)' == 'DebugForCheck-in'">
<Message Importance="high" Text="-------------------------------- Running Unit Tests for category 1 only--------------------------------" />
<Message Importance="high" Text=" "/>
<Exec Command='"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /testcontainer:"..\BLTest\bin\Debug\BLTest.dll" /category:cat1' />
</Target>
<Target Name="Chekin-pendingChange" AfterTargets="UnitTests" >
<Message Importance="high" Text ="-------------------------------- start Check-in process-------------------------------- "></Message>
<Message Importance="high" Text=" "/>
<Exec Command='"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\TF.exe" checkin $/AutoDBand/AutomateDatabaseAndTest/AutomateDatabaseAndTest /recursive /login:YourUsername,YourPassword' ContinueOnError='false'/>
</Target>
For more information you can see this article with source code
http://mohamedradwan.wordpress.com/2010/11/13/automate-the-best-practice-for-check-in-including-get-latest-deploy-db-run-test-check-in/
Have a look at something like:
TeamCity
Jenkins
Team Foundation Server
all are Continous Integration Servers, which are good in doing the jobs you like to have done.
I need to know how to tell MSTEST to run all test projects in a solution file. This needs to be done from the command line. Right now I have to pass it a specific project file, I'm trying to get it to run from a SOLUTION file.
I'm hoping this is possible, because in Visual Studio, hitting Ctrl+R, A, runs ALL tests in the currently opened solution.
The way I've interpretted the help files, you have to pass in each DLL specifically.
I want to run this from the command line from my CruiseControl.NET server, so I can write other utilities to make this happen. If there is a wierd way of getting this to happen through some OTHER method, let me know.
How do I tell MSTEST to run all test projects for a solution?
<exec>
<!--MSTEST seems to want me to specify the projects to test -->
<!--I should be able to tell it a SOLUTION to test!-->
<executable>mstest.exe</executable>
<baseDirectory>C:\projects\mysolution\</baseDirectory>
<buildArgs>/testcontainer:testproject1\bin\release\TestProject1.dll
/runconfig:localtestrun.Testrunconfig
/resultsfile:C:\Results\testproject1.results.trx</buildArgs>
<buildTimeoutSeconds>600</buildTimeoutSeconds>
</exec>
To elaborate on VladV's answer and make things a bit more concrete, following the suggested naming convention running your tests can be easily be automated with MSBuild. The following snippet from the msbuild file of my current project does exactly what you asked.
<Target Name="GetTestAssemblies">
<CreateItem
Include="$(WorkingDir)\unittest\**\bin\$(Configuration)\**\*Test*.dll"
AdditionalMetadata="TestContainerPrefix=/testcontainer:">
<Output
TaskParameter="Include"
ItemName="TestAssemblies"/>
</CreateItem>
</Target>
<!-- Unit Test -->
<Target Name="Test" DependsOnTargets="GetTestAssemblies">
<Message Text="Normal Test"/>
<Exec
WorkingDirectory="$(WorkingDir)\unittest"
Command="MsTest.exe #(TestAssemblies->'%(TestContainerPrefix)%(FullPath)',' ') /noisolation /resultsfile:$(MSTestResultsFile)"/>
<Message Text="Normal Test Done"/>
</Target>
Furthermore integrating MsBuild with CruiseControl is a piece of cake.
Edit
Here's how you can 'call' msbuild from your ccnet.config.
First if you do not already use MSBuild for your build automation add the following xml around the snippet presented earlier:
<Project DefaultTargets="Build"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
..... <insert snippet here> .....
</Project>
Save this in e.g. RunTests.proj next to your solution in your source tree. Now you can modify the bit of ccnet.config above to the following:
<msbuild>
<executable>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe</executable>
<workingDirectory>C:\projects\mysolution\</workingDirectory>
<baseDirectory>C:\projects\mysolution\</baseDirectory>
<projectFile>RunTests.proj</projectFile>
<targets>Test</targets>
<timeout>600</timeout>
<logger>C:\Program Files\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll</logger>
</msbuild>
This is an old thread, but I have been struggling with the same issue and I realized that you can really just run MSTest on every dll in the whole solution and it doesn't really cause any problems. MSTest is looking for methods in the assemblies marked with the [TestMethod] attribute, and assemblies that aren't "test" assemblies just won't have any methods decorated with that attribute. So you get a "No tests to execute." message back and no harm done.
So for example in NAnt you can do this:
<target name="default">
<foreach item="File" property="filename">
<in>
<items>
<include name="**\bin\Release\*.dll" />
</items>
</in>
<do>
<echo message="${filename}" />
<exec program="C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe">
<arg value="/testcontainer: ${filename}" />
<arg value="/nologo" />
</exec>
</do>
</foreach>
</target>
and it will run all the test methods in every dll in every bin\Release folder in the solution. Those which are not test dlls will return a "No tests to execute." and those that have tests will have the tests run. The only part I haven't figured out yet is that (in NAnt) execution stops the first time a command returns a non-zero value. So if any unit tests fail it doesn't keep going to execute any tests in subsequent assemblies. That is not great, but if all the tests pass, then they will all run.
I just resolve this problem recently. Here is my proposal: Use testmetadata + testlist option of mstest
First you should create a testlist in testmetadata file(vsmdi)
the commandline should be mstest /testmetadata:....vsmdi /testlist:<name>
Then use ccnet config to run mstest
i know this thread is quite Old, but its still high on Google so i thought i might help one or two.
Anyway, since there is no satisfactory solution for this.
I've written an msbuild task for this.
details can be found here:
http://imistaken.blogspot.com/2010/08/running-all-tests-in-solution.html
You could enforce some convention on the naming and location of test projects, then you could run MSTest on, say, all *Test.dll below the location of your solution.
As far as I know, there is no way to tell a test project from a 'normal' DLL project based soleley on a solution file. So, an alternative could be to analyze the project files and/or .vsmdi files to find the test projects, but that could be rather tricky.
I don't know directly but this is where VSMDI [fx:spits in a corner] can help. In your solution add all the tests to the VSMDI. And then pass the VSMDI to mstest using /testmetadata.
However I would suggest that you follow the conventions above. And use a naming convention and dump that out from the SLN file using say a for loop in the command script
I would just write a target that calls it the way you want, then whip up a batch file that calls the target that contains all the DLL's to be tested.
Unless you're adding test projects all the time, you'll very rarely need to modify it.
Why not just have msbuild output all the test assemblies to a folder.
Try setting OutputPath,OutputDir,OutDir properties in msbuild to accomplish this.
then have mstest execute against all assemblies in that folder.