How to integrate MSTest in your TeamCity build process - teamcity

How do I run MSTest as part of my build process in TeamCity? What are the pitfalls?

This answer is specifically for TeamCity 7.1 on Windows, but may apply to other environments.
In your TeamCity build configuration, on the General Settings page
Artifact paths: Artifacts\MSTest => MSTest
Create a new Command Linebuild step
Custom script: if not exist Artifacts\MSTest mkdir Artifacts\MSTest
Create a new MSTestbuild step
List assembly files: **\bin\**\*.Tests.dll
Results file: Artifacts\MSTest\testResults.trx
Pitfalls
Using wildcards when specifying which test assemblies to run
You can use wildcards when specifying which test assemblies to run in the MSTest build step, although it is unclear exactly how they work. A bug report has been filed.
The build process doesn't stop when tests fail
Be aware that if some of your tests fail and the build is marked as failed, the MSTest build step itself does not fail. This causes problems if you have build steps after the MSTest build step which you don't want to run if you have test failures (e.g. it may not make sense to produce an installer or documentation of a build you know has bugs). The problem will hopefully be fixed in later versions of TeamCity.
If you want your build process to stop when you have test failures, you can create a new build step that uses the TeamCity REST API to detect if the current build has been marked as failed (remember that when tests fail, the build step is not marked as failed, but the build is), and then fail the current build step explicitly. Example:
Create a new Powershell build step
Script: Source code
Source code: See script below
Make sure your newly created build step comes immediately after your MSTest build step
Make sure every build step after this one has Execute step set to Only if all previous steps were successful
Script:
$xml = [xml](curl --request GET http://USERNAME:PASSWORD#HOSTNAME/httpAuth/app/rest/builds/%teamcity.build.id%)
Microsoft.PowerShell.Utility\Select-Xml $xml -XPath "/build" | % { $status = $_.Node.status }
if ($status -eq "FAILURE") {
throw "Failing build step on purpose"
}

Related

Run NUnit tests in Appcenter?

I have a Xamarin project that I regularly push to Appcenter, where it's build. That works fine, but now I added NUnit 3 tests to my solution and while they execute fine locally, they seem not to be executed during Appcenter build.
How do I configure my solution so that my test project is executed on Appcenter? Seems like one has to combine it with Xamarin.UITest, but I don't really understand what steps are necessary for that. Note that my NUnit tests are not UI tests, they are normal unit tests.
Update:
Quote from https://devblogs.microsoft.com/appcenter/faster-android-tests-and-nunit-3/
Now you can update your NUnit package along with UITest to the latest versions and run tests both locally and in App Center,
So I would expect NUnit tests to be run during App Center build without an additional script. Ca anyone shed some light on this? Adding a bounty now.
The easiest way I found to have AppCenter run a project's NUnit tests was by adding a post-build script for each app and install NunitXml.TestLogger Nuget package into your NUnit project, which will output an Xml file of your test results.
To create a post-build script it needs to be in the root directory of the Android/iOS .csproj and named appcenter-post-build.sh. Then your script should look something like this:
#Android post build script
#Make sure the directly to the NUnit csproj is correct
ProjectPath="$APPCENTER_SOURCE_DIRECTORY\YourProject.NUnit\YourProject.NUnit.csproj"
echo "$ProjectPath"
#To generate the xml file it requires nuget NunitXml.TestLogger
dotnet test "$APPCENTER_SOURCE_DIRECTORY" --logger:"nunit;LogFilePath=TestResults.xml"
echo "NUnit tests result:"
pathOfTestResults=$(find $APPCENTER_SOURCE_DIRECTORY -name 'TestResults.xml')
cat $pathOfTestResults
echo
#Looks for a failing test and causes the build to fail if found
grep -q 'result="Failed"' $pathOfTestResults
if [[ $? -eq 0 ]]
then
echo "A test Failed"
exit 1
else
echo "All tests passed"
fi
The last part should cause your AppCenter build to fail if a test does. Also, you may need to try building it twice in AppCenter before it picks up that a post build script has been added to your repo.
My understanding of the blog is that the UITest used to based on nunit 2.x. So, previously if you accidentally updated to nunit3.x your UITests wouldn't work.
They have now updated it so that you can use nunit 3.x to run your UI tests.
I believe #Nick Peppers give the correct approach. Sample post build script here:
https://github.com/microsoft/appcenter/blob/master/sample-build-scripts/xamarin/nunit-test/appcenter-post-build.sh

DevOps Services tests not working in BlueMix/hub.jazz with Go

I'm trying to run tests on my software as it's about to be deployed via git to blueMix. Currently the pipeline is 'build stage' -> 'deploy stage', and now I'm trying to implement the 'test stage' in the middle of that process.
My test stage has a test job, and the Tester Type is simple. The command I have currently is:
#!/bin/bash
# invoke tests here
go test
, however the test stage fails and returns:
_build.sh: line 3: go: command not found
Build step 'Execute shell' marked build as failure
Evidently go isn't installed at the testing stage. Do I get the test server to install Go on each new git push to test it, or am I sorely mistaken?
Thanks!
That's correct. There are only certain tools pre-installed in the enviroment that runs the jobs. You can find them here: https://hub.jazz.net/docs/deploy_var/#resources - Everything else would have to be installed each time the job is run.

TeamCity pass codebase changes to next build

I have a TeamCity build that sometimes fails too early.
What I mean by that is that the first few steps are for "provisioning" (setting up the testing environment) and the testing of my code itself comes later.
Sometimes (for whatever reason) the build fails during one of the "provisioning" steps. This is not a problem since running the build again usually works fine.
But - the "changes" are not passed along to the next run of the build.
I am using this command as part of my build to output the "changes" that came from my codebase:
copy "%system.teamcity.build.changedFiles.file%" changelog.txt
So I need a way to tell TeamCity "hey, ignore the last run, that failure doesn't count because it didn't test my code, I want the next run to contain the same 'changes' in system.teamcity.build.changedFiles.file"
How can I do that?
Have you tried build chains with dependencies? They can be set up to only execute if the build (including tests) is successful: http://blog.jetbrains.com/teamcity/2012/04/teamcity-build-dependencies-2/

Is it possible to stop a failing TeamCity build from a build configuration?

We have a TeamCity build configuration which does a deploy and then runs integration tests.
Deploy system
Run test suite A
Run test suite B
Run test suite C
If test suite A fails, B and C should still be run (likewise C should run if B fails). To satisfy this, the build steps are set to run "Even if some of the previous steps failed". However, I don't want any of the tests to run if the first step to deploy the system fails.
Is there a way of terminating the build if the deployment fails, but to keep running all tests of there are individual tests which fail?
You could chain the builds together so have a build for 'Deploy the system' and then have a separate build for 'Run the tests' which has your 3 steps A,B and C in it. The second build takes a snapshot dependency on the first build which means that it will kick off when the 'Deploy' build has completed, but it won't kick off if the build fails.
The steps in the second build could then be set to run even if the previous steps fail as you have it now and they would all run.

Failed build trigger in team city TeamCity

Is it possible to Trigger an exe to run on a failed build? Can you do this within Team City?
If you specifically want the failed builds, you can set up the dependent build as Eric said, and have that secondary buildscript use the REST API to pull up a list of the failed builds for the actual project.
If the latest build is in that failed builds list, then tell the build script to run the executable. If not, then you're all done!
http://confluence.jetbrains.net/display/TW/REST+API+Plugin
I don't think it's possible to trigger an executable to run only on failed TeamCity builds. TeamCity usually allows you to do things either always or only upon successful builds.
It would be possible to trigger an executable to run after this build is finished (failed or successful).
If that would work for you, you could set up a new build configuration that runs the executable. The new build configuration would have a "finish build" trigger. This would cause the executable to be run whenever the other build is completed.
You should add another build step with the exe you want to run and set the correct option to execute.execution options

Resources