I have a Xamarin project that I regularly push to Appcenter, where it's build. That works fine, but now I added NUnit 3 tests to my solution and while they execute fine locally, they seem not to be executed during Appcenter build.
How do I configure my solution so that my test project is executed on Appcenter? Seems like one has to combine it with Xamarin.UITest, but I don't really understand what steps are necessary for that. Note that my NUnit tests are not UI tests, they are normal unit tests.
Update:
Quote from https://devblogs.microsoft.com/appcenter/faster-android-tests-and-nunit-3/
Now you can update your NUnit package along with UITest to the latest versions and run tests both locally and in App Center,
So I would expect NUnit tests to be run during App Center build without an additional script. Ca anyone shed some light on this? Adding a bounty now.
The easiest way I found to have AppCenter run a project's NUnit tests was by adding a post-build script for each app and install NunitXml.TestLogger Nuget package into your NUnit project, which will output an Xml file of your test results.
To create a post-build script it needs to be in the root directory of the Android/iOS .csproj and named appcenter-post-build.sh. Then your script should look something like this:
#Android post build script
#Make sure the directly to the NUnit csproj is correct
ProjectPath="$APPCENTER_SOURCE_DIRECTORY\YourProject.NUnit\YourProject.NUnit.csproj"
echo "$ProjectPath"
#To generate the xml file it requires nuget NunitXml.TestLogger
dotnet test "$APPCENTER_SOURCE_DIRECTORY" --logger:"nunit;LogFilePath=TestResults.xml"
echo "NUnit tests result:"
pathOfTestResults=$(find $APPCENTER_SOURCE_DIRECTORY -name 'TestResults.xml')
cat $pathOfTestResults
echo
#Looks for a failing test and causes the build to fail if found
grep -q 'result="Failed"' $pathOfTestResults
if [[ $? -eq 0 ]]
then
echo "A test Failed"
exit 1
else
echo "All tests passed"
fi
The last part should cause your AppCenter build to fail if a test does. Also, you may need to try building it twice in AppCenter before it picks up that a post build script has been added to your repo.
My understanding of the blog is that the UITest used to based on nunit 2.x. So, previously if you accidentally updated to nunit3.x your UITests wouldn't work.
They have now updated it so that you can use nunit 3.x to run your UI tests.
I believe #Nick Peppers give the correct approach. Sample post build script here:
https://github.com/microsoft/appcenter/blob/master/sample-build-scripts/xamarin/nunit-test/appcenter-post-build.sh
Related
I have created Xamarin.UITest that can run locally on my desktop. My goal is to execute these test as a part of a post-build script to run UITest after the app has built as mentioned in this article below:
https://tomsoderling.github.io/AppCenter-Automated-UI-tests-on-build/
Below is my script
appcenter test run uitest --app "MY-APP" --devices 168683d9 --app-path $APPCENTER_OUTPUT_DIRECTORY/MyApp.Mobile.Droid.apk --test-series "myapp-mobile-test" --locale "en_US" --build-dir $APPCENTER_SOURCE_DIRECTORY/MyApp.Mobile.UITests/bin/Release --token MY-TOKEN --uitest-tools-dir $APPCENTER_SOURCE_DIRECTORY/packages/Xamarin.UITest.*/tools
When the script above is apart of my appcenter post build script, I get the following error:
Error: Cannot find test-cloud.exe, the path specified by "--uitest-tools-dir" was not found.
Please check that "/Users/vsts/agent/2.141.1/work/1/s/packages/Xamarin.UITest.2.2.6/tools" is a valid directory and contains test-cloud.exe.
Minimum required version is "2.2.0".
I think a lot of people are having trouble dealing with this actually and I know it has something to do with --uitest-tools-dir OR --build-dir variables.
Keep in mind this I am first trying to do this with Xamarin.Android, if successful I will try the Xamarin.iOS
One clue i do see is when Tom says "I had to chose to build the app solution file in my App Center CI build - not simply the iOS project like I normally would" in the noted article, but I am not quite sure how to do that or if is connected to why AppCenter cannot locate my test-cloud.exe I will also say that test-cloud.exe somehow comes from the Xamarin.UITest nuget, but I do not see any test-cloud.exe file in my Xamarin.Forms project.
This answer works, but it's pretty fragile.
The test-cloud.exe can't be located at packages/Xamarin.UITest.2.X.X/tools in projects that uses the old project structure (projects that use packages.config). For new projects (new .csproj formats), there is no such file in the path of the project. The only way I found to make it work on AppCenter is to use it from the NuGet package cache (/Users/vsts/.nuget/packages/xamarin.uitest/2.X.X)
Kudos to AppCenter Agents for helping me to resolve this. 2 things were required as indicated below:
Agent Anvesh says
Hi there, Thanks for the details, So seems like you are using a nuget as a PackageReference in your project(this means that there will be no package folder in your project, packages will be there at user profile).
So when you are trying to run the test as part of the app center build. Then in the shell script used the --uitest-tools-dir value as below
/Users/vsts/.nuget/packages/xamarin.uitest/2.2.6/lib/tools
So I modified my above script to add the below:
--uitest-tools-dir /Users/vsts/.nuget/packages/xamarin.uitest/2.2.6/tools \
Agent Shawn says
So I added the below
msbuild $APPCENTER_SOURCE_DIRECTORY/MyApp.Mobile.UITestProject.csproj
I'm trying to run SoapUI tests as unit tests as described here:
http://blog.simplecode.eu/post/Soap-UI-testing-with-MsTest
Everything works nice when I'm running test locally in visual studio.
But when I'm trying to run those tests during the build process on tfs2015 i get "Error: Could not create the Java Virtual Machine":
My other unit tests during that build are performed correctly.
Anyone had similar issue?
It's not a really build error in TFS. You can see the result in your screenshot, build partially succeeded.
Currently, by default the build result is "Failed" if anything failed to compile, "Partially Succeeded" if there are any unit test failures, and "Succeeded" otherwise.
Build is reported as "Partially Succeeded" when the "Test Success" property is set to "False" and "CompilationSuccess"="True"
So your error is more like something in your code fails the test or some configuration related to test in the build definition or build agent environment. Since everything works nice when you are running test locally in visual studio.
Seems disable code coverage fixed the issue.
Manage, to fix that - problem was enabled code coverage. Disabling it fixed to problem.
I have a number of unit tests written for my project, executed with CTest. I would like to integrate the results into my TeamCity build. I've downloaded and set up the plugin for my testing framework (Boost Test).
The problem that I have run into is that the tests run with CTest output to Testing/Temporary/LastTest.log, whereas TeamCity is trying to read the results from standard out. To get around this, my testing step is.
make test
cat Testing/Temporary/LastTest.log
which works, but feels like a hack.
Is there any way to get TeamCity to read from this file in addition to standard out? Alternatively, is there any way to tell ctest to output to standard out in addition to this LastTest.log file?
This question is similar, but I would like it to work for all output rather than just on failure: CMake: setting an environmental variable for ctest (or otherwise getting failed test output from ctest/make test automatically)
Teamcity has additional build features which allow to process CTest reports. I am not sure if it'll work or not but you could try adding an additional build feature in your build step to read CTest report.
So I have a go project with go-gettable dependencies, tests, etc.
I want to integrate it into Jenkins. Is there an automated build system for go that anyone recommends for this use case, other than writing makefiles?
I need:
automatic installation of go-get deps (they can be in a spec file of course)
recursive build.
running tests.
GOPATH/GOROOT management (to isolate SDKs and paths)
I've used godag in the past for this sort of job but it seems a bit unmaintained.
EDIT: For the time being I'm living with the following script entered directly into Jenkins as a build step:
#this gets the dependencies but doesn't install them, avoiding permission problems
go get -d
#build the packages, -x outputs the compiler command line
go build -x
#this was tricky - ./... means "for each sub-package recursively"
go test ./...
You can do it with teamcity as well.
Here is an example for building terraform.
Teamcity agent setup:
Install Go
Add Go to path
Don't forget to restart agent as path was changed
Teamcity build setup:
Use agent side checkout (if we want to include the .git folder, which a lot of build scripts make use of)
Use checkout rule (we want to use the Go convention):
+:. => src/github.com/mitchellh/terraform
Build steps:
echo cd %system.teamcity.build.checkoutDir%
cd "%system.teamcity.build.checkoutDir%"
path=C:\Program Files (x86)\Git\bin;%env.Path%;%system.teamcity.build.checkoutDir%\bin;
echo path=C:\Program Files (x86)\Git\bin;%env.Path%;%system.teamcity.build.checkoutDir%\bin;
set GOPATH=%system.teamcity.build.checkoutDir%
echo set GOPATH=%system.teamcity.build.checkoutDir%
echo "Getting dependancies"
go get golang.org/x/tools/cmd/vet
go get golang.org/x/tools/cmd/cover
go get golang.org/x/tools/cmd/stringer
go get github.com/mitchellh/gox
echo cd %system.teamcity.build.checkoutDir%\src\github.com\mitchellh\terraform
cd "%system.teamcity.build.checkoutDir%\src\github.com\mitchellh\terraform"
echo "Update resources"
go get ./...
echo "Run static code analysis"
go tool vet -asmdecl -atomic -bool -buildtags -copylocks -methods -nilfunc -printf -rangeloops -shift -structtags -unsafeptr .
echo "Build"
cd scripts
sh build.sh
echo "Run unit tests"
go test -timeout=30s -parallel=4
echo "Run code coverage"
go test -coverprofile=coverage.out
go tool cover -html=coverage.out
rm coverage.out
I am using a Team City build server on a Mac that runs a rake file, in the rake file I do all the commands to get dependancies, (go gets), tests and builds to the correct environment.
let me know if you want any pointers in writing the Rake files,
As a side note, i have been creating functional tests for REST Api's using this frame work. This has saved my code many times over. http://github.com/DigitalInnovation/cucumber_rest_api
Since August 2019, TeamCity 2019.1 now supports Go directly.
See "Building Go programs in TeamCity"
To enable Go support in TeamCity,
go to Build Configuration Settings | Build Features,
click Add build feature, and
select Golang from the list.
Support for Go is provided by TeamCity out of the box, there are no external plugins required.
TeamCity parses results of go test command execution. The results are persisted and it is possible to review the figures in a historical perspective.
Consequently, all the TeamCity features that are related to test reporting are now available for Go developers.
How do I run MSTest as part of my build process in TeamCity? What are the pitfalls?
This answer is specifically for TeamCity 7.1 on Windows, but may apply to other environments.
In your TeamCity build configuration, on the General Settings page
Artifact paths: Artifacts\MSTest => MSTest
Create a new Command Linebuild step
Custom script: if not exist Artifacts\MSTest mkdir Artifacts\MSTest
Create a new MSTestbuild step
List assembly files: **\bin\**\*.Tests.dll
Results file: Artifacts\MSTest\testResults.trx
Pitfalls
Using wildcards when specifying which test assemblies to run
You can use wildcards when specifying which test assemblies to run in the MSTest build step, although it is unclear exactly how they work. A bug report has been filed.
The build process doesn't stop when tests fail
Be aware that if some of your tests fail and the build is marked as failed, the MSTest build step itself does not fail. This causes problems if you have build steps after the MSTest build step which you don't want to run if you have test failures (e.g. it may not make sense to produce an installer or documentation of a build you know has bugs). The problem will hopefully be fixed in later versions of TeamCity.
If you want your build process to stop when you have test failures, you can create a new build step that uses the TeamCity REST API to detect if the current build has been marked as failed (remember that when tests fail, the build step is not marked as failed, but the build is), and then fail the current build step explicitly. Example:
Create a new Powershell build step
Script: Source code
Source code: See script below
Make sure your newly created build step comes immediately after your MSTest build step
Make sure every build step after this one has Execute step set to Only if all previous steps were successful
Script:
$xml = [xml](curl --request GET http://USERNAME:PASSWORD#HOSTNAME/httpAuth/app/rest/builds/%teamcity.build.id%)
Microsoft.PowerShell.Utility\Select-Xml $xml -XPath "/build" | % { $status = $_.Node.status }
if ($status -eq "FAILURE") {
throw "Failing build step on purpose"
}