TeamCity: how to create custom statiscics from a build - teamcity

I would like to add my own statistics graph to the build. For example, how many tests passed and how many failed.
[==========] 85 tests from 16 test suites ran. (1372 ms total)
[ PASSED ] 85 tests.
Yes, there is a TeamCity plugin available to interface with google test. But it hasn't been updated for a while and I would like to avoid another plugin.
Is there some syntax that will allow me to add another line to the build script to grep for the number of tests and update some sort of TeamCity variable that can be used to create a custom chart?

You can import any test results with help of service messages: https://www.jetbrains.com/help/teamcity/build-script-interaction-with-teamcity.html#BuildScriptInteractionwithTeamCity-ServiceMessages
Basically your build process should produce special stdout which will be parsed by TeamCity and imported into the TeamCity database.
For example, the following lines in stdout will report a single passed test with name 'test1' to TeamCity:
##teamcity[testStarted name='test1']
##teamcity[testFinished name='test1']

Related

Run Surefire tests in Maven with HTML output & a nonzero exit code on failure

I'm working on a Gitlab CI/CD pipeline for various Dockerized Java apps, using Maven to build them. The first step of the pipeline builds the app and pushes it to our registry. The second step is supposed to run unit tests and ensure they all pass before allowing the pipeline to continue to the deployment stage.
The problem I'm having is finding a way to invoke Maven which does what I want. What I need the test step to do is:
Execute the tests once with Surefire
Print to STDOUT or STDERR the test results (so they are visible in the job log)
Output XML test reports which will be uploaded to Gitlab as job artifacts (which will be parsed by Gitlab to populate the test results in the web GUI)
Output an HTML test report which will be uploaded to Gitlab as a job artifact (to give a downloadable, human-readable, all-in-one summary of the test results)
Exit with a nonzero return code if and only if any tests failed (so that the pipeline job will be marked as failed in the Gitlab pipeline)
1-3 are handled automatically by Surefire. The problem is 4 & 5; I'm using the Surefire-report plugin to generate the HTML report, but I can't find a way to invoke it which does what I want:
If I invoke mvn 'test' 'surefire-report:report', then, if any tests fail, Maven exits with a nonzero exit code but the HTML report is not generated.
I think this is because, in the case of a failed test, Maven considers the test goal to have failed and exits without running the surefire-report goal at all
If there are no failed tests, then the surefire-report goal does run and the report is generated, but a report that's only generated when everything passed is of limited use.
If I invoke mvn 'surefire-report:report' or mvn 'surefire-report:report' 'test', then, if any tests fail, an HTML report is still generated, but the return code is 0.
I think this is because Maven considers the surefire-report goal to be successful as long as it successfully generates a report, and then considers the test goal to be redundant because it already ran the tests in order to generate the report
My current workaround is mvn 'surefire-report:report' && mvn 'test', which works, but also runs the tests twice (or at least prints their results to STDOUT twice), and also requires the Docker container in which the tests are run to use a shell script instead of running a single Maven command
Someone suggested mvn 'test' ; result=$? ; mvn 'surefire-report:report-only' && exit ${result}, which seems to work, but I'd much rather have a single command rather than a shell script

How to set name of the artifacts produced from Gradle to values passed from Jenkins pipeline

I am trying to setup a Jenkins pipeline to trigger builds using gradle for multiple environments.My requirement is that the artifacts produced when I run gradlew clean build should produce artifacts with name indicating the environment for which the pipeline was run. Example my-application-dev.jar
The value of the environment would be selected by the user when build will be triggered.
What is the optimal way to achieve this ? Does build allow to configure any such property via command line or do I need to define a task in my build.gradle and define properties within that task for which I will pass value from command line
There are basically two ways.
The first one is to pass these naming-relevant pieces of information to the gradlew process, e.g. via -D or -P properties.
Then you need the Gradle build to be aware of these parameters and craft the artifact names it produces accordingly.
The second one is arguably better and more contained. You simply rename the artifacts produced by the gradlew command after it completes, in the Jenkinsfile. This works well if the pipeline decides what to do with these artifacts (e.g. publish to a repository) as opposed to the gradle script doing it (in which case you would most likely be better off using the first method).

How do I make Bamboo understand phpspec tests?

Bamboo gives us the ability to run phpunit tests, but likely given a lower popularity, nothing for phpspec.
I suspect however (Googling) that it must be possible!
https://revive.beccati.com/bamboo/browse/PHP-PHPSPEC-813/test
Has anyone successfully fed phpspec tests into Bamboo?
Thanks.
One option is to run the phpspec test as a Script Task. If the task returns a non-0 exit code, the build fails. phpspec obligingly returns a 0 exit code only when all your tests pass.
In order to do this you would need to ensure that phpspec is available to your Bamboo build. If you're using Composer, you can add it to the require-dev section of your composer.json file.
Then, in the Plan Configuration, under Default Job (or some other job), you can add a new task of type Script. This Script Task can then make a call to phpspec:
vendor/bin/phpspec run 1>&2
You may also want (as above) to redirect the output to stderr because Bamboo seems to suppress any output on stdout. This will then allow you to see the output of phpspec in your Bamboo log.
The answer was to run the test with the junit formatter. Bamboo has built-in support for the former, and this made the tests run smoothly.

Android gradle skip build before running test?

I would like to skip the build process and directly run tests when I do
./gradlew connectedDevDebug
I could also use adb command but it will run test on only one deivce at a time.
There is currently no gradle tasks that just runs the tests.
But once you have installed (and run) the tests, you can (re-)run them directly with the am instrument command. To start them from your development machine's command line just run:
adb shell am instrument -w <test_package_name>/<runner_class>
You can copy the actual command from AndroidStudio's output. Just run the tests from AndroidStudio, and then scroll to the top of the test log view.
There is one caveat, am instrument does not create any test-reports. All test results are written to stdout. However you could pipe stdout into a file and create a report yourself. I.e. this tool can create an xml JUnit test report from the output of am instrument.
If you have made changes to the tests you can rebuild and install them with:
./gradlew installDebugAndroidTest
For detailed information and instructions about starting tests from the command line you can refer to the official article Test from the Command Line

TeamCity and CTest test results

I have a number of unit tests written for my project, executed with CTest. I would like to integrate the results into my TeamCity build. I've downloaded and set up the plugin for my testing framework (Boost Test).
The problem that I have run into is that the tests run with CTest output to Testing/Temporary/LastTest.log, whereas TeamCity is trying to read the results from standard out. To get around this, my testing step is.
make test
cat Testing/Temporary/LastTest.log
which works, but feels like a hack.
Is there any way to get TeamCity to read from this file in addition to standard out? Alternatively, is there any way to tell ctest to output to standard out in addition to this LastTest.log file?
This question is similar, but I would like it to work for all output rather than just on failure: CMake: setting an environmental variable for ctest (or otherwise getting failed test output from ctest/make test automatically)
Teamcity has additional build features which allow to process CTest reports. I am not sure if it'll work or not but you could try adding an additional build feature in your build step to read CTest report.

Resources