i use MSTest with CruiseControl. When I run my compile and tests and all tests pass then I can see the mstest results in my dashboard.
for example: 5 tests ran. 5 tests passed. 0 tests failed.
when tests fail I would expect to see something like this:
5 tests ran. 1 failed. 4 passed.
but if one test fails I get this:
0 tests ran. 0 passed. 0 failed.
It looks like cruisecontrol has a display problem if tests fail.
bye bembi
solved it.
i had to move the file merge block out of the task block and into a publishers block:
<project>
...
<publishers>
<merge>
<files>
<file>C:\cc\mstestResult.trx</file>
</files>
</merge>
<xmllogger />
</publishers>
</project>
Related
I'm trying to configure a Maven job with tests that should do two things:
Generate HTML report with test results.
Fail if at least 1 test was failed during the run.
I thought it was a simple case and was stuck because one of the conditions always blocks the second. For example:
mvn surefire-report:report site - Maven result is always successful, not depending on the tests failures.
mvn test surefire-report:report site - the 'test' step generates an error, the following steps are ignored, and the report isn't generated. '--fail-at-end' flag didn't change anything in this case.
Is there any way to keep both my conditions in a single run: to generate a report and have some exit code on the tests failures?
So, I didn't find any way to do this except by using two different calls to maven:
mvn clean test <my params>
result=$?
mvn surefire-report:report-only site
exit $result
In this case, the first call gets and saves the exit code, which equals 1 if at least one failed test. The second call publishes reports and always returns the successful result, which I ignore.
I'm working on a Gitlab CI/CD pipeline for various Dockerized Java apps, using Maven to build them. The first step of the pipeline builds the app and pushes it to our registry. The second step is supposed to run unit tests and ensure they all pass before allowing the pipeline to continue to the deployment stage.
The problem I'm having is finding a way to invoke Maven which does what I want. What I need the test step to do is:
Execute the tests once with Surefire
Print to STDOUT or STDERR the test results (so they are visible in the job log)
Output XML test reports which will be uploaded to Gitlab as job artifacts (which will be parsed by Gitlab to populate the test results in the web GUI)
Output an HTML test report which will be uploaded to Gitlab as a job artifact (to give a downloadable, human-readable, all-in-one summary of the test results)
Exit with a nonzero return code if and only if any tests failed (so that the pipeline job will be marked as failed in the Gitlab pipeline)
1-3 are handled automatically by Surefire. The problem is 4 & 5; I'm using the Surefire-report plugin to generate the HTML report, but I can't find a way to invoke it which does what I want:
If I invoke mvn 'test' 'surefire-report:report', then, if any tests fail, Maven exits with a nonzero exit code but the HTML report is not generated.
I think this is because, in the case of a failed test, Maven considers the test goal to have failed and exits without running the surefire-report goal at all
If there are no failed tests, then the surefire-report goal does run and the report is generated, but a report that's only generated when everything passed is of limited use.
If I invoke mvn 'surefire-report:report' or mvn 'surefire-report:report' 'test', then, if any tests fail, an HTML report is still generated, but the return code is 0.
I think this is because Maven considers the surefire-report goal to be successful as long as it successfully generates a report, and then considers the test goal to be redundant because it already ran the tests in order to generate the report
My current workaround is mvn 'surefire-report:report' && mvn 'test', which works, but also runs the tests twice (or at least prints their results to STDOUT twice), and also requires the Docker container in which the tests are run to use a shell script instead of running a single Maven command
Someone suggested mvn 'test' ; result=$? ; mvn 'surefire-report:report-only' && exit ${result}, which seems to work, but I'd much rather have a single command rather than a shell script
I've configured and am executing mocha tests in WebStorm, so I know the module is working properly. But I can't seem to make it run from a Bamboo task. The task runs with Success but there are 0 tests executed.
This is my configuration atm:
app/ is my working dir. Tried also with app/node_modules/mocha/bin/ and other possibilities. I am not sure which exactly is the Mocha executable of all the mocha named files in the module...
Or maybe the problem lies in the tests dir? I've got test files, respectively in app/test/unit/models/ and app/test/unit/services/. But in WebStorm I configured it with the general test dir - just /app/test. Configuring the Mocha task in Bamboo with the specific test folders did not yield result...
I believe the problem comes from wrong directory configurations in the task, but I've tried writing whatever paths already and I've got no idea what's missing or wrong...
I noticed from your screenshot that the "Parse test results produced by this task" box isn't checked. This is what tells Bamboo to parse the output of the tests that you run.
I have a suite of tests that are launched in continuous integration using parallel test. After the deploys the environment is a little unstable, so when the suite is finished, the failed tests are relaunched (by this moment the environment is stable again).
Basically, there is only one jenkins build, and it will execute this:
task :jenkins1_with_rerun do
selenium_successful = system "bundle exec parallel_cucumber features/web/ -o \"-p jenkins1\" -n 3"
p 'start rerun'
rerun_successful = run_rake_task("features:jenkins_rerun")
unless selenium_successful || rerun_successful
raise 'Cucumber tests failed'
end
end
The first execution launches the suite with parallel tests. Here is the cucumber profile:
jenkins1:
<%= std_opts %>
HEADLESS=true
--tags #part1
--tags ~#wip
--format junit
--out log/junit
--format ParallelTests::Cucumber::FailuresLogger
--out parallel_cucumber_failures.log
After this is finished. The second execution starts, launching the failed tests recorded in a file. The cucumber profile for this execution is this:
jenkins_rerun:
ENVIRONMENT=master
SCREENSHOT=true
HEADLESS=true
--format pretty
--no-source
--format junit
--out log/junit_rerun
#parallel_cucumber_failures.log
Now. Everything works. The only problem is that I use the junit report in jenkins to make graphs of failed and successful tests. I may have reports like this:
-------------------Tests---KO---OK
First execution---75----10----65
Rerun-------------10-------0----10
This is 100% green, because all the problems were caused by instability after the deploy. So I want the junit report in jenkins to said that 75 tests have been launched, 75 OK, 0 KO, or something equivalent.
Right now, the junit report of the first execution says that of 75 tests, I have 10 KOs; and the second junit says that of 10 tests, there are 0 KOs.
What may be a good solution to this? It possible to mix the results of both Junit reports?
I will also accept to be able to display in jenkins both junit reports, each one with a graph. But I think that jenkins only allows to show one junit report graph.
I finally resolved this problem.
The solution was to create a "parser" that searches (with RegEx) through the Junit files generated by the rerun, and then goes to the Jnuit files of the original/first run and rewrites the results of those tests that have passed during the rerun.
In the end, if everything is ok in the rerun, the junit graph is 100% green and there are not alerts of failures.
As part of the continuous integration task I'm owning, I have been able to add lots of selenium WebDriver scripts in Java and Eclipse. Basically, using a maven project, I have written various java scripts using Selenium and am now able to run the automated tests using Jenkins after the nightly build is deployed. In addition, I am able to successfully create a surefire-report.html. Howoever, this report only contains the following matrix:
Tests Errors Failures Skipped Success Rate Time
38 4 2 18 36.842% 746.477
The development manager has requested to see the following matrix as well:
TRR = (Number of tests run) / (Total number of tests available)
TPR = (Number of test passed) / (number of tests run)
Is there any way I can modify the surefire report in order to add the above matrix results as well?
Look for PerfPublisher plugin in Jenkins.
https://wiki.jenkins-ci.org/display/JENKINS/PerfPublisher+Plugin
This will by default give you the following things:
1. Number of tests run
2. Number of total tests.
3. Number of tests passed and failed.
So you have all required values there :)
The only change you would need to do is to create an XML file of particular format:
<report name="{REPORT_NAME}" categ="{CATEGORY_NAME}">
<test name="{TEST_NAME1}" executed="{EXECUTED_STATUS}">
<result>
<success passed="{RESULT_STATUS}" state="{RESULT_STATE}"/>
</result>
</test>
<test name="{TEST_NAME2}" executed="{EXECUTED_STATUS}">
<result>
<success passed="{RESULT_STATUS}" state="{RESULT_STATE}"/>
</result>
</test>
</report>
Hope this is useful.