When a TestNG test fails I am retrying it once. If the retry is successful, then the build should be successful. If both the retry and the original test fail, the build should fail. The ignoreFailures option will ignore all failed tests and mark the build successful. I would like a more selective option. If such a feature doesn't exist, I can calculate in the afterSuite hook whether the suite was a success or failure; I would just need a way to manually mark the build as such.
You have to do it manually.
If your are using IRetryAnalyzer it will mark all your failed tries as skipped. For gradle test report it would be looks like ignored tests and final build result is successful.
Suite state is private property, so changing is not be a good idea. Also it has only one option - hasFailures.
Related
I'm trying to configure a Maven job with tests that should do two things:
Generate HTML report with test results.
Fail if at least 1 test was failed during the run.
I thought it was a simple case and was stuck because one of the conditions always blocks the second. For example:
mvn surefire-report:report site - Maven result is always successful, not depending on the tests failures.
mvn test surefire-report:report site - the 'test' step generates an error, the following steps are ignored, and the report isn't generated. '--fail-at-end' flag didn't change anything in this case.
Is there any way to keep both my conditions in a single run: to generate a report and have some exit code on the tests failures?
So, I didn't find any way to do this except by using two different calls to maven:
mvn clean test <my params>
result=$?
mvn surefire-report:report-only site
exit $result
In this case, the first call gets and saves the exit code, which equals 1 if at least one failed test. The second call publishes reports and always returns the successful result, which I ignore.
I'm working on a Gitlab CI/CD pipeline for various Dockerized Java apps, using Maven to build them. The first step of the pipeline builds the app and pushes it to our registry. The second step is supposed to run unit tests and ensure they all pass before allowing the pipeline to continue to the deployment stage.
The problem I'm having is finding a way to invoke Maven which does what I want. What I need the test step to do is:
Execute the tests once with Surefire
Print to STDOUT or STDERR the test results (so they are visible in the job log)
Output XML test reports which will be uploaded to Gitlab as job artifacts (which will be parsed by Gitlab to populate the test results in the web GUI)
Output an HTML test report which will be uploaded to Gitlab as a job artifact (to give a downloadable, human-readable, all-in-one summary of the test results)
Exit with a nonzero return code if and only if any tests failed (so that the pipeline job will be marked as failed in the Gitlab pipeline)
1-3 are handled automatically by Surefire. The problem is 4 & 5; I'm using the Surefire-report plugin to generate the HTML report, but I can't find a way to invoke it which does what I want:
If I invoke mvn 'test' 'surefire-report:report', then, if any tests fail, Maven exits with a nonzero exit code but the HTML report is not generated.
I think this is because, in the case of a failed test, Maven considers the test goal to have failed and exits without running the surefire-report goal at all
If there are no failed tests, then the surefire-report goal does run and the report is generated, but a report that's only generated when everything passed is of limited use.
If I invoke mvn 'surefire-report:report' or mvn 'surefire-report:report' 'test', then, if any tests fail, an HTML report is still generated, but the return code is 0.
I think this is because Maven considers the surefire-report goal to be successful as long as it successfully generates a report, and then considers the test goal to be redundant because it already ran the tests in order to generate the report
My current workaround is mvn 'surefire-report:report' && mvn 'test', which works, but also runs the tests twice (or at least prints their results to STDOUT twice), and also requires the Docker container in which the tests are run to use a shell script instead of running a single Maven command
Someone suggested mvn 'test' ; result=$? ; mvn 'surefire-report:report-only' && exit ${result}, which seems to work, but I'd much rather have a single command rather than a shell script
I have several build steps that need to run regardless of (test failure, in this case).
However when tests in one step fail, the whole build fails and quits. To overcome this problem I unchecked the "build process exit code is not zero".
After this the build fails more quickly at step 2, which is install of grunt-cli
Is there a (better) way to have my build continue, even on a non-zero exit from a previous step? I tried muting, but this is not what I was hoping for.
TeamCity Enterprise 9.1.7 (build 37573)
If this is a commandline build step, you can add "exit 0" to force the exit code to always be 0. Since you are reporting tests, you will still see the failed tests in TeamCity.
When you configure the build stteps, you can set an execution condition of Execution policy to: Even if some of previous steps failed
You can also set to: Always, even if build stop command was issued if you want to do some cleanup on your agent after the execution of your tests.
We are using Maven for script execution. For reporting we have used TestNG.
In case test failed for first time we want to re execute those failed tests again.
Is there any option available?
Implement IRetryAnalyzer interface to rerun failed test cases with a count to re execute. Add #Test(retryAnalyzer=<classname>) to test case.
Let's pretend that we have a jenkins job called functional-tests. I am looking for a way to configure this job, so that when it fails due to one or more of the tests failing, on the next run (e.g. triggered by a commit) it only runs the failed tests.
Now if this run was successful, it automatically triggers another build of the job to run all the tests.
However if the run was unsuccessful, it will continue to only run the tests that were failed in the previous run.
I think the only thing I need, is to pass the state of the previous build as well as the tests that were failed to the current job.
We are using Maven and Surefire for the build.