We are using Maven for script execution. For reporting we have used TestNG.
In case test failed for first time we want to re execute those failed tests again.
Is there any option available?
Implement IRetryAnalyzer interface to rerun failed test cases with a count to re execute. Add #Test(retryAnalyzer=<classname>) to test case.
Related
I'm trying to configure a Maven job with tests that should do two things:
Generate HTML report with test results.
Fail if at least 1 test was failed during the run.
I thought it was a simple case and was stuck because one of the conditions always blocks the second. For example:
mvn surefire-report:report site - Maven result is always successful, not depending on the tests failures.
mvn test surefire-report:report site - the 'test' step generates an error, the following steps are ignored, and the report isn't generated. '--fail-at-end' flag didn't change anything in this case.
Is there any way to keep both my conditions in a single run: to generate a report and have some exit code on the tests failures?
So, I didn't find any way to do this except by using two different calls to maven:
mvn clean test <my params>
result=$?
mvn surefire-report:report-only site
exit $result
In this case, the first call gets and saves the exit code, which equals 1 if at least one failed test. The second call publishes reports and always returns the successful result, which I ignore.
I'm working on a Gitlab CI/CD pipeline for various Dockerized Java apps, using Maven to build them. The first step of the pipeline builds the app and pushes it to our registry. The second step is supposed to run unit tests and ensure they all pass before allowing the pipeline to continue to the deployment stage.
The problem I'm having is finding a way to invoke Maven which does what I want. What I need the test step to do is:
Execute the tests once with Surefire
Print to STDOUT or STDERR the test results (so they are visible in the job log)
Output XML test reports which will be uploaded to Gitlab as job artifacts (which will be parsed by Gitlab to populate the test results in the web GUI)
Output an HTML test report which will be uploaded to Gitlab as a job artifact (to give a downloadable, human-readable, all-in-one summary of the test results)
Exit with a nonzero return code if and only if any tests failed (so that the pipeline job will be marked as failed in the Gitlab pipeline)
1-3 are handled automatically by Surefire. The problem is 4 & 5; I'm using the Surefire-report plugin to generate the HTML report, but I can't find a way to invoke it which does what I want:
If I invoke mvn 'test' 'surefire-report:report', then, if any tests fail, Maven exits with a nonzero exit code but the HTML report is not generated.
I think this is because, in the case of a failed test, Maven considers the test goal to have failed and exits without running the surefire-report goal at all
If there are no failed tests, then the surefire-report goal does run and the report is generated, but a report that's only generated when everything passed is of limited use.
If I invoke mvn 'surefire-report:report' or mvn 'surefire-report:report' 'test', then, if any tests fail, an HTML report is still generated, but the return code is 0.
I think this is because Maven considers the surefire-report goal to be successful as long as it successfully generates a report, and then considers the test goal to be redundant because it already ran the tests in order to generate the report
My current workaround is mvn 'surefire-report:report' && mvn 'test', which works, but also runs the tests twice (or at least prints their results to STDOUT twice), and also requires the Docker container in which the tests are run to use a shell script instead of running a single Maven command
Someone suggested mvn 'test' ; result=$? ; mvn 'surefire-report:report-only' && exit ${result}, which seems to work, but I'd much rather have a single command rather than a shell script
We have close to 100 test scenarios based on QAF in feature files. I have a testng file which I invoke from command prompt through "mvn test" command.
Some tests fail occasionally, but when I run them again they pass. Is there some way where I can run only the failed test cases again?
There are two ways to run failed tests:
run failed test immediately it failed by setting retry.count property. Refer list of properties
Run failed test after completion of entire execution by running generated failed test xml file under test results.
Let's pretend that we have a jenkins job called functional-tests. I am looking for a way to configure this job, so that when it fails due to one or more of the tests failing, on the next run (e.g. triggered by a commit) it only runs the failed tests.
Now if this run was successful, it automatically triggers another build of the job to run all the tests.
However if the run was unsuccessful, it will continue to only run the tests that were failed in the previous run.
I think the only thing I need, is to pass the state of the previous build as well as the tests that were failed to the current job.
We are using Maven and Surefire for the build.
I was wondering whether it's possible to obtain information about test execution order somehow.
I have a project in Maven and all the tests are passing. After I migrated the project to Gradle, one of the tests started failing. The test itself is working: when I execute gradle test -Dtest.single=..., it passes. However, when I run the tests for the whole project, the test fails.
It's possible that some tests that run before the failing test do not release resources correctly and therefore the test fails. But I need to somehow find out which tests are causing this problem.
Tell Gradle to log more events about test processing. There is a documentation how to do that http://www.gradle.org/docs/current/dsl/org.gradle.api.tasks.testing.logging.TestLoggingContainer.html
The beforeTest method can be used to output details about each test before it is run, based on a TestDescriptor for each test:
test {
beforeTest { testDescriptor ->
println "${testDescriptor.className} > ${testDescriptor.name} STARTED"
}
}