Let's pretend that we have a jenkins job called functional-tests. I am looking for a way to configure this job, so that when it fails due to one or more of the tests failing, on the next run (e.g. triggered by a commit) it only runs the failed tests.
Now if this run was successful, it automatically triggers another build of the job to run all the tests.
However if the run was unsuccessful, it will continue to only run the tests that were failed in the previous run.
I think the only thing I need, is to pass the state of the previous build as well as the tests that were failed to the current job.
We are using Maven and Surefire for the build.
Related
I have run the cypress suite then I got the pass and fail result on Cypress Dashboard. After that, I fixed the issue for those failed test cases on the new branch. I am not able to run the workflow for failed test cases of the last run on the new branch through the GitHub action
My requirement - How to run only those failed test cases of the last run on the new branch through GitHub action or any other ways?
I'm working on a Gitlab CI/CD pipeline for various Dockerized Java apps, using Maven to build them. The first step of the pipeline builds the app and pushes it to our registry. The second step is supposed to run unit tests and ensure they all pass before allowing the pipeline to continue to the deployment stage.
The problem I'm having is finding a way to invoke Maven which does what I want. What I need the test step to do is:
Execute the tests once with Surefire
Print to STDOUT or STDERR the test results (so they are visible in the job log)
Output XML test reports which will be uploaded to Gitlab as job artifacts (which will be parsed by Gitlab to populate the test results in the web GUI)
Output an HTML test report which will be uploaded to Gitlab as a job artifact (to give a downloadable, human-readable, all-in-one summary of the test results)
Exit with a nonzero return code if and only if any tests failed (so that the pipeline job will be marked as failed in the Gitlab pipeline)
1-3 are handled automatically by Surefire. The problem is 4 & 5; I'm using the Surefire-report plugin to generate the HTML report, but I can't find a way to invoke it which does what I want:
If I invoke mvn 'test' 'surefire-report:report', then, if any tests fail, Maven exits with a nonzero exit code but the HTML report is not generated.
I think this is because, in the case of a failed test, Maven considers the test goal to have failed and exits without running the surefire-report goal at all
If there are no failed tests, then the surefire-report goal does run and the report is generated, but a report that's only generated when everything passed is of limited use.
If I invoke mvn 'surefire-report:report' or mvn 'surefire-report:report' 'test', then, if any tests fail, an HTML report is still generated, but the return code is 0.
I think this is because Maven considers the surefire-report goal to be successful as long as it successfully generates a report, and then considers the test goal to be redundant because it already ran the tests in order to generate the report
My current workaround is mvn 'surefire-report:report' && mvn 'test', which works, but also runs the tests twice (or at least prints their results to STDOUT twice), and also requires the Docker container in which the tests are run to use a shell script instead of running a single Maven command
Someone suggested mvn 'test' ; result=$? ; mvn 'surefire-report:report-only' && exit ${result}, which seems to work, but I'd much rather have a single command rather than a shell script
We have close to 100 test scenarios based on QAF in feature files. I have a testng file which I invoke from command prompt through "mvn test" command.
Some tests fail occasionally, but when I run them again they pass. Is there some way where I can run only the failed test cases again?
There are two ways to run failed tests:
run failed test immediately it failed by setting retry.count property. Refer list of properties
Run failed test after completion of entire execution by running generated failed test xml file under test results.
We have a TeamCity build configuration which does a deploy and then runs integration tests.
Deploy system
Run test suite A
Run test suite B
Run test suite C
If test suite A fails, B and C should still be run (likewise C should run if B fails). To satisfy this, the build steps are set to run "Even if some of the previous steps failed". However, I don't want any of the tests to run if the first step to deploy the system fails.
Is there a way of terminating the build if the deployment fails, but to keep running all tests of there are individual tests which fail?
You could chain the builds together so have a build for 'Deploy the system' and then have a separate build for 'Run the tests' which has your 3 steps A,B and C in it. The second build takes a snapshot dependency on the first build which means that it will kick off when the 'Deploy' build has completed, but it won't kick off if the build fails.
The steps in the second build could then be set to run even if the previous steps fail as you have it now and they would all run.
I'm trying to integrate fitnesse with our cruisecontrol setup.
I'd like to have a set of acceptance tests that we develop before the features for the release are worked on. Develop our acceptance tests, run them against our nightly build (not on every check in, we have a job for that but I suspect our acceptance tests would slow that down too much).
So I want to be able to run the fitnesse test suite and not have it fail the build when some tests fail (the expectation is that some of the tests will fail, until we have finished the release).
I have the acceptance tests building on the integration server and the the fitnesse suite running from the command line (as an exec task in the integration job).
At the moment it is failing the build (runner.exe has a non-zero exit code when any test fails).
So... does anyone have a list of exit codes for fitsharp runner.exe? Is there any way to tell a cruisecontrol exec task that I really don't care about the return value from the job? Is there another cc.net task I should use instead?
Edit:
Current best idea is to wrap the fitsharp runner in a batch file or powershell script and swallow the return code from fitness
Edit2:
Return code from fitsharp runner.exe is the number of tests that failed (making setting the success return codes element for the cruisecontrol.net exec task difficult)
I think the best way to do it is with nant, which integrates very nicely with ccnet. You can simply tell a nant exec task to not fail the build, see http://nant.sourceforge.net/release/latest/help/tasks/exec.html
When you're close to release, simply set the failonerror property to true.