My Maven build uses JUnit tests with the Surefire plugin. If a test fails, an mvn install will fail, which is normally desirable, but sometimes we wish to check in all the tests in advance for a feature which is not fully implemented. We would like be able to specify certain tests as "known failures" which will log their failure but not stop the build. Is there a way to do this?
(I'm aware we can set testFailureIgnore to true to not fail on any test failures, but we'd still like to fail on 'unexpected' test failures)
Related
Suppose I have a multi-module maven project. I want to run it in such a way that:
If a test fails, I still want to compile and test the next module. This sounds like a job for --fail-never, but see below.
If there are test failures, I don't really care if the build succeeds or fails. Ideally this would be configurable, but whatever.
If there are compiler errors, then the build should fail. I think this rules out "fail-never"?
You can use surefire parameter: maven.test.failure.ignore
mvn test -Dmaven.test.failure.ignore
With this options failing test will not fail build a module and finally whole build will have success result.
When generating a skeleton Maven plugin from archetype, the new project includes a Maven project under the src/it directory. It is an integration it (hinted at by the it dir name) and fresh out-of-the-box it passes when run during Maven's integration-test phase.
There are nearly 10 such IT Maven projects, a subset of which intentionally result in BUILD FAILURE, and attendant verify.groovy scripts that ensure those builds fail for the correct reason. Ideally each IT test sub-build that fails for the correct reason results in that IT test passing, but by including any of these failing IT tests as part of the whole integration test suite causes the overall Maven run to fail as well, which is incorrect in my case.
How do I coax Maven to run those failing Maven sub-builds, ignore their build results, but honor the results of their Groovy verification scripts?
Edit: One IT test (disabled) is committed here.
If you like to write an integration test which is intended to fail as a result
you have to express this via the invoker.properties file like this:
invoker.buildResult=failure
The full description of the file can be found in the documentation.
Some of test cases are failed so due to that maven build is also failed .Just because of maven failure , Jenkins build is also failed.
So is it right or something is wrong in it?
In your Jenkins job you could run Maven with either of the following parameters:
-DskipTests=true to skip test execution entirely
-Dmaven.test.failure.ignore=true to ignore test failures
But if the tests are important (and they certainly should be :) then perhaps ignoring them (or not running them) is the wrong approach, perhaps you should instead fix the failed tests.
In a multi module mvn 3.0 build, I set forkedProcessTimeoutInSeconds to one hour. If tests in module A hang, I hit surefire threshold which will fails the build and skips remaining modules. We use a junit timeouts which should kill hung tests prior to this but some scenarios encounter this problem. I'd like to keep my builds running while I investigate enhancements to our junit runner.
Current Command Line: mvn clean install -Dmaven.test.failure.ignore
How I can change the surefire timeout behavior to move to next module and continue the build?
Does anyone have tips for testing one's junit framework?
I was going to review the surefire project's integration tests for ideas for both items. I plan to have a junitsystest module which activated by a specific profile so I can test various problematic situations outside of my builds.
According to Mastering The Maven Command Line – Managing failures --fail-at-end or --fail-never will provide the desired results. Surefire timeout continues to be reported but doesn't halt the build.
Not recommending in the long run as it lengthens bad build times
mvn clean install -Dmaven.test.failure.ignore --fail-never
I'd like to configure jasmine-maven-plugin to make jenkins unstable if a test fails but the only options appear to be:
set haltOnFailure true and have failures break the build
set haltOnFailure false and have failures reported in the logs but the build succeeds.
Is there a way to check the logs post-test and mark the build unstable?
Sam Hasler's answer only works for freestyle Jenkins jobs. We use Maven jobs and this configuration option of the JUnit Jenkins plugin is unavailable for Maven jobs. So I was looking for a more universal solution.
What we did in order to get it working is to reconfigure the Jasmine Maven plugin so as to
no longer halt the build upon test failures and
write the Jasmine test reports to target/surefire-reports where Jenkins expects to find them. This has the additional advantage that we now also see the failed Jasmine test in the build job alongside our Java tests.
<haltOnFailure>false</haltOnFailure>
<jasmineTargetDir>${project.build.directory}/surefire-reports</jasmineTargetDir>
Now our build jobs are yellow (unstable) as expected, no longer red (failed).
Found the answer myself!
I had to configure jenkins to also look at the jasmine junit report:
under Publish JUnit test result report add **/TEST-jasmine.xml to Test report XMLs, comma separated if there is something there already:
**/TESTS-TestSuites.xml,**/TEST-jasmine.xml