I have a Jenkins job which runs certain SoapUI test cases using the maven plugin.The Jenkins build fails even if one of the test cases fail.I do not want this to be the case and want the job to fail only when there is an issue with the job(like it terminates in between due to some exception) or if the server is down.How can I do that?
Have you tried to skip test cases in your Maven run? Use the code below in maven properties section of maven plugin:
maven.test.failure.ignore=true
Related
Some of test cases are failed so due to that maven build is also failed .Just because of maven failure , Jenkins build is also failed.
So is it right or something is wrong in it?
In your Jenkins job you could run Maven with either of the following parameters:
-DskipTests=true to skip test execution entirely
-Dmaven.test.failure.ignore=true to ignore test failures
But if the tests are important (and they certainly should be :) then perhaps ignoring them (or not running them) is the wrong approach, perhaps you should instead fix the failed tests.
I am using Maven and the failsafe plugin to execute a set of runners that run integration tests.
One of these runners execute tests that involve altering my configuration. For this reason, I would like this runner to be executed only when the other tests have finished executing. Otherwise, there is the possibility of unrepresentative tests.
How can I go about this?
To expand on #khmarbaise comment, in Testng you have annotations that can help you set and reset config files
http://testng.org/doc/documentation-main.html#annotations
At the moment I have a project, lets call it mightymouse. Mightymouse POM is a multi-module project for mightymouse-web and mightmouse-backend.
Both of the sub modules have tests. In my situation, the mightymouse-web project compiles, test, package, install, deploys first. Then then -backend project goes through, only to fail during the integration-test stage. (Yet, it still goes through to the deploy stage).
The Jenkins job is setup to execute mvn clean deploy.
Question: Is there anyway to prevent Maven from going all the way to the deploy step on any module failing it's tests?
By default, a maven multi module build also uses the fail-fast option
--fail-fast - the default behavior - whenever a module build fails, stop the overall build immediately
Moreover, by default the maven-failsafe-plugin already fails as soon as an integration test fails via the testFailureIgnore option
Set this to true to ignore a failure during testing. Its use is NOT RECOMMENDED, but quite convenient on occasion.
User Property: maven.test.failure.ignore
Default: false
Note that the maven-surefire-plugin has the same for unit testings.
Hence, by default a multi-module build should stop as soon as a test fails (unit test or integration test), no further module would be built and not further maven phase should be invoked.
However, a Maven Jenkins job (and not a freestyle Jenkins job invoking Maven) set this option to true by default and hence the build would not fail, it will keep on and deploy, while the job would be set to UNSTABLE and not SUCCESSFUL (but still deploying).
As such, you could change your maven invocation in your Jenkins job to:
mvn clean deploy -Dmaven.test.failure.ignore=false
Overriding thus Jenkins default settings and meeting your requirements.
I have a Continuous integration setup using jenkins. I need to run Junit test cases and selenium test cases and commit the jar files in svn, only if all the junit and selenium scripts are passed.
Junit and selenium test cases are separate maven projects so i have used two invoke top level maven targets (One for selenium and other for Junit). I came across following issues,
Selenium scripts will be executed first and if any of the selenium test cases are failed, the jenkins build is marked as Failed and it will start to execute the post-build actions skipping all the build steps including junit. I want to run the Junit test cases also even if selenium scripts are failed. So i have added -Dmaven.test.failure.ignore=true in maven goals to execute the junit test cases even if selenium scripts are failed. This worked perfectly as i expected and continued to execute junit test caes and the build is marked as UNSTABLE.
Now i got another issue, i have a build step to invoke ant at the end (To commit jars in svn only if selenium and junit are passed). After adding -Dmaven.test.failure.ignore=true, the jars are getting commited to svn even if there is any failures.
Can anyone help me to solve this problem?
The build should run both Selenium and Junit, even if there is any failure in any one of project.
Jars should be committed in svn only if everything is successful (This is done by using ant target, so i have placed invoke ant at the end in Build step.
Is there any other way to accomplish this ? Thanks in advance.
I'd like to configure jasmine-maven-plugin to make jenkins unstable if a test fails but the only options appear to be:
set haltOnFailure true and have failures break the build
set haltOnFailure false and have failures reported in the logs but the build succeeds.
Is there a way to check the logs post-test and mark the build unstable?
Sam Hasler's answer only works for freestyle Jenkins jobs. We use Maven jobs and this configuration option of the JUnit Jenkins plugin is unavailable for Maven jobs. So I was looking for a more universal solution.
What we did in order to get it working is to reconfigure the Jasmine Maven plugin so as to
no longer halt the build upon test failures and
write the Jasmine test reports to target/surefire-reports where Jenkins expects to find them. This has the additional advantage that we now also see the failed Jasmine test in the build job alongside our Java tests.
<haltOnFailure>false</haltOnFailure>
<jasmineTargetDir>${project.build.directory}/surefire-reports</jasmineTargetDir>
Now our build jobs are yellow (unstable) as expected, no longer red (failed).
Found the answer myself!
I had to configure jenkins to also look at the jasmine junit report:
under Publish JUnit test result report add **/TEST-jasmine.xml to Test report XMLs, comma separated if there is something there already:
**/TESTS-TestSuites.xml,**/TEST-jasmine.xml