Jenkins + selenium tests with failsafe plugin - maven

Question

I have a Jenkins platform which calls maven to make unit tests (with surefire plugin) and integration tests (with failsafe plugin). When there is an error in the integration tests, Jenkins considers the build as successfull. Is this behavior normal? I'd prefer it considers the build as unstable. More generally, do you know how Jenkins read and interprets the result of the build to consider a build as successfull or unstable? I read somewhere on the net that the failsafe reports must be redirected to the surefire report path. I did id but the problem is still here.
pom.xml :
[...]
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.10</version>
<configuration>
<disableXmlReport>false</disableXmlReport>
</configuration>
<executions>
<execution>
<id>default-test</id>
<phase>test</phase>
<configuration>
<includes>
<include>**/tests/**</include>
</includes>
<excludes>
<exclude>**/testsIntegration/**</exclude>
</excludes>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.7.2</version>
<configuration>
<disableXmlReport>false</disableXmlReport>
<reportsDirectory>${basedir}/target/surefire-reports</reportsDirectory>
<includes>
<include>com/acelys/conventionsJuridiques/*.java</include>
<!-- ... inclure les tests Selenium -->
</includes>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
</goals>
<configuration>
<includes>
<include>**/testsIntegration/**</include>
</includes>
<excludes>
<exclude>**/tests/**</exclude>
</excludes>
</configuration>
</execution>
</executions>
</plugin>
[...]
output of jenkins :
[...]
mojoStarted org.apache.maven.plugins:maven-failsafe-plugin:2.7.2(integration-test)
[INFO]
[INFO] --- maven-failsafe-plugin:2.7.2:integration-test (integration-test) # BaseContrats ---
[INFO] Failsafe report directory: C:\jenkins_home\workspace\Base Contrats EXT JS MAVEN\target\surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running com.acelys.conventionsJuridiques.testsIntegration.connexion.TestConnexion
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 23.971 sec <<< FAILURE!
Results :
Failed tests:
testHomePage(com.acelys.conventionsJuridiques.testsIntegration.connexion.TestConnexion)
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is platform dependent!
mojoSucceeded org.apache.maven.plugins:maven-failsafe-plugin:2.7.2(integration-test)
mojoStarted org.apache.tomcat.maven:tomcat6-maven-plugin:2.1-SNAPSHOT(tomcat-shutdown)
[INFO]
[INFO] --- tomcat6-maven-plugin:2.1-SNAPSHOT:shutdown (tomcat-shutdown) # BaseContrats ---
25 févr. 2013 09:32:08 org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080
25 févr. 2013 09:32:08 org.apache.catalina.core.ApplicationContext log
INFO: Closing Spring root WebApplicationContext
25 févr. 2013 09:32:08 org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
mojoSucceeded org.apache.tomcat.maven:tomcat6-maven-plugin:2.1-SNAPSHOT(tomcat-shutdown)
projectSucceeded BaseContrats:BaseContrats:1.0-SNAPSHOT
sessionEnded
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:07.408s
[INFO] Finished at: Mon Feb 25 09:32:08 CET 2013
[INFO] Final Memory: 13M/51M
[INFO] ------------------------------------------------------------------------
Projects to build: [MavenProject: BaseContrats:BaseContrats:1.0-SNAPSHOT # C:\jenkins_home\workspace\Base Contrats EXT JS MAVEN\pom.xml]
[JENKINS] Archiving C:\jenkins_home\workspace\Base Contrats EXT JS MAVEN\pom.xml to C:\jenkins_home\jobs\Base Contrats EXT JS MAVEN\modules\BaseContrats$BaseContrats\builds\2013-02-25_09-29-58\archive\BaseContrats\BaseContrats\1.0-SNAPSHOT\BaseContrats-1.0-SNAPSHOT.pom
[JENKINS] Archiving C:\jenkins_home\workspace\Base Contrats EXT JS MAVEN\target\ConventionsJuridiques.war to C:\jenkins_home\jobs\Base Contrats EXT JS MAVEN\modules\BaseContrats$BaseContrats\builds\2013-02-25_09-29-58\archive\BaseContrats\BaseContrats\1.0-SNAPSHOT\BaseContrats-1.0-SNAPSHOT.war
channel stopped
Finished: SUCCESS

Answer 1

According to the documentation:
failsafe:verify verifies that the integration tests of an application passed.
You need to add the goal to the maven-failsafe-plugin execution:
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
This will allow Jenkins to interpret the test results and mark the build as unstable.

Answer 2

You can see that maven finished with SUCCESS. This will have Jenkins also finish with success.
To mark build as unstable, you need a post build action that will analyse your test results and mark build as failed or unstable. I suggest you search for more details on Jenkins post build actions for processing the test results and marking build as needed.
Hope this helps.

Answer 3

Under the post build actions the Publish JUnit test result report changes the report XML's to search for.
The assumption is the standard surefire and failsafe plugin output directories are used.
changes to

Answer 4

You should add a step in your build : "publish Junit report"
You will have graph showing all your test result, but it will also set the correct build result if tests are failing.

Answer 5

As said by https://stackoverflow.com/users/709863/stephane-piette:
Add a post build action 'Publish Performance test result report'. This action is provided by the plugin named 'Performance plugin'. You probably do not have this plugin installed, which is why it does not exist in the list.

Answer 6

The goal "integration-test" of the failsafe plugin won't mark the build as error when tests fail.
You have to add the goal "verify" to run after "integration-test" in the plugin execution. This will look for errors or failures and mark the build as "error". Then Jenkins will see that Maven did not succeed.

Answer 7

You have to set the plugin's configuration so that the build fails upon failed test(s):
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
....
<configuration>
<testFailureIgnore>false</testFailureIgnore>
</configuration>
<executions>
....
</executions>
</plugin>