Ensure Maven build fails when tests fail to run - maven

Question

I've noticed that sometimes when running maven builds on Jenkins the number of Jbehave tests that are run vary from one run to another. When analyzing the logs I see the following snippet:
Failed to run story stories/cancel.story
java.lang.InterruptedException: stories/cancel.story
at org.jbehave.core.embedder.StoryRunner$RunContext.interruptIfCancelled(StoryRunner.java:616)
at org.jbehave.core.embedder.StoryRunner.runStepsWhileKeepingState(StoryRunner.java:514)
at org.jbehave.core.embedder.StoryRunner.runScenarioSteps(StoryRunner.java:479)
at org.jbehave.core.embedder.StoryRunner.runStepsWithLifecycle(StoryRunner.java:445)
at org.jbehave.core.embedder.StoryRunner.runCancellable(StoryRunner.java:305)
at org.jbehave.core.embedder.StoryRunner.run(StoryRunner.java:220)
at org.jbehave.core.embedder.StoryRunner.run(StoryRunner.java:181)
at org.jbehave.core.embedder.StoryManager$EnqueuedStory.call(StoryManager.java:235)
at org.jbehave.core.embedder.StoryManager$EnqueuedStory.call(StoryManager.java:207)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
The problem is that when tests are skipped or fail to run in this way the build is still considered a success.
Is there a maven surefire plugin configuration that will ensure that whenever tests fail to run the build results in a failure? Here are the maven surefire build configurations
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.11</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.11</version>
<configuration>
<includes>
<include>**/*TestSuite.java</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9</version>
</plugin>
<plugin>
<groupId>net.thucydides.maven.plugins</groupId>
<artifactId>maven-thucydides-plugin</artifactId>
<version>${thucydides.version}</version>
<executions>
<execution>
<id>thucydides-reports</id>
<phase>post-integration-test</phase>
<goals>
<goal>aggregate</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<version>3.2</version>
<configuration>
<reportPlugins>
<plugin>
<groupId>net.thucydides.maven.plugins</groupId>
<artifactId>maven-thucydides-plugin</artifactId>
<version>${thucydides.version}</version>
</plugin>
</reportPlugins>
</configuration>
</plugin>
</plugins>
</build>

Answer 1

Your maven-surefire-plugin is set to skip tests completely (with <skip>true</skip>), so tests are running with maven-failsafe-plugin. That plug-in is supposed to not stop on failure during integration-test, and then only fail on verify phase.
So if you really want this question answered:
Is there a maven surefire plugin configuration that will ensure that whenever tests fail to run the build results in a failure?
That is: you want maven-surefire-plugin to run the tests, and not the maven-failsafe-plugin, then the answer is: remove
<configuration>
<skip>true</skip>
</configuration>
from your POM. In this case you also don't need maven-failsafe-plugin configuration, because it would just make your tests run twice.
But if your goal is to get maven-failsafe-plugin to work, then I think you may have one of the following issues:
Not running the right goal. As plug-in help states, you should invoke it as
mvn verify
An old plug-in, which is not compatible with test framework you are using (current version is 2.19.1)
Or this help recommendation:
For very complex builds, it may be better to separate the executions for the integration-test and verify goals:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.19.1</version>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
<execution>
<id>verify</id>
<goals>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>