Multiple test configurations for surefire plugin - maven

I have an issue with my pom:
There are two types of tests running within the project:
Main set of tests using TestNG and JUnit code coverage tests. What I would like is to run JUnit tests on test-compile phase and do not run TestNG tests (which are run on test phase) if previous failed.
As of now I have the following which runs only TestNG:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<argLine>
-javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/1.8.4/aspectjweaver-1.8.4.jar
</argLine>
<properties>
<property>
<name>listener</name>
<value>org.uncommons.reportng.HTMLReporter, org.uncommons.reportng.JUnitXMLReporter</value>
</property>
</properties>
<suiteXmlFiles>
<suiteXmlFile>src/test/_all.xml</suiteXmlFile>
</suiteXmlFiles>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
I tried to play with <execution> tag but I can't provide path to TestNG suite then. Any ideas how to combine these tests using surefire plugin and specify different goals?

Solution was to add maven-failsafe-plugin separately from maven-surefire-plugin.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.17</version>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>2.17</version>
</dependency>
</dependencies>
<configuration>
<includes>
<include>junitcodecoverage/**Test.java</include>
</includes>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<phase>test-compile</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>

Related

Configuring jacoco for integration and unit test reporting in Sonarqube with Powermock

I am using Sonarqube to keep track of both unit and integration test coverage for a multi-module Maven project.
This was the existing profile in the parent pom.xml that was used to generate the Sonarqube report locally before I made the change:
Profile that generates all unit test coverage locally in Sonarqube
<profiles>
<profile>
<id>coverage</id>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<sonar.jacoco.reportPaths>${project.basedir}/../target/jacoco.exec</sonar.jacoco.reportPaths>
<sonar.projectName>plan-advantage-serverless-${project.artifactId}</sonar.projectName>
<sonar.projectKey>${project.groupId}-MPA-${project.artifactId}</sonar.projectKey>
<sonar.exclusions>file:**/generated-sources/**,**/*Model.java,**/models/**/*</sonar.exclusions>
<sonar.test.exclusions>**/test/*</sonar.test.exclusions>
<sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
<sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
<sonar.language>java</sonar.language>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.2</version>
<configuration>
<append>true</append>
<excludes>
<exclude>**/test/*</exclude>
</excludes>
</configuration>
<executions>
<execution>
<id>default-instrument</id>
<goals>
<goal>instrument</goal>
</goals>
</execution>
<execution>
<id>default-restore-instrumented-classes</id>
<goals>
<goal>restore-instrumented-classes</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<showWarnings>true</showWarnings>
<compilerArgs>
<arg>-Xlint:all</arg>
<arg>-Xlint:-processing</arg>
</compilerArgs>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.0</version>
<configuration>
<argLine>-XX:-UseSplitVerifier</argLine>
<systemPropertyVariables>
<jacoco-agent.destfile>${sonar.jacoco.reportPaths}</jacoco-agent.destfile>
</systemPropertyVariables>
</configuration>
<dependencies>
<!-- needed for powermock to run correctly with surefire-->
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>2.22.0</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.sonarsource.scanner.maven</groupId>
<artifactId>sonar-maven-plugin</artifactId>
<version>3.5.0.1254</version>
</plugin>
</plugins>
</build>
</profile>
This is generating the expected test coverage (sans integration tests) in
Sonarqube locally when I run mvn clean install -P coverage sonar:sonar.
I've so far been able to get integration coverage added as a proof of concept using the following addition
to the parent pom.xml:
pom.xml that includes integration test coverage in Sonarqube but excludes some unit tests
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<jacoco.version>0.7.9</jacoco.version> . <sonar.jacoco.reportPaths>${project.basedir}/../target/jacoco.exec</sonar.jacoco.reportPaths>
<sonar.jacoco.itReportPath>${project.basedir}/../target/jacoco-it.exec</sonar.jacoco.itReportPath>
<sonar.language>java</sonar.language>
<sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
</properties>
...
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>3.0.0-M3</version>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>3.0.0-M3</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<execution>
<id>agent-for-ut</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<append>true</append>
<destFile>${sonar.jacoco.reportPaths}</destFile>
</configuration>
</execution>
<execution>
<id>agent-for-it</id>
<phase>package</phase>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
<configuration>
<append>true</append>
<destFile>${sonar.jacoco.itReportPath}</destFile>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
This was inspired by the example found here.
However, when I run it with the command mvn clean install failsafe:integration-test sonar:sonar, it causes some of the unit tests which were previously being covered to not show up in the Sonarqube output. I believe that the prepare-agent and prepare-agent integrationgoals are using on-the-fly instrumentation. According to JaCoCo's docs, on-the-fly instrumentation is not possible while using PowerMock (which my project is utilizing), so we have to use the offline instrumentation for JaCoCo.
I looked at this example for using offline instrumentation and used the following pom.xml with the command mvn clean install test sonar:sonar:
parent pom.xml that fails to build due to NoClassDefFound errors
<build>
...
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<execution>
<id>default-instrument</id>
<goals>
<goal>instrument</goal>
</goals>
</execution>
<execution>
<id>default-restore-instrumented-classes</id>
<goals>
<goal>restore-instrumented-classes</goal>
</goals>
</execution>
</executions>
</plugin>
</build>
And here's the resulting error:
Any ideas for the proper pom.xml configuration to enable offline instrumentation to get integration and unit test coverage to show up in Sonarqube?
I know this question is 10 months old, but for anyone else coming across this, you need to add this to your dependencies in your pom.xml file.
<dependency>
<groupId>org.jacoco</groupId>
<artifactId>org.jacoco.agent</artifactId>
<classifier>runtime</classifier>
<version>${jacoco.version}</version>
<scope>test</scope>
</dependency>

Get Arquillians Code Coverage Using JaCoCo

I have a Maven build that runs Arquillian tests on a Wildfly. What I want to do is run JaCoCo as well so that I get a test coverage.
What I did to my working Arquillian setup: I changed the parent's pom.xml the following way:
<properties>
<sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
<sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
<jacoco.out.ut.file>../target/jacoco/jacoco-ut.exec</jacoco.out.ut.file>
<jacoco.out.it.file>../target/jacoco/jacoco-it.exec</jacoco.out.it.file>
<sonar.jacoco.reportPaths>${jacoco.out.ut.file},${jacoco.out.it.file}</sonar.jacoco.reportPaths>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<execution>
<id>prepare-ut-agent</id>
<phase>process-test-classes</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${jacoco.out.ut.file}</destFile>
<propertyName>jacoco.agent.ut.arg</propertyName>
<append>true</append>
</configuration>
</execution>
<execution>
<id>prepare-it-agent</id>
<phase>package</phase>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
<configuration>
<destFile>${jacoco.out.it.file}</destFile>
<propertyName>jacoco.agent.it.arg</propertyName>
<append>true</append>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<argLine>${test.argLine} ${jacoco.agent.it.arg}</argLine>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>${test.argLine} ${jacoco.agent.ut.arg}</argLine>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
That's a setup that works in multiple other projects (without Arquillian), so I assume it must work now, too.
For the module hosting the integration project I added the following:
<dependencies>
<dependency>
<groupId>org.jboss.arquillian.extension</groupId>
<artifactId>arquillian-jacoco</artifactId>
<version>1.0.0.Alpha9</version>
<scope>test</scope>
</dependency>
</dependencies>
The maven-surefire-plugin is called there (which is weird to me, but I tried maven-failsafe-plugin and it didn't work either).
When executed the module produces a jacoco.exec (which is 70kb for only the one module, so at least not empty). Sonar uses these files in the report generation. Still the code coverage gets displayed as 0%.
I found a couple tutorials on this topic, but they all seem to be missing a step (at least the setup is always identical to mine).
How do I get Arquillian to use JaCoCo to report the code coverage?

Ensure Maven build fails when tests fail to run

I've noticed that sometimes when running maven builds on Jenkins the number of Jbehave tests that are run vary from one run to another. When analyzing the logs I see the following snippet:
Failed to run story stories/cancel.story
java.lang.InterruptedException: stories/cancel.story
at org.jbehave.core.embedder.StoryRunner$RunContext.interruptIfCancelled(StoryRunner.java:616)
at org.jbehave.core.embedder.StoryRunner.runStepsWhileKeepingState(StoryRunner.java:514)
at org.jbehave.core.embedder.StoryRunner.runScenarioSteps(StoryRunner.java:479)
at org.jbehave.core.embedder.StoryRunner.runStepsWithLifecycle(StoryRunner.java:445)
at org.jbehave.core.embedder.StoryRunner.runCancellable(StoryRunner.java:305)
at org.jbehave.core.embedder.StoryRunner.run(StoryRunner.java:220)
at org.jbehave.core.embedder.StoryRunner.run(StoryRunner.java:181)
at org.jbehave.core.embedder.StoryManager$EnqueuedStory.call(StoryManager.java:235)
at org.jbehave.core.embedder.StoryManager$EnqueuedStory.call(StoryManager.java:207)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
The problem is that when tests are skipped or fail to run in this way the build is still considered a success.
Is there a maven surefire plugin configuration that will ensure that whenever tests fail to run the build results in a failure? Here are the maven surefire build configurations
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.11</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.11</version>
<configuration>
<includes>
<include>**/*TestSuite.java</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9</version>
</plugin>
<plugin>
<groupId>net.thucydides.maven.plugins</groupId>
<artifactId>maven-thucydides-plugin</artifactId>
<version>${thucydides.version}</version>
<executions>
<execution>
<id>thucydides-reports</id>
<phase>post-integration-test</phase>
<goals>
<goal>aggregate</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<version>3.2</version>
<configuration>
<reportPlugins>
<plugin>
<groupId>net.thucydides.maven.plugins</groupId>
<artifactId>maven-thucydides-plugin</artifactId>
<version>${thucydides.version}</version>
</plugin>
</reportPlugins>
</configuration>
</plugin>
</plugins>
</build>
Your maven-surefire-plugin is set to skip tests completely (with <skip>true</skip>), so tests are running with maven-failsafe-plugin. That plug-in is supposed to not stop on failure during integration-test, and then only fail on verify phase.
So if you really want this question answered:
Is there a maven surefire plugin configuration that will ensure that whenever tests fail to run the build results in a failure?
That is: you want maven-surefire-plugin to run the tests, and not the maven-failsafe-plugin, then the answer is: remove
<configuration>
<skip>true</skip>
</configuration>
from your POM. In this case you also don't need maven-failsafe-plugin configuration, because it would just make your tests run twice.
But if your goal is to get maven-failsafe-plugin to work, then I think you may have one of the following issues:
Not running the right goal. As plug-in help states, you should invoke it as
mvn verify
An old plug-in, which is not compatible with test framework you are using (current version is 2.19.1)
Or this help recommendation:
For very complex builds, it may be better to separate the executions for the integration-test and verify goals:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.19.1</version>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
<execution>
<id>verify</id>
<goals>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>

maven-failsafe-plugin not seeing my tests (seleniumHQ)

i'm writing tests via selenium web driver here's my code :
Pom.xml
<project>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>2.24.1</version>
</dependency>
</dependencies>
<build>
<finalName>SeleniumebDriverProject</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.11</version>
<configuration>
<!-- Skip the normal tests, we'll run them in the integration-test phase-->
<skip>false</skip>
</configuration>
<executions>
<execution>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>false</skip>
<includes>
<include>**/*Test.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
My test class named GoogleTest.java.
after reading this post : failsafe plugin won't run on one project but will run on another -- why?
I changed the name of my class so it's:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.11</version>
<configuration>
<!-- Skip the normal tests, we'll run them in the integration-test phase-->
<skip>false</skip>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
<executions>
</plugin>
But the problem persist.
The goal of the maven-failsafe-plugin is named integration-test instead of test. Furthermore if you changed your naming convention to the convention of maven-failsafe-plugin than you don't need any configuration which includes etc. files. Just use the defaults.
Furthermore i assume you have started running the integration tests via:
mvn verify

How to run concordion test with maven?

I am wondering how to set up maven to run concordion tests having a FooFixture.java naming convention. The tests are on the classpath in src/test/specs. I would like to do it in a separate profile.
Thanks for the help.
I finally set the tests up is by creating a different profile to run the acceptance tests.
An example given here:
<profile>
<id>acceptance-test</id>
<activation>
<property>
<name>type</name>
<value>acceptance</value>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.7</version>
<executions>
<execution>
<id>add-specs-source</id>
<phase>compile</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>${basedir}/src/test/specs</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.1</version>
<configuration>
<encoding>UTF-8</encoding>
<source>1.5</source>
<target>1.5</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.11</version>
<configuration>
<forkMode>pertest</forkMode>
<argLine>-Xms1024m -Xmx1024m</argLine>
<testFailureIgnore>false</testFailureIgnore>
<skip>false</skip>
<testSourceDirectory>src/test/specs</testSourceDirectory>
<includes>
<include>**/*Fixture.java</include>
</includes>
<systemPropertyVariables>
<propertyName>concordion.output.dir</propertyName>
<buildDirectory>target/concordion</buildDirectory>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
</build>
</profile>
You can use the maven failsafe plugin to run unit Concordion tests in the integration-test phase. Its similar to the surefire plugin.

Resources