Why 0% in Jacoco coverage for integration tests - spring-boot

I have setup maven project to get the integration test coverage in my it profile. But the jacoco report shows 0% for all the classes.
My main POM Jacoco integration setup section in the profile and failsafe plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${maven-failsafe-plugin.version}</version>
<executions>
<!--
Invokes both the integration-test and the verify goals of the
Failsafe Maven plugin
-->
<execution>
<id>integration-tests</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<!--
Skips integration tests if the value of skip.it.tests
property is true
-->
<argLine>#{failsafeArgLine}</argLine>
<skipTests>${skip.it.tests}</skipTests>
<testFailureIgnore>true</testFailureIgnore>
<excludes>
<exclude>none</exclude>
</excludes>
<includes>
<!-- Include your Cucumber tests, as an example -->
<include>**/*CucumberTests.java</include>
<include>**/*IT.java</include>
</includes>
<systemPropertyVariables>
<it.server.port>${random.http.port}</it.server.port>
<it.jmx.port>${random.jmx.port}</it.jmx.port>
</systemPropertyVariables>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
.....
<profiles>
<profile>
<id>it</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<packaging.type>jar</packaging.type>
<run.profiles>it</run.profiles>
<skip.it.tests>true</skip.it.tests>
<skipTests>true</skipTests>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<configuration>
<append>true</append>
<!-- Sets the path to the file which contains the execution data. ;-->
<destFile>${user.dir}/target/jacoco-it.exec</destFile>
<!-- Sets the path to the file which contains the execution data.-->
<dataFile>${user.dir}/target/jacoco-it.exec</dataFile>
<!-- Sets the name of the property containing the settings for JaCoCo runtime agent.-->
<propertyName>failsafeArgLine</propertyName>
<excludes>
<exclude>**/*App*</exclude>
<exclude>**/test/**/*</exclude>
<exclude>**/it/**/*</exclude>
<exclude>**/config/*Config*</exclude>
</excludes>
<!-- Sets the output directory for the code coverage report.-->
<outputDirectory>${user.dir}/target/coverage-reports/jacoco-it</outputDirectory>
</configuration>
<executions>
<execution>
<id>pre-integration-test</id>
<phase>pre-integration-test</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<!--
Ensures that the code coverage report for integration tests after
integration tests have been run.
-->
<execution>
<id>post-integration-test</id>
<phase>post-integration-test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
The profile setting of the module POM which has api tests where the command is executed
<profile>
<id>it</id>
<properties>
<skip.it.tests>false</skip.it.tests>
<spring.profile>it</spring.profile>
<skipTests>true</skipTests> -- This is for unit tests
</properties>
</profile>
However I can see jacoco-it.exec file has been generated and the report file as below.
Version:
<jacoco.plugin.version>0.8.3</jacoco.plugin.version>
<maven-failsafe-plugin.version>3.0.0-M5</maven-failsafe-plugin.version>
Command to execute tests: mvn clean verify -P it. This will spin up a server and run all integration tests (api end point tests) there.
Further when setup for unit tests that work as expected with surefire plugin with showing proper coverage in jacoco reports.

Related

Maven install is running failsafe

I am attempting to have my integration tests separated from the usual lifecycle, meaning that I do not want them to be executed during a mvn install. I have the following configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<excludes>
<exclude>**/*IT.java</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
<goals>
<goal>integration-test</goal>
</goals>
</plugin>
And my integration tests are all suffixed with IT. I was able to confirm that surefire does exclude the *IT.java tests, however it seems that failsafe is being triggered during mvn install anyway. What am I missing?
Thank you for your help
A typical configuration to handle the issue with running integration tests only if you like is a profile like the following:
<profiles>
<profile>
<id>run-its</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
By using the above configuration you can run
mvn clean verify
which will execute the unit tests. Using the following you can activate the integration tests:
mvn -Prun-its clean verify
Based on the default naming conventions the integration tests which will be executed by maven-failsafe-plugin should be named like *IT.java where as the unit tests can be name like *Test.java (will be executed via maven-surefire-plugin).

Configuring jacoco for integration and unit test reporting in Sonarqube with Powermock

I am using Sonarqube to keep track of both unit and integration test coverage for a multi-module Maven project.
This was the existing profile in the parent pom.xml that was used to generate the Sonarqube report locally before I made the change:
Profile that generates all unit test coverage locally in Sonarqube
<profiles>
<profile>
<id>coverage</id>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<sonar.jacoco.reportPaths>${project.basedir}/../target/jacoco.exec</sonar.jacoco.reportPaths>
<sonar.projectName>plan-advantage-serverless-${project.artifactId}</sonar.projectName>
<sonar.projectKey>${project.groupId}-MPA-${project.artifactId}</sonar.projectKey>
<sonar.exclusions>file:**/generated-sources/**,**/*Model.java,**/models/**/*</sonar.exclusions>
<sonar.test.exclusions>**/test/*</sonar.test.exclusions>
<sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
<sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
<sonar.language>java</sonar.language>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.2</version>
<configuration>
<append>true</append>
<excludes>
<exclude>**/test/*</exclude>
</excludes>
</configuration>
<executions>
<execution>
<id>default-instrument</id>
<goals>
<goal>instrument</goal>
</goals>
</execution>
<execution>
<id>default-restore-instrumented-classes</id>
<goals>
<goal>restore-instrumented-classes</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<showWarnings>true</showWarnings>
<compilerArgs>
<arg>-Xlint:all</arg>
<arg>-Xlint:-processing</arg>
</compilerArgs>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.0</version>
<configuration>
<argLine>-XX:-UseSplitVerifier</argLine>
<systemPropertyVariables>
<jacoco-agent.destfile>${sonar.jacoco.reportPaths}</jacoco-agent.destfile>
</systemPropertyVariables>
</configuration>
<dependencies>
<!-- needed for powermock to run correctly with surefire-->
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>2.22.0</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.sonarsource.scanner.maven</groupId>
<artifactId>sonar-maven-plugin</artifactId>
<version>3.5.0.1254</version>
</plugin>
</plugins>
</build>
</profile>
This is generating the expected test coverage (sans integration tests) in
Sonarqube locally when I run mvn clean install -P coverage sonar:sonar.
I've so far been able to get integration coverage added as a proof of concept using the following addition
to the parent pom.xml:
pom.xml that includes integration test coverage in Sonarqube but excludes some unit tests
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<jacoco.version>0.7.9</jacoco.version> . <sonar.jacoco.reportPaths>${project.basedir}/../target/jacoco.exec</sonar.jacoco.reportPaths>
<sonar.jacoco.itReportPath>${project.basedir}/../target/jacoco-it.exec</sonar.jacoco.itReportPath>
<sonar.language>java</sonar.language>
<sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
</properties>
...
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>3.0.0-M3</version>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>3.0.0-M3</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<execution>
<id>agent-for-ut</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<append>true</append>
<destFile>${sonar.jacoco.reportPaths}</destFile>
</configuration>
</execution>
<execution>
<id>agent-for-it</id>
<phase>package</phase>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
<configuration>
<append>true</append>
<destFile>${sonar.jacoco.itReportPath}</destFile>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
This was inspired by the example found here.
However, when I run it with the command mvn clean install failsafe:integration-test sonar:sonar, it causes some of the unit tests which were previously being covered to not show up in the Sonarqube output. I believe that the prepare-agent and prepare-agent integrationgoals are using on-the-fly instrumentation. According to JaCoCo's docs, on-the-fly instrumentation is not possible while using PowerMock (which my project is utilizing), so we have to use the offline instrumentation for JaCoCo.
I looked at this example for using offline instrumentation and used the following pom.xml with the command mvn clean install test sonar:sonar:
parent pom.xml that fails to build due to NoClassDefFound errors
<build>
...
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<execution>
<id>default-instrument</id>
<goals>
<goal>instrument</goal>
</goals>
</execution>
<execution>
<id>default-restore-instrumented-classes</id>
<goals>
<goal>restore-instrumented-classes</goal>
</goals>
</execution>
</executions>
</plugin>
</build>
And here's the resulting error:
Any ideas for the proper pom.xml configuration to enable offline instrumentation to get integration and unit test coverage to show up in Sonarqube?
I know this question is 10 months old, but for anyone else coming across this, you need to add this to your dependencies in your pom.xml file.
<dependency>
<groupId>org.jacoco</groupId>
<artifactId>org.jacoco.agent</artifactId>
<classifier>runtime</classifier>
<version>${jacoco.version}</version>
<scope>test</scope>
</dependency>

SonarQube not reading Integration JaCoCo Test coverage in parent pom target directory of a multi-module project

I have followed the instructions for enabling Unit and Integration tests for our Sonar Project here:
http://docs.sonarqube.org/display/PLUG/Code+Coverage+by+Integration+Tests+for+Java+Project
My pom settings are almost identical to the settings given in this example:
https://github.com/SonarSource/sonar-examples/tree/master/projects/languages/java/code-coverage/combined%20ut-it/combined-ut-it-multimodule-maven-jacoco
parent pom settings:
<profile>
<id>code-coverage</id>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<configuration>
<append>true</append>
</configuration>
<executions>
<execution>
<id>default-prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>default-agent-for-it</id>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
</execution>
<execution>
<id>default-report</id>
<phase>verify</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<excludes>
<exclude>static/**/*.jar</exclude>
<exclude>static/**/*.class</exclude>
</excludes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
We have a specific module that will run the integration tests:
<profile>
<id>code-coverage</id>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<configuration>
<!-- The destination file for the code coverage report has to be set to the same value
in the parent pom and in each module pom. Then JaCoCo will add up information in
the same report, so that, it will give the cross-module code coverage. -->
<destFile>${project.basedir}/../target/jacoco-it.exec</destFile>
</configuration>
</plugin>
</plugins>
</build>
</profile>
I have configured Sonar to look for UT and IT:
Unit tests are being reported fine, but not Integration tests. In the Bamboo logs I can see that sonar is looking for integration tests and unit tests in all the submodules, but when it gets to the parent module it never runs the JaCoCoSensor or the JaCoCoItSensor. Both of those Sensors are run for each of the module projects. I also noticed there is a JaCoCoOverallSensor run for each module project but not the parent project.
How do I get the JaCoCoItSensor to run for the parent project?
I added a few configuration properties and it started working.
parent pom.xml added:
<properties>
...
<sonar.jacoco.itReportPath>${project.basedir}/../target/jacoco-it.exec</sonar.jacoco.itReportPath>
<sonar.language>java</sonar.language>
</properties>
IT Tests module I changed:
<destFile>${sonar.jacoco.itReportPath}/../target/jacoco-it.exec</destFile>
To this:
<destFile>${sonar.jacoco.itReportPath}</destFile>

Multiple test configurations for surefire plugin

I have an issue with my pom:
There are two types of tests running within the project:
Main set of tests using TestNG and JUnit code coverage tests. What I would like is to run JUnit tests on test-compile phase and do not run TestNG tests (which are run on test phase) if previous failed.
As of now I have the following which runs only TestNG:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<argLine>
-javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/1.8.4/aspectjweaver-1.8.4.jar
</argLine>
<properties>
<property>
<name>listener</name>
<value>org.uncommons.reportng.HTMLReporter, org.uncommons.reportng.JUnitXMLReporter</value>
</property>
</properties>
<suiteXmlFiles>
<suiteXmlFile>src/test/_all.xml</suiteXmlFile>
</suiteXmlFiles>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
I tried to play with <execution> tag but I can't provide path to TestNG suite then. Any ideas how to combine these tests using surefire plugin and specify different goals?
Solution was to add maven-failsafe-plugin separately from maven-surefire-plugin.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.17</version>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>2.17</version>
</dependency>
</dependencies>
<configuration>
<includes>
<include>junitcodecoverage/**Test.java</include>
</includes>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<phase>test-compile</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>

Sonar: measuring code coverage when integration tests are in a different project

Having following Maven project structure:
-project1 <-- parent pom with two children.
|--module1 <-- web services
\--module1-itest <-- integration tests written with TestNG
What we do today:
Run mvn sonar:sonar in module1, which shows code coverage from
the Unit tests in Sonar's dashboard.
Run mvn jetty:run on module1, and immediately after that run mvn test on module1-itests to test it.
I know this is far from the ideal scenario... it's more like an intermediate step while we try to improve a legacy project with almost no tests at all...
My question is: what would be the best way to get the code coverage done by the execution of the integration tests in Sonar's dashboard of module1?
Initially, I'm inclined to move the code in module1-itest to module1, and run them using the Failsafe plugin and the well-documented integration with JaCoCo. In this way, Module1 will have a mix of JUnit unit tests and TestNG integration tests, each group to be run by Surefire and Failsafe, respectively, starting up a Jetty server in the pre-integration phase.
However, we have reasons to keep both projects separated, so I'm wondering:
Is the approach above a good one?
Is there any other recommended way we can use to keep both projects separated, but including the code coverage done by module1-itest in module1 Sonar's dashboard?
Thanks,
This is how we solved it:
Summary:
Service and Service-itest projects are two independent modules.
Service has unit tests which coverage is reported by Sonar.
Service-itests uses Cargo to load the service WAR and run the integration tests. The code coverage collected in this execution is reported one level above. In this way, the coverage for the ITests is collected and merged cross-module and reported at the parent project pom level. We also like this approach becasue it allows us to have different ITests projects (for example, one with Cargo loading the app and another using JBehave), and they can evolve independently.
In the parent pom:
<modules>
<module>service</module>
<module></module>
<module>service-itest</module>
</modules>
<!-- Sonar -->
<sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
<sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
<!-- The destination file for the code coverage report has to be set to
the same value in the parent pom and in each module pom. Then JaCoCo will
add up information in the same report, so that, it will give the cross-module
code coverage. -->
<sonar.jacoco.itReportPath>${project.basedir}/../target/jacoco-itests.exec</sonar.jacoco.itReportPath>
...
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.6.3.201306030806</version>
</plugin>
</plugins>
</pluginManagement>
...
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<configuration>
<includes>
<include>our.project.packages.*</include>
</includes>
</configuration>
<executions>
<execution>
<id>pre-test</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>post-test</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
And in the pom.xml file of the -itest project:
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<configuration>
<!-- The destination file for the code coverage report has to be set
to the same value in the parent pom and in each module pom. Then JaCoCo will
add up information in the same report, so that, it will give the cross-module
code coverage. -->
<destFile>${project.basedir}/../target/jacoco-itests.exec</destFile>
<includes>
<include>our.packages.*</include>
</includes>
</configuration>
<executions>
<execution>
<id>post-test</id>
<configuration>
<skip>true</skip>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.cargo</groupId>
<artifactId>cargo-maven2-plugin</artifactId>
<version>1.2.2</version>
<configuration>
<skip>${cargo.skip}</skip>
<container>
<containerId>jetty7x</containerId>
<artifactInstaller>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-distribution</artifactId>
<version>7.6.12.v20130726</version>
<type>zip</type>
</artifactInstaller>
</container>
<configuration>
<type>standalone</type>
<properties>
<cargo.servlet.port>${jetty.port}</cargo.servlet.port>
<cargo.jvmargs>${argLine}</cargo.jvmargs>
</properties>
<deployables>
<deployable>
<artifactId>pam_filetask_manager_service</artifactId>
<groupId>${project.groupId}</groupId>
<type>war</type>
<pingURL>http://server:22000/ping</pingURL>
<properties>
</properties>
</deployable>
</deployables>
</configuration>
</configuration>
<executions>
<execution>
<id>start-server</id>
<phase>pre-integration-test</phase>
<goals>
<goal>install</goal>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop-server</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>

Resources