Sonar configuration in multimodule maven project using tycho for unit tests and jacoco for coverage - maven

We're using maven to run a sonar analysis and it works well except for the code coverage results with jacoco. We have an eclipse project that uses tycho-surefire-plugin for testing. I've not overriden the argLines properties so solutions involving that line may not be appropiate.
Facts :
Maven structure structure:
parent
master
module 1
module ...
module n
Testing structure:
client.admin (eclipse-plugin packaging)
client.admin.test.fragment (eclipse-test-plugin packaging)
Properties that are correctly set and identified
sonar.junit.reportsPath
sonar.jacoco.reportPath,
sonar.jacoco.itReportPath
sonar.core.codeCoveragePlugin
sonar.language
The main problem is with the following properties
sonar.test
sonar.sources
sonar.java.binaries
As seen in the Testing structure in the client.admin.test.fragment tests are contained in the /src folder and the sources are located in the project client.admin in the /src folder too.
When we run the analysis we get the following error :
[WARN] Coverage information was not collected. Perhaps you forget to include
debug information into compiled classes?
I believe this has to do with the properties sonar.java.binaries that goes looking for the sources in target/classes of the fragment project (client.admin.project) that are in fact located in the host project (client.admin). In the fragment project we've configured sonar.tests and sonar.sources properties so that they call the /src folder of the corresponding projects.
In the sonar Analysis Parameters page there says that only sonar.sources is a maven valid property, sonar.tests and sonar.java.binaries cannot apparently be configured in maven. How then could I attach the binaries to the project. I've tried copying the folder target/classes from the host project but I got the same message. Is there any workaround in maven ?
Edit 1
There is one jacoco.exec file that is generated for the whole project that can be found at the parent folder. This was done configuring the jacoco.destFile and sonar.jacoco.reportPath properties
Jacoco plugin in main pom :
<!-- Jacoco Plugin -->
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<executions>
<execution>
<id>prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${sonar.jacoco.reportPath}</destFile>
<append>true</append>
</configuration>
</execution>
<execution>
<id>default-report</id>
<goals>
<goal>report</goal>
</goals>
<configuration>
<dataFile>${sonar.jacoco.reportPath}</dataFile>
<outputDirectory>${jacoco.reports.outputDirectory}</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
Maven Plugin Versions:
sonar: 2.4
jacoco: 0.7.1.201405082137
Properties
<sonar.language>java</sonar.language>
<sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
<sonar.junit.reportsPath>${project.build.directory}/surefire-reports/</sonar.junit.reportsPath>
<sonar.jacoco.reportPath>${basedir}/../../../main/**.master/target/jacoco.exec</sonar.jacoco.reportPath>
<jacoco.reports.outputDirectory>${basedir}/../../../main/**.master/target/site/jacoco</jacoco.reports.outputDirectory>
<sonar.sources>src</sonar.sources>
In the test projects (eclipse-test-plugin) we changed added the property sonar.sources to go find the sources from the src folder of the project that we're testing for example in client.admin.test.fragment we go search the src from the client.admin
The following properties were commented in code because they're not supported in maven according to documentation and to the debug output.
<!--<sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>-->
<!--<sonar.tests></sonar.tests>-->
<!--<sonar.java.binaries></sonar.java.binaries>-->

First, you must tell the JaCoCo agent to report all coverage data into one common file. Second, you tell the Sonar JaCoCo plugin to read the coverage data from the aggregated file.
To do so, set the properties "jacoco.destFile" and "sonar.jacoco.reportPath" in your parent pom.xml to the same absolute path, e.g.:
<properties>
<jacoco.destFile>/home/jenkins/jobs/my.project/workspace/parent/target/jacoco.exec</jacoco.destFile>
<sonar.jacoco.reportPath>/home/jenkins/jobs/my.project/workspace/parent/target/jacoco.exec</sonar.jacoco.reportPath>
</properties>
Note that these properties will be inherited to all child poms, so you can't use Maven expressions like ${project.build.directory} because this would evaluate to a different directory for each pom.
You could create a small helper Mojo which automatically resolves an absolute path on the current build machine and then injects the properties into the Maven model.

Related

Differing behavior in maven multi-module projects when run in TeamCity vs locally

I have a maven multi-module project on TeamCity. I'm using TeamCity's built in maven 3.5 tooling.
In one of the child projects, in the section of its pom.xml I set "<target.env>dev</target.env>".
Later in the pom I use the properties-maven-plugin to load a file with the name "${target.env}.env.properties"
Locally if I run "mvn package -Dtarget.env=prod" in the parent project, the child project loads prod.env.properties as expected.
If I configure my teamcity build with param("system.target.env", "prod"), I can see "-Dtarget.env=prod" passed to the maven execution in the build log (where teamcity invokes the plexus-classworlds launcher to do so), the child project loads dev.env.properties, breaking the build.
Here's my questions:
Why does the behavior differ?
How do I reconcile this?
Update including some of the information #khmarbaise asked for:
The properties-maven-plugin is being used to load an environment specific set of properties based on which environment the application will run in.
It is set up to choose which file to load based on a system property, and a default value is set in the properties block to avoid having to constantly
add -Dtarget.env=dev during development. The properties-maven-plugin configuration for the child project is as follows:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>properties-maven-plugin</artifactId>
<executions>
<execution>
<id>load-environment-properties</id>
<phase>validate</phase> <!-- Bound to validate phase to ensure it comes before loading of local.build.properties --> # No local.build.properties on TeamCity, so nothing clobbers this in practice
<goals>
<goal>read-project-properties</goal>
</goals>
<configuration>
<files>
<file>src/main/targetEnvironment/default.env.properties</file>
<file>src/main/targetEnvironment/${target.env}.env.properties</file>
</files>
</configuration>
</execution>
<execution>
<id>write-properties</id>
<phase>generate-resources</phase>
<goals>
<goal>write-project-properties</goal>
</goals>
<configuration>
<outputFile>${project.build.directory}/effective.build.properties</outputFile>
</configuration>
</execution>
</executions>
</plugin>
Version info:
Teamcity version: 2020.2.1 (build 85633)
Maven version: 3.3.9
Java version:
openjdk version "1.8.0_272"
OpenJDK Runtime Environment Corretto-8.272.10.3 (build 1.8.0_272-b10)
OpenJDK 64-Bit Server VM Corretto-8.272.10.3 (build 25.272-b10, mixed mode)
This is the line TeamCity is using to invoke maven, I've only included property definitions that seemed relevant since there were so many that are definitely irrelevant (build numbers, names, timestamps, and other TC specifc, maven agnostic, config)
/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64/bin/java
-Dclassworlds.conf=/home/ec2-user/BuildAgent/temp/buildTmp/teamcity.m2.conf
-Dmaven.home=/home/ec2-user/BuildAgent/tools/maven3_3
-DskipTests=true
-Dteamcity.build.properties.file=/home/ec2-user/BuildAgent/temp/buildTmp/teamcity.build4380238471360533686.properties
-Dtarget.env=prod
-classpath /home/ec2-user/BuildAgent/tools/maven3_3/boot/plexus-classworlds-2.5.2.jar: org.codehaus.plexus.classworlds.launcher.Launcher -f /home/ec2-user/BuildAgent/work/4508a7116faa21f3/pom.xml -B clean package
The contents of teamcity.m2.conf is as follows:
main is org.apache.maven.cli.MavenCli from plexus.core
set maven.home default ${user.home}/m2
[plexus.core]
load ${teamcity.maven.watcher.home}/*.jar
optionally ${maven.home}/lib/ext/*.jar
load ${maven.home}/lib/*.jar
load ${maven.home}/conf/logging
teamcity.build4380238471360533686.properties contains many properties, the value of target.env within that file is 'prod' as expected
This ended up being a known bug in TeamCity
The underlying issue seems to be that TeamCity uses the MAVEN_OPTS environment variable to pass system properties into maven by default, but properties in MAVEN_OPTS are treated differently from properties passed as arguments to the maven command itself.
The workaround is that for any property "foo" that gets set in a POM section, that you want to override in a TeamCity build, you have to specify it in the "Additional Maven command line parameters" with -Dfoo=value, or, if you're setting the value in a system or build property within the TeamCity build -Dfoo=%system.foo%.

how to create maven mojo plugin to overwrite files directories in target project

I am writing a maven plugin which generates java source code based on input text file and some additional configuration.
e.g. User creates maven project and adds my plugin in his project's pom.xml as below -
<plugin>
<groupId>abc.plugin</groupId>
<artifactId>abc-maven-plugin</artifactId>
<version>1.0.0</version>
<executions>
<execution>
<goals>
<goal>abc</goal>
</goals>
</execution>
</executions>
<configuration>
<schemaPath>${basedir}\input</schemaPath>
<package>com.svc.xyz</package>
<fileNametxt>${basedir}\input.txt</fileNametxt>
</configuration>
</plugin>
As part of executing the goal of my plugin, i would like the plugin delete all the pre existing java source files/directories in user's project as well as rewrite pom.xml of user project.Plugin should regenerate user project's pom.xml taking existing contents and also keep some other existing user project resources like schema Directory.
Also,I don't want user of plugin to do any editing after generation of source code and would prefer to directly 'compile package' the generated source code in user project by the plugin.
Since plugin will regenerate users' pom.xml itself, may be need to create a new maven project altogether with generated source code, compile, package and have jar created?

maven surefire reports in default lifecyle

How can i generate surefire reports in the maven's default lifecycle. We have jobs setup in teamcity with goals as
mvn clean install -Pprod
This job runs all the junits, i want to create HTML report of all the tests running in the project. I came across the sure-fire-reports plugin but it generates the report only in site phase it does not generates the report during the default clean install phase.
Can one please help how can i generate report default lifecycle
I tried including the surefire-reports plugin, in test phase as below but doesnot not works
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-report-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<phase>test</phase>
<goals>
<goal>report-only</goal>
</goals>
</execution>
</executions>
</plugin>
If you need the very report generated by maven-surefire-report-plugin, I see no other way than execute mvn site, because a report is executed only within the site phase of the build lifecycle.
Instead, if you just need an HTML-readable report, I'd suggest you this walk-around:
Take advantage of the XML files generated by the maven-surefire-plugin in the target/surefire-reports directory.
Code your own transformation sheet (XSL) to transform them to the desired HTML format.
In the pom, set a transformation in the next phase (for example, prepare-package) through the xml-maven-plugin.
If you put the XSL in the parent project and set this transformation in the parent pom, all the submodule projects should inherit it and produce the HTML reports during the corresponding build.
And last: How to browse the child HTML reports from the parent project? Hum... I'd say to code an Ant script to browse all the submodules and list the HTML files and produce an HTML index with them. This script should be executed only from the parent project.

IT Code Coverage for multi-module Maven projects that have differing parents

I've spent the day yesterday up and down StackOverflow and Google and have come close to some solutions, but nothing is working. I hope someone could just tell me whether this is even possible.
We have a multi-module Maven project. I just learned that the structure is like this:
Web Service XYX
+- pom.xml
+- Web Module
+- pom.xml (parent pom is WebService 123)
+- API Module
+- pom.xml (parent pom is API ABC)
+- API Implementation
+- pom.xml (parent pom is API ABC)
What I have done is setup UT and IT Code Coverage from Jacoco to Sonar via Jenkins. The UT Coverage gets sent to Sonar correctly for the UT combined.
The IT Coverage is always 0%. My IT is run against the Web Module after starting the WAR up using the Maven Tomcat plugin. The code that it should be exercising lives inside the module API Implementation. Even though a jacoco-it.exec file was generated and analyzed by Sonar, it always shows me that no lines of code were hit. Is there a way to actually do this?
BTW - I know it can be done because this article indicates exactly what I need:
http://www.sonarqube.org/measure-code-coverage-by-integration-tests-with-sonar/
However, the article fails to mention if the modules are referred to the same parent in the project. I'm making an assumption that it does.
After searching up and down the interwebs, the best solution seem to have been add another module for ITs that actually referred to the main pom as the parent. Used Cargo to grab and run the WAR from the IT module using the Tomcat Maven Module. The Code Coverage now has all the classes, not just the ones from the Web Module.
I doubt it's due to using different parent poms.
When analyzing the coverage sonar checks the code of each module against the coverage file that is specified in the sonar.jacoco.itReportPath property. The default is target/jacoco-it.exec. So when analyzing WebModule it checks for coverage info in WebModule/target/jacoco-it.exec.
So what I do for my projects, is to use a central file in the root module for the the IT coverage data instead.
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.7.4.201502262128</version>
<executions>
<execution>
<id>prepare-it-agent</id>
<phase>pre-integration-test</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${session.executionRootDirectory}/target/jacoco-it.exec</destFile>
<append>true</append>
<propertyName>failsafeArgLine</propertyName>
</configuration>
</execution>
</executions>
</plugin>
The ${session.executionRootDirectory} property is the root of execution, in your case the WebServiceXYZ if you run the build of that. This also works if you have multi-module with more than one level of nesting.
Now you need to point sonar to use that file when analyzing IT coverage. So you have to set the sonar.jacoco.itReportPath to that file. Unfortunately, this does not work with the session.executionRootDirectory property and you have to set the absolute path to the file manually. I do not recommend to specify the absolute path in the pom.xml as this path is specific to your build environment. So either set the path in Sonar or as System property of your build environment. I set it directly in the Sonar Project Settings (Java > Jacoco), for example /opt/buildroot/myProject/target/jacoco-it.exec.
Now sonar will check that file for the IT coverage analysis of each module.

maven can't add files in generated-sources for compilation phase

I use Apache Thrift to generate code in target/generated-sources.
The Thrift compiler produces a directory named gen-java which contains all the Java code. When I execute mvn compile, the code is generated correctly in target/generated-source/gen-java, but in compilation phase, it complains can't find the classes which defined in gen-java.
In my understanding, Maven 2 automatically adds generated sources, is that right?
And what if my testing code also depends on the generated-sources, do I have to manually specified the compiler includes?
In my understanding, maven 2 automatically add generated sources, is that right?
Nothing automatic, plugins generating source code typically handle that by adding their output directory (something like target/generated-sources/<tool> by convention) as source directory to the POM so that it will be included later during the compile phase.
Some less well implemented plugins don't do that for you and you have to add the directory yourself, for example using the Build Helper Maven Plugin.
And since you didn't provide any POM snippet, any link, I can't say anything more.
And what if my testing code also depends on the generated-sources, do I have to manually specified the compiler includes?
As I said, generated sources are usually added as source directory and compiled and are thus available on the test classpath without you having to do anything.
Generated sources are not compiled or packaged automatically. Some IDEs (i.e. IntelliJ) will however show them as source folders.
To make generated sources visible to maven add a add-source-step to the build/plugins node of your pom.xml:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>${project.build.directory}/generated-sources/gen-java</source><!-- adjust folder name to your needs -->
</sources>
</configuration>
</execution>
</executions>
</plugin>

Resources