Multi module project analysis with SonarQube - sonarqube

SonarQube Server 5.1.2, Sonar-Runner 2.4
As provide in Multi-moduleProject i have created a project structure as
Accounts
|
->invoice
|
->src
->receipt
|
->src
->sonar.properties
File:sonar.properties
sonar.projectKey=org.mycompany.acc
sonar.projectName=Account
sonar.projectVersion=1.0
sonar.sources=src
sonar.modules=invoice,receipt
invoice.sonar.projectName=Invoice
receipt.sonar.projectName=Receipt
When execute with above configuration in sonar-runner i encountered with error "src" folder is missing in "Account" directory, hope this configuration is same as the conf available in that link. As per the understanding if the configuration is fine then the Invoice and Receipt will be listed as sub project under Account Project, so what are the changes are required in above configuration to achieve multi module / project under one project.
ERROR
ERROR: Error during Sonar runner execution
ERROR: Unable to execute Sonar
ERROR: Caused by: The folder 'src' does not exist for 'org.mycompany.acc' (base
directory = C:\Users\xyz\Accounts\.)
ERROR:
ERROR: To see the full stack trace of the errors, re-run SonarQube Runner with t
he -e switch.
ERROR: Re-run SonarQube Runner using the -X switch to enable full debug logging.

try this:
sonar.projectKey=org.mycompany.acc
sonar.projectName=Account
sonar.projectVersion=1.0
sonar.sources=src # try to remove this by the way if you don't have suchdirectory under root folder of project
sonar.modules=invoice,receipt
invoice.sonar.projectName=Invoice
invoice.sonar.sources=invoice/src
receipt.sonar.projectName=Receipt
receipt.sonar.sources=receipt/src

You can try this option, If you want scan multiple projects at same time OR in same build then try this option.
sonar.sources=invoice,receipt

sonar.modules property was deprecated long time ago. Use sonar.sources parameter to perform a scan.
In short, you have to specify in sonar:sonar goal parameters where the sources and reportPaths are located for the required modules and declare jacoco configuration for each module respectively. As a result, it will look something like this:
Jacoco plugin declaration in each module:
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.7</version>
<executions>
<execution>
<id>default-prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
Sonar goal execution:
mvn clean verify sonar:sonar /
-Dsonar.projectKey=... /
-Dsonar.host.url=... /
-Dsonar.login=... /
-Dsonar.branch.name=... /
-Dsonar.sources=../module1/src/main,../module2/src/main /
-Dsonar.jacoco.reportPath=module1/target/jacoco.exec,module2/target/jacoco.exec
Important! It's required to specify .. before module name in -Dsonar.sources parameter because jacoco plugin in each module can only read content by relative path from it's location.

Related

Differing behavior in maven multi-module projects when run in TeamCity vs locally

I have a maven multi-module project on TeamCity. I'm using TeamCity's built in maven 3.5 tooling.
In one of the child projects, in the section of its pom.xml I set "<target.env>dev</target.env>".
Later in the pom I use the properties-maven-plugin to load a file with the name "${target.env}.env.properties"
Locally if I run "mvn package -Dtarget.env=prod" in the parent project, the child project loads prod.env.properties as expected.
If I configure my teamcity build with param("system.target.env", "prod"), I can see "-Dtarget.env=prod" passed to the maven execution in the build log (where teamcity invokes the plexus-classworlds launcher to do so), the child project loads dev.env.properties, breaking the build.
Here's my questions:
Why does the behavior differ?
How do I reconcile this?
Update including some of the information #khmarbaise asked for:
The properties-maven-plugin is being used to load an environment specific set of properties based on which environment the application will run in.
It is set up to choose which file to load based on a system property, and a default value is set in the properties block to avoid having to constantly
add -Dtarget.env=dev during development. The properties-maven-plugin configuration for the child project is as follows:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>properties-maven-plugin</artifactId>
<executions>
<execution>
<id>load-environment-properties</id>
<phase>validate</phase> <!-- Bound to validate phase to ensure it comes before loading of local.build.properties --> # No local.build.properties on TeamCity, so nothing clobbers this in practice
<goals>
<goal>read-project-properties</goal>
</goals>
<configuration>
<files>
<file>src/main/targetEnvironment/default.env.properties</file>
<file>src/main/targetEnvironment/${target.env}.env.properties</file>
</files>
</configuration>
</execution>
<execution>
<id>write-properties</id>
<phase>generate-resources</phase>
<goals>
<goal>write-project-properties</goal>
</goals>
<configuration>
<outputFile>${project.build.directory}/effective.build.properties</outputFile>
</configuration>
</execution>
</executions>
</plugin>
Version info:
Teamcity version: 2020.2.1 (build 85633)
Maven version: 3.3.9
Java version:
openjdk version "1.8.0_272"
OpenJDK Runtime Environment Corretto-8.272.10.3 (build 1.8.0_272-b10)
OpenJDK 64-Bit Server VM Corretto-8.272.10.3 (build 25.272-b10, mixed mode)
This is the line TeamCity is using to invoke maven, I've only included property definitions that seemed relevant since there were so many that are definitely irrelevant (build numbers, names, timestamps, and other TC specifc, maven agnostic, config)
/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64/bin/java
-Dclassworlds.conf=/home/ec2-user/BuildAgent/temp/buildTmp/teamcity.m2.conf
-Dmaven.home=/home/ec2-user/BuildAgent/tools/maven3_3
-DskipTests=true
-Dteamcity.build.properties.file=/home/ec2-user/BuildAgent/temp/buildTmp/teamcity.build4380238471360533686.properties
-Dtarget.env=prod
-classpath /home/ec2-user/BuildAgent/tools/maven3_3/boot/plexus-classworlds-2.5.2.jar: org.codehaus.plexus.classworlds.launcher.Launcher -f /home/ec2-user/BuildAgent/work/4508a7116faa21f3/pom.xml -B clean package
The contents of teamcity.m2.conf is as follows:
main is org.apache.maven.cli.MavenCli from plexus.core
set maven.home default ${user.home}/m2
[plexus.core]
load ${teamcity.maven.watcher.home}/*.jar
optionally ${maven.home}/lib/ext/*.jar
load ${maven.home}/lib/*.jar
load ${maven.home}/conf/logging
teamcity.build4380238471360533686.properties contains many properties, the value of target.env within that file is 'prod' as expected
This ended up being a known bug in TeamCity
The underlying issue seems to be that TeamCity uses the MAVEN_OPTS environment variable to pass system properties into maven by default, but properties in MAVEN_OPTS are treated differently from properties passed as arguments to the maven command itself.
The workaround is that for any property "foo" that gets set in a POM section, that you want to override in a TeamCity build, you have to specify it in the "Additional Maven command line parameters" with -Dfoo=value, or, if you're setting the value in a system or build property within the TeamCity build -Dfoo=%system.foo%.

SonarQube Scanner for Maven and SonarTsPlugin Cannot Run node

I am using SonarQube 5.6.1 for code analysis (code written in Spring Boot and Angular (using TypeScript)). SonarQube Scanner for Maven 3.4.0.905 and SonarTsPlugin 1.1.0 are used for code scanning in Java and TypeScript, respectively.
In my pom.xml, I have the following properties set:
<frontend.directory>frontend</frontend.directory>
<sonar.host.url>http://sonarqube.some-url.local/</sonar.host.url>
<sonar.sources>src/main/java,${frontend.directory}/src/app,pom.xml</sonar.sources>
<sonar.exclusions>${frontend.directory}/node_modules/**</sonar.exclusions>
<sonar.tests>${frontend.directory}/src/app,src/test/java</sonar.tests>
<sonar.test.inclusions>**/*.spec.ts,src/test/java/**/*.java</sonar.test.inclusions>
<sonar.testExecutionReportPaths>${frontend.directory}/reports/ut_report.xml</sonar.testExecutionReportPaths>
<sonar.ts.coverage.lcovReportPath>${frontend.directory}/coverage/lcov.info</sonar.ts.coverage.lcovReportPath>
<sonar.ts.tslint.configPath>${frontend.directory}/tslint.json</sonar.ts.tslint.configPath>
<sonar.ts.tslint.nodePath>${frontend.directory}/node/node</sonar.ts.tslint.nodepath>
<sonar.ts.tslint.path>${frontend.directory}/node_modules/.bin/tslint</sonar.ts.tslint.path>
Please note that I am pointing sonar.ts.tslint.nodePath to the local node installed as following.
I use frontend-maven-plugin 1.6 to install npm and node:
<plugin>
<groupId>com.github.eirslett</groupId>
<artifactId>frontend-maven-plugin</artifactId>
<version>1.6</version>
<configuration>
<workingDirectory>${frontend.directory}</workingDirectory>
</configuration>
<executions>
<execution>
<id>install node and npm</id>
<goals>
<goal>install-node-and-npm</goal>
</goals>
<configuration>
<nodeVersion>v10.1.0</nodeVersion>
<npmVersion>6.0.1</npmVersion>
</configuration>
</execution>
...
</executions>
</plugin>
I run ./mvnw sonar:sonar to start the SonarQube scanner. However, when the SonarTsPlugin runs and goes to run tslint, the nodePath is ignored and the global node is used. This causes problems in my Jenkins box where I get the following error:
[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.4.0.905:sonar (default-cli) on project nsgra-web: java.io.IOException: Cannot run program "node": error=2, No such file or directory -> [Help 1]
This is odd to me since node is installed via frontend-maven-plugin and I am pointing to that node binary in sonar.ts.tslint.nodePath.
Does anyone have any insight?
Thanks Aerus for your response.
I found the same issue inside docker pod using jenkins.
I had maven installed but not node, then maven create a local installation of node at ./node/ folder. When the TS sonar was trying to execute node to know the current version, the commented error arise and all .ts files was ignored.
I solved using your suggestion and adding this to the command line execution
-Dsonar.typescript.node=./node/node

Using maven-deploy-plugin to deploy Eclipse product

I am trying to deploy an Eclipse product to Nexus repository. I am somewhat successful, but still get some errors that I am not sure how to handle.
I am using maven-deploy-plugin in the feature (F) that builds product:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<phase>deploy</phase>
<goals>
<goal>deploy-file</goal>
</goals>
<configuration>
<repositoryId>snapshots</repositoryId>
<packaging>zip</packaging>
<generatePom>true</generatePom>
<url>http://repo:8081/nexus/content/repositories/snapshots</url>
<groupId>my.group</groupId>
<artifactId>my.artifact</artifactId>
<version>1.0.0-SNAPSHOT</version>
<file>
target/products/product.zip
</file>
</configuration>
</execution>
</executions>
</plugin>
I execute 'mvn clean install' to build the product and then 'mvn deploy' on F. If I do that then I get error
Exception while publishing product /home/akravets/dev/workspaces/trunk/my.repository.feature/myProduct.product: publishing result: [Included element my.product.feature.feature.group 8.0.1.R20180301-1431 is missing. Cannot determine filter for requirement to this element.] -> [Help 1]
I've read discussions about this issue, but nothing seems relevant to my issue because I don't have any defined deploy phases besides the one in maven-deploy-plugin.
If I change phase of maven-deploy-plugin to 'install' I get almost satisfactory results. The zip file and generated pom get deployed to repository, but the build fails with error
product.qualifier-p2artifacts.xml. Return code is: 400, ReasonPhrase: Bad Request.
Why does this file even gets considered to be deployed to repository when it's not in file element of maven-deploy-plugin and it's not in the directory structure where zip file is: target/ vs target/product/product.zip? Is this because maven-deploy-plugin is executed during the install phase and it picks up all generated resources as candidates for deployment?
I simply recommend you to use Maven-Tycho to deploy Eclipse RCP product.Tycho determines the dependencies of a plug-in via the MANIFEST.MF file of the plug-in and fetches the required plugins-bundles through the online repositories.
You can check this git repository as an example; rcp - tycho

Maven Release Plugin - release:branch - Version Parameter in -DbranchName

When executing the mvn release:branch command, I'd like to try and configure the -DbranchName argument with a version parameter. But I haven't been able to find a way to successfully do this yet.
For example, if I have a project in trunk with version 1.0-SNAPSHOT, I want to create a branch using the --batch-mode option and specify a -DbranchName with the current project version (1.0-SNAPSHOT) as a parameter in the within the branchName.
I've tried doing mvn release:branch -DbranchName=VERSION-#{project.version}, but it doesn't seem to work as I had hoped. Instead of creating a branch with the name VERSION-1.0, I get the following error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:2.5.3:branch (default-cli) on project project-a: Unable to branch SCM
[ERROR] Provider message:
[ERROR] The svn branch command failed.
[ERROR] Command output:
[ERROR] svn: E205000: Try 'svn help copy' for more information
[ERROR] svn: E205000: Syntax error parsing peg revision '{project.version}'
Is there a valid way to do this? Or is this not possible when doing a release:branch with the maven release plugin?
Edit
Basically what I'm looking for is something similar to the tagNameFormat property but for branches. When doing a release:perform goal, I can have <tagNameFormat>VERSION-#{project.version}</tagNameFormat> specified in the pom, and I don't need to include any tag name property when doing a release using --batch-mode. Is there any similar property for formatting of branch names when doing a release:branch?
You can use the built-in ${project.version} (prefixed with $ instead of # -- the latter seems only available in the tagNameFormat parameter). Note that if you use ${project.version}, the value will probably contain the substring SNAPSHOT in it. There may be several ways of removing that substring. One way is to apply the helper plugin mentioned in the answer to this question which gives you access to the version's semantic components:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.8</version>
<executions>
<execution>
<id>parse-version</id>
<goals>
<goal>parse-version</goal>
</goals>
</execution>
</executions>
</plugin>
Then you can use the properties on the command line:
mvn release:branch -DbranchName=VERSION-${parsedVersion.majorVersion}.${parsedVersion.minorVersion}.${parsedVersion.incrementalVersion}
Or better to include the branchName parameter as part of the plugin configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>2.5.3</version>
<executions>
<execution>
<configuration>
<branchName>VERSION-${parsedVersion.majorVersion}.${parsedVersion.minorVersion}.${parsedVersion.incrementalVersion}</branchName>
</configuration>
<goals>
<goal>branch</goal>
</goals>
</execution>
</executions>
</plugin>
In this case you don't need to pass these magic properties on the command line.

Sonar configuration in multimodule maven project using tycho for unit tests and jacoco for coverage

We're using maven to run a sonar analysis and it works well except for the code coverage results with jacoco. We have an eclipse project that uses tycho-surefire-plugin for testing. I've not overriden the argLines properties so solutions involving that line may not be appropiate.
Facts :
Maven structure structure:
parent
master
module 1
module ...
module n
Testing structure:
client.admin (eclipse-plugin packaging)
client.admin.test.fragment (eclipse-test-plugin packaging)
Properties that are correctly set and identified
sonar.junit.reportsPath
sonar.jacoco.reportPath,
sonar.jacoco.itReportPath
sonar.core.codeCoveragePlugin
sonar.language
The main problem is with the following properties
sonar.test
sonar.sources
sonar.java.binaries
As seen in the Testing structure in the client.admin.test.fragment tests are contained in the /src folder and the sources are located in the project client.admin in the /src folder too.
When we run the analysis we get the following error :
[WARN] Coverage information was not collected. Perhaps you forget to include
debug information into compiled classes?
I believe this has to do with the properties sonar.java.binaries that goes looking for the sources in target/classes of the fragment project (client.admin.project) that are in fact located in the host project (client.admin). In the fragment project we've configured sonar.tests and sonar.sources properties so that they call the /src folder of the corresponding projects.
In the sonar Analysis Parameters page there says that only sonar.sources is a maven valid property, sonar.tests and sonar.java.binaries cannot apparently be configured in maven. How then could I attach the binaries to the project. I've tried copying the folder target/classes from the host project but I got the same message. Is there any workaround in maven ?
Edit 1
There is one jacoco.exec file that is generated for the whole project that can be found at the parent folder. This was done configuring the jacoco.destFile and sonar.jacoco.reportPath properties
Jacoco plugin in main pom :
<!-- Jacoco Plugin -->
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<executions>
<execution>
<id>prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${sonar.jacoco.reportPath}</destFile>
<append>true</append>
</configuration>
</execution>
<execution>
<id>default-report</id>
<goals>
<goal>report</goal>
</goals>
<configuration>
<dataFile>${sonar.jacoco.reportPath}</dataFile>
<outputDirectory>${jacoco.reports.outputDirectory}</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
Maven Plugin Versions:
sonar: 2.4
jacoco: 0.7.1.201405082137
Properties
<sonar.language>java</sonar.language>
<sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
<sonar.junit.reportsPath>${project.build.directory}/surefire-reports/</sonar.junit.reportsPath>
<sonar.jacoco.reportPath>${basedir}/../../../main/**.master/target/jacoco.exec</sonar.jacoco.reportPath>
<jacoco.reports.outputDirectory>${basedir}/../../../main/**.master/target/site/jacoco</jacoco.reports.outputDirectory>
<sonar.sources>src</sonar.sources>
In the test projects (eclipse-test-plugin) we changed added the property sonar.sources to go find the sources from the src folder of the project that we're testing for example in client.admin.test.fragment we go search the src from the client.admin
The following properties were commented in code because they're not supported in maven according to documentation and to the debug output.
<!--<sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>-->
<!--<sonar.tests></sonar.tests>-->
<!--<sonar.java.binaries></sonar.java.binaries>-->
First, you must tell the JaCoCo agent to report all coverage data into one common file. Second, you tell the Sonar JaCoCo plugin to read the coverage data from the aggregated file.
To do so, set the properties "jacoco.destFile" and "sonar.jacoco.reportPath" in your parent pom.xml to the same absolute path, e.g.:
<properties>
<jacoco.destFile>/home/jenkins/jobs/my.project/workspace/parent/target/jacoco.exec</jacoco.destFile>
<sonar.jacoco.reportPath>/home/jenkins/jobs/my.project/workspace/parent/target/jacoco.exec</sonar.jacoco.reportPath>
</properties>
Note that these properties will be inherited to all child poms, so you can't use Maven expressions like ${project.build.directory} because this would evaluate to a different directory for each pom.
You could create a small helper Mojo which automatically resolves an absolute path on the current build machine and then injects the properties into the Maven model.

Resources