SonarQube - Multimodule project based on filesystem hierarchy - sonarqube

Some cvsrepository named Foo has several modules, f.e. module1, module2, ...
A buildscript does cvs checkout, compile and building the module.jar for all modules.
After the build has finished the filesystem looks like that :
Foo/module1/src
/bin
/module2/src
/bin
...
Now i want to use the sonartask like that :
Define the global sonar properties like sonar.username ... sonar.projektName=Foo for MainProject and sonar.modules=module1.module2,... for the modules.
Afterwards looping over the module folders, setting module specific sonar properties,
like sonar.projectName (for modulename) sonar.sources, sonar.binaries etc.
Looping works fine, but it seems as sonar wants a single build.xml for every submodule.
So i need another possibility for grouping modules.
I only found a commercial plugin => Portfolio Management plugin but need another solution.
-- EDIT --
I already found and tried to use the examples from github and documentation, but it seems theyrequire one main build file and a build.xml in every module folder.
Whereas my situation is different. There's only one build.xml that contains a javac + jar target for every module of my project. When build is done, the resulting filesystem looks like mentioned above.
The build is part of the deployment workflow which is driven by a xmlproperty file, that contains the projectname and all modulenames of that deployment job among other things.
Now after the build is done, in the next antfile i want to loop over the modules in that xmlproperty file via xmltask, pointing the module specific properties to filesystem, f.e. sonar.sources=/Foo/module1/src sonar.binaries=/Foo/module1/bin and call <sonar:sonar/> . Overriding of module specific properties in the loop is done via Flaka <let>
After all it's just looping over an existing filesystem setting the sonar poperties and calling the sonar task with approiate properties.
-- EDIT after comment --
Tried it like that :
build.xml
<project name="sonar" xmlns:fl="antlib:it.haefelinger.flaka" xmlns:sonar="antlib:org.sonar.ant">
<fl:install-property-handler/>
<!-- Import Sonar -->
<taskdef uri="antlib:org.sonar.ant" resource="org/sonar/ant/antlib.xml"/>
<!-- Import XMLTask -->
<taskdef name="xmltask" classname="com.oopsconsultancy.xmltask.ant.XmlTask"/>
<xmltask source="C:/WKS/Eclipse/372/sonar/props.xml">
<copy path="//project/name/text()" property="projectname"/>
<copy path="//module/*/label/text()" property="projectlabel"/>
<copy path="//module/*/name/text()" property="modules" append="true" propertyseparator="," />
</xmltask>
<property name="sonar.projectKey" value="${projectname}"/>
<!-- Set modules IDs -->
<property name="sonar.modules" value="#{replace('${modules}', '', ' ')}"/>
<property name="sonar.language" value="java" />
<property name="sonar.sourceEncoding" value="UTF-8" />
<property name="sonar.host.url" value="http://somehost:9000" />
<property name="sonar.login" value="admin" />
<property name="sonar.password" value="admin" />
<xmltask source="C:/WKS/Eclipse/372/sonar/props.xml">
<call path="//module/*">
<param path="label/text()" name="modullabel"/>
<param path="name/text()" name="modulname"/>
<actions>
<fl:let>
sonar.projectName ::= replace('#{modulname}', '', ' ')
sonar.projectVersion ::= '#{modullabel}'
sonar.#{replace('#{modulname}', '', ' ')}.projectBaseDir ::= '${ant.working.dir}/build/projects/${projectname}/#{modulname}'
sonar.sources ::= '${ant.working.dir}/build/projects/${projectname}/#{modulname}/src'
sonar.binaries ::= '${ant.working.dir}/build/results/compile/${projectname}/#{modulname}'
</fl:let>
<!-- Execute Sonar -->
<sonar:sonar />
</actions>
</call>
</xmltask>
</project>
output :
Setting project property: modules -> Modul 1,Modul 2,Modul 3,Modul 4
Setting project property: sonar.projectKey -> Foobar
Setting project property: sonar.modules -> Modul1,Modul2,Modul3,Modul4
Setting project property: ant.working.dir -> c:\WKS\5513
Setting project property: sonar.language -> java
Setting project property: sonar.sourceEncoding -> UTF-8
Setting project property: sonar.host.url -> http://somehost:9000
Setting project property: sonar.login -> admin
Setting project property: sonar.password -> admin
Setting project property: sonar.projectName -> Modul1
Setting project property: sonar.projectVersion -> l_5_04_52_cd
Setting project property: sonar.Modul1.projectBaseDir -> c:\WKS\5513/build/projects/foobar/Modul 1
Setting project property: sonar.sources -> c:\WKS\5513/build/projects/foobar/Modul 1/src
Setting project property: sonar.binaries -> c:\WKS\5513/build/results/compile/foobar/Modul 1
parsing buildfile jar:file:/C:/WKS/Eclipse/372/DevJars/sonar/sonar-ant-task-2.1.jar!/org/sonar/ant/antlib.xml with URI = jar:file:/C:/WKS/Eclipse/372/DevJars/sonar/sonar-ant-task-2.1.jar!/org/sonar/ant/antlib.xml from a zip file
Class org.sonar.ant.SonarTask loaded from parent loader (parentFirst)
[sonar:sonar] Apache Ant(TM) version 1.9.1 compiled on May 15 2013
[sonar:sonar] Sonar Ant Task version: 2.1
[sonar:sonar] Loaded from: file:/C:/WKS/Eclipse/372/DevJars/sonar/sonar-ant-task-2.1.jar
[sonar:sonar] INFO: Default locale: "de_DE", source code encoding: "UTF-8"
[sonar:sonar] INFO: Work directory: C:\WKS\sonar
[sonar:sonar] INFO: Sonar Server 3.7.2
BUILD FAILED
C:\WKS\Eclipse\372\sonar\sonar.xml:41: The following error occurred while executing this line:
C:\WKS\Eclipse\372\sonar\sonar.xml:54: org.sonar.runner.impl.RunnerException: Unable to execute Sonar
...
Caused by: java.lang.IllegalStateException: The base directory of the module 'Modul1' does not exist: C:\WKS\Eclipse\372\sonar\Modul 1
I tried also to set the basedir attribute to the value of my ${ant.working.dir}, where all binariues and src folders reside, didn't work either.

See documentation : http://docs.codehaus.org/display/SONAR/Analyzing+with+SonarQube+Ant+Task#AnalyzingwithSonarQubeAntTask-Multi-moduleProject
And example : https://github.com/SonarSource/sonar-examples/tree/master/projects/multi-module/ant

I had a similar problem. When you run the ant task from eclipse, sonar errors are not showed. so, I figure out the problem when the ant ask was executed from command console.
I had two problems:
The sonar.host.url property is mandatory
The sonar.projectKey could not contain special characters, for instance space

Related

maven error "Property ${samedir} has not been set"

How to fix the maven checkstyle plugin error
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:check (validate) on project yourproject:
Failed during checkstyle execution:
Failed during checkstyle configuration: unable to parse configuration stream:
com.puppycrawl.tools.checkstyle.api.CheckstyleException:
Property ${samedir} has not been set -> [Help 1]
${samedir} is a property working well with eclipse plugin and is required to get related files mentioned in the checkstyle configuration working well with eclipse IDE. So either I need a consistent replacement, or I can tell maven to define the property
Any ideas how to fix this issue?
You will have to set these values via the plugin configuration in your pom.
Example configuration of pom.xml if you have a directory named checks in your project root:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.1.2</version>
<configuration>
<configLocation>${basedir}/checks/checkstyle.xml</configLocation>
<propertiesLocation>${basedir}/checks/checkstyle_maven.properties</propertiesLocation>
</configuration>
</plugin>
And the checkstyle_maven.properties contents:
samedir=checks
As an alternative to the <propertiesLocation> you can also use the following, in which case you don't need the .properties file:
<propertyExpansion>samedir=${basedir}/checks</propertyExpansion>
This will make sure any configuration in checkstyle.xml will work like this:
<module name="Checker">
<module name="SuppressionFilter">
<property name="file" value="${samedir}/suppress.xml"/>
</module>
</module>
See for reference:
Using ${samedir} in Checkstyle-plugins for Gradle and Eclipse
http://rolf-engelhard.de/2011/04/using-the-same-suppression-filter-for-checkstyle-in-eclipse-and-maven/

TeamCity REST API: Triggering Build with Custom Artifact Dependency

I'm trying to trigger a custom build using the TeamCity 8.1 REST API (see https://confluence.jetbrains.com/display/TCD8/REST+API#RESTAPI-TriggeringaBuild). My builds run fine and I'm able to specify custom branches and properties with no issue. My goal is now to specify a custom artifact dependency for the builds I'm triggering.
The response I receive when creating a build is something like the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<build taskId="1234" buildTypeId="buildConfig1" state="queued" ...>
<buildType id="buildConfig1" name="Build Config Name" ... />
...
<properties count="1">
<property name="testParam" value="Test 123" own="true" />
</properties>
<custom-artifact-dependencies />
</build>
The "custom-artifact-dependencies" tag in this response leads me to believe that there is a way to specify custom dependencies, but I have not found anything in the TeamCity documentation, the TeamCity forums, or from Google explaining how this can be accomplished. Is there something I'm overlooking here or another way of accomplishing this?
Adding the following to as a child of the "build" tag results in an "Artifact dependency should have type 'artifact_dependency'." error:
<custom-artifact-dependencies>
<artifact-dependency buildId="5432" buildTypeId="parentBuildConfig"/>
</custom-artifact-dependencies>
The server may be confusing my intention with the build configuration API for setting and viewing artifact dependencies (e.g. http://teamcity:8111/httpAuth/app/rest/buildTypes/<buildTypeLocator>/artifact-dependencies/)
I had the same problem
This is what worked for me:
<build>
<triggeringOptions queueAtTop="true"/>
<buildType id="buildConfig1"/>
<custom-artifact-dependencies count="1">
<artifact-dependency id="0" type="artifact_dependency">
<properties>
<property name="pathRules" value="Artifacts_1.zip
Artifacts_2.zip
Artifacts_To_Unzip.zip!/**
"/>
<property name="cleanDestinationDirectory" value="true"/>
<property name="revisionName" value="buildId"/>
<property name="revisionValue" value="5432"/>
</properties>
<source-buildType id="parentBuildConfig" />
</artifact-dependency>
</custom-artifact-dependencies>
</build>
If the 'parentBuildConfig' build is still running, replace the buildId parameter with taskId

Mandatory properties missing when migrating to new 2.0 ant task sonar

I have a multi module project (with many sub-modules) defined in Sonar and it was working great until i've upgraded sonar to the newer version.
My previous settings were:
a single parent build.xml that contain all of the general project properties like jdbc connection etc. Also it contained the <sonar:sonar /> task to run Sonar. This parent project does not contain any source / binaries, just the instruction to build the sub-modules.
many sub-modules build.xml files that contained only the sub-module specific properties like sonar.projectKey, sonar.sources and sonar.binaries
Now in the new Sonar ant task I need to define everything in the parent buid.xml. I have defined it in the following way (I only show one sub-module here, I assume it should work the same if I add more modules later) :
<target name="sonar">
<taskdef uri="antlib:org.sonar.ant" resource="org/sonar/ant/antlib.xml">
<classpath path="build-utils/lib/sonar-ant-task-2.0.jar" />
</taskdef>
<!-- list of Sonar database related properties -->
<property name="sonar.jdbc.url" value="jdbc:oracle:thin:#localhost/DB11g" />
<property name="sonar.jdbc.driverClassName" value="oracle.jdbc.driver.OracleDriver" />
<!-- for security reasons, pass these parameters from outside -->
<property name="sonar.jdbc.username" value="" />
<property name="sonar.jdbc.password" value="" />
<property name="sonar.host.url" value="http://localhost:9000" />
<!-- list of Sonar project related properties -->
<property name="sonar.projectName" value="My Project" />
<property name="sonar.projectKey" value="com.my.project" />
<property name="sonar.projectVersion" value="7.3" />
<property name="sonar.language" value="java" />
<property name="sonar.libraries" value="server/third-party-jars"/>
<property name="sonar.modules" value="admin-api" />
<!-- all of the sub-modules -->
<property name="admin-api.sonar.projectName" value="admin-api" />
<property name="admin-api.sonar.projectBaseDir" location="server/admin/admin-api"/>
<property name="admin-api.sonar.sources" value="src" />
<property name="admin-api.sonar.binaries" value="build" />
<sonar:sonar xmlns:sonar="antlib:org.sonar.ant" />
</target>
When I run the target I get the following error:
You must define the following mandatory properties for 'com.orca.rightv': sonar.sources
Another question, is there any way to keep the specific definition of the sub-modules in each module's build.xml like I did in the previous version ? That would save me a lot of work.
Thanks
There's a bug in the current version of the Ant Task, which is related to the version of the Sonar Runner API embedded. This will be fixed in the next version: http://jira.codehaus.org/browse/SONARPLUGINS-2818
Meanwhile, you can define "sonar.sources" at the top, its value will be inherited in to modules (and obviously you can still override it in modules).
As for the definition of properties at module level, you can define those properties in a "sonar-project.properties" file that you put at the root of the module, like:
sonar.projectName=My Project
# following needed only to override the values defined in top build.xml
sonar.sources=src
sonar.binaries=build
, and just leave "admin-api.sonar.projectBaseDir" property (like you did) in the top build.xml file.

Maven / IDEA setting the Resource patterns

My project is using Maven to compile both Java and Groovy classes.
My project compiles and builds just fine using maven but I would like IDEA (IntelliJ) to build it correctly as well. I use maven to generate the IDEA project file with the following command:
mvn idea:idea
I noticed it doesn't build the groovy files, only the java files.
I found out the issue was that by default IDEA was considering all the files not having a .java extension as resources.
When I generate the project file with the maven IDEA plugin I get the following in the ipr file
<component name="CompilerConfiguration">
<option name="DEFAULT_COMPILER" value="Javac" />
<resourceExtensions />
<wildcardResourcePatterns>
<entry name="!?*.java" />
</wildcardResourcePatterns>
...
I have to modify it this way in order to have my groovy files compiling (and few others kind of files):
<component name="CompilerConfiguration">
<option name="DEFAULT_COMPILER" value="Javac" />
<resourceExtensions />
<wildcardResourcePatterns>
<entry name="!?*.java" />
<entry name="!?*.groovy" />
<entry name="!?*.scala" />
<entry name="!?*.flex" />
</wildcardResourcePatterns>
...
Note: this information can be found/edited as well in IDEA -> Settings -> Compiler -> Resource patterns
I can generate it correctly using the following command:
mvn idea:idea -DwildcardResourcePatterns="!?*.java;!?*.groovy;"
Does anyone know how to configure this in the pom.xml?
This need to be added in the pom.xml
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-idea-plugin</artifactId>
<configuration>
<wildcardResourcePatterns>!?*.java;!?*.groovy;</wildcardResourcePatterns>
</configuration>
</plugin>

How to stop Maven from expanding env. variable w/Jenkins?

I'm familiar with Jenkins, but new to Maven, and I'm trying to figure out how to stop our Maven jobs in Jenkins from expanding an environment variable inside of the ApplicationContext.xml file.
Inside our ApplicationContext.xml file we reference a ${DeployMode} environment variable (that we created) which Tomcat expands at load/run time:
<!-- SPRING CONTEXT static accessor -->
<beans:bean id="contextApplicationContextProvider"
class="com.dartneuroscience.compserv.rest.appcontext.AppContextProvider">
<beans:constructor-arg>
<beans:value>${DeployMode}</beans:value>
</beans:constructor-arg>
</beans:bean>
<beans:bean id="placeholderConfig"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<beans:property name="location"
value="/WEB-INF/App_${DeployMode}.properties" />
</beans:bean>
The problem is that in a Maven build run by Jenkins, the built WEB-INF/ApplicationContext.xml looks like this if there's an environment variable set on the build machine (let's say to 'Prod' in this example):
<!-- SPRING CONTEXT static accessor -->
<beans:bean id="contextApplicationContextProvider"
class="com.dartneuroscience.compserv.rest.appcontext.AppContextProvider">
<beans:constructor-arg>
<beans:value>Prod</beans:value> <!-- Expanded env. var -->
</beans:constructor-arg>
</beans:bean>
<beans:bean id="placeholderConfig"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<beans:property name="location"
value="/WEB-INF/App_Prod.properties" /> <!-- Expanded env. var -->
</beans:bean>
And so the value is now "hard coded" into the WAR, and will always act like "Prod" even if the target web server has its DeployMode environment variable set to something else, like 'Staging'.
This does not happen when I run Maven manually on the same build server--it only happens when Jenkins builds the job.
Is there a setting I can pass to Jenkins to stop this behavior?
I've looked at options like the EnvInject Plugin to UNSET all environment variables as a job is run, but I am really puzzled by this behavior and would like to get to the bottom of it.
Thank you.
Update 1
I found the following resource filtering block in the top-level POM and added the <excludes/> block to skip our AppContext.xml file:
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.1.1</version>
<configuration>
<webResources>
<webResource>
<directory>${basedir}/src/main/webapp/WEB-INF</directory>
<includes>
<include>*.xml</include>
</includes>
<excludes>
<exclude>*AppContext.xml</exclude>
</excludes>
<targetPath>WEB-INF</targetPath>
<filtering>true</filtering>
</webResource>
</webResources>
But this still doesn't answer why the environment variables are added to the properties list when run by Jenkins, and ignored when run from the command line; though I did find this Hudson issue from a couple of years ago: Resource filtering fails when run from Hudson
Also, EnvInjectPlugin does what it advertises, but in removing at least the PATH var, the build broke because maven could not find the ls command.
Update 2
Changing the job in Jenkins from a "Maven 2/3 project" to a "Free-style software project" and using the Invoke top-level Maven targets build step produces the desired result without having to modify the POM.
Jenkins may be enabling resource filtering somehow. You have lots of choices for configuring filtering. You may disable it entirely, limit filtering to files with certain extensions, tell it not to include build properties in the filters, or choose different delimiters to be filtered entirely. See the resources:resources docs or the description of resource filtering concepts.

Resources