I'm struggling with sonar server and sonar runner (standalone via commandline). My project sonar file is:
# Required metadata
sonar.projectKey=org.codehaus.sonar:calculator
sonar.projectName=Calculator
sonar.projectVersion=1.0
# Comma-separated paths to directories with sources (required)
sonar.sources=src
# Language
sonar.language=py
# Encoding of the source files
sonar.sourceEncoding=UTF-8
And my command is
/opt/sonar-runner/bin/sonar-runner -Dsonar.projectBaseDir=/var/lib/jenkins/workspace/Servidor-RDS/serv/rdsweb/WebContent/fccma/js/fcc/tests/python-sonar-runner/ -Dproject.settings=/var/lib/jenkins/workspace/Servidor-RDS/serv/rdsweb/WebContent/fccma/js/fcc/tests/python-sonar-runner/sonar-project.properties
Nothing exotic. I haven't any project in sonar (I've delete all), and my project structure is
src/
myCaluclator/
calculator.py
init.py
Calculator code is:
class Calculator(object):
def add(self, x, y):
return x + y
I always get an:
Caused by: org.sonar.api.resources.DuplicatedSourceException: Duplicate source for resource: org.sonar.api.resources.File#7d3e8935[key=myCalculator/calculator.py,dir=myCalculator,filename=calculator.py,language=Python]
at org.sonar.batch.index.SourcePersister.saveSource(SourcePersister.java:45)
at org.sonar.batch.index.DefaultPersistenceManager.setSource(DefaultPersistenceManager.java:78)
at org.sonar.batch.index.DefaultIndex.setSource(DefaultIndex.java:456)
at org.sonar.batch.DefaultSensorContext.saveSource(DefaultSensorContext.java:161)
at org.sonar.api.batch.AbstractSourceImporter.parseDirs(AbstractSourceImporter.java:88)
Do I need to clean some cache or something?
Thanks in advance.
I found the solution:
The sonar.sources is present in my sonar-runner properties and in my project properties, so all files are declared twice.
My error
Related
I have following part of the build.gradle:
def external = file('../external').absolutePath
repositories {
maven {
url file(external).toURI()
}
}
dependencies {
api 'ch.qos.cal10n:cal10n-api:0.7.7'
}
Third-party library that my project uses is stored in the ../external/ch.qos.cal10n/cal10n-api/0.0.7/(.jar and .pom).
However when running gradle build I got following:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
> Could not find ch.1qos.cal10n:cal10n-api:0.7.7.
Searched in the following locations:
- file:/home/a/b/external/ch/1qos/cal10n/cal10n-api/0.7.7/cal10n-api-0.7.7.pom
Why /home/a/b/external/ch/1qos/cal10n/cal10n-api/0.7.7/cal10n-api-0.7.7.pom
and not /home/a/b/external/ch.1qos.cal10n/cal10n-api/0.7.7/cal10n-api-0.7.7.pom ?
UPD: .pom and .jar files are downloaded manually from Maven Repository and placed into /home/a/b/external/ch.1qos.cal10n/cal10n-api/0.7.7/`
Because that is the layout of local-maven repositories, i.e., the group will be turned to directories. This is in part for performance and usability reasons. Consider the alternative, you'd have one huge directory containing all the groupIds. Working with directories with lots of entries slows down processing a lot. Instead, this layout is hierarchical, reducing the number of files/sub-directories per directory.
So to solve your problem, just change your directory layout accordingly.
/home/a/b/external/ch.1qos.cal10n/cal10n-api/0.7.7/cal10n-api-0.7.7.pom
->
/home/a/b/external/ch/1qos/cal10n/cal10n-api/0.7.7/cal10n-api-0.7.7.pom
I have a gradle project with 1 submodule, defined in the following file structure (+- refers to a directory):
+- <root>
build.gradle.kts
+- graph-commons
+- core
build.gradle.kts
The only submodule was included using the following kotlin script:
val graphCommons = project(File("./graph-commons/core"))
includeBuild(graphCommons)
When I execute ./gradlew clean assembly, I got the following error:
FAILURE: Build failed with an exception.
* Where:
Settings file '/home/peng/git/shapesafe/settings.gradle.kts' line: 2
* What went wrong:
Project with path './graph-commons/core' could not be found.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 492ms
FAILURE: Build failed with an exception.
* Where:
Settings file '/home/peng/git/shapesafe/settings.gradle.kts' line: 2
* What went wrong:
Project with path './graph-commons/core' could not be found.
What went wrong? Why the valid path "./graph-commons/core" cannot be identified by gradle?
The project is uploaded and tested on github:
https://github.com/tribbloid/shapesafe/runs/4280805005?check_suite_focus=true
Gradle does not work this way. A project path refers to a gradle project path, not to a file path. See https://docs.gradle.org/current/userguide/multi_project_builds.html#multi_project_builds
Edit: As mentioned in the comments, the project(File) method that is available in the settings.gradle.kts is a special method allowing to receive a ProjectDescriptor whose directory points to the given file. The project must be present already e.g. by including it via include(String...) first.
I first thought you tried to use the DependencyHandler#project(Map) method in some way, which is the usual way to refer to project dependencies. Gradle separates between dependencies and multi-project setup. In the settings.gradle.kts you usually setup the project structure while you declare dependencies in each build.gradle.kts. When using includeBuild you merely depend on the build of another completely separate project. When you then want to declare a dependency to a project from that included build you usually use the project's artifact coordinates to do so. This way the build still works when removing the includeBuild declaration.
If you want to use composite builds see here for basic usage: https://docs.gradle.org/current/samples/sample_composite_builds_basics.html
You will have to coordinate the artifact publishing and corresponding dependencies to make it work like a normal multi-project. Something like this:
graph-commons
|build.gradle.kts -> group = "org.scala-lang"; version = "1.0";
|settings.gradle.kts -> include(":graph-commons-core")
|graph-commons-core
||build.gradle.kts
shapesafe
|settings.gradle.kts -> includeBuild("../graph-commons")
|core
||build.gradle.kts -> dependencies { implementation("org.scala-lang:graph-commons-core:1.0") }
I got some source code and was asked to build it. It was a Gradle project. So I changed to the project directory and ran:
$ gradle clean assemble
and the following error came up:
...
* What went wrong:
A problem occurred evaluating root project 'pcase'.
> Could not get unknown property 'postgresql.jdbc' for root project 'pcase' of type org.gradle.api.Project.
...
There is a settings.gradle file in the project folder too. It contains:
rootProject.name = 'pcase'
I took a look at build.gradle and found lots of occurrences like
${project['x']}
For example:
buildscript {
dependencies {
...
// FlywayDB, JOOQ.
classpath "org.postgresql:postgresql:${project['postgresql.jdbc']}"
classpath "org.flywaydb:flyway-gradle-plugin:${project['flywaydb.plugin.version']}"
classpath "nu.studer:gradle-jooq-plugin:${project['jooq.plugin.version']}"
...
What could be ${project['x']}? Looks like associative array in bash and the build script tries to get the value of the key 'x'.
But I didn't find the place in code where this array would be declared and initialized.
The question is: Is the project buildable or is it better to consult the company that worked at it before me?
From the information provided, the project is perfectly buildable, to some certain extend. First of all, project['a.b.c'] is Groovy syntax to access properties from the project object. They're referred to as project properties.
They can be set via
Project properties via command line: gradle -Ppostgresql.jdbc=x.y.z
System properties via command line: gradle -Dorg.gradle.project.postgresql.jdbc=x.y.z
System properties via gradle.properties: org.gradle.project.postgresql.jdbc=x.y.z
All 3 properties (postgresql.jdbc, flywaydb.plugin.version, jooq.plugin.version) denote the version numbers of the particular build script dependencies. However, which versions to use best is beyond my knowledge. I would certainly consult the respective project websites, Maven artifact search or simply ask the company.
org.postgresql:postgresql is the database JDBC driver and certainly depends on the database version.
org.flywaydb:flyway-gradle-plugin is for database migrations. Try with the latest version.
I wasn't able to find gradle-jooq-plugin on Maven central. It's most likely available on the Gradle Plugin Portal.
We have projects with a structure like this ( a root pom and 2 sub modules ) :
local-module
implementation-module
pom.xml
By default sonar seems to use these sources :
<sonar.sources>pom.xml,src/main,src/test</sonar.sources>
We want to analyse YML files or other files in others locations.
But after trying various configuration according to https://docs.sonarqube.org/latest/project-administration/narrowing-the-focus/
This one :
<sonar.sources>.</sonar.sources>
<sonar.tests>src/test/java</sonar.tests>
<sonar.test.inclusions>src/test/java</sonar.test.inclusions>
<sonar.exclusions>src/test/java</sonar.exclusions>
produces the following error :
[main] ERROR org.apache.maven.cli.MavenCli - Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.6.0.1398:sonar (default-cli) on project sample: null: MojoExecutionException: NullPointerException
and this one :
<sonar.sources>.</sonar.sources>
<sonar.test.exclusions>src/test/java</sonar.test.exclusions>
produces :
[main] ERROR org.apache.maven.cli.MavenCli - Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.6.0.1398:sonar (default-cli) on project pew-fwk-sample: File sample-impl/src/test/java/com/xxx/sample/FeaturesTest.java can't be indexed twice. Please check that inclusion/exclusion patterns produce disjoint sets for main and test files -> [Help 1]
Do you have any ideas on the solutions ?
I would not change to many parameters at once. Some of these properties are provided by the maven plugin and should not need to be changed. I would start just using the exclusions parameter. Note that it uses some ant style pattern to exclude files. If you don't provide it it will only apply to that specific file or folder. The page you linked contains some example at the bottom:
# Exclude all classes ending by "Bean"
# Matches org/sonar.api/MyBean.java, org/sonar/util/MyOtherBean.java, org/sonar/util/MyDTO.java, etc.
sonar.exclusions=**/*Bean.java,**/*DTO.java
# Exclude all classes in the "src/main/java/org/sonar" directory
# Matches src/main/java/org/sonar/MyClass.java, src/main/java/org/sonar/MyOtherClass.java
# But does not match src/main/java/org/sonar/util/MyClassUtil.java
sonar.exclusions=src/main/java/org/sonar/*
# Exclude all COBOL programs in the "bank" directory and its sub-directories
# Matches bank/ZTR00021.cbl, bank/data/CBR00354.cbl, bank/data/REM012345.cob
sonar.exclusions=bank/**/*
# Exclude all COBOL programs in the "bank" directory and its sub-directories whose extension is .cbl
# Matches bank/ZTR00021.cbl, bank/data/CBR00354.cbl
sonar.exclusions=bank/**/*.cbl
So something like <sonar.exclusions>src/main/java/**/*</sonar.exclusions> should exclude all sources. Above is an example to only exclude Dto classes. Usually the exclusions are enough to narrow the focus down. But includes work the same way should you want to go that way.
I have the next problem. I'm developing a project in java and create sonar.properties file to use SonarQube (v6.7.2). My project have two modules. The project structur is:
my-project
|--core-utils
|--src
|--target
|--pdf-utils
|--src
|--target
And sonar.properties is:
First module
sonar.moduleKey=com.app.core.pdf:pdf-utils
sonar.projectBaseDir=/var/lib/jenkins/workspace/MyProject/pdf-utils
sonar.projectKey=com.app.core.pdf:pdf-utils
sonar.projectName=pdf-utils
sonar.projectVersion=0.90.0-SNAPSHOT
sonar.sources=/var/lib/jenkins/workspace/MyProject/pdf-utils/src/main/java
sonar.working.directory=/var/lib/jenkins/workspace/MyProject/pdf-utils/target/sonar
sonar.java.binaries=/var/lib/jenkins/workspace/MyProject/pdf-utils/target/classes
Second module
sonar.moduleKey=com.app.core:core-utils
sonar.projectBaseDir=/var/lib/jenkins/workspace/MyProject/core-utils
sonar.projectKey=com.app.core:core-utils
sonar.projectName=core-utils
sonar.projectVersion=0.90.0-SNAPSHOT
sonar.sources=/var/lib/jenkins/workspace/MyProject/core-utils/src/main/java
sonar.working.directory=/var/lib/jenkins/workspace/MyProject/core-utils/target/sonar
sonar.java.binaries=/var/lib/jenkins/workspace/MyProject/core-utils/target/classes
Main module
sonar.host.url=http://localhost:9000/sonar
sonar.java.source=1.6
sonar.java.target=1.6
sonar.moduleKey=com.app.core:core-all
sonar.modules=com.app.core:core-utils,com.app.core.pdf:pdf-utils
sonar.projectBaseDir=/var/lib/jenkins/workspace/MyProject
sonar.projectKey=com.app.core:core-all
sonar.projectName=core-all
sonar.projectVersion=DEV-SNAPSHOT
sonar.sourceEncoding=UTF-8
Then runs SonarScanner and the result is:
The base directory of the module 'com.app.core:core-utils' does not exist: /var/lib/jenkins/workspace/MyProject/com.app.core:core-utils
I don't know why do this if I put base directory for each module.
Thanks!!
sonar.modules stores identifiers of the children modules. You have to use child identifier as prefix of all child's sonar properties. Example:
# Main module
sonar.host.url=http://localhost:9000/sonar
sonar.java.source=1.6
sonar.java.target=1.6
sonar.modules=coreutils,pdfutils
sonar.projectBaseDir=/var/lib/jenkins/workspace/MyProject
sonar.projectKey=com.app.core:core-all
sonar.projectName=core-all
sonar.projectVersion=DEV-SNAPSHOT
sonar.sourceEncoding=UTF-8
# First module
pdfutils.sonar.projectBaseDir=/var/lib/jenkins/workspace/MyProject/pdf-utils
pdfutils.sonar.projectKey=com.app.core.pdf:pdf-utils
pdfutils.sonar.projectName=pdf-utils
pdfutils.sonar.sources=/var/lib/jenkins/workspace/MyProject/pdf-utils/src/main/java
pdfutils.sonar.working.directory=/var/lib/jenkins/workspace/MyProject/pdf-utils/target/sonar
pdfutils.sonar.java.binaries=/var/lib/jenkins/workspace/MyProject/pdf-utils/target/classes
# Second module
coreutils.sonar.projectBaseDir=/var/lib/jenkins/workspace/MyProject/core-utils
coreutils.sonar.projectKey=com.app.core:core-utils
coreutils.sonar.projectName=core-utils
coreutils.sonar.sources=/var/lib/jenkins/workspace/MyProject/core-utils/src/main/java
coreutils.sonar.working.directory=/var/lib/jenkins/workspace/MyProject/core-utils/target/sonar
coreutils.sonar.java.binaries=/var/lib/jenkins/workspace/MyProject/core-utils/target/classes
I also cleanup your properties:
sonar.moduleKey - removed from all modules
sonar.projectVersion - removed from children modules