Gradle task output not reused in gitlab for kotlin build - gradle

I have created simple kotlin project with gradle build tool and pushed it to gitlab. To speed up pipeline I want to reuse tasks output. So I configured gitlab to cache build folder. Here is gitlab-ci.yml:
stages:
- build
build:
stage: build
image: gradle:7.5.1-jdk17
variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
cache:
- key: global-cache # gradle thirdparty artifacts cache
paths:
- /home/gradle/.gradle
- key: $CI_COMMIT_REF_NAME # gradle previous build task output cache
paths:
- build
script:
- ls build/libs
- gradle -version
- gradle build --console=plain
artifacts:
paths:
- build/libs/*.jar
reports:
junit:
- build/test-results/test/TEST-*.xml
expire_in: 1 week
when: always
In pipeline I have executed build job twice(second job run). Since build folder cached my expectation was all tasks will be UP-TO-DATE for second job. Unfortunately, both jobs have same task state:
> Task :compileKotlin
> Task :compileJava NO-SOURCE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE
> Task :jar
> Task :inspectClassesForKotlinIC
> Task :assemble
> Task :compileTestKotlin
> Task :compileTestJava NO-SOURCE
> Task :processTestResources NO-SOURCE
> Task :testClasses UP-TO-DATE
> Task :test
> Task :check
> Task :build
To debug "jar" task I printed build/lib/ files to console:
$ ls build/libs
gradle-reuse-build-example.jar
You can notice build/libs have cached file inside. Do you have any ideas why jar task isn't UP-TO-DATE? How should I configurate gitlab/gradle to reuse previous job run task output?

Your build fetches the git repo again when running each time(because gitlab uses different shared runner for job execution). That means that the timestamps of the kotlin source files are later than the classes compiled in earlier builds. As a result, the compileKotlin task is not UP-TO-DATE, causing the jar task to also be out of date.
Documentation here: https://docs.gradle.org/current/userguide/more_about_tasks.html#sec:how_does_it_work
As to how to configure the build to do what you want, you may need to look into how to reuse an existing git clone instead of cloning fresh each time. I don't even know if that's possible.

Related

Ignore file when calculating gradle cache fingerprint

In my project I access the build-info.properties generated by the gradle springboot plugin buildInfo() task during runtime to include my project version in log metadata.
My problem is that this file is included in the fingerprint calculation for gradle tasks such as tests via the classpath fingerprint, but the version in that file changes with every build in my pipelines. Therefore I can never reuse that cache.
I saw this question on how to exclude that file from runtime, but if I follow that advise,
the file's not available during runtime anymore, naturally.
the file /BOOT-INF/classes/META-INF/build-info.properties is empty. Therefore my application fails on startup.
Can I somehow exclude it from the cache fingerprint calculation only?
Edits:
I'm currently on Gradle 6.8.1 and Spring Bot 2.2.2.
This is how I generate the file:
springBoot {
buildInfo()
}
And this is how I add the normalization:
normalization {
runtimeClasspath {
ignore "**/build-info.properties"
}
}
Update: As stated in the comment, this problem appeared due to a missconfiguration of my Gradle build scripts in another location. The normalization approach linked in the question and explained in the accepted answer is the solution to the initial question.
Gradle input normalization should be a solution for it.
normalization {
runtimeClasspath {
ignore '**/build-info.properties'
}
}
Not sure why you are saying " if I follow that advise, the file's not available during runtime anymore". According to documentation
The effect of this configuration would be that changes to build-info.properties would be ignored for up-to-date checks and build cache key calculations. Note that this will not change the runtime behavior of the test task — i.e. any test is still able to load build-info.properties and the runtime classpath is still the same as before.
Here are some tests that proves the above
Running build first time
./gradlew build -Pversion=0.0.1 --console=plain
> Task :bootBuildInfo
> Task :compileJava UP-TO-DATE
> Task :processResources UP-TO-DATE
> Task :classes
> Task :resolveMainClassName
> Task :bootJar
> Task :jar
> Task :assemble
> Task :compileTestJava UP-TO-DATE
> Task :processTestResources UP-TO-DATE
> Task :testClasses UP-TO-DATE
> Task :test
> Task :check
> Task :build
test task was executed because there is no build cache.
Running build second time with different version
./gradlew build -Pversion=0.0.2 --console=plain
> Task :bootBuildInfo
> Task :compileJava UP-TO-DATE
> Task :processResources UP-TO-DATE
> Task :classes
> Task :resolveMainClassName UP-TO-DATE
> Task :bootJar
> Task :jar
> Task :assemble
> Task :compileTestJava UP-TO-DATE
> Task :processTestResources UP-TO-DATE
> Task :testClasses UP-TO-DATE
> Task :test UP-TO-DATE
> Task :check UP-TO-DATE
> Task :build
As you may see only build tasks were executed but test task is still UP-TO-DATE.
build-info.properties is still available under build/resources/main/META-INF/
build.artifact=demo
build.group=com.example
build.name=demo
build.time=2023-02-01T18\:32\:03.871040Z
build.version=0.0.2
and could be accessed using Spring Boot actuator endpoint /actuator/info in case it's enabled
{
"build": {
"artifact": "demo",
"name": "demo",
"version": "0.0.2",
"group": "com.example"
}
}
Consider excluding build time
You could optimize even more by excluding time from the build info.
springBoot {
buildInfo {
excludes = ['time']
}
}
Usually it's a good idea for optimizing local builds. Otherwise build tasks will be always executed. By excluding time all tasks will be cached
./gradlew build --console=plain
> Task :bootBuildInfo UP-TO-DATE
> Task :compileJava UP-TO-DATE
> Task :processResources UP-TO-DATE
> Task :classes UP-TO-DATE
> Task :resolveMainClassName UP-TO-DATE
> Task :bootJar UP-TO-DATE
> Task :jar UP-TO-DATE
> Task :assemble UP-TO-DATE
> Task :compileTestJava UP-TO-DATE
> Task :processTestResources UP-TO-DATE
> Task :testClasses UP-TO-DATE
> Task :test UP-TO-DATE
> Task :check UP-TO-DATE
> Task :build UP-TO-DATE
but build.time will not be part of the build-info.properties
build.artifact=demo
build.group=com.example
build.name=demo
build.version=0.0.1-SNAPSHOT

Gitlab CICD: Tests are not being run as part of the build process, during build stage for Gradle Project resulting in 0% coverage in SonarQube

I have created a simple project to run a gitlab pipeline, including build, test, coverage, and exporting results in SonarQube. It fails to run the test tasks and therefore no coverage reports are generated.
I have included a build.gradle and a sonar-project.properties file in my gitlab repository as well as the src/main/HelloWorld.java and src/test/HelloWorldTest.java folder. No test-results folder is being generated when running gradle build or gradle test command.
The gitlab-ci.yml file used is the following:
image gradle:latest
stages:
- build
- test
- coverage
- scan
before_script:
- echo $CI_PROJECT_DIR
- echo $CI_BUILDS_DIR
- echo $CI_COMMIT_REF_NAME
build:
stage: build
script:
- gradle clean build
after_script:
- ls
- cd build
artifacts:
paths:
- build/libs
test:
stage: test
before_script:
- echo $CI_PROJECT_DIR
- echo $CI_BUILDS_DIR
- echo $CI_COMMIT_REF_NAME
script:
- gradle test --debug
after_script:
- ls
artifacts:
paths:
- build/test-results
report:
stage: coverage
before_script:
- echo $CI_PROJECT_DIR
- echo $CI_BUILDS_DIR
- echo $CI_COMMIT_REF_NAME
script:
- gradle jacocoTestReport
after_script:
- ls
artifacts:
paths:
- build/reports/jacoco/test/html
- build/reports/jacoco/test/xml
sonarqube-check:
image:
name: sonarsource/sonar-scanner-cli:latest
entrypoint: [""]
stage: scan
variables:
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"
GIT_DEPTH: "0"
cache:
key: "${CI_JOB_NAME}"
paths:
- .sonar/cache
before_script:
- echo $CI_PROJECT_DIR
- echo $CI_BUILDS_DIR
- echo $CI_COMMIT_REF_NAME
- echo $CI_JOB_NAME
script:
- sonar-scanner
allow_failure: true
only:
- main
The result following gradle build is:
$ ls
build
build.gradle
README.md
sonar-project.properties
src
No test results folder is created.
The associated build.gradle file is:
plugins {
id 'java'
id 'org.sonarqube' version '2.7'
id 'jacoco'
}
repositories {
mavenCentral()
}
dependencies {
testImplementation 'org.junit.jupiter:junit-jupiter:5.7.1'
testImplementation 'org.jacoco:org.jacoco.core:0.8.6'
}
test {
useJUnitPlatform()
reports {
junitXml.outputLocation.set(layout.buildDirectory.dir("$buildDir/test-
results/test-junit-xml"))
}
jacoco {
destinationFile = file("$buildDir/jacoco/jacocoTest.exec")
}
}
sonarqube {
properties {
property "sonar.projectName", "imagehelloworld"
property "sonar.projectKey", "imagehelloworld"
property "sonar.sources", "src/main"
property "sonar.tests", "src/test"
property "sonar.junit.reportsPath", "$buildDir/test-results/test"
property "sonar.jacoco.reportPaths",
"$buildDir/jacoco/jacocoTest.exec"
}
}
The output of the build job is as follows:
gradle clean build
Welcome to Gradle 7.6!
Here are the highlights of this release:
- Added support for Java 19.
- Introduced `--rerun` flag for individual task rerun.
- Improved dependency block for test suites to be strongly typed.
- Added a pluggable system for Java toolchains provisioning.
For more details see https://docs.gradle.org/7.6/release-notes.html
Starting a Gradle Daemon (subsequent builds will be faster)
> Task :clean UP-TO-DATE
> Task :compileJava NO-SOURCE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE
> Task :jar
> Task :assemble
> Task :compileTestJava NO-SOURCE
> Task :processTestResources NO-SOURCE
> Task :testClasses UP-TO-DATE
> Task :test NO-SOURCE
> Task :check UP-TO-DATE
> Task :build
BUILD SUCCESSFUL in 17s

Gradle features were used in this build, making it incompatible with Gradle 7.0

This is my spring boot project. When I going to run gradle task in spring tool suit I can get error like below. I am new to sprint boot can anyone help me. I used java 11 and gradle version 6.8.3
Gradle Distribution: Specific Gradle version 6.8.3
Gradle Version: 6.8.3
Java Home: /usr/lib/jvm/java-11-openjdk-amd64
JVM Arguments: None
Program Arguments: None
Build Scans Enabled: false
Offline Mode Enabled: false
Gradle Tasks: clean build
> Task :clean
> Task :compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :processResources
> Task :classes
> Task :bootJarMainClassName
> Task :bootJar
> Task :jar SKIPPED
> Task :assemble
> Task :compileTestJava NO-SOURCE
> Task :processTestResources NO-SOURCE
> Task :testClasses UP-TO-DATE
> Task :test NO-SOURCE
> Task :check UP-TO-DATE
> Task :build
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
You can use ./gradlew build --warning-mode all to find out deprecated features.
For example
This problem can be for several reasons, To avoid going into so much explanation If the application worked at some point
Delete de app in the device and run yarn android or npm android
or
Delete de build folder and delete the app in the device and run yarn android or npm android
That's works for me

gradle submodules of submodules not built with :submodule:build syntax

I have the following project structure correctly found by gradle:
$ ./gradlew projects
Root project 'test-project'
\--- Project ':sub-1'
\--- Project ':sub-1:sub-2'
Which makes me believe that my setups is correct.
Now, I have found that the following gradle syntax:
$ ./gradlew clean :sub-1:build
is not equivalent to:
$ cd sub-1
$ ../gradlew clean build
$ cd ..
The above is equivalence is stated in many places on the gradle website.
Like HERE
Running the former, the result is incorrect:
> Task :sub-1:test
test.project.LibrarySuite > someLibraryMethod is always true PASSED
Running the latter, the result is correct:
> Task :sub-1:test
test.project.LibrarySuite > someLibraryMethod is always true PASSED
> Task :sub-1:sub-2:test
test.project.LibrarySuite > someLibraryMethod is always true PASSED
Please help me understand if I might have assumed or done anything wrong, or if it's a bug that should be raised to the gradle team.
You will find detailed description of how Gradle handles execution of tasks in a multiproject build setup here : https://docs.gradle.org/current/userguide/intro_multi_project_builds.html#sec:executing_a_multiproject_build , specially:
From a user’s perspective, multi-project builds are still collections
of tasks you can run. The difference is that you may want to control
which project’s tasks get executed. You have two options here:
Change to the directory corresponding to the subproject you’re
interested in and just execute gradle as normal.
Use a qualified task name from any directory, although this is usually
done from the root. For example: gradle :services:webservice:build
will build the webservice subproject and any subprojects it depends
on.
The first approach is similar to the single-project use case, but
Gradle works slightly differently in the case of a multi-project
build. The command gradle test will execute the test task in any
subprojects, relative to the current working directory, that have that
task. So if you run the command from the root project directory,
you’ll run test in api, shared, services:shared and
services:webservice. If you run the command from the services project
directory, you’ll only execute the task in services:shared and
services:webservice.
This explains how Gradle behaves in the two examples you gave in your question :
$ ./gradlew clean :sub-1:build
from the root project directory: you execute task clean, which will be executed for current project and each subprojects below, then :sub-1:build (with qualified task name) which executes build tasks ONLY for subproject sub1
Gradle execution log:
> Task :clean UP-TO-DATE
> Task :sub-1:clean
> Task :sub-1:sub-2:clean UP-TO-DATE
> Task :sub-1:compileJava NO-SOURCE
> Task :sub-1:processResources NO-SOURCE
> Task :sub-1:classes UP-TO-DATE
> Task :sub-1:jar
> Task :sub-1:assemble
> Task :sub-1:compileTestJava NO-SOURCE
> Task :sub-1:processTestResources NO-SOURCE
> Task :sub-1:testClasses UP-TO-DATE
> Task :sub-1:test NO-SOURCE
> Task :sub-1:check UP-TO-DATE
> Task :sub-1:build
EDIT to answer #Guido's comment: this will also build any other projects sub-1 depends on, so ./gradlew clean :sub-1:build will also trigger build of sub-2 if sub-1 project dependsOn sub-2:
sub-1 build.gradle
dependencies {
implementation project(":sub-1:sub-2")
}
$ cd sub-1 && ../gradlew clean build
From sub-1 subproject directory , you trigger tasks clean then build, without using qualified names, so these two tasks will both be executed on current project and sub-projects` :
Gralde output:
$ ../gradlew clean build --console=plain
> Task :sub-1:clean
> Task :sub-1:sub-2:clean
> Task :sub-1:compileJava NO-SOURCE
> Task :sub-1:processResources NO-SOURCE
> Task :sub-1:classes UP-TO-DATE
> Task :sub-1:jar
> Task :sub-1:assemble
> Task :sub-1:compileTestJava NO-SOURCE
> Task :sub-1:processTestResources NO-SOURCE
> Task :sub-1:testClasses UP-TO-DATE
> Task :sub-1:test NO-SOURCE
> Task :sub-1:check UP-TO-DATE
> Task :sub-1:build
> Task :sub-1:sub-2:compileJava NO-SOURCE
> Task :sub-1:sub-2:processResources NO-SOURCE
> Task :sub-1:sub-2:classes UP-TO-DATE
> Task :sub-1:sub-2:jar
> Task :sub-1:sub-2:assemble
> Task :sub-1:sub-2:compileTestJava NO-SOURCE
> Task :sub-1:sub-2:processTestResources NO-SOURCE
> Task :sub-1:sub-2:testClasses UP-TO-DATE
> Task :sub-1:sub-2:test NO-SOURCE
> Task :sub-1:sub-2:check UP-TO-DATE
> Task :sub-1:sub-2:build

Gradle : Copy different properties file depending on the environment and create jar

I am evaluating gradle for my spring boot project. Everything seems to work but here is where I am stuck. I have 2 properties file. One for prod i.e.:
application_prod.properties
and another for qa i.e.:
application_qa.properties
My requirement is such that while I build (create jar file) the project from gradle, I've to rename the properties file to
application.properties
and then build the jar file. As far as I know, gradle has a default build task. So here I've to override it such that it considers only the required properties file and rename it and then build depending on the environment.
How can I achieve this?
What you need to do is to override processResources configuration:
processResources {
def profile = (project.hasProperty('profile') ? project.profile : 'qa').toLowerCase()
include "**/application_${profile}.properties"
rename {
'application.properties'
}
}
With the following piece of code changed you will get the output below:
$ ./gradlew run -Pprofile=PROD
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:run
LOL
Profile: PROD
BUILD SUCCESSFUL
Total time: 3.63 secs
$ ./gradlew run -Pprofile=QA
:compileJava UP-TO-DATE
:processResources
:classes
:run
LOL
Profile: QA
BUILD SUCCESSFUL
Total time: 3.686 secs
$ ./gradlew run
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:run
LOL
Profile: QA
BUILD SUCCESSFUL
Total time: 3.701 secs
Demo is here.

Resources