I am new to gitlab CI and trying to build my maven project using gitlab.
I created my own docker image for builds and I am using my local runner.
My .gitlab-ci.yml looks as below: Edited:
image: registry.gitlab.com/girishenoy2003/docker-java-8-mvn-3.6.3:latest
services:
- docker:dind
stages:
- compile
- test
before_script:
- export MVN_USER_HOME=`pwd`/.m2
cache:
paths:
- $MVN_USER_HOME/.m2/repository/
- target/
maven-compile:
stage: compile
script:
- mvn compile
tags:
- my-local-runner
only:
- master
maven-test:
stage: test
script:
- mvn test
tags:
- my-local-runner
only:
- master
During the compile stage I expect all the dependencies to download and cache it and while running test I expect it to get it from cache as I am using my local runner for both jobs.
How can we avoid download of dependent jars from downloading in different jobs?
P.S.:
I already looked at this question - but didn't help
I know that if I use mvn test-compile it will do the trick but I wanted to make it in different jobs to segregate the stages
Edit:
Some runner logs:
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/aws-learning-path/spring-boot-rest/.git/
Checking out f86f9c63 as master...
Removing "..\\..\\..\\cache\\aws-learning-path\\spring-boot-rest\\default\\cache.zip"
Removing target/
Skipping Git submodules setup
Restoring cache
00:00
Checking cache for default...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
Executing "step_script" stage of the job script
$ export MVN_USER_HOME=/root
$ mvn test
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/2.2.2.RELEASE/spring-boot-starter-parent-2.2.2.RELEASE.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/2.2.2.RELEASE/spring-boot-starter-parent-2.2.2.RELEASE.pom (8.1 kB at 3.3 kB/s)
Your issue is related to your /root folder
The gitlab-runner do not cache any paths outside of project directory, you need to configure your CI to download and store the dependencies inside your project path
Related
I have a GitLab pipeline setup that has a package step to do a maven build during the tag event and a release to upload the jar to the GitLab generic package registry using curl and GitLab-release cli.
What I'm expecting to happen is a cache of the .m2 to be loaded into the package step to allow the mvn clean package to do its thing. Then archive the created jar and test results only.
The release step should begin clean with no git clone, no cache and only the jar and test results.
Instead the 'find .' shows the release step contains everything including
Git directory (.git)
Full checked out repository
.m2 cache
target (fully built as the Package step produced)
From the cache documentation (https://docs.gitlab.com/ee/ci/caching/) on GitLab it states
Archive: 'dependencies' keyword to control which job fetches the artifacts
Disable Cache uses the 'cache: []'
Why is GitLab putting so much content into the release job? The release job fails at times because its finding multiple Jar files from previous tags (IE the clean and the archiving are holding past version).
gitlab-ci.yml
variables:
MAVEN_CLI_OPTS: "-s $CI_PROJECT_DIR/.m2/settings.xml"
MAVEN_VERSION_PLUGIN_VERSION: 2.11.0
MAVEN_ARTIFACT_NAME: test-component
GIT_CLEAN_FLAGS: -ffd
PACKAGE_REGISTRY_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${MAVEN_ARTIFACT_NAME}"
cache:
key: primary
paths:
- .m2/repository
stages:
- package
- release
package:
stage: package
image: maven:latest
script:
- mvn ${MAVEN_CLI_OPTS} clean package
artifacts:
paths:
- target/*.jar
- target/surefire-reports
only:
- tags
- merge_requests
- branches
except:
- main
release:
stage: release
image: alpine:latest
cache: []
variables:
GIT_STRATEGY: none
dependencies:
- package
script:
- |
apk add curl gitlab-release-cli
find .
JAR_NAME=`basename target/${MAVEN_ARTIFACT_NAME}-${CI_COMMIT_TAG}.jar`
'curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file target/${JAR_NAME} ${PACKAGE_REGISTRY_URL}/${CI_COMMIT_TAG}/${JAR_NAME}'
release-cli create --name "Release $CI_COMMIT_TAG" --description "$TAG_MESSAGE" --tag-name ${CI_COMMIT_TAG} --assets-link "{\"name\":\"jar\",\"url\":\"${PACKAGE_REGISTRY_URL}/${CI_COMMIT_TAG}/${JAR_NAME}\"}"
only:
- tags
See the GitLab docs on GIT_STRATEGY:
A Git strategy of none also re-uses the local working copy, but skips all Git operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, if present. This strategy could mean you need to add fetch and checkout commands to your .gitlab-ci.yml script.
It can be used for jobs that operate exclusively on artifacts, like a deployment job. Git repository data may be present, but it’s likely out of date. You should only rely on files brought into the local working copy from cache or artifacts.
So GitLab documentation is pretty clear that you should always expect the git repository to be present. When you want to work exclusively with artifacts, I you can create a new temporary directory and reference the path to the artifacts explicitly rather than relying on a totally clean working directory.
We have a complex build system, with a many to many relationship between our libraries and our applications. We put each library and application in it's own repository, and use the output of the library builds in our application builds.
On our old Jenkins server, we simply set up a custom workspace and checked out the projects into standardized relative paths so they could find each other. Post build steps assured that only successful builds copied to the central bin folder at the expected relative path.
On our Bamboo server, our repository was fetched to a Checkout Directory at the expected relative path, and we could fetch artifacts from other builds and put them in the central bin folder at the expected relative path.
Now I'm trying to set up some Bitbucket Pipelines builds, and I can't see an obvious way to do a similar thing. The working folder is set automatically by pipelines, I can't push that repository into a subfolder that is relative to other build outputs. I can create artifacts, but I can't seem to import them into other pipelines. I can create caches, but again I can't seem to import them into other pipelines.
Library bitbucket-pipelines.yml
image: mcr.microsoft.com/dotnet/sdk:5.0
pipelines:
branches:
master:
eCRF2:
- step:
name: Build and Test
caches:
- dotnetcore
- platform2
script:
- dotnet restore ./NET5/Platform2.sln
- dotnet build ./NET5/Platform2.sln --no-restore --configuration Release
artifacts:
- NET5/Platform2/bin/**
definitions:
caches:
platform2: NET5/Platform2/bin
App bitbucket-pipelines.yml
image: mcr.microsoft.com/dotnet/sdk:5.0
pipelines:
default:
- step:
name: Build and Test
caches:
- dotnetcore
- platform2
script:
- export PROJECT_NAME=./PlatformDataService.sln
- dotnet restore ${PROJECT_NAME}
- dotnet build ${PROJECT_NAME} --no-restore --configuration Release
artifacts:
- PlatformDataService/bin/**
https://support.atlassian.com/bitbucket-cloud/docs/deploy-build-artifacts-to-bitbucket-downloads/ did get me to upload a file to the Downloads section of the repository, but how do I pull it into the other pipeline?
Is there a way to solve this within bitbucket pipelines itself or do I have to get a nuget server that's available outside my VPN?
I hope someone can help me with a simple setup of maven CI scripts for GitLab.
I tried to search stackoverflow and google, which results in several questions and answers, but either they seem to be completely different or not that I understand them.
I have a simple setup of two projects. project B depends on project A (= pom packaging).
I have in the runner configuration /etc/gitlab-runner/config.toml the line with the volumes added
[[runners]]
...
[runners.docker]
...
volumes = ["/cache", "/.m2"]
...
my .gitlab-ci.yml for both projects look like this
image: maven:3.6.1-jdk-12
cache:
paths:
- /.m2/repository
- target/
variables:
MAVEN_OPTS: "-Dmaven.repo.local=/.m2/repository"
maven_job:
script:
- mvn clean install
with this - the first project builds correctly and I can see that the caching is working, as it does not download all maven related plugins for building the project, when executed again and again.
It also states
[INFO] Installing /builds/end2end/projectA/pom.xml to /.m2/repository/de/end2end/projectA/0.4.4-SNAPSHOT/projectA-0.4.4-SNAPSHOT.pom
It reports though at the end
WARNING: /.m2/repository: not supported: outside build directory
WARNING: /.m2/repository/classworlds: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds/1.1-alpha-2: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds/1.1-alpha-2/_remote.repositories: not supported: outside build directory
[...]
When executing projectB, the job fails with the info, that it cannot find projectA.
So - what is wrong with the configuration of the runner / .gitlab-ci.yml files ?
I tried
cache:
paths:
- .m2/repository
which removes the warnings, but then the projectA gets in its local .m2 installed
[INFO] Installing /builds/end2end/projectA/pom.xml to /builds/end2end/projectAt/.m2/repository/de/end2end/projectA/0.4.4-SNAPSHOT/projectA-0.4.4-SNAPSHOT.pom
and projectB fails with the same error as above.
In fact, as described in gitlab doc, you use the dynamic storage so the volume is shared between subsequent runs of the same concurrent job for one project. I you want to share data between projects you must use the host-bound storage.
For the warning, the cache is only for working directory, so absolute path like /.m2/repository is not supported. In your case, you don't have to use cache for maven repository because you use a volume.
(Please note i'm a total beginner in all Docker and CI in general)
I am trying to set up a simple CI environment with GitLab. I'm using a local GitLab runner which is configured to build and test in Docker. In registering the runner I chose the 'shell' option. This installation succeeded.
After pushing my code I got an email saying 'build failed'. In the build log I found the following:
on desktop-docker-runner cDD_yf4V
Using Shell executor...
Running on b567d1ba4654...
DEPRECATION: this GitLab server doesn't support refspecs, gitlab-runner 12.0 will no longer work with this version of GitLab
Fetching changes...
Checking out fd20ca86 as dev...
Skipping object checkout, Git LFS is not installed.
Skipping Git submodules setup
Checking cache for default...
Runtime platform arch=amd64 os=linux pid=722 revision=1f513601 version=11.10.1
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ mvn $MAVEN_CLI_OPTS test-compile
bash: line 74: mvn: command not found
ERROR: Job failed: exit status 1
How can I add Maven to resolve this?
(EDIT: added gitlab-ci.yml file)
# Build JAVA applications using Apache Maven (http://maven.apache.org)
# For docker image tags see https://hub.docker.com/_/maven/
#
# For general lifecycle information see https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
#
# This template will build and test your projects as well as create the documentation.
#
# * Caches downloaded dependencies and plugins between invocation.
# * Verify but don't deploy merge requests.
# * Deploy built artifacts from master branch only.
# * Shows how to use multiple jobs in test stage for verifying functionality
# with multiple JDKs.
# * Uses site:stage to collect the documentation for multi-module projects.
# * Publishes the documentation for `master` branch.
variables:
# This will suppress any download for dependencies and plugins or upload messages which would clutter the console log.
# `showDateTime` will show the passed time in milliseconds. You need to specify `--batch-mode` to make this work.
MAVEN_OPTS: "-Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
# As of Maven 3.3.0 instead of this you may define these options in `.mvn/maven.config` so the same config is used
# when running from the command line.
# `installAtEnd` and `deployAtEnd` are only effective with recent version of the corresponding plugins.
MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"
# Cache downloaded dependencies and plugins between builds.
# To keep cache across branches add 'key: "$CI_JOB_NAME"'
cache:
paths:
- .m2/repository
# This will only validate and compile stuff and run e.g. maven-enforcer-plugin.
# Because some enforcer rules might check dependency convergence and class duplications
# we use `test-compile` here instead of `validate`, so the correct classpath is picked up.
.validate: &validate
stage: build
script:
- 'mvn $MAVEN_CLI_OPTS test-compile'
# For merge requests do not `deploy` but only run `verify`.
# See https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
.verify: &verify
stage: test
script:
- 'mvn $MAVEN_CLI_OPTS verify site site:stage'
except:
- master
# Validate merge requests using JDK7
validate:jdk7:
<<: *validate
image: maven:3.3.9-jdk-7
# Validate merge requests using JDK8
validate:jdk8:
<<: *validate
image: maven:3.3.9-jdk-8
# Verify merge requests using JDK7
verify:jdk7:
<<: *verify
image: maven:3.3.9-jdk-7
# Verify merge requests using JDK8
verify:jdk8:
<<: *verify
image: maven:3.3.9-jdk-8
# For `master` branch run `mvn deploy` automatically.
# Here you need to decide whether you want to use JDK7 or 8.
# To get this working you need to define a volume while configuring your gitlab-ci-multi-runner.
# Mount your `settings.xml` as `/root/.m2/settings.xml` which holds your secrets.
# See https://maven.apache.org/settings.html
deploy:jdk8:
# Use stage test here, so the pages job may later pickup the created site.
stage: test
script:
- 'mvn $MAVEN_CLI_OPTS deploy site site:stage'
only:
- master
# Archive up the built documentation site.
artifacts:
paths:
- target/staging
image: maven:3.3.9-jdk-8
pages:
image: busybox:latest
stage: deploy
script:
# Because Maven appends the artifactId automatically to the staging path if you did define a parent pom,
# you might need to use `mv target/staging/YOUR_ARTIFACT_ID public` instead.
- mv target/staging public
dependencies:
- deploy:jdk8
artifacts:
paths:
- public
only:
- master
I'm using a local GitLab runner which is configured to build and test in Docker. In registering the runner I chose the 'shell' option
From the sounds of it, you have registered the gitlab-runner incorrectly for the mode you are after. You said you want to use a runner to build and test in Docker, however you have registered the runner in shell mode.
To use the runner in Docker, follow the instructions set in the documentation. Make sure to set the runner executor to Docker when registering.
You'd also ideally remove the shell runner you have created.
If you have issues registering the runner, see this answer, which may also help with setting up your environment.
I have the following build configuration for a multi-project gradle file:
stages:
- test
before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle
cache:
paths:
- .gradle/wrapper
- .gradle/caches
- build
test :
dependencies: []
image: openjdk:x
stage: test
script:
- ./gradlew test --debug
On the GitLab, between builds with no changes to source files, I get:
Up-to-date check for task ':x:compileJava' took 1.117 secs. It is not up-to-date because:
No history is available.
I'm not sure why it says this, as I would expected the task history to be restored from cache. I see this in the logs between runs:
Creating cache default...
.gradle/wrapper: found 207 matching files
.gradle/caches: found 5058 matching files
build: found 2743 matching files
When I re-run on my local machine, I can see the tests are not being re-run:
> Skipping task ':x:compileJava' as it is up-to-date (took 0.008 secs).
More confusing is dependencies are cached perfectly, it just keeps rerunning tests when I have made no code changes.
As far as I know the history that gradle is missing is also stored in the .gradle folder, but not in the caches or wrapper subfolder. If you tell Gitlab to cache the complete .gradle folder the problem should go away.
See also this example:
https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Gradle.gitlab-ci.yml