I hope someone can help me with a simple setup of maven CI scripts for GitLab.
I tried to search stackoverflow and google, which results in several questions and answers, but either they seem to be completely different or not that I understand them.
I have a simple setup of two projects. project B depends on project A (= pom packaging).
I have in the runner configuration /etc/gitlab-runner/config.toml the line with the volumes added
[[runners]]
...
[runners.docker]
...
volumes = ["/cache", "/.m2"]
...
my .gitlab-ci.yml for both projects look like this
image: maven:3.6.1-jdk-12
cache:
paths:
- /.m2/repository
- target/
variables:
MAVEN_OPTS: "-Dmaven.repo.local=/.m2/repository"
maven_job:
script:
- mvn clean install
with this - the first project builds correctly and I can see that the caching is working, as it does not download all maven related plugins for building the project, when executed again and again.
It also states
[INFO] Installing /builds/end2end/projectA/pom.xml to /.m2/repository/de/end2end/projectA/0.4.4-SNAPSHOT/projectA-0.4.4-SNAPSHOT.pom
It reports though at the end
WARNING: /.m2/repository: not supported: outside build directory
WARNING: /.m2/repository/classworlds: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds/1.1-alpha-2: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds/1.1-alpha-2/_remote.repositories: not supported: outside build directory
[...]
When executing projectB, the job fails with the info, that it cannot find projectA.
So - what is wrong with the configuration of the runner / .gitlab-ci.yml files ?
I tried
cache:
paths:
- .m2/repository
which removes the warnings, but then the projectA gets in its local .m2 installed
[INFO] Installing /builds/end2end/projectA/pom.xml to /builds/end2end/projectAt/.m2/repository/de/end2end/projectA/0.4.4-SNAPSHOT/projectA-0.4.4-SNAPSHOT.pom
and projectB fails with the same error as above.
In fact, as described in gitlab doc, you use the dynamic storage so the volume is shared between subsequent runs of the same concurrent job for one project. I you want to share data between projects you must use the host-bound storage.
For the warning, the cache is only for working directory, so absolute path like /.m2/repository is not supported. In your case, you don't have to use cache for maven repository because you use a volume.
Related
I have configured and working following setup
gitlab-ci, which uses docker-machine runner and uploads cache to S3
maven build with configured caching
caching correctly loads and uploads on each job
But the problem is, that every time I run mvn install, something in the local maven repository changes (I assume it updates pom metadata) and gitlab runner keeps uploading new versions of the cache, on every single build.
It is still faster and more reliable to use this "busted" cache, than to download the deps from internet every time, but the upload can take a long time and I would like to shave off this extra time.
How can I modify my build to force maven, to generate cacheable local repository?
Simplified version of my .gitlab-ci.yml:
variables:
# we have a custom java+maven image, that uses this ENV variable,
# to auto-configure path where to put the local maven repository
MAVEN_LOCAL_REPOSITORY: $CI_PROJECT_DIR/.cache/maven
job-build:
stage: build
image: internal-gitlab/java/maven:3.6-jdk8-alpine
script:
- mvn -B clean package
cache:
key: backend-dependencies
paths:
- .cache/
You have a constant as a cache key. Maybe a more fine grained cache would help.
See the link here
In short - prepare your own maven image with required dependencies and use it instead of internal-gitlab/java/maven:3.6-jdk8-alpine.
Some details:
First of all, you need to create a maven docker image where all (or most of) required for your project dependencies are presented. Publish it to your registry (gitlab has one) and use it instead of internal-gitlab/java/maven:3.6-jdk8-alpine.
To create such an image I usually create an additional job in CI triggered manually. You need to trigger it at initial stage and when project dependencies are heavily modified.
Working sample can be found here:
https://gitlab.com/alexej.vlasov/syncer/blob/master/.gitlab-ci.yml
- this project is using the prepared image and also it has a job to prepare this image.
https://gitlab.com/alexej.vlasov/maven/blob/master/Dockerfile
- dockerfile to run maven and download dependencies once.
The pros:
don't need to download dependencies each time - they are inside a
docker image (and docker layers are cached on the runners)
don't need to upload artifacts when job is finished
I'm configuring my sonarqube to run on my project, following the instructions given to run from the cloud I have my file like this:
sonar.projectKey=yisera_aaswtest
sonar.projectName=aaswtest
sonar.projectVersion=1.0
sonar.sourceEncoding=UTF-8
sonar.language=js
sonar.profile=node
sonar.exclusions=node_modules, migrations, models_old, seeders, .gitignore
I downloaded sonarqube CLI, copied the bin file to %path% on windows but for some reason, after running the test, I get the following error:
Property "sonar.cfamily.build-wrapper-output" was not specified
As you can see, my project is in javascript(nodejs specifically) and not C++, and even then, for some reason it's scanning my node_modules folder while I explicitly state in my properties file to exclude scanning there.
Any ideas what I might be doing wrong? Here is the command I run:
sonar-scanner.bat -Dsonar.projectKey=yisera_aaswtest -Dsonar.organization=yisera-bitbucket -Dsonar.sources=. -Dsonar.host.url=https://sonarcloud.io -Dsonar.login=cc10c0f538a6b50c876c2c53ed479eadd3e60988
I'm using GitLab CI to build a C# solution and try to pass some build artifacts from one build stage to another.
The problem is, that the artifacts are not located in a single directory but in different subdirectories, which however all have the same names bin/ or obj/.
My .gitlab-ci.yml looks like the following:
...
stages:
- build
- test
build:
stage: build
script:
CALL %MSBuild% ...
artifacts:
paths:
- /**/bin/
- /**/obj/
expire_in: 6 hrs
test:
stage: test
dependencies:
- build
...
I tried to capture the artifacts using different ways, e.g.
**/bin/
**/obj/
(invalid syntax), or
.*/bin/
.*/obj/
but that one did not find any artifacts, just as /**/bin/ and /**/obj/, giving me following errors:
Uploading artifacts...
WARNING: /**/bin/: no matching files
WARNING: /**/obj/: no matching files
How can I specify a subdirectory pattern to be scanned for artifacts? Or is this even possible at all?
Simply using
artifacts:
untracked: true
is not an option, because of a huge untracked packages/ subdirectory, which causes artifacts upload to fail because of a too large archive:
Uploading artifacts...
untracked: found 4513 files
ERROR: Uploading artifacts to coordinator... too large archive id=36 responseStatus=413 Request Entity Too Large token=...
FATAL: Too large
The gitlab-ci-multi-runner build runner is built using Go and currently uses filepath.Glob() to scan for any specified artifacts in file_archiver.go.
Go doesn't seem to support the double star glob expression as discussed in another question here at SO. So there seem to be no way to use a full-featured **/bin expression at the moment.
Because however all my projects are located at the same level below the solution root, it is still possible to use something like
artifacts:
paths:
- "*/bin"
- "*/obj"
Note that the quotes (") seem to be required, as well as no trailing path separator at the end.
It should also be possible to explicitly add more levels by adding more globbing expressions (as described here):
paths:
...
- "*/obj"
- "*/*/bin"
- "*/*/obj"
...
GitLab is tracking this issue here, and will possibly be fixed in a future version.
UPD: due to mentioned issue is closed and due to documentation, since GitLab runner 13.0 it supports doublestar.Glob.
So it is now possible to write something like this:
paths:
- "**/bin"
- "**/obj"
And it will include all nested directories with names bin and obj in artifacts
I am trying to use caching in gitlab runner, which builds a Maven Java project. Currently Gitlab runner only allow caching specific paths defined in gitlab yaml file in the cache: clause. When maven builds projects, it generate everything inside target/ folder, which are untracked files in git. So I can simply use untracked: true option to cache everything under target/ folder. The purpose of caching is to skip compiling the files, which have already been compiled by maven under the target/ folder.
However this cache amounts to about 6GB, which is completely unreasonable for its size and time required to create and restore such a giant cache. It caches all jar and war artifacts built during compiling multi-module maven project. However, maven only needs .class files to check changes for re-compilation
So if their was some way using which I can cache only *.class files, and make them available in subsequent builds, then maven could check the .class files and skip re-compiling unchanged files and cache size would also be pretty small. Currently gitlab-runner only allow specifying absolute paths for caching. It does not support regex patterns for paths such as \.class$ (which would have been very useful).
Is there any way I could cache only specific file types using gitlab runner yaml settings?
So based on cascaval's comment, I was able to figure out a solution.
At the end of maven build I ran a command to clean all build artifacts created by maven, which are not used for checking stale status of .java resources. Here is what I wrote -
cd ./projects/directory
find . | grep --perl-regexp --regexp='\/target\/(?!classes|maven)' | xargs rm --recursive --force
This saves all .class files in target/classes folder, including folder structure and also files in maven-status folder, which are probably used by maven to check file status for recompilation.
Gradle created a ?/.gradle/ in the directory that gradle was run in. We would expect the cache directory to be created at ~/.gradle.
Example:
/project # Project root and cwd when running gradle command
/.gradle # Expected - project-specific gradle folder
/? # Directory literally named with a question mark
/.gradle # Unexpected - Global gradle folder with wrappers and cached artifacts
The user running the scripts did not have a home directory, giving the user a home directory or specifying a gradle-user-home solved the issue:
gradle --gradle-user-home=/foo/bar ...
or
GRADLE_USER_HOME=/foo/bar gradle ...
There are two different folders gradle stores information. ~/.gradle is used to store downloaded artifacts, gradle wrappers, etc. Basically everything that can be shared between multiple builds. The .gradle folder in your project is used to store project specific information used for example by the gradle up-to-date check mechanism.
let's find it out why it behaves like this.
As gradle use following code to get user home:
System.getProperty("user.home");
Follow the link for openjdk 8 source code.
It comes to conclusion: When JVM can not found user name in os, it will use ? as a return. So gradle will create ?/.gradle for usage.