I have a GitLab pipeline setup that has a package step to do a maven build during the tag event and a release to upload the jar to the GitLab generic package registry using curl and GitLab-release cli.
What I'm expecting to happen is a cache of the .m2 to be loaded into the package step to allow the mvn clean package to do its thing. Then archive the created jar and test results only.
The release step should begin clean with no git clone, no cache and only the jar and test results.
Instead the 'find .' shows the release step contains everything including
Git directory (.git)
Full checked out repository
.m2 cache
target (fully built as the Package step produced)
From the cache documentation (https://docs.gitlab.com/ee/ci/caching/) on GitLab it states
Archive: 'dependencies' keyword to control which job fetches the artifacts
Disable Cache uses the 'cache: []'
Why is GitLab putting so much content into the release job? The release job fails at times because its finding multiple Jar files from previous tags (IE the clean and the archiving are holding past version).
gitlab-ci.yml
variables:
MAVEN_CLI_OPTS: "-s $CI_PROJECT_DIR/.m2/settings.xml"
MAVEN_VERSION_PLUGIN_VERSION: 2.11.0
MAVEN_ARTIFACT_NAME: test-component
GIT_CLEAN_FLAGS: -ffd
PACKAGE_REGISTRY_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${MAVEN_ARTIFACT_NAME}"
cache:
key: primary
paths:
- .m2/repository
stages:
- package
- release
package:
stage: package
image: maven:latest
script:
- mvn ${MAVEN_CLI_OPTS} clean package
artifacts:
paths:
- target/*.jar
- target/surefire-reports
only:
- tags
- merge_requests
- branches
except:
- main
release:
stage: release
image: alpine:latest
cache: []
variables:
GIT_STRATEGY: none
dependencies:
- package
script:
- |
apk add curl gitlab-release-cli
find .
JAR_NAME=`basename target/${MAVEN_ARTIFACT_NAME}-${CI_COMMIT_TAG}.jar`
'curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file target/${JAR_NAME} ${PACKAGE_REGISTRY_URL}/${CI_COMMIT_TAG}/${JAR_NAME}'
release-cli create --name "Release $CI_COMMIT_TAG" --description "$TAG_MESSAGE" --tag-name ${CI_COMMIT_TAG} --assets-link "{\"name\":\"jar\",\"url\":\"${PACKAGE_REGISTRY_URL}/${CI_COMMIT_TAG}/${JAR_NAME}\"}"
only:
- tags
See the GitLab docs on GIT_STRATEGY:
A Git strategy of none also re-uses the local working copy, but skips all Git operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, if present. This strategy could mean you need to add fetch and checkout commands to your .gitlab-ci.yml script.
It can be used for jobs that operate exclusively on artifacts, like a deployment job. Git repository data may be present, but it’s likely out of date. You should only rely on files brought into the local working copy from cache or artifacts.
So GitLab documentation is pretty clear that you should always expect the git repository to be present. When you want to work exclusively with artifacts, I you can create a new temporary directory and reference the path to the artifacts explicitly rather than relying on a totally clean working directory.
I hope someone can help me with a simple setup of maven CI scripts for GitLab.
I tried to search stackoverflow and google, which results in several questions and answers, but either they seem to be completely different or not that I understand them.
I have a simple setup of two projects. project B depends on project A (= pom packaging).
I have in the runner configuration /etc/gitlab-runner/config.toml the line with the volumes added
[[runners]]
...
[runners.docker]
...
volumes = ["/cache", "/.m2"]
...
my .gitlab-ci.yml for both projects look like this
image: maven:3.6.1-jdk-12
cache:
paths:
- /.m2/repository
- target/
variables:
MAVEN_OPTS: "-Dmaven.repo.local=/.m2/repository"
maven_job:
script:
- mvn clean install
with this - the first project builds correctly and I can see that the caching is working, as it does not download all maven related plugins for building the project, when executed again and again.
It also states
[INFO] Installing /builds/end2end/projectA/pom.xml to /.m2/repository/de/end2end/projectA/0.4.4-SNAPSHOT/projectA-0.4.4-SNAPSHOT.pom
It reports though at the end
WARNING: /.m2/repository: not supported: outside build directory
WARNING: /.m2/repository/classworlds: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds/1.1-alpha-2: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds/1.1-alpha-2/_remote.repositories: not supported: outside build directory
[...]
When executing projectB, the job fails with the info, that it cannot find projectA.
So - what is wrong with the configuration of the runner / .gitlab-ci.yml files ?
I tried
cache:
paths:
- .m2/repository
which removes the warnings, but then the projectA gets in its local .m2 installed
[INFO] Installing /builds/end2end/projectA/pom.xml to /builds/end2end/projectAt/.m2/repository/de/end2end/projectA/0.4.4-SNAPSHOT/projectA-0.4.4-SNAPSHOT.pom
and projectB fails with the same error as above.
In fact, as described in gitlab doc, you use the dynamic storage so the volume is shared between subsequent runs of the same concurrent job for one project. I you want to share data between projects you must use the host-bound storage.
For the warning, the cache is only for working directory, so absolute path like /.m2/repository is not supported. In your case, you don't have to use cache for maven repository because you use a volume.
Between 1 and 2 minutes of my AWS CodeBuilds are spent downloading dependencies from Maven Central.
Short of building a pre-provisioned Docker container, is there any way to cache these between builds?
CodeBuild now provides a cache feature you can use to pre-load your dependencies.
Unsigned's answer is good but is a tad outdated. As of February 2019, CodeBuild allows both caching in an S3 bucket and allows the user to cache locally. You can now specify cache at 3 different layers of a build:
Docker Layer Caching
Git Layer Cahing (cache the last build and then only build from git diff)
Custom caching - specified within the cache: portion of your buildspec.yml file. Personally, I cache my node_modules/ here and then cache at the Git Layer.
Source: https://aws.amazon.com/blogs/devops/improve-build-performance-and-save-time-using-local-caching-in-aws-codebuild/
I am trying to use caching in gitlab runner, which builds a Maven Java project. Currently Gitlab runner only allow caching specific paths defined in gitlab yaml file in the cache: clause. When maven builds projects, it generate everything inside target/ folder, which are untracked files in git. So I can simply use untracked: true option to cache everything under target/ folder. The purpose of caching is to skip compiling the files, which have already been compiled by maven under the target/ folder.
However this cache amounts to about 6GB, which is completely unreasonable for its size and time required to create and restore such a giant cache. It caches all jar and war artifacts built during compiling multi-module maven project. However, maven only needs .class files to check changes for re-compilation
So if their was some way using which I can cache only *.class files, and make them available in subsequent builds, then maven could check the .class files and skip re-compiling unchanged files and cache size would also be pretty small. Currently gitlab-runner only allow specifying absolute paths for caching. It does not support regex patterns for paths such as \.class$ (which would have been very useful).
Is there any way I could cache only specific file types using gitlab runner yaml settings?
So based on cascaval's comment, I was able to figure out a solution.
At the end of maven build I ran a command to clean all build artifacts created by maven, which are not used for checking stale status of .java resources. Here is what I wrote -
cd ./projects/directory
find . | grep --perl-regexp --regexp='\/target\/(?!classes|maven)' | xargs rm --recursive --force
This saves all .class files in target/classes folder, including folder structure and also files in maven-status folder, which are probably used by maven to check file status for recompilation.
How can I tell gradle to redownload dependencies from repositories?
Generally, you can refresh dependencies in your cache with the command line option --refresh-dependencies. You can also delete the cached files under ~/.gradle/caches. With the next build Gradle would attempt to download them again.
What is your specific use case? Do you use dynamic dependency versions or SNAPSHOT versions?
On Unix systems, you can delete all the existing artifacts (artifacts and metadata) Gradle has downloaded using:
rm -rf $HOME/.gradle/caches/
Note that --refresh-dependencies won't always re-download every artifact; it will use existing copies if they match what exists in the repository. From the Gradle User Guide, refreshing dependencies:
The --refresh-dependencies option tells Gradle to ignore all cached entries for resolved modules and artifacts. A fresh resolve will be performed against all configured repositories, with dynamic versions recalculated, modules refreshed, and artifacts downloaded. However, where possible Gradle will check if the previously downloaded artifacts are valid before downloading again. This is done by comparing published SHA1 values in the repository with the SHA1 values for existing downloaded artifacts.
[...]
It’s a common misconception to think that using --refresh-dependencies will force download of dependencies. This is not the case: Gradle will only perform what is strictly required to refresh the dynamic dependencies. This may involve downloading new listing or metadata files, or even artifacts, but if nothing changed, the impact is minimal.
If you are using a recent version of Gradle, you can use --refresh-dependencies option.
./gradlew build --refresh-dependencies
you can refer to the Gradle manual.
The --refresh-dependencies option tells Gradle to ignore all cached entries for resolved modules and artifacts. A fresh resolve will be performed against all configured repositories, with dynamic versions recalculated, modules refreshed, and artifacts downloaded.
You can tell Gradle to re-download some dependencies in the build script by flagging the dependency as 'changing'. Gradle will then check for updates every 24 hours, but this can be configured using the resolutionStrategy DSL. I find it useful to use this for for SNAPSHOT or NIGHTLY builds.
configurations.all {
// Check for updates every build
resolutionStrategy.cacheChangingModulesFor 0, 'seconds'
}
Expanded:
dependencies {
implementation group: "group", name: "projectA", version: "1.1-SNAPSHOT", changing: true
}
Condensed:
implementation('group:projectA:1.1-SNAPSHOT') { changing = true }
I found this solution at this forum thread.
For MAC
./gradlew build --refresh-dependencies
For Windows
gradlew build --refresh-dependencies
Can also try gradlew assembleDevelopmentDebug --refresh-dependencies
For Windows...in order to make gradle re-download specific dependencies:
delete the dependencies you want to re-download from the directory below:
C:\Users\%USERNAME%\.gradle\caches\modules-2\files-2.1
delete all metadata directories at the path:
C:\Users\%USERNAME%\.gradle\caches\modules-2\metadata-*
run gradle build (or gradlew build if using gradle wrapper) in the project's root directory.
note: the numbers in the file paths above might be different for you.
None of the solutions above worked for me.
If you use IntelliJ, what resolved it for me was simply refreshing all Gradle projects:
One can remove folder with cached jars.
In my case, on Mac the library was cached at path:
/Users/MY_NAME/.gradle/caches/modules-2/files-2.1/cached-library-to-remove
I removed the cached library folder ("cached-library-to-remove" in above example), deleted the build folder of my project and compiled again.
Fresh library was downloaded then.
To refresh cached 'release' version the only option is to clear local cache.
rm -rf $HOME/.gradle/caches/
To refresh cached 'snapshot' version you can:
./gradlew build --refresh-dependencies
For those who are wondering where to run gradle commands:
Open Android Studio
Click on Terminal(You will find it in the base of Android Studio)
The command tool will open
Type your command gradlew build --refresh-dependencies
Instead of removing your entire gradle cache, like some answers here are suggesting, you can delete the cache for a specific group or artifact id. I added the following function to my .bash_profile:
deleteGradleCache() {
local id=$1
if [ -z "$id" ]; then
echo "Please provide an group or artifact id to delete"
return 1
fi
find ~/.gradle/caches/ -type d -name "$id" -prune -exec rm -rf "{}" \; -print
}
Usage:
$ deleteGradleCache com.android.support
Then, on the next build or if you resync, gradle will re-download dependencies.
There is 2 ways to do that:
Using command line option to refresh dependenices cashe.
You can delete local cache where artefasts are caches by Gradle and trigger build
Using --refresh-dependencies option:
./gradlew build --refresh-dependencies
Short explanation --refresh-dependencies option tells Gradle to ignore all cached entries for resolved modules and artifacts.
Long explanantion
WIth –refresh-dependencies’ Gradle will always hit the remote server to check for updated artifacts: however, Gradle will avoid downloading a file where the same file already exists in the cache.
First Gradle will make a HEAD request and check if the server reports the file as unchanged since last time (if the ‘content-length’ and ‘last-modified’ are unchanged). In this case you’ll get the message: "Cached resource is up-to-date (lastModified: {})."
Next Gradle will determine the remote checksum if possible (either from the HEAD request or by downloading a ‘.sha1’ file)..
If this checksum matches another file already downloaded (from any repository), then Gradle will simply copy the file in the cache,
rather than re-downloading. In this case you’ll get the message: "“Found locally available resource with matching checksum: [{}, {}]”.
Using delete:
When you delete caches
rm -rf $HOME/.gradle/caches/
You just clean all the cached jars and sha1 sums and Gradle is in situation where there is no artifacts on your machine and has to download everything. Yes it will work 100% for the first time, but when another SNAPSHOT is released and it is part of your dependency tree you will be faced again in front of the choice to refresh or to purge the caches.
For Android Studio 3.4.1
Simply open the gradle tab (can be located on the right) and right-click on the parent in the list (should be called "Android"), then select "Refresh dependencies".
This should resolve your issue.
Seems change is changed to isChange for gradle version 6.3, kotlin version 1.3.70, Groovy 2.5.10
The working configuration is
implementation("com.sample:commons:1.0.0-SNAPSHOT") {
isChanging = true
}
Also, run this command to fetch the latest
./gradlew assemble --refresh-dependencies
This worked for me.
Make sure Gradle is not set to offline by unchecking button at File>Settings>Gradle>Offline Work.
Add this to the top level of your build.gradle, nice to have above dependencies
configurations.all {
resolutionStrategy.cacheChangingModulesFor 0, 'seconds'
}
I made sure my dependencies are written like this:
implementation('com.github.juanmendez:ThatDependency:ThatBranch-SNAPSHOT') {
changing = true
}
Thereafter, I open the Gradle panel in Android Studio and click the blue circle arrows button. I can always see my updates getting a new fresh copy.
If you are using Intellij, you can right click the root project and then select refresh gradle dependencies.
Mb I'm too late however my solution is for single repository. I think deleting ~/.gradle/* is overkill.
The problmem I've bumped into was that I was deleting directory where sources were and gradle was getting another version not from nexus.
To avoid that I run the next:
~/.gradle$ find . -type d -name 'group.plugins.awssdk'
./caches/modules-2/files-2.1/group.plugins.awssdk
./caches/modules-2/metadata-2.23/descriptors/group.plugins.awssdk
~/.gradle$ rm -r ./caches/modules-2/files-2.1/group.plugins.awssdk ./caches/modules-2/metadata-2.23/descriptors/group.plugins.awssdk
After that gradle is dragging files from nexus.
In my case none of the above worked, what I did was:
In build.gradle, commenting the dependencies related to the unresolved imports I had
Clicking "Sync Now"
Uncommenting what I just commented
Clicking "Sync Now" again
Then my imports were properly resolved again.
Deleting all the caches makes download all the dependacies again. so it take so long time and it is boring thing wait again again to re download all the dependancies.
How ever i could be able to resolve this below way.
Just delete groups which need to be refreshed.
Ex : if we want to refresh com.user.test group
rm -fr ~/.gradle/caches/modules-2/files-2.1/com.user.test/
then remove dependency from build.gradle and re add it.
then it will refresh dependencies what we want.
I think gradle 2.14.1 fixes the issue. The accepted answer is correct, but there is a bug in gradle with –refresh-dependencies. 2.14.1 fixes that.
See https://discuss.gradle.org/t/refresh-dependencies-should-use-cachechangingmodulesfor-0s/556
For the majority of cases, just simply re-building the project should do the trick. Sometimes you have to run ./gradlew build --refresh-dependencies as several answers have already mentioned (takes a long time, depending on how much dependencies you have). How ever, sometimes none of those will work: the dependency just won't get updated. Then, you can do this:
Remove dependency from your gradle file
Run / debug your project and wait for it to fail (with NonExistingClass reason)
Hit "build project" and wait for it to finish successfully
Run / debug once again
This is ridiculous and seems like madness, but I actually do use this procedure daily, simply because the dependency I need can be updated dozens of times and none of adequate solutions would have any effect.
If you are using eclipse and if you want force eclipse to re load dependencies you could try below command
gradlew clean cleaneclipse build eclipse --refresh-dependencies
Only a manual deletion of the specific dependency in the cache folder works... an artifactory built by a colleague in enterprise repo.
You can do it like this
https://marschall.github.io/2017/04/17/disabling-gradle-cache.html
To quote from Disabling the Gradle Build Cache
The Gradle build cache may be a great thing when you’re regularly building >large projects with Gradle. However when only occasionally building open source >projects it can quickly become large.
To disable the Gradle build cache add the following line to ~/.gradle/gradle.properties
org.gradle.caching=false
You can clean the existing cache with
rm -rf $HOME/.gradle/caches/
rm -rf $HOME/.gradle/wrapper/
delete this directory:
C:\Users\[username]\.gradle
You need to redownload it, so you can either manually download and replace the corrupted file and again sync your project . Go to this location
C:\users[username].gradle\wrapper\dist\gradle3.3-all\55gk2rcmfc6p2dg9u9ohc3hw9\gradle-3.3-all.zip
Here delete gradle3.3allzip and replace it by downloading again from this site
https://services.gradle.org/distributions/
Find the same file and download and paste it to that location
Then sync your project.
Hope it works for you too.