Can I have gradle cache dependencies w/out building a specific project - gradle

I regularly reformat my entire machine, wiping Gradle's dependency cache in the process. As part of my re-setup script I want Gradle to repopulate the dependency cache. Is there a way to tell Gradle to download and cache a dependency without having to build the project which depends on that dependency?
I suppose I could just have a dummy project that lists a bunch of dependencies and then build that project once in my script. But that seems a bit hacky.

Gradle does not have a built-in task to cache a dependency, which is in fact a set of files: metadata, JAR and potentially more. In addition, for resolution, the origin repository matters.
An alternative option could be to backup and restore the Gradle dependency cache.

Related

When working on a project and one of it's dependency in parallel how to load it from the file system

I'm working on a java dependency that I publish on GitLab via gradle. In parallel I also work on some projects dependent of it.
When I need to do a change in the parent I have to wait for my CI/CD to be over before I can keep developing the childs. There is most certainly a way to tell gradle 'check there before online' but all I've found until now is to do that with local jar but not raw source files.
I tried most of the things in How to make Gradle repository point to local directory but without success as gradle is excepting a maven repo structure with some pom.xml files.
So how could I achieve something like this ?
After rethinking the problem and studying a bit more gradle/maven I found the solution.
Just execute the task gradle publishToMavenLocal in the parent project. Then in the dependent project add mavenLocal() to your list of repo. That's all you have to do.
I'm still looking for a way to make gradle build and publish the parent automatically on the child's build. But it's already much more practical like this?

Does "build with local dependencies" exist in Maven without multi-module?

I have a set of applications, all use Maven and the local repository. The applications form a dependency tree using <dependency> in their pom.xml. All of these projects have -SNAPSHOT in their version.
Is it possible for Maven (or some compatible dependency manager) to build an application together with all of its local dependencies whose source changed?
I do not want to create a multi-module project, because:
the projects are exactly libraries, not modules;
I do not want an additional complexity just to have a form of build which is already precisely defined;
I want the process to be dynamic: if a library is mature enough to be put into a remote repository, it would be no more rebuilt with the main project and that's ok.
For now, there is a lot of refactoring, moving code from one library to another etc. and it happens often that substantial parts of the dependency tree need to be rebuilt. I thus need to manually write mvn install in several projects in order to assure that there is no stale code.
No, it doesn't work. Even with a multi-module project, maven does not detect which modules have changed sources in it and which do not.
There was a (flaky) implementation in Maven 2, but it was not continued in 3.x, see How to get maven 3.0 to only build modules with local scm changes
I hoped they would include it again in maven 4, but I didn't see it yet: https://maarten.mulders.it/2020/11/whats-new-in-maven-4/
I once did a similar setup, but had to use shell scripts with some git magic to get it working.
You can also decide to put your libraries in separate repo's from the start, and use the repo tool that google uses for android development: https://github.com/GerritCodeReview/git-repo/blob/main/README.md
Once you run mvn install on the particular Maven project, it will be accessible for all other Maven projects, which are on the same workstation, during dependency collection (before the compile phase).
Official Maven Build Lifecycle description:
install - install the package into the local repository, for use as a dependency in other projects locally
It's not necessary to keep libraries as part of the same project(or have it as a multi-module project). But once you want to share those libraries with your teammates, you would need either to force them installing libraries locally (as you did), or store those libraries at some external repo, like Artifactory or Nexus

Same Gradle files, same code base, different dependency graphs for different developers

This problem seems like a real mystery to me. We are working on a Spring REST app. On my team, we have Jenkins running a build on all checkins (CI build). In addition, there is an official build done elsehwere nightly. Some integration tests recently started failing during the CI build, and we traced the cause to jar conflicts. However, we had successfully excluded the transitive dependencies that caused those conflicts earlier, and for the official build, and for some developers on the team, that is still true. For others it no longer is.
We have the same code base, and the same Gradle files. When I run the dependency task on my system, I have a long tree of transitive dependencies in one subproject (as does the Jenkins machine), whereas others have no such tree. In addition, the size of the WAR is several orders larger for some developers than for others. The transitive dependencies come from another related project uploaded to Nexus.
I have refreshed dependencies both on my machine and the Jenkins machine, and even wiped out .gradle, but to no avail.
It sounds like something environmental, but the Gradle versions and Java versions are all roughly the same, a couple minor versions off at most.
Any ideas about what could cause this discrepancy?
You could force your dependencies to specific versions across the board
def gsonModule = 'com.google.code.gson:gson'
configurations.all {
resolutionStrategy {
// add dependency substitution rules
dependencySubstitution {
substitute module(gsonModule) with module("$gsonModule:2.6.1")
}
}
}
// note: configurations block must be above the dependencies
// or build will error
dependencies {
compile "$gsonModule:2.7"
}
Then when any developer/CI/test/CD runs the build your resolutionStrategy forces consistency across all compilations. Default behavior in gradle is upon version conflict the newer version will be used. This will force the all versions to the same regardless of their declared version.
$ ./gradlew dependencies --configuration compile -q
------------------------------------------------------------
Root project
------------------------------------------------------------
compile - Dependencies for source set 'main'.
\--- com.google.code.gson:gson:2.7 -> 2.6.1
Reference: https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.ResolutionStrategy.html
You should use gradle wrapper to ensure the same gradle version throughout, in order to exclude some possibility of that being a factor.
Also, add --refresh-dependencies to the gradle command-line. It may be that some of your dependencies are not being versioned correctly, and this flag will verify each time that only the latest are pulled.
Finally, take a look at workstations that have different results, and ensure that you have identical gradle.properties in your gradle user home folder and the project folder.
First, thanks for the quick responses, but I have an update:
In short, one of my colleagues identified one of the problems. I can't get into details, but the gradle files have convenience functions for either downloading another project from Nexus (such as the related project that I referred to above) or using a locally-built artifact if it exists.
Doing the latter resolves the issue. Downloading from Nexus (as Jenkins did, and as you should be able to do) allows the issue to occur, exposing it. I don't believe this effects the build order, but it does effect the transitive dependencies that are pulled in.
There are also some problems in the dependencies of the various sub-projects: e.g., one project intended not to have dependencies has one, and that one is a big culprit in bringing transitive dependencies. So when that base project is pulled in by other projects, it has ripple effects. And some of those dependency blocks need to be cleaned up.
Again, thanks for the responses!

Gradle dependency caching mechanism

Under my user-home/.gradle/caches I am seeing multiple artifacts directories e.g. artifacts-14,artifacts-24, modules-2.
all these folders are storing duplicate artifacts. so my questions is, why and in what conditions gradle has to create multiple artifacts folders? can gradle also be configured to lookup and store artifacts in on directory. doing so I can save disk space from storing duplicate artifacts.
highly sophisticated dependency caching, that does not work (3.4). Build file has a new dependency version yet Gradle still fetches artifact from some weird intermediary dependency cache instead of taking the new version from local maven repo
Gradle contains a highly sophisticated dependency caching mechanism.
You can check a first folder .gradle located under project directory.
Gradle recreates every time the tasks are run.
Also there is a folder .gradle under home directory.
Gradle creates it and uses it to store the depdendencies to reduce the download-time.
The Gradle dependency cache consists of 2 key types of storage:
A file-based store of downloaded artifacts, including binaries like jars as well as raw downloaded meta-data like POM files and Ivy files. The storage path for a downloaded artifact includes the SHA1 checksum, meaning that 2 artifacts with the same name but different content can easily be cached.
A binary store of resolved module meta-data, including the results of resolving dynamic versions, module descriptors, and artifacts.
Separating the storage of downloaded artifacts from the cache metadata permits us to do some very powerful things with our cache that would be difficult with a transparent, file-only cache layout.
You can read more info here. Check the The dependency cache chapter.

Automating Leiningen local dependency management

I am using a local maven repository to house some code I am using to develop a project. I have cited this repository in my project.clj file, and am now able to rely on local jars in this way (how to do this in a previous question of mine).
Since I am actively developing these projects, I have my project.clj file looking for the LATEST version. But, in order to update a dependency, I still have to increment the version number of that dependency and then run lein install to build it to the maven repository.
Does leiningen have a way to do this where this is automatically done for me when I build the project that depends on things from the maven repo? Can lein just look for those things and rebuild them as needed?
If you want to develop two projects in parallel, with one depending on the other, you can use symlinks in a checkouts directory to avoid having to install snapshots all the time.
To quote from the Leiningen README:
Q: I want to hack two projects in parallel, but it's annoying to switch between them.
A: Use a feature called checkout dependencies. If you create a directory called checkouts in your project root and symlink some other project roots into it, Leiningen will allow you to hack on them in parallel. That means changes in the dependency will be visible in the main project without having to go through the whole install/switch-projects/deps/restart-repl cycle. Note that this is not a replacement for listing the project in :dependencies; it simply supplements that for tighter change cycles.
Are your dependencies version snapshots? Maven should update all *-SNAPSHOT dependencies on build automatically.

Resources