Under my user-home/.gradle/caches I am seeing multiple artifacts directories e.g. artifacts-14,artifacts-24, modules-2.
all these folders are storing duplicate artifacts. so my questions is, why and in what conditions gradle has to create multiple artifacts folders? can gradle also be configured to lookup and store artifacts in on directory. doing so I can save disk space from storing duplicate artifacts.
highly sophisticated dependency caching, that does not work (3.4). Build file has a new dependency version yet Gradle still fetches artifact from some weird intermediary dependency cache instead of taking the new version from local maven repo
Gradle contains a highly sophisticated dependency caching mechanism.
You can check a first folder .gradle located under project directory.
Gradle recreates every time the tasks are run.
Also there is a folder .gradle under home directory.
Gradle creates it and uses it to store the depdendencies to reduce the download-time.
The Gradle dependency cache consists of 2 key types of storage:
A file-based store of downloaded artifacts, including binaries like jars as well as raw downloaded meta-data like POM files and Ivy files. The storage path for a downloaded artifact includes the SHA1 checksum, meaning that 2 artifacts with the same name but different content can easily be cached.
A binary store of resolved module meta-data, including the results of resolving dynamic versions, module descriptors, and artifacts.
Separating the storage of downloaded artifacts from the cache metadata permits us to do some very powerful things with our cache that would be difficult with a transparent, file-only cache layout.
You can read more info here. Check the The dependency cache chapter.
Related
I have maven-base ear file.
The ear contains lot of jars the built by same CI-CD process - monolith.
I have implementation of incremental build, that improve the CI-CD times by build the artifacts that changed or affected only. and take the other artifacts from artifactory.
The problem is - when I start new version, I have to build all the artifacts in order to create the artifacts in artifactory the first time. This takes long of time.
Is there a quick way to copy all the artifacts from version 0.1 to version 0.2 in artifactory?
In the beginning of version 0.2, The artifacts are the same.
Thanks
TL;DR
It's not that straight forward.
Personally I think it's not the right way to go. You should probably break your internal dependencies based on their lifecycles, and reuse existing versions instead of rebuilding them.
In more details:
Technically, you can use the JFrog CLI's copy command to copy files in Artifactory. You can also use the Copy Item API directly if you prefer. Copy is a cheap and quick operation in Artifactory because of its checksum based storage (the content is stored only once).
But, that's not enough, mainly because of Maven.
Maven uses a file path format which contains the version - both as a folder and in the file names. The Maven artifacts file path format is (simplified): /<group>/<artifact>/<version>/<artifact>-<version>.<ext>. For example, the jar file of the artifact org.acme:foo:0.1 will have a file like: /org/acme/foo/0.1/foo-0.1.jar.
It means that copying files from folder of 0.1 to 0.2 is not enough, you also need to rename them accordingly.
And that's still not enough - Maven also embeds the the version in files, mainly in the pom file. It means that copying it is also not enough, you need to modify the content of that xml file.
But, it also depends on how you use Artifactory. The best practice is to publish also a build-info when you upload to Artifactory. That usually means that Artifactory stores properties on the files, properties which are specific to the build as part of which they were created and uploaded. These properties are also copied as part of the copy operation mentioned above, and you should remove them from the new version.
To do all of that you can probably write a simple script which lists the files, copy and rename the files, adjust the content of the pom file, and remove the properties.
I regularly reformat my entire machine, wiping Gradle's dependency cache in the process. As part of my re-setup script I want Gradle to repopulate the dependency cache. Is there a way to tell Gradle to download and cache a dependency without having to build the project which depends on that dependency?
I suppose I could just have a dummy project that lists a bunch of dependencies and then build that project once in my script. But that seems a bit hacky.
Gradle does not have a built-in task to cache a dependency, which is in fact a set of files: metadata, JAR and potentially more. In addition, for resolution, the origin repository matters.
An alternative option could be to backup and restore the Gradle dependency cache.
That directory has some directories, such as jars-3, modules-2, transforms-1, etc.
In the modules-2, there exist files-2.1, metadata-2.53/metadata-2.59, etc
In the files-2.1, there exists the various downloaded packages.
In the metadata-2.53, there is the direcriptors directory, where various packages description info (descriptor.bin) resides.
Where can I find some user manual about this? What's the rule where one package is put? What if I want to copy one package from one computer to another? What's the content of descriptor.bin?
I have read the gradle user manual and gradle user guide, there is nearly nothing about this.
The gradle user guide you linked is quite outdated. As of gradle 6.1 there is a relocatable dependency cache and as of gradle 6.2 there is support for a shareable read-only dependency cache.
From the gradle 6.2.2 user guide:
Copying and reusing the cache
The dependency cache, both the file and
metadata parts, are fully encoded using relative paths. This means
that it is perfectly possible to copy a cache around and see Gradle
benefit from it.
The path that can be copied is $GRADLE_HOME/caches/modules-.
The only constraint is placing it using the same structure at the
destination, where the value of GRADLE_HOME can be different.
I have a large maven project with several dependencies. There is also an assembly that contains all files I want to deploy - which is not necessarily identical to the dependencies.
My question: How can I generate a textfile that contains all URLs to JAR-files maven usually downloads the artifacts from? Ideally only from those files in the assembly, but of all dependencies would do as well.
[Update]
My project consists of a root project which has several artifacts from subprojects and external artifacts as dependencies. All dependencies are either stored in my private Artifactory repository or can at least accessed via it using Artifactory as a proxy.
When I roll out a new release, I want to be able that the root project writes all it dependencies in a file, suitable that a client can download those dependencies. The client has no idea about repository structures (I'd like to avoid that), so I need full URLs or at least full pathes within an repository root.
I'd like to make sure I'm not including duplicate libraries in my Maven project. I'm pulling in quite a few dependencies from a few different repositories including main, Spring's repo, and JBoss' repository. How can I see a graph of the dependencies so I can make sure I'm not depending on multiple versions of a single library?
It would be kind of a bummer if I was including something like org.springframework:spring-core:3.0.1.RELEASE and org.springframework:org.springframework.spring-core:3.0.5.RELEASE which actually do show up as different dependencies, as weird as it sounds, because Spring's repository names things weirdly. What can I do to make sure I don't have duplicate dependencies?
The dependency:tree maven plugin should do what you want and show the tree.
Alternatively force Maven2 to copy dependencies into target/lib to copy the the artifacts in the target/lib directory which should also show duplicates but you would still need to grep all the POMs in the ~/.m2/repository to see which library is pulling in which version manually and that would be time consuming for large local repositories.
Duplicate libraries can be a problem, but duplicate classes can exist even if no libraries are duplicated. JBoss Tattletale can analyze a set of jar files and produce a report which lists all duplicated classes. There’s a Maven plugin for running the report - see http://docs.jboss.org/tattletale/userguide/1.2/en-US/html/maven.html
If you'd like to get a graphical, searchable representation of the dependency tree (including all modules from your project, transitive dependencies and eviction information), check out UpdateImpact: https://app.updateimpact.com (free service).
Using the search, you can find out how many times a given library is imported, by what modules and in which versions, as well as verify if older versions of a library are evicted by newer ones.
Disclaimer: I'm one of the developers of the site