How to manually download file from Maven repository in Gradle - gradle

We have a huge monolith application which is build by multiple tools (shell scripts, Ant and Maven). The build process is quite complex:
a lot of manually steps
hidden dependencies between Ant targets
different steps must be executed depending on the used Operating System
We decided to simplify it by creating Gradle scripts which wraps all this logic (it is quite impossible to fix it, so we create a wrapper which standardize the way of executing all the logic). We have to download some files from the Maven repository, but we cannot use the dependencies syntax:
we don't need to always download all files
the versions of the downloaded artifacts are dynamic (depends on configuration located in completely different place)
we need a path to the downloaded files (e.g. we have to unpack an artifact distributed as zip)
How we can achieve it?

The easiest way to achieve it is to create a dynamic configuration with dependencies, and next resolve it. The resolve method returns paths to the dependencies on the local disk. It is important to use a unique name for every configuration. If not, executing the logic twice would fail (cannot overwrite the configuration with XYZ name).
Here is an example method which returns a path to an artifact. If the artifact is already available in the Gradle cache it won't be downloaded for the second time, but of course the path will be returned. In this example all artifacts are downloaded from Maven Central.
Method:
ext.resolveArtifact = { CharSequence identifier ->
def configurationName = "resolveArtifact-${UUID.randomUUID()}"
return rootProject.with {
configurations.create(configurationName)
dependencies.add(configurationName, identifier)
return configurations.getByName(configurationName, {
repositories {
mavenCentral()
}
}).resolve()[0]
}
}
Usage:
def jaCoCoZip = resolveArtifact('org.jacoco:jacoco:0.8.6')
def jaCoCoAgent = resolveArtifact('org.jacoco:org.jacoco.agent:0.8.6')

Related

Gradle resolves "." in dependecies as "/"?

I have following part of the build.gradle:
def external = file('../external').absolutePath
repositories {
maven {
url file(external).toURI()
}
}
dependencies {
api 'ch.qos.cal10n:cal10n-api:0.7.7'
}
Third-party library that my project uses is stored in the ../external/ch.qos.cal10n/cal10n-api/0.0.7/(.jar and .pom).
However when running gradle build I got following:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
> Could not find ch.1qos.cal10n:cal10n-api:0.7.7.
Searched in the following locations:
- file:/home/a/b/external/ch/1qos/cal10n/cal10n-api/0.7.7/cal10n-api-0.7.7.pom
Why /home/a/b/external/ch/1qos/cal10n/cal10n-api/0.7.7/cal10n-api-0.7.7.pom
and not /home/a/b/external/ch.1qos.cal10n/cal10n-api/0.7.7/cal10n-api-0.7.7.pom ?
UPD: .pom and .jar files are downloaded manually from Maven Repository and placed into /home/a/b/external/ch.1qos.cal10n/cal10n-api/0.7.7/`
Because that is the layout of local-maven repositories, i.e., the group will be turned to directories. This is in part for performance and usability reasons. Consider the alternative, you'd have one huge directory containing all the groupIds. Working with directories with lots of entries slows down processing a lot. Instead, this layout is hierarchical, reducing the number of files/sub-directories per directory.
So to solve your problem, just change your directory layout accordingly.
/home/a/b/external/ch.1qos.cal10n/cal10n-api/0.7.7/cal10n-api-0.7.7.pom
->
/home/a/b/external/ch/1qos/cal10n/cal10n-api/0.7.7/cal10n-api-0.7.7.pom

Git repository with Maven project as a Gradle source dependency

This article describes an interesting feature of Gradle 4.10+ called a source dependency:
https://blog.gradle.org/introducing-source-dependencies
It allows to use a Git (for example a GitHub) source code repository to build a dependency from it. However it seems like it supports only Gradle projects as source dependencies. Is it possible to use a Maven project as well and if it's possible, please show an example.
When I try to use this feature with Maven project Gradle tries to find the build.gradle file there anyway (I see it when run Gradle with the --info option) and fails with an error message like:
Git repository at https://github.com/something/something.git did not contain a project publishing the specified dependency.
The short answer
... is: "no".
Under the hood, source dependencies are composite builds. These needs to be Gradle projects as the external projects are sort of merged with the main project.
The long answer
... is: "yes but it is hard".
It is actually mentioned in the same blog post you linked to (emphasis mine):
Source dependencies make these use cases simpler to implement. Gradle takes care of automatically checking out the correct versions of dependencies, making sure the binaries are built when required. It does this everywhere that the build is run. The checked out project doesn’t even need to have an existing Gradle build. This example shows a Gradle build consuming two source dependencies that have no build system by injecting a Gradle build via plugins. The injected configuration could do anything a regular Gradle plugin can do, such as wrapping an existing CMake or Maven build.
Because it sounded like it wasn't the biggest thing in the world to create bridge between a Maven and a Gradle project in source dependencies, I gave it a shot. And I have it working except for transitive dependencies. You will basically need to do what is shown in the examples linked to above, but instead of building native libraries, you make a call-out to Maven (e.g. using a Maven plugin for Gradle).
However, the scripts I ended up with are complex enough that I would suggest you instead build the Maven project yourself, deploy it to a local Maven repository and then add that repository to the Gradle project.
<edit>
The loooooooong answer
Alright, so here is how to actually do it. The feature is poorly documented, and appears to be mostly targeted towards native projects (like C++ or Swift).
Root project setup
Take a normal Gradle project with the Java plugin applied. I did a "gradle init" in an empty folder. Assume that in this project, you are depending on a library called `` that you later want to include as a source dependency:
// [root]/build.gradle
dependencies {
implementation 'org.example:my-maven-project:1.1'
}
Note that the version number defined here must match a Git tag in the repository. This is the code revision that will be checkout out.
Then in the settings file, we define a source dependency mapping for it:
// [root]/settings.gradle
rootProject.name = 'my-project'
includeBuild('plugins') // [1]
sourceControl {
gitRepository("https://github.com/jitpack/maven-simple") { // [2]
producesModule("org.example:my-maven-project") // [3]
plugins {
id "external-maven-build" // [4]
}
}
}
[1]: This includes a Gradle project called plugins that will be explained later.
[2]: This is just an arbitrary Maven project that I found, which was relatively simple. Substitute with the actual repository you have.
[3]: This is the name of the Maven module (the same as in the dependency block) that we are defining a source build for
[4]: This defines a custom settings plugin called external-maven-build that is defined in the plugins project, which will be explained later.
Plugins project structure
Inside the root project, we define a new Gradle project. Again, you can use gradle init to initialize it as a Groovy (or whatever you like) project. Delete all generated sources and tests.
// [root]/plugins/settings.gradle
// Empty, but used to mark this as a stand-alone project (and not part of a multi-build)
// [root]/plugins/build.gradle
plugins {
id 'groovy'
id 'java-gradle-plugin' // [1]
}
repositories {
gradlePluginPortal() // [2]
}
dependencies {
implementation "gradle.plugin.com.github.dkorotych.gradle.maven.exec:gradle-maven-exec-plugin:2.2.1" // [3]
}
gradlePlugin {
plugins {
"external-maven-build" { // [4]
id = "external-maven-build"
implementationClass = "org.example.ExternalMavenBuilder"
}
}
}
[1]: In this project, we are defining a new Gradle plugin. This is a standard way to do that.
[2]: To invoke Maven, I am using another 3rd party plugin, so we need to add the Gradle plugin portal as a repository.
[3]: This is the plugin used to invoke Maven. I am not too familiar with it, and I don't know how production ready it is. One thing I noticed is that it does not model inputs and outputs, so there are no built-in support for up-to-date checking. But this can be added retrospectively.
[4]: This defines the custom plugin. Notice that it has the same ID as used in the settings file in the root project.
Plugin implementation class
Now comes the fun stuff. I chose to do it in Groovy, but it can be done in any supported JVM languages of cause.
The plugin structure is just like any other Gradle plugin. One thing to note is that it is a Settings plugin, whereas you normally do Project plugins. This is needed as it we are basically defining a Gradle project at run-time, which needs to be done as part of the initialization phase.
// [root]/plugins/src/main/groovy/org/example/ExternalMavenBuilder.groovy
package org.example
import com.github.dkorotych.gradle.maven.exec.MavenExec
import org.gradle.api.Plugin
import org.gradle.api.artifacts.ConfigurablePublishArtifact
import org.gradle.api.initialization.Settings
class ExternalMavenBuilder implements Plugin<Settings> {
void apply(Settings settings) {
settings.with {
rootProject.name = 'my-maven-project' // [1]
gradle.rootProject {
group = "org.example" //[2]
pluginManager.apply("base") // [3]
pluginManager.apply("com.github.dkorotych.gradle-maven-exec") // [4]
def mavenBuild = tasks.register("mavenBuild", MavenExec) {
goals('clean', 'package') // [5]
}
artifacts.add("default", file("$projectDir/target/maven-simple-0.2-SNAPSHOT.jar")) { ConfigurablePublishArtifact a ->
a.builtBy(mavenBuild) // [6]
}
}
}
}
}
[1]: Must match the Maven module name
[2]: Must match the Maven module group
[3]: Defines tasks like "build" and "clean"
[4]: The 3rd party plugin that makes it more easy to invoke Maven
[5]: For options, see https://github.com/dkorotych/gradle-maven-exec-plugin
[6]: Adds the Maven output as an artifact in the "default" configuration
Be aware that it does not model transitive dependencies, and it is never up-to-date due to missing inputs and outputs.
This is as far as I got with a few hours of playing around with it. I think it can be generalized into a generic plugin published to the Gradle portal. But I think I have too much on my plate as it is already. If anyone would like to continue on from here, you have my blessing :)

Missing checksum files when using Gradle maven-publish and signing plugins

I have a Java project that makes use of Gradle to build and package. My purpose is to create artifacts that are published to Maven Central.
As a first step, I configured my Gradle project as shown in the following example from the documentation:
https://docs.gradle.org/current/userguide/publishing_maven.html#publishing_maven:complete_example
When I run gradle publishToMavenLocal, I get the following files installed in my local repository:
maven-metadata-local.xml
my-library-1.0.2-SNAPSHOT.jar
my-library-1.0.2-SNAPSHOT.jar.asc
my-library-1.0.2-SNAPSHOT-javadoc.jar
my-library-1.0.2-SNAPSHOT-javadoc.jar.asc
my-library-1.0.2-SNAPSHOT.pom
my-library-1.0.2-SNAPSHOT.pom.asc
my-library-1.0.2-SNAPSHOT-sources.jar
my-library-1.0.2-SNAPSHOT-sources.jar.asc
The files are all OK. The only issue I have is that checksum files (md5 and sha1) are not generated. However, checksum files are a requirement to have artifacts deployed on Maven Central via OSS Sonatype.
How can I generate the missing checksum files? It seems the maven-publish or signing plugins do not have an option for this purpose? what is wrong?
The solution I found was to use shadow along with ant.checksum:
tasks.withType(Jar) { task ->
task.doLast {
ant.checksum algorithm: 'md5', file: it.archivePath
ant.checksum algorithm: 'sha1', file: it.archivePath
ant.checksum algorithm: 'sha-256', file: it.archivePath, fileext: '.sha256'
ant.checksum algorithm: 'sha-512', file: it.archivePath, fileext: '.sha512'
}
}
Invoking gradle publishShadowPublicationToMavenLocal will generate the signatures as needed, although won't publish them to ~/.m2.
At first I thought those signatures should have been automatic, so I opened https://github.com/johnrengelman/shadow/issues/718 to discuss.
I thought this was a bug in Gradle and I opened an issue, but as described here this actually mimics mvn install behavior. It sounds like Maven Local works a little different than a Maven Repository.
The proper way to test this locally is to use a file based repository. Since you're only using it to test (and not to actually share things with other projects) I think putting that into the build directory is best. Add the repositories section from below to the publishing block. Then when you ./gradlew publish it will publish to your build directory.
Kotlin
repositories {
maven {
// change URLs to point to your repos, e.g. http://my.org/repo
val releasesRepoUrl = uri(layout.buildDirectory.dir("repos/releases"))
val snapshotsRepoUrl = uri(layout.buildDirectory.dir("repos/snapshots"))
url = if (version.toString().endsWith("SNAPSHOT")) snapshotsRepoUrl else releasesRepoUrl
}
}
Groovy
repositories {
maven {
// change URLs to point to your repos, e.g. http://my.org/repo
def releasesRepoUrl = layout.buildDirectory.dir('repos/releases')
def snapshotsRepoUrl = layout.buildDirectory.dir('repos/snapshots')
url = version.endsWith('SNAPSHOT') ? snapshotsRepoUrl : releasesRepoUrl
}
}
These two samples are actually from the link you shared. It's possible they were added later or you (like me) thought that publishToMavenLocal should behave the same as publish (apart from where the files actually go).

Resolve a dependency dynamically inside a gradle task

I am trying to build a gradle plugin, which does the following:
As part of one its tasks, it creates a new configuration
It adds a DefaultExternalModuleDependency to this configuration - more specifically, it constructs a dependency to the application server zip file (available on Nexus). This information can be overridden by the invoking project as well.
Tries to resolve this newly added dependency and then unpacks the file to a local folder
All of this was working well when I had the details hard coded in a build file, but it looks like adding dependencies as part of a task are not treated the same way as having that information available at the parsing time.
So my question is, how do I get the project to reload the configurations / dependencies?
The code looks like the following:
#TaskAction
void installAppserver() {
Dependency dependency = new DefaultExternalModuleDependency(group,name,version)
Configuration configuration = project.configurations.detachedConfiguration(dependency)
configuration.setTransitive(false)
configuration.files.each { file ->
if (file.isFile() && file.name.endsWith('.zip')) {
println 'Attempting to unzip: ' + file + ' into folder: ' + appServerFolder
new Copy().from(project.zipTree(file)).into(appServerFolder).execute()
}
}
}
The problem is that the actual artifacts are not getting resolved!
A task can't configure the build model (that's what plugins do). It's fine to create and resolve a detached configuration in a task. If this doesn't work, there is likely a problem with the task's code, or the dependency it tries to resolve. Note that dependencies can only be resolved if the correct repository(s) are defined.
Instead of new DetaultExternalModuleDependency() (which is an internal class), project.dependencies.create() should be used. Instead of new Copy().execute() (Task#execute must not be called from user code), project.copy should be used.

In Gradle, How do ignore the cache for a local ivy repository?

In my Gradle script (version 0.8), I have a local ivy repository which I use to publish a library, and I can't seem to get Gradle to ignore its cache.
I'm accessing the local ivy repo. based on the recommendation in the Gradle manual. Here's there relevant code (with some variables renamed).
dependencies{
// [several remote dependencies]
// ...
compile('myorg:mylib:0.1.0')
}
def ivy_repo = new org.apache.ivy.plugins.resolver.FileSystemResolver()
ivy_repo.name = 'ivy-repo'
ivy_repo.addIvyPattern local_repository + '/ivy/[organisation]/'+
'[module]-ivy-[revision].xml'
ivy_repo.addArtifactPattern local_repository + '/ivy/[organisation]/'+
'[module]-[revision](-[classifier]).[ext]'
ivy_repo.descriptor = 'optional'
ivy_repo.checkmodified = true
repositories{
// [several remote repositories]
// ...
add(ivy_repo)
}
My problem is that when I run my script, once the local libraries are resolved, subsequent calls to the build script make use of the ~/.gradle/cache/myorg:mylib/, rather than the location of my local ivy repository. The Gradle manual has only a few lines mentioning that default local repositories should not use the cache, but I have found now way to actually indiciate that my repository is 'default' and local and thus shouldn't use the cache.
I've tried setting ivy_repo.local = true, or using compile('myorg:mylib:0.1.0'){changing = true}, but neither seems to work.
I don't want to use a flatDir repository because the local library has dependencies which I want my project to find (and I'm using the ivy file associated with the local library to do that now, and it appears that a flatDir repository ignores these sort of files... or maybe I was doing something wrong there?)
Anyone know a solution? Or maybe a better way to accomplish my task with Gradle?
Any ivy resolver has a Resolver.setRepositoryCacheManager() method. It takes a RepositoryCacheManager as an argument. You can create you own specially configured instance of DefaultRepositoryCacheManager and pass it to the method. With useOrigin you indicate that the cache shouldn't be used for artifacts.
DefaultRepositoryCacheManager cacheManager = new DefaultRepositoryCacheManager();
cacheManager.setName(name);
cacheManager.setUseOrigin(true);
cacheManager.setLockStrategy(new NoLockStrategy());
cacheManager.setIvyPattern(ResolverContainer.DEFAULT_CACHE_IVY_PATTERN);
We want to make this more convenient in 1.0.

Resources