How to declare a dependency between 2 non-custom Gradle tasks - gradle

I have a Gradle build in which I am publishing the output to the Maven local repo (its used by other modules of a larger non Gradle build). But doing so generates this warning:
Gradle detected a problem with the following location:
'display\build\outputs\apk\release\display-release.apk'. Reason: Task
':display:publishMavenAndroidPublicationToMavenLocal' uses this output
of task ':display:packageRelease' without declaring an explicit or
implicit dependency. This can lead to incorrect results being
produced, depending on what order the tasks are executed. Please refer
to
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
for more details about this problem.
The link suggests that I declare a dependency between the 2 tasks.
But neither of the tasks are custom tasks, they are part of the included plugins. And I despite spending hours trying all kinds of syntax I can't work out how to declare the dependency from the publishToMavenLocal to the package task.
How do I declare it?
plugins {
id 'com.android.application'
id 'maven-publish' // So that we can publish the APKs
}
android {
...
}
publishing {
publications {
mavenAndroid(MavenPublication) {
groupId = 'com.mygroup.foo'
artifactId = project.name
version = 1.2.3'
artifact "$buildDir/outputs/apk/release/${project.name}-release.apk"
// Explicitly not generating a pom dependencies node
// 1) we don't need it because the APK includes all its deps.
on Artifactory.
}
}
}
build.finalizedBy publishToMavenLocal

I believe the problem stems from this line:
artifact "$buildDir/outputs/apk/release/${project.name}-release.apk"
The APK at that location is created by some task. Instead of referencing that tasks' output, you have given a direct path. Because of that, Gradle is not able to accurately deduce task dependencies.
I am not an Android developer, but judging from the error:
uses this output of task ':display:packageRelease' without declaring an explicit or implicit dependency.
The project display has a task named packageRelease which may or may not come from the Android Gradle plugin. Again, I am not an Android developer, so do not know for certain. The packageRelease is what produces the APK at the aforementioned path.
The publishMavenAndroidPublicationToMavenLocal is created when you create a publication named mavenAndroid; this is documented in the maven-publish documentation
With that said, you must declare task dependencies between the two. Something like:
tasks.withType(PublishToMavenRepository::class).configureEach {
inputFile.from(packageRelease)
}
tasks.withType(PublishToMavenLocal::class).configureEach {
inputFile.from(packageRelease)
}
Without knowing the task type of packageRelease and unfamiliarity with Android projects, best I can do above.

Related

Copy Gradle dependencies from another subproject without deprecation warning

In a Gradle project I have multiple sub-projects. I need to copy the dependencies from the configuration of one sub-project to that of another. I can achieve this using the following, adapted from this answer here:
task copyDependencies(type: Copy) {
from project(":sub").configurations.compile
into "${buildDir}/libraries"
}
Unfortunately, with Gradle 5.1, this generates the following deprecation warning (truncated for brevity):
The configuration :sub:compile was resolved without accessing the project in a safe manner. ... This behaviour has been deprecated and is scheduled to be removed in Gradle 6.0.
What's the best way to do this now for Gradle 5.1 and later?
The warning appeared as reaching across project boundaries can cause issues when the build runs in parallel.
The proper way of doing this is to let Gradle know that the project declaring the copyDependencies task will need the project :sub's dependencies.
You can achieve this by doing the following:
configurations {
dependenciesToCopy
}
dependencies {
dependenciesToCopy project(':sub')
}
task copyDependencies(type: Copy) {
from configurations.dependenciesToCopy
into "${buildDir}/libraries"
}
This will cause Gradle to properly create an execution dependency between your project and the resolution of the dependencies of the sub project.
Relevant discussion on the Gradle forums.

Boilerplate project configuration in Gradle with Gradle Kotlin DSL

I'm currently trying to improve the way our projects share their configuration. We have lots of different multi-module gradle projects for all of our libraries and microservices (i.e. many git repos).
My main goals are:
To not have my Nexus repository config duplicated in every project (also, I can safely assume that the URL won't change)
To make my custom Gradle plugins (published to Nexus) available to every project with minimal boilerplate / duplication (they should be available to every project, and the only thing the project cares about is the version it's using)
No magic - it should be obvious to developers how everything is configured
My current solution is a custom gradle distribution with an init script that:
adds mavenLocal() and our Nexus repository to the project repos (very similar to the Gradle init script documentation example, except it adds repos as well as validating them)
configures an extension that allows our gradle plugins to be added to the buildscript classpath (using this workaround). It also adds our Nexus repo as a buildscript repo as that's where the plugins are hosted. We have quite a few plugins (built upon Netflix's excellent nebula plugins) for various boilerplate: standard project setup (kotlin setup, test setup, etc), releasing, publishing, documentation, etc and it means our project build.gradle files are pretty much just for dependencies.
Here is the init script (sanitised):
/**
* Gradle extension applied to all projects to allow automatic configuration of Corporate plugins.
*/
class CorporatePlugins {
public static final String NEXUS_URL = "https://example.com/repository/maven-public"
public static final String CORPORATE_PLUGINS = "com.example:corporate-gradle-plugins"
def buildscript
CorporatePlugins(buildscript) {
this.buildscript = buildscript
}
void version(String corporatePluginsVersion) {
buildscript.repositories {
maven {
url NEXUS_URL
}
}
buildscript.dependencies {
classpath "$CORPORATE_PLUGINS:$corporatePluginsVersion"
}
}
}
allprojects {
extensions.create('corporatePlugins', CorporatePlugins, buildscript)
}
apply plugin: CorporateInitPlugin
class CorporateInitPlugin implements Plugin<Gradle> {
void apply(Gradle gradle) {
gradle.allprojects { project ->
project.repositories {
all { ArtifactRepository repo ->
if (!(repo instanceof MavenArtifactRepository)) {
project.logger.warn "Non-maven repository ${repo.name} detected in project ${project.name}. What are you doing???"
} else if(repo.url.toString() == CorporatePlugins.NEXUS_URL || repo.name == "MavenLocal") {
// Nexus and local maven are good!
} else if (repo.name.startsWith("MavenLocal") && repo.url.toString().startsWith("file:")){
// Duplicate local maven - remove it!
project.logger.warn("Duplicate mavenLocal() repo detected in project ${project.name} - the corporate gradle distribution has already configured it, so you should remove this!")
remove repo
} else {
project.logger.warn "External repository ${repo.url} detected in project ${project.name}. You should only be using Nexus!"
}
}
mavenLocal()
// define Nexus repo for downloads
maven {
name "CorporateNexus"
url CorporatePlugins.NEXUS_URL
}
}
}
}
}
Then I configure each new project by adding the following to the root build.gradle file:
buildscript {
// makes our plugins (and any others in Nexus) available to all build scripts in the project
allprojects {
corporatePlugins.version "1.2.3"
}
}
allprojects {
// apply plugins relevant to all projects (other plugins are applied where required)
apply plugin: 'corporate.project'
group = 'com.example'
// allows quickly updating the wrapper for our custom distribution
task wrapper(type: Wrapper) {
distributionUrl = 'https://com.example/repository/maven-public/com/example/corporate-gradle/3.5/corporate-gradle-3.5.zip'
}
}
While this approach works, allows reproducible builds (unlike our previous setup which applied a build script from a URL - which at the time wasn't cacheable), and allows working offline, it does make it a little magical and I was wondering if I could do things better.
This was all triggered by reading a comment on Github by Gradle dev Stefan Oehme stating that a build should work without relying on an init script, i.e. init scripts should just be decorative and do things like the documented example - preventing unauthorised repos, etc.
My idea was to write some extension functions that would allow me to add our Nexus repo and plugins to a build in a way that looked like they were built into gradle (similar to the extension functions gradleScriptKotlin() and kotlin-dsl() provided by the Gradle Kotlin DSL.
So I created my extension functions in a kotlin gradle project:
package com.example
import org.gradle.api.artifacts.dsl.DependencyHandler
import org.gradle.api.artifacts.dsl.RepositoryHandler
import org.gradle.api.artifacts.repositories.MavenArtifactRepository
fun RepositoryHandler.corporateNexus(): MavenArtifactRepository {
return maven {
with(it) {
name = "Nexus"
setUrl("https://example.com/repository/maven-public")
}
}
}
fun DependencyHandler.corporatePlugins(version: String) : Any {
return "com.example:corporate-gradle-plugins:$version"
}
With the plan to use them in my project's build.gradle.kts as follows:
import com.example.corporateNexus
import com.example.corporatePlugins
buildscript {
repositories {
corporateNexus()
}
dependencies {
classpath(corporatePlugins(version = "1.2.3"))
}
}
However, Gradle was unable to see my functions when used in the buildscript block (unable to compile script). Using them in the normal project repos/dependencies worked fine though (they are visible and work as expected).
If this worked, I was hoping to bundle the jar into my custom distribution , meaning my init script could just do simple validation instead of hiding away the magical plugin and repo configuration. The extension functions wouldn't need to change, so it wouldn't require releasing a new Gradle distribution when plugins change.
What I tried:
adding my jar to the test project's buildscript classpath (i.e. buildscript.dependencies) - doesn't work (maybe this doesn't work by design as it doesn't seem right to be adding a dependency to buildscript that's referred to in the same block)
putting the functions in buildSrc (which works for normal project deps/repos but not buildscript, but is not a real solution as it just moves the boilerplate)
dropping the jar in the lib folder of the distribution
So my question really boils down to:
Is what I'm trying to achieve possible (is it possible to make custom classes/functions visible to the buildScript block)?
Is there a better approach to configuring a corporate Nexus repo and making custom plugins (published to Nexus) available across lots of separate projects (i.e. totally different codebases) with minimal boilerplate configuration?
If you want to benefit from all the Gradle Kotlin DSL goodness you should strive to apply all plugins using the plugins {} block. See https://github.com/gradle/kotlin-dsl/blob/master/doc/getting-started/Configuring-Plugins.md
You can manage plugin repositories and resolution strategies (e.g. their version) in your settings files. Starting with Gradle 4.4 this file can be written using the Kotlin DSL, aka settings.gradle.kts. See https://docs.gradle.org/4.4-rc-1/release-notes.html.
With this in mind you could then have a centralized Settings script plugin that sets things up and apply it in your builds settings.gradle.kts files:
// corporate-settings.gradle.kts
pluginManagement {
repositories {
maven {
name = "Corporate Nexus"
url = uri("https://example.com/repository/maven-public")
}
gradlePluginPortal()
}
}
and:
// settings.gradle.kts
apply(from = "https://url.to/corporate-settings.gradle.kts")
Then in your project build scripts you can simply request plugins from your corporate repository:
// build.gradle.kts
plugins {
id("my-corporate-plugin") version "1.2.3"
}
If you want your project build scripts in a multi-project build to not repeat the plugin version you can do so with Gradle 4.3 by declaring versions in your root project. Note that you also could set the versions in settings.gradle.kts using pluginManagement.resolutionStrategy if having all builds use the same plugins version is what you need.
Also note that for all this to work, your plugins must be published with their plugin marker artifact. This is easily done by using the java-gradle-plugin plugin.
I promised #eskatos that I would come back and give feedback on his answer - so here it is!
My final solution consists of:
Gradle 4.7 wrapper per project (pointed at a mirror of http://services.gradle.org/distributions setup in Nexus as a raw proxy repository, i.e. it's vanilla Gradle but downloaded via Nexus)
Custom Gradle plugins published to our Nexus repo along with plugin markers (generated by the Java Gradle Plugin Development Plugin)
Mirroring the Gradle Plugin Portal in our Nexus repo (i.e. a proxy repo pointing at https://plugins.gradle.org/m2)
A settings.gradle.kts file per project that configures our maven repo and gradle plugin portal mirror (both in Nexus) as plugin management repositories.
The settings.gradle.kts file contains the following:
pluginManagement {
repositories {
// local maven to facilitate easy testing of our plugins
mavenLocal()
// our plugins and their markers are now available via Nexus
maven {
name = "CorporateNexus"
url = uri("https://nexus.example.com/repository/maven-public")
}
// all external gradle plugins are now mirrored via Nexus
maven {
name = "Gradle Plugin Portal"
url = uri("https://nexus.example.com/repository/gradle-plugin-portal")
}
}
}
This means that all plugins and their dependencies are now proxied via Nexus, and Gradle will find our plugins by id as the plugin markers are published to Nexus as well. Having mavenLocal in there as well facilitates easy testing of our plugin changes locally.
Each project's root build.gradle.kts file then applies the plugins as follows:
plugins {
// plugin markers for our custom plugins allow us to apply our
// plugins by id as if they were hosted in gradle plugin portal
val corporatePluginsVersion = "1.2.3"
id("corporate-project") version corporatePluginsVersion
// 'apply false` means this plugin can be applied in a subproject
// without having to specify the version again
id("corporate-publishing") version corporatePluginsVersion apply false
// and so on...
}
And configures the gradle wrapper to use our mirrored distribution, which when combined with the above means that everything (gradle, plugins, dependencies) all come via Nexus):
tasks {
"wrapper"(Wrapper::class) {
distributionUrl = "https://nexus.example.com/repository/gradle-distributions/gradle-4.7-bin.zip"
}
}
I was hoping to avoid the boilerplate in the settings files using #eskatos's suggestion of applying a script from a remote URL in settings.gradle.kts. i.e.
apply { from("https://nexus.example.com/repository/maven-public/com/example/gradle/corporate-settings/1.2.3/corporate-settings-1.2.3.kts" }
I even managed to generate a templated script (published alongside our plugins) that:
configured the plugin repos (as in the above settings script)
used a resolution strategy to apply the version of the plugins associated with the script if the requested plugin id was one of our plugins and the version wasn't supplied (so you can just apply them by id)
However, even though it removed the boilerplate, it meant our builds were reliant on having a connection to our Nexus repo, as it seems that even though scripts applied from a URL are cached, Gradle does a HEAD request anyway to check for changes. It also made it annoying to test plugin changes locally, as I had to point it manually at the script in my local maven directory. With my current config, I can simply publish the plugins to maven local and update the version in my project.
I'm quite happy with the current setup - I think it's far more obvious to developers now how the plugins are applied. And it's made it far easier to upgrade Gradle and our plugins independently now that there's no dependency between the two (and no custom gradle distribution required).
I've been doing something like this in my build
buildscript {
project.apply {
from("${rootProject.projectDir}/sharedValues.gradle.kts")
}
val configureRepository: (Any) -> Unit by extra
configureRepository.invoke(repositories)
}
In my sharedValues.gradle.kts file I have code like this:
/**
* This method configures the repository handler to add all of the maven repos that your company relies upon.
* When trying to pull this method out of the [ExtraPropertiesExtension] use the following code:
*
* For Kotlin:
* ```kotlin
* val configureRepository : (Any) -> Unit by extra
* configureRepository.invoke(repositories)
* ```
* Any other casting will cause a compiler error.
*
* For Groovy:
* ```groovy
* def configureRepository = project.configureRepository
* configureRepository.invoke(repositories)
* ```
*
* #param repoHandler The RepositoryHandler to be configured with the company repositories.
*/
fun repositoryConfigurer(repoHandler : RepositoryHandler) {
repoHandler.apply {
// Do stuff here
}
}
var configureRepository : (RepositoryHandler) -> Unit by extra
configureRepository = this::repositoryConfigurer
I follow a similar patter for configuring the resolution strategy for plugins.
The nice thing about this pattern is that anything you configure in sharedValues.gradle.kts can also be used from your buildSrc project meaning that you can reuse repository declarations.
Updated:
You can apply another script from a URL, for example doing this:
apply {
// This was actually a plugin that I used at one point.
from("http://dl.bintray.com/shemnon/javafx-gradle/8.1.1/javafx.plugin")
}
Simply host your script that you want all your builds to share on some http server (would highly recommend using HTTPS so your build can't be targeted by a man in the middle attack).
The downside of this is that I don't think that scripts applied from urls aren't cached so they will be re-downloaded every time you run your build.
This may have been fixed by now, I'm not certain.
A solution offered to me by Stefan Oehme when I was having a similar problem was to vendor my own custom distribution of Gradle. According to him this is a common thing to do at large companies.
Simply create a custom fork of the gradle repo, add your companies special sauce to every project using this custom version of gradle.
I encountered a similar problem when common config is replicated in each and every project. Solved it by a custom gradle distribution with the common settings defined in init script.
Created a gradle plugin for preparing such custom distributions - custom-gradle-dist. It works perfectly for my projects, e.g. a build.gradle for a library project looks like this (this is a complete file):
dependencies {
compile 'org.springframework.kafka:spring-kafka'
}

Run task before repositories are added and dependencies are resolved

I'm working on compiling python bindings using gradle.
There is a plugin by linkedin that facilitates that.
They also include a project called the pivy-importer that converts python dependencies into an ivy repository.
I've created a gradle plugin that wraps the pivy-importer so that it can run as a python task.
My repositories are declared like this:
repositories {
pyGradlePyPi()
ivy {
name 'pypi-local' //optional, but nice
url "${project.buildDir.path}/pythonIvy"
layout "pattern", {
ivy "[organisation]/[module]/[revision]/[module]-[revision].ivy"
artifact "[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"
m2compatible = true
}
}
}
The problem, however, is that the repositories are being loaded before the plugin executes.
The first task that the python task runs is pinRequirements so I was adding my cusom pythonImporter task before that like this:
pinRequirements.dependsOn pythonImporter
However, even when I do that, the console shows that the pythonImporter task is running before but as soon as it tries to assemble the dependencies, it can't find them even though they do exist in the file system.
If you rerun the task again, however, it passes because the first run added the repository to the file system.
TL;DR
I need a way to run a task before dependencies are looked up under using a project's configured repositories are loaded.
I moved the tasks execution to my buildSrc subproject and made the build task depend upon its execution.
This works because the buildSrc project is always evaluated before the rest of the projects so you can do "before build" logic there.

Gradle artifactory plugin saying "Cannot cast object 'org.jfrog.gradle.plugin.artifactory.dsl.ArtifactoryPluginConvention'..."

Here's the configuration to get the artifactory plugin:
buildscript {
repositories {
mavenCentral()
maven { url 'http://jcenter.bintray.com' }
}
dependencies {
classpath group:'org.jfrog.buildinfo', name: 'build-info-extractor-gradle', version: '3.0.1'
}
}
apply plugin:'com.jfrog.artifactory'
apply plugin:'ivy-publish'
...some publish spec stuff...
I run gradle (2.3) and I get:
> Failed to apply plugin [id 'com.jfrog.artifactory']
> Cannot cast object 'org.jfrog.gradle.plugin.artifactory.dsl.ArtifactoryPluginConvention#6b6c7be4' with class 'org.jfrog.gradle.plugin.artifactory.dsl.ArtifactoryPluginConvention' to class 'org.jfrog.gradle.plugin.artifactory.dsl.ArtifactoryPluginConvention'
Certainly looks like a classpath issue, but I literally have this project and a sibling project using this same set of gradle/artifactory configurations and one works and the other does not. Both are part of the same top level project. Same JDK (1.8.0_20). Same Gradle. Same everything.
I'm baffled...
The problem was that when I added the various bits to the sibling project that meant I had two projects defining the buildscript {} section.
buildscript {
...
dependencies {
classpath group:'org.jfrog.buildinfo', name: 'build-info-extractor-gradle', version: '3.0.1'
}
}
Apparently that caused two different versions of the dependency to exist in the classpath, hence the error.
The solution was to move the buildscript bit into the master project so those dependencies are only defined once:
buildscript {
repositories {
maven { url "https://plugins.gradle.org/m2/" }
}
dependencies {
classpath group:'org.jfrog.buildinfo', name: 'build-info-extractor-gradle', version: '3.0.1'
}
}
Here's another potential cause. All of this looks to be a problem with rival classloaders defining the class. The full qualified classes include the loader. so, load A foo.bar is not loader B foo.bar and crossing that divide is a complex dance requiring interfaces and careful definition.
So, when using the Jenkins artifactory plugin to build your gradle project with the gradle artifactory plugin, you must add the usesPlugin or jenkins plugin will generate an init script which adds the gradle plugin on to a class loader.
def server = Artifactory.server "artifactory"
def rtGradle = Artifactory.newGradleBuild()
rtGradle.usesPlugin = true // Artifactory plugin already defined in build script
...
My problem was, desktop build OK, jenkins build shows this post's problem
I was getting a similar exception when building with Jenkins. For me the conflict was with Jenkin's version and the version in the Build script:
To address this the Artifactory section of the build has a flag you can check specifying that you want to use the version from the gradle file:
This fixed my issue. Hope it helps.
I had a similar problem. Gradle seems to try to reach across and do some checking or evaluation across siblings. I have a top level settings.gradle with 10 or so subprojects.
The fix for me was to put the buildscript block and dependencies at the top level build.gradle and put it in each of the individual subprojects build.gradle files where needed.
My guess as to the reason this works is that the plugin gets loaded in the parent which will be a parent classloader, then each child project inherits that classloader such that the declaration in the lower child script uses that classloaders class and CCE does not occur. The problem is they are the same class, but not assignable since the different classloaders per subproject if nothing is declared at the top. This was Gradle 2.4, and using IntelliJ 14.
In case it helps someone, I got the same error, but for a different reason.
I had the following in my build.gradle:
dependencies {
classpath "org.jfrog.buildinfo:build-info-extractor-gradle:+"
}
At some point the artifactory plugin updated itself from version 3.x to version 4.x while building, because no specific version was specified for the dependency. After it updated I got the error (Could not find any convention object of type ArtifactoryPluginConvention).
I guess the problem was that the rest of the configuration in my build script doesn't work with the new plugin version. Setting the dependency to use version 3.x fixed the problem for me:
dependencies {
classpath "org.jfrog.buildinfo:build-info-extractor-gradle:3.+"
}
While the currently accepted answer correctly identifies the cause of this issue, the proposed solution doesn't work when you still need to be able to build individual subprojects (because then of course they no longer have access to the buildscript defined repositories and dependencies). The solution that worked for me was to have identical buildscript blocks in each of my subprojects, that seemed to be the key. Any variations would cause the original error.
I got the same exception thrown by bamboo:
'org.jfrog.gradle.plugin.artifactory.dsl.ArtifactoryPluginConvention#18eb2827' with class 'org.jfrog.gradle.plugin.artifactory.dsl.ArtifactoryPluginConvention' to class 'org.jfrog.gradle.plugin.artifactory.dsl.ArtifactoryPluginConvention'
Since the bamboo Bamboo Artifactory Plugin by default looks for the gradle.propeties file in each sub-project module, it has to be provided there.
There is no need for publishing logic in the build.gradle file since the Bamboo Artifactory plugin will read the gradle.properties file for each module respectively, containing:
group=com.example
artifactId=your-project
version=1.0.0
The reason that I got the ArtifactoryPluginConvention exception thrown was that my configured build plan on Bamboo was misconfigured.
With misconfigured, the build ordered of the tasks was not correct. Have a look at your bamboo building tasks/preferably clone a Bamboo plan that is already working.

Gradle - Determine which dependency a jar is from

I've recently run into a situation where I have a conflict because my Gradle dependency resolve results in 2 different-name jar files that are actually having the same contents. It's taken me a while to determine which dependency declaration a jar is from so I can exclude one of them. Is there an easy way that can allow gradle to generate some kind of report so that I can see which jars a dependency is resolved to?
Thanks!
You could use something like the following task which will list out all the files brought in by each of your dependencies.
task dependencyFiles << {
configurations.compile.dependencies.all {
println "${it}: "
println configurations.compile.files(it).collect { it.name }
}
}
Yes, there is. The project report plugin is what you're looking for, especially the dependencyReport and htmlDependencyReport tasks it adds to the project, as well as the dependencyInsight command it allows to execute to get information about a specific dependency.

Resources