I know you can access different modules (included using include) in a project via org.gradle.api.Project#getSubprojects(), and I know you can get the name and directories of separate builds that have been included (using includeBuild) via org.gradle.api.invocation.Gradle#getIncludedBuilds().
But how can my plugin get information such as the locations of Java source files and class files for projects included using includeBuild?
My goal here is to determine which files have changed in the current git branch (which I can do), and then collect their corresponding class files into a jar file that's used for our patching mechanism that inserts the patch jars at the front of the classpath rather than redeploying the whole application.
I don’t think it is a goal of Gradle to provide including builds with detailed information on included builds. Currently, the Gradle docs basically only state two goals for such composite builds:
combine builds that are usually developed independently, […]
decompose a large multi-project build into smaller, more isolated chunks […]
Actually, isolation between the involved builds seems to be an important theme in general:
Included builds do not share any configuration with the composite build, or the other included builds. Each included build is configured and executed in isolation.
For that reason, it also doesn’t seem to be possible or even desired to let an including build consume any build configurations (like task outputs) of an included build. That would only couple the builds and hence thwart the isolation goal.
Included builds interact with other builds only via dependency substitution:
If any build in the composite has a dependency that can be satisfied by the included build, then that dependency will be replaced by a project dependency on the included build.
So, if you’d like to consume specific parts of an included build from the including build, then you have to do multiple things:
Have a configuration in the included build which produces these “specific parts” as an artifact.
Have a configuration in the including build which consumes the artifact as a dependency.
Make sure that both configurations are compatible wrt. their capabilities so that dependency substitution works.
Let some task in the including build use the dependency artifact in whatever way you need.
Those things happen kind of automatically when you have a simple dependency between two Gradle projects, like a Java application depending on a Java library. But you can define your own kinds of dependencies, too.
The question is: would that really be worth the effort? Can’t you maybe solve your goal more easily or at least without relying on programmatically retrieved information on included builds? For example: if you know that your included build produces class files under build/classes/java/main, then maybe just take the classes of interest from there via org.gradle.api.initialization.IncludedBuild#getProjectDir().
I know, this may not be the answer you had hoped to get. I still hope it’s useful.
Related
I have the following modules with the top one depending on the next depending on the next ( These links have VERY SIMPLE build.gradle and settings.gradle files)
https://github.com/deanhiller/webpieces/tree/master/core/core-ssl
https://github.com/deanhiller/webpieces/tree/master/core/core-datawrapper
https://github.com/deanhiller/webpieces/tree/master/core/core-util
https://github.com/deanhiller/webpieces/tree/master/core/core-logging
I temporarily added an throw new RuntimeException to a test in core-datawrapper and core-util and built the project core-ssl (in the repo, ../../gradlew build)
The settings of core-ssl (found in above link and pasted here) is
includeBuild '../core-datawrapper'
includeBuild '../core-mock'
The settings of core-datawrapper (again in above links)
includeBuild '../core-util'
I clear out core-util/build and I see these targets run
> Task :core-util:compileJava
> Task :core-util:classes
> Task :core-util:jar
That is it. Why is the tests not running? I thought build depended on assemble and test separately?
The same for ../../gradlew clean and ../../gradlew publish
Ideally, I want my target to affect all transitive projects as well. As developers add projects, I don't want to have to add code to each gradle project in the transitive deps list either.
Yes, that’s correct.
This is most probably due to the original use-case of composite builds.
You have some binary dependency on some library, and then you want to change something in the library and test it in your project before even committing or publishing the library.
So the composite build result replaces the binary dependeny and only as much work as really necessary is done, meaning the jar is built.
If you use composite builds as normal structuring element, you either need to wire the lifecycle tasks you want to have wired yourself or search for a plugin that maybe does it.
As developers add projects, I don't want to have to add code to each gradle project in the transitive deps list either.
You can programmatically iterate through the included builds using gradle.includedBuilds, so you can do the wiring in a generic way.
We have a directory structure like so
java
build/build.gradle (This does NOT exist yet, but we want this)
servers
server1/build.gradle
server2/build.gradle
libraries
lib1/build.gradle
lib2/build.gradle
We have 11 servers and 14 libraries with varying uses of dependencies. EACH server is a composite build ONLY depending on libraries (we don’t allow servers to depend on each other). In this way, as our mono-repo grows, opening up server1 does NOT get slower and slower as more and more gradle code is added(ie. gradle only loads server1 and all it’s libraries and none of the other libraries OR servers are loaded keeping things FAST).
Ok, so one problem we are running into is duplication now which is why we need build/build.gradle file AND we want EVERY module in our mono repo to include that somehow for a few goals(each goal may need a different solution)
GOAL 1: To have an ext { … } section containing a Map of Strings to gradle dependencies much like so
deps = [
'web-webserver': "org.webpieces:http-webserver:${webpiecesVersion}",
'web-webserver-test': "org.webpieces:http-webserver-test:${webpiecesVersion}",
'web-devrouter': "org.webpieces:http-router-dev:${webpiecesVersion}"
]
In this way, we want ALL our projects to them import dependencies like so
compile deps['web-webserver']
GOAL 2: We want to 'include' a standard list of plugins so we are versioning all gradle plugins the same across the repo. While the above configures all jars to avoid jar hell in a mono-repo, we would like to do the same with just this section
plugins {
id 'com.github.sherter.google-java-format' version '0.9'
}
Of course, it each project may also want to add a few more plugins OR even not depend on this section(in case of an emergency and trying to just get the job done).
GOAL 3: We want checkstyle configuration (or any plugin config) to be defined the SAME for all projects (eventually!!!). We would like the checkstyle gradle to live in a common area but have all libraries somehow pull it in. Again, it would be nice for it to be optional in that, I can pull the gradle section into my build.gradle OR can create a new one in case of emergencies so I don't have to fix all projects in the monorepo right away.
IDEALLY, perhaps I kind of want configuration injection where when I run server1/build.gradle, it actually runs java/build/build.grade as it’s parent somehow but with overrides (IF I declare 'extends xxx.gradle' maybe) then all libraries it uses also use java/build/build.gradle as their parent. I am not sure this is possible or feasible. I am pretty sure 'extends xxx' doesn't exist in gradle.
Are any of these GOALS possible?
thanks,
Dean
I have been working on a monorepo with the exact same requirement as you, using gradle composite builds as well. The way we have solved this problem is by using pre compiled plugins
You need to do a new gradle project with only the code you want to share. This will create a plugin, that you can just add as a composite build and apply to the other projects.
I'm a bit confused by why you don't just use a "standard" gradle top level build file and compose the others as subprojects.
This solves all 3 of your goals
If you are concerned by build speed, you can target each server individually simply by running
./gradlew :server1:build
But if you are not able to do this for some reason you can use the apply from: syntax as described here
I have the following Java projects structure:
Util
|
-- Core
|
-- Services
|
-- Tools
The projects: Tools and Services references to Core and Util projects, the thing is that I ended up writing the same dependency over each project, there must be a better way to inherit the dependencies of the referenced projects and add new ones if needed.
I know about multi projects in Gradle, but this is not like a multi project, since I can basically take the Core library, compile it (which will then contain Core + Util libs) and use it in another project.
I wonder what would be the best way to approach this?
Repeating the same dependencies in every project is usually reasonable because in a bigger project you'll never know when they become different, and you don't want to deal with compilation/runtime problems when someone changes common dependencies list.
I believe that it is more pragmatic to add dependency analyser plugin to your build. It will help you to remove unnecessary dependencies and explicitly add transitive dependencies. And if you add this plugin to your build chain, it will help you to keep your dependencies healthy in the future. Pick this plugin here gradle-dependency-analyze, or maybe there is a better fork or equivalent somewhere.
You are actually out of options in your case because there are only two kinds of dependencies: (1) external (some other jar artefact) or (2) internal (another module in a multimodule build).
2.1 When you use an external maven-like dependency it will come to you with own dependencies (they are named "transitive dependencies"). It means that if you do compile 'yourgroup:Core:1.0' then you will get Util as a transitive dependency. But as I mentioned above, it is better to list transitive dependencies explicitly if they are used during compilation or to prevent them from being accidentally removed and crash your application in runtime.
2.2. If your projects live in the same version control repository and usually change and build together, then the multimodule layout is your best choice. In this case, you will refer to Core dependency like compile project(':Util:Core') and it will grab Util as a transitive dependency as well. And you will be able to do what you asked for and define dependencies for Services and Tools once - inside subprojects {} closure in the Core/build.gradle.
Having multimodule built doesn't limit you from using Core library elsewhere. No matter if it is a multimodule build or not, you can always add maven-publish plugin to Core/build.gradle, execute publishToMavenLocal task and reference to Core.jar from another project the same way you do for external dependencies.
You can always put your common code (like the one which will add common dependencies) in the external gradle file or custom plugin and apply it in Services and Tools.
This is not related to update the pom dependencies using a maven goal. I already have that sorted out.
So I am responsible for creating, packaging and maintaining common utilities. These common utilities are in turn used by all the teams in the org. Problem is that the teams using these utilities do not update the dependencies unless it is the last resort. We would like them to use the latest release version of our common utilities barring very few.
Now I have come across this Maven Versions Plugin by mojohaus which I think serves my need by using two goals - versions:update-properties and versions:use-latest-releases. It serves my purpose except two things:
I do not see a way to exclude certain groupid:artifactid from the update dependency/property
We really want this to be a compulsory thing (maybe part of the Test execution - this is for test automation utilities mainly) rather than a maven goal. Because if it is a maven goal, it needs to be invoked separately and hence becomes optional for teams.
We know that forcefully updating to latest version might cause some issues with defect re-produciblity, but we are willing to take that risk. Our utilities are really test products.
Any direction/help on this is appreciated.
Edit: We run our tests using maven goals clean install already. So they use existing pom. We want dependencies update to happen before the tests run. Also it is desirable to commit the changes to source control (bitbucket) if possible.
We have our tests setup using Jenkins but teams also run multiple test on local machines.
Edit: Found the answer to #1. The plugin provides to exlude regex for group and artifact id. using tags excludes and excludesList
Sorry for insisting. From the Maven point of view I see two main solutions:
Your projects and your utility jar are tightly coupled and every project always needs to use the latest version. Then you can bundle all projects and your utility in one multi-module project. This makes sure that everything is up to date all the time, but it requires that all projects and your utilities are always build together (not separately).
You distribute your utilities to different projects which build and release at different times. Then it is up to the projects to decide when to update. There is unfortunately no standard way to deprecate jars.
If I understand you correctly, you want something "in the middle". This may be hard to achieve.
I was trying to build Maven pom in something similar to the following hierarchical form:
root
+-- A-POM
+-- B-POM
+-- C-POM
+---D-POM
I was hoping that this could take care of my changed module problem. That is, if C is changed, then A must be rebuilt, etc.
But I ran into the issue that it seems the packaging at root is "pom," and after that I can't have A as packaging "war" then continue to drill in to have A include B, C as its modules. It seems to me that any POM which does not have "pom" in the then it can't have child modules. Is my understanding correct? Is there a way to do what I wanted to do?
In addition, I don't seem about to chain the "changed" mechanism in Maven (must due to my lack of knowledge). I like to have Maven detect a dependent project has changed and rebuild all the affected projects.
Thanks so much!
the reactor project (the root of the multimodule project) must have pom packaging. So your nested structure is invalid since A is not of type pom and I'm pretty sure you won't get it to work this way.
Second point is that Maven is a modularized build system and uses repository mechanisms to locate pre-built artifacts instead of checking out all modules from version control and building them in a monolithic way like in the old days ;) This means that Maven cannot know what to rebuild when you change something at your module since it simple does not have all the other module there at this time.
I think this is more a CI task than that should be handled by the build system itself. I know that your can achieve such a behavior with an appropriate build/CI Server like Jenkins that supports upstream and downstream projects. This means it is able to detect dependencies between the projects and trigger other builds as soon as a dependency has been built. This comes close to the behavior you are trying to achieve.
Btw. rebuilding other projects is only required for SNAPSHOT dependencies. Jenkins with the maven plugin supports this behavior but, depending on the number of SNAPSHOT dependencies of your project, this can cause long chains of project builds on the server. Some folks are of the opinion that in general SNAPSHOT versions are hell for CI tasks since these artifacts can change over time and are not reproducible. You could think over completely omitting SNAPSHOT versions and building final versions each time. This would also obviate your requirement to rebuild other modules as soon as a module changes. There are simply no changes until you upgrade dependency versions.