We have a directory structure like so
java
build/build.gradle (This does NOT exist yet, but we want this)
servers
server1/build.gradle
server2/build.gradle
libraries
lib1/build.gradle
lib2/build.gradle
We have 11 servers and 14 libraries with varying uses of dependencies. EACH server is a composite build ONLY depending on libraries (we don’t allow servers to depend on each other). In this way, as our mono-repo grows, opening up server1 does NOT get slower and slower as more and more gradle code is added(ie. gradle only loads server1 and all it’s libraries and none of the other libraries OR servers are loaded keeping things FAST).
Ok, so one problem we are running into is duplication now which is why we need build/build.gradle file AND we want EVERY module in our mono repo to include that somehow for a few goals(each goal may need a different solution)
GOAL 1: To have an ext { … } section containing a Map of Strings to gradle dependencies much like so
deps = [
'web-webserver': "org.webpieces:http-webserver:${webpiecesVersion}",
'web-webserver-test': "org.webpieces:http-webserver-test:${webpiecesVersion}",
'web-devrouter': "org.webpieces:http-router-dev:${webpiecesVersion}"
]
In this way, we want ALL our projects to them import dependencies like so
compile deps['web-webserver']
GOAL 2: We want to 'include' a standard list of plugins so we are versioning all gradle plugins the same across the repo. While the above configures all jars to avoid jar hell in a mono-repo, we would like to do the same with just this section
plugins {
id 'com.github.sherter.google-java-format' version '0.9'
}
Of course, it each project may also want to add a few more plugins OR even not depend on this section(in case of an emergency and trying to just get the job done).
GOAL 3: We want checkstyle configuration (or any plugin config) to be defined the SAME for all projects (eventually!!!). We would like the checkstyle gradle to live in a common area but have all libraries somehow pull it in. Again, it would be nice for it to be optional in that, I can pull the gradle section into my build.gradle OR can create a new one in case of emergencies so I don't have to fix all projects in the monorepo right away.
IDEALLY, perhaps I kind of want configuration injection where when I run server1/build.gradle, it actually runs java/build/build.grade as it’s parent somehow but with overrides (IF I declare 'extends xxx.gradle' maybe) then all libraries it uses also use java/build/build.gradle as their parent. I am not sure this is possible or feasible. I am pretty sure 'extends xxx' doesn't exist in gradle.
Are any of these GOALS possible?
thanks,
Dean
I have been working on a monorepo with the exact same requirement as you, using gradle composite builds as well. The way we have solved this problem is by using pre compiled plugins
You need to do a new gradle project with only the code you want to share. This will create a plugin, that you can just add as a composite build and apply to the other projects.
I'm a bit confused by why you don't just use a "standard" gradle top level build file and compose the others as subprojects.
This solves all 3 of your goals
If you are concerned by build speed, you can target each server individually simply by running
./gradlew :server1:build
But if you are not able to do this for some reason you can use the apply from: syntax as described here
Related
I want to publish a common build script which i will include across various projects in my application.
This will contain only the common set of dependencies, i.e dependencies with particular versions that will be common across all the artifacts in my enterprise application..
My applications will refer to this file from the url.
How can i achieve this?
EDIT1: my exploration in this direction is based on this answer on SO:
How to share a common build.gradle via a repository?
There are a few different options for this.
One is to publish a project with the dependencies you want to share defined as API dependencies. Projects that depend on this will inherit the dependencies.
Or you could write and publish a Gradle plugin that will configure your projects with the common dependencies. Projects can apply the plugin, and will automatically be configured in a certain way. (You don't need to publish a plugin to do this - first try creating a project-local buildSrc convention plugin.)
I would actually recommend neither of these approaches.
It's easy to get into a tangled web of dependency hell when transitive dependencies are inherited. It's likely that at some point some dependency will clash, and excluding dependencies can be a big headache, and will easily cancel out any benefit in trying to reduce a little duplication.
Additionally, it's nice when a project is explicit about its dependencies. Being able to look at a build.gradle.kts and understand exactly what dependencies are set is very convenient.
Instead, what I would recommend is controlling the versions of common dependencies in a central location. This can be achieved with the Java Platform plugin. This plugin can be applied to a single build.gradle.kts file, and it lists all versions of all possible dependencies. (It can also import existing Maven BOMs, like the Spring Boot BOM).
Now, all subprojects can add a platform dependency on the 'Java Platform' project.
dependencies {
// import the platform from a Maven repo
implementation(platform("my.company:my-shared-platform:1.2.3"))
// or import a platform from a local project
implementation(platform(":my-project:version-platform"))
// no need to define a version, if it's defined in the platform
implementation("com.fasterxml.jackson.core:jackson-databind")
}
This is the best of both worlds. Projects can be explicit about their dependencies, retain autonomy, while the versions can be aligned across independent projects.
I know you can access different modules (included using include) in a project via org.gradle.api.Project#getSubprojects(), and I know you can get the name and directories of separate builds that have been included (using includeBuild) via org.gradle.api.invocation.Gradle#getIncludedBuilds().
But how can my plugin get information such as the locations of Java source files and class files for projects included using includeBuild?
My goal here is to determine which files have changed in the current git branch (which I can do), and then collect their corresponding class files into a jar file that's used for our patching mechanism that inserts the patch jars at the front of the classpath rather than redeploying the whole application.
I don’t think it is a goal of Gradle to provide including builds with detailed information on included builds. Currently, the Gradle docs basically only state two goals for such composite builds:
combine builds that are usually developed independently, […]
decompose a large multi-project build into smaller, more isolated chunks […]
Actually, isolation between the involved builds seems to be an important theme in general:
Included builds do not share any configuration with the composite build, or the other included builds. Each included build is configured and executed in isolation.
For that reason, it also doesn’t seem to be possible or even desired to let an including build consume any build configurations (like task outputs) of an included build. That would only couple the builds and hence thwart the isolation goal.
Included builds interact with other builds only via dependency substitution:
If any build in the composite has a dependency that can be satisfied by the included build, then that dependency will be replaced by a project dependency on the included build.
So, if you’d like to consume specific parts of an included build from the including build, then you have to do multiple things:
Have a configuration in the included build which produces these “specific parts” as an artifact.
Have a configuration in the including build which consumes the artifact as a dependency.
Make sure that both configurations are compatible wrt. their capabilities so that dependency substitution works.
Let some task in the including build use the dependency artifact in whatever way you need.
Those things happen kind of automatically when you have a simple dependency between two Gradle projects, like a Java application depending on a Java library. But you can define your own kinds of dependencies, too.
The question is: would that really be worth the effort? Can’t you maybe solve your goal more easily or at least without relying on programmatically retrieved information on included builds? For example: if you know that your included build produces class files under build/classes/java/main, then maybe just take the classes of interest from there via org.gradle.api.initialization.IncludedBuild#getProjectDir().
I know, this may not be the answer you had hoped to get. I still hope it’s useful.
I have the following modules with the top one depending on the next depending on the next ( These links have VERY SIMPLE build.gradle and settings.gradle files)
https://github.com/deanhiller/webpieces/tree/master/core/core-ssl
https://github.com/deanhiller/webpieces/tree/master/core/core-datawrapper
https://github.com/deanhiller/webpieces/tree/master/core/core-util
https://github.com/deanhiller/webpieces/tree/master/core/core-logging
I temporarily added an throw new RuntimeException to a test in core-datawrapper and core-util and built the project core-ssl (in the repo, ../../gradlew build)
The settings of core-ssl (found in above link and pasted here) is
includeBuild '../core-datawrapper'
includeBuild '../core-mock'
The settings of core-datawrapper (again in above links)
includeBuild '../core-util'
I clear out core-util/build and I see these targets run
> Task :core-util:compileJava
> Task :core-util:classes
> Task :core-util:jar
That is it. Why is the tests not running? I thought build depended on assemble and test separately?
The same for ../../gradlew clean and ../../gradlew publish
Ideally, I want my target to affect all transitive projects as well. As developers add projects, I don't want to have to add code to each gradle project in the transitive deps list either.
Yes, that’s correct.
This is most probably due to the original use-case of composite builds.
You have some binary dependency on some library, and then you want to change something in the library and test it in your project before even committing or publishing the library.
So the composite build result replaces the binary dependeny and only as much work as really necessary is done, meaning the jar is built.
If you use composite builds as normal structuring element, you either need to wire the lifecycle tasks you want to have wired yourself or search for a plugin that maybe does it.
As developers add projects, I don't want to have to add code to each gradle project in the transitive deps list either.
You can programmatically iterate through the included builds using gradle.includedBuilds, so you can do the wiring in a generic way.
I am using gradle and its local repository is at \.gradle\caches\modules-2\files-2.1 which has all the downloaded jar but not my modules.Is there any specific place I should be searching it for ?
I need it as is in settings.gradle I am having a dependency path specified like :
include ':model'
project (':model').projectDir = new File(settingsDir, './model')
in a new project. Also I don't want to give path in that way because if I have a dependency from multiple projects on this project then mentioning path will be difficult and weird.
How can I make gradle search it from local maven or gradle repositories.
I'm still not sure what is being asked here, and I suspect there is some confusion over how multi-project builds work. So I'm going to attempt to provide a general-purpose answer.
The first question you need to answer is whether you're interested in dependencies between projects that are part of the same build — as in part of a multi-project build — or in separate builds.
Project dependencies (multi-project builds)
Project dependencies are covered in the user manual and only apply to multi-project builds. They use a logical path, using colons as 'path' separators, to specify the location of the target module, like so:
dependencies {
implementation project(":model")
}
At this point, Gradle needs to know where ":model" exists on the file system. There's no getting around that. You have a few options:
Follow the convention of directory structure matching the logical path structure, i.e. have a MyBigProject/model directory containing the ":model" child project
Specify the file path of ":model" in settings.gradle, e.g. with project(":model").projectDir = new File(rootDir, "unusual/path/to/model")
Automate the discovery of projects
The most common approach is the first one. The second is not unusual, particularly if you want to put child projects into a separate directory, like subprojects — something the build of Gradle itself does. I haven't seen the last option done, and I don't know whether it runs into problems.
For the sake of completeness, and at your own risk if you use something like it, here's an example of automatic discovery of projects in the settings.gradle file:
rootDir.eachDir { File dir ->
if ("build.gradle" in dir.listFiles()*.name) {
include dir.name
}
}
This fragment basically looks for directories within the root project folder that have a build.gradle file in them and adds them as child projects. The child projects' directory names become the projects' names.
It's not particularly clever, and you should really use different names for the build files, but it may give you some ideas to work with.
Non-project dependencies
As with project dependencies, Gradle needs to know where to get the corresponding JAR or other form of artifact for a specified module. You normally specify Maven Central or something similar for this, but there are other useful, but less common, options:
Copy a project's artifacts into the local Maven repository — both the Maven Plugin and Maven Publish Plugin support this
Publish to a Maven-compatible repository using a file:// URL rather than an HTTP/HTTPS one, which protects your projects from corruption of Maven Local
Worth noting is that Gradle supports composite builds that allow you to substitute a normal dependency with (effectively) a project dependency from another build. So if model were part of a separate build but you had the source code and build locally, you could make changes and immediately test them in another build's project without going through the whole "install" intermediate step that's common in the Maven world (and Gradle pre-composite-builds).
Hope all this makes sense.
I have a multi-module project, i.e.
parent
module1
module2
In one dev cycle, I added a class mod1.A to module1. Class mod2.B in module2 depends on it.
I do not have the artifacts in my local .m2/repository. Running this:
$ cd prj/module2
$ mvn -o exec:java -Dexec.mainClass=mod2.B
results in an error along the lines of:
The following artifacts could not be resolved: com.example:module1:jar:1.0-SNAPSHOT
After I install the artifacts via mvn install while in the prj folder, it all works as expected.
However, this presents an issue in at least two ways:
I have to go through the slower install phase instead of the faster compile phase
I have two versions of the same project and conflicting modifications in these. I cannot run the same Java class with their respective modifications, only the currently installed modifications, considering they are both the same SNAPSHOT version
There are workaround for both (skip parts of the build for the first, different snapshot versions for the second), but they are far from usable in practice.
Is there a way to make maven use the local modules, instead of using artifacts from local maven repository?
If I understand your question correctly, it seems like you are living a bit outside the norm here: you have two local "copies" of the project with different modifications, that you want to work with alternately when running "exec:java". And Maven is getting in your way: it expects your local .m2 repository area to be in play, but the version strings in each copy are the same, so you end up with the changes interfering among the copies.
To me, it sounds like what you are trying to do is to test your changes. I suggest you just write an actual JUnit or TestNG test in module2 that tests what you want (it can just call mod2.B Main if you want). Then, from your chosen project directory, you can run mvn test -Dtest=MyTestName. It won't "install" anything and it will find the dependencies the way you want it to.
Otherwise, I can see three options.
Change the version string locally in one of the copies (mvn versions:set -DnewVersion=B-SNAPSHOT can do this for you). That way any "installed" jars from your work on that copy will not be considered by the other copy, and vice-versa. You refer to this as being "far from usable" ... I think it should be fine? These are different versions of the project! They should have different version strings! I strongly recommend this option out of the three. (You can do mvn versions:revert when done if you used :set, or you can rely on version control to undo the change.)
Select a different local repository used by Maven when working on one of the projects, with a command-line flag as per https://stackoverflow.com/a/7071791/58549. I don't really think this is a good solution, since you would have to be very careful about using the right flags every time with both projects. Also you'd end up having to re-download Maven plugins and any other dependencies into your new local repository anyway, which is kind of a waste of time.
Try to avoid using any local repository at all. You seem to be trying to make this option work. I don't think this is a great approach either; you're fighting against Maven's expectations, and it limits your flexibility a lot. Maven will indeed find dependencies from the "reactor" (i.e., the executing mvn process) first, but this means all of the required modules must be available in the reactor to be found, which means you can only run mvn at the top level. So if instead you want to just do "mvn exec:java" inside a single module, mvn needs to find that module's dependencies somewhere ... and that's what the local repo is generally used for.
If you're dead set on going with option 3 (instead of option 1), then I suggest you follow the comments on your question and create a profile that runs your exec selectively against module2 and binds it to a lifecycle phase. But this is in practice very close to just wrapping it with a test.
For IntelliJ users:
I solved this problem using IntelliJ's Run configuration. It has the options Resolve workspace artifacts and Add before launch task -> Build. See this picture for clarification:
Run configuration example
The whole point of modules in Maven is to create decoupling between them. You either build each module independently, so that you can work on one module without touching the other, or include both modules as sub-modules in the parent pom and build the parent, which will resolve dependencies between its sub-modules and trigger their builds.
It looks like you have two options here:
Review the structure of your project. Do you really need to split it into two separate modules, if you change code in both of them simultaneously?
Import the project into a Maven-aware IDE (IntelliJ IDEA is very good at working with Maven), and let the IDE handle the compilation. Once finished and stabilized the code-base, build normally with Maven.