I have 3 projects A, B and C where C is dependent on A and B.
Project A has some core API which changes very frequently (eg: every couple of weeks) and almost all the transitive dependencies needed for Project C which are pretty much constant all the time.
Project B has only core API which changes very rarely (eg: once every 2-3 months).
Project C has some custom implementations which use the core API's from both A and B.
Project C generates a runnable fat/uber/shade jar which is the end product. The problem is, overtime the jar became humongous and it takes atleast couple of hours just to build and publish it. My goal is to bring down this build time by only generating the necessary components as and when needed.
Our current gradle dependencies section looks like:
dependencies {
implementation 'com:project-A:aaaa'
implementation 'com:project-B:bbbb'
implementation 'com:project-C-dependency-1:xxxx'
implementation 'com:project-C-dependency-2:yyyy'
.
.
.
}
I am then using a Zip task to get all these dependencies included in a single distribution.
My goal is, if possible, to be able to create custom configurations such that I can create 4 seperate distributions - Project A core API, Project A transitive dependencies, Project B core API and Project C API. My issue is how to seperate core API and transitive dependencies from Project A after I am able to resolve it in Project C which should look like following:
`dependencies {
projectAcore ('com:project-A:aaaa') {
transitive = false // This is exactly what I need
}
projectATD ('com:project-A:aaaa') {
// how to achieve transitiveOnly = true?
}
projectBcore ('com:project-B:bbbb')
projectC .....`
One way I am able to get it to run is downloading everything from project-A and then manually deleting all the core API files. This is cumbersome and not productive.
Related
What is the key difference between includeBuild(...) and implementation(project(...)) in the Gradle build system? I cannot really see the usecase difference after reading the documentation:
https://docs.gradle.org/current/userguide/declaring_dependencies.html#sub:project_dependencies
https://docs.gradle.org/current/userguide/composite_builds.html#separate_composite
What I am trying to do: Share same codebase for two separate projects: data classes (kotlix.serialization), external database dao, interfaces. It is not a full library but just some code snippets.
How can I connect the two projects in Intellij so that type hinting works?
Composite Build (by using includeBuild) is a way to create a dependency between autonomous Gradle Projects.
Project import, is a way to create a dependency between two modules within a same Gradle Project.
Composite Build is far more powerful and is also meant to be the new way of factorizing gradle configuration between multiple projects that you traditionally do with the buildSrc technique.
I found this "Structuring Large Projects" article to be easier to read than the "Composite Builds" doc.
An excellent sample project that demonstrates the power of Composite Builds can be found in Gradle sample_structuring_software_projects.
Project dependency case
The tree would look like this:
settings.gradle.kts
module1/build.gradle.kts
module2/build.gradle.kts
And you are declaring a dependency in module1/build.gradle.kts like this:
dependencies {
implementation(project("com.domain:module2"))
}
The dependency will be resolved only if both projects are declared as sub-modules of a common root project.
It means you have a root settings.gradle.kts like this:
rootProject.name = "rootProject"
include(":module1")
include(":module2")
Composite build case
The projects do not need to have common "umbrella" root project.
Each project is a totally independent project.
One project can simply declare a dependency on the other project (without even the target project knowing about it).
Tree:
project1/build.gradle.kts
project1/settings.gradle.kts
project2/build.gradle.kts
project2/settings.gradle.kts
In project1/settings.gradle.kts:
rootProject.name = "project1"
includeBuild("../project2") //No more ':' as it is not a module
In project2/settings.gradle.kts:
rootProject.name = "project2"
In project1/build.gradle.kts like this:
dependencies {
implementation("com.domain:project2")
}
I have the same problem. Reading on the first link, the next para says:
Local forks of module dependencies
A module dependency can be substituted by a dependency to a local fork
of the sources of that module, if the module itself is built with
Gradle. This can be done by utilising composite builds. This allows
you, for example, to fix an issue in a library you use in an
application by using, and building, a locally patched version instead
of the published binary version. The details of this are described in
the section on composite builds.
So, it should be implementation project according to me.
P.S. The code completion is working on one of my sub project but not on the other. I am still trying to figure that out
We have a few full gradle projects A, B, C, D. These are microservices that are going to start sharing the protobuf generated java files. we are thinking a structure like this
A
build.gradle (this is a full on gradle build)
B
build.gradle (this is B's full on gradle)
common
build.gradle (build the protobuf that is used by A and B)
Now, the question is how do we make sure that when a developer builds A, it also builds common in case it changed on his git pull. Same goes for B. The settings.gradle file didn't seem to hav a ../../:project or something like that.
I do remember gradle came out with a way to build multiple gradle projects though as well.
Ideally, when someone changes common, multiple jenkins builds would be kicked off as well verifying that changing core code didn't break any of the services that use it. I am not quite sure how to
1. document the things that depend on common
2. use the document to kick off builds of all things depending on common
Then if this were to grow, and you have D depends on C depends on common, each build needs to be kicked off feeding the binary upstream from common to C and then C's jar and common's jar to D. I know 'pants' is used at twitter to do this. Google is using bazel. Perhaps I look into that instead of gradle? or can we intermingle them?
Simply declaring a dependency on common should be sufficient enough:
// Project A's build.gradle
dependencies {
implementation(project(":common"))
}
In order to build a, the build of common would need to succeed. If the build of common failed for whatever reason, then then build of a will also fail. Example:
$ ./gradlew project-a:build
> Task :common:compileJava FAILED
/Users/cisco/code/example-multi-project/common/src/main/java/common/ExampleCommon.java:6: error: incompatible types: int cannot be converted to String
return 1;
^
1 error
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':common:compileJava'.
> Compilation failed; see the compiler error output for details.
You can see in the above that when I tried to build project a (project-a:build), the common's build task was invoked (:common:compileJava).
Both projects a/b/etc should have thorough tests (unit, integration, smoke, etc) to make sure that any incompatible changes are detected early/often.
You can read more about multi-project builds in the official guide: https://docs.gradle.org/current/userguide/multi_project_builds.html
I have the following Java projects structure:
Util
|
-- Core
|
-- Services
|
-- Tools
The projects: Tools and Services references to Core and Util projects, the thing is that I ended up writing the same dependency over each project, there must be a better way to inherit the dependencies of the referenced projects and add new ones if needed.
I know about multi projects in Gradle, but this is not like a multi project, since I can basically take the Core library, compile it (which will then contain Core + Util libs) and use it in another project.
I wonder what would be the best way to approach this?
Repeating the same dependencies in every project is usually reasonable because in a bigger project you'll never know when they become different, and you don't want to deal with compilation/runtime problems when someone changes common dependencies list.
I believe that it is more pragmatic to add dependency analyser plugin to your build. It will help you to remove unnecessary dependencies and explicitly add transitive dependencies. And if you add this plugin to your build chain, it will help you to keep your dependencies healthy in the future. Pick this plugin here gradle-dependency-analyze, or maybe there is a better fork or equivalent somewhere.
You are actually out of options in your case because there are only two kinds of dependencies: (1) external (some other jar artefact) or (2) internal (another module in a multimodule build).
2.1 When you use an external maven-like dependency it will come to you with own dependencies (they are named "transitive dependencies"). It means that if you do compile 'yourgroup:Core:1.0' then you will get Util as a transitive dependency. But as I mentioned above, it is better to list transitive dependencies explicitly if they are used during compilation or to prevent them from being accidentally removed and crash your application in runtime.
2.2. If your projects live in the same version control repository and usually change and build together, then the multimodule layout is your best choice. In this case, you will refer to Core dependency like compile project(':Util:Core') and it will grab Util as a transitive dependency as well. And you will be able to do what you asked for and define dependencies for Services and Tools once - inside subprojects {} closure in the Core/build.gradle.
Having multimodule built doesn't limit you from using Core library elsewhere. No matter if it is a multimodule build or not, you can always add maven-publish plugin to Core/build.gradle, execute publishToMavenLocal task and reference to Core.jar from another project the same way you do for external dependencies.
You can always put your common code (like the one which will add common dependencies) in the external gradle file or custom plugin and apply it in Services and Tools.
Very new to Maven, can someone please explain to me the difference between using maven modules vs just adding a dependency to your maven project to another maven project in your workspace? When would you use one over the other?
A dependency is a pre-built entity. You get the artifact for that dependency from Maven Central (or Nexus or the like.) It is common to use dependencies for code that belongs to other teams or projects. For example, suppose you need a CSV library in Android. You'd pull it as a dependency.
A Maven module gets built just like your project does. It is common to use Maven modules for components that the project owns. For example, maybe your project creates three jar files.
A dependency can be thought of as a lib/jar (aka Artifact in Maven parlance) that you need to use for building and/or running your code.
This artifact can either be built by your one of the modules of your multi module project or a third party pre-build library (for example log4j).
One of the concepts of maven is that each module is going to output a single artifact (say a jar). So in case of a complex project it is good idea to split your project to multiple modules. And these modules can be dependent on each other via declared dependencies.
See http://books.sonatype.com/mvnex-book/reference/multimodule-sect-intro.html for example of how a web app is split to parent and child modules and how they are linked.
One of the most confusing aspects of Maven is the fact that the parent pom can act as both a parent and as an aggregator.
99% of the functionality you think about in Maven is the parent pom aspect, where you inherit things like repositories, plugins, and most importantly, dependencies.
Dependencies are hard, tangible relationships between your libs that are evaluated during each build. If you think of your software as a meal, it's basically saying A requires ingredient B.
So let's say you're preparing lasagne. Then your dependency chain would look something like this:
lasagne
<- meatSauce
<- groundBeef
<- tomatoPaste
<- cheese
<- noodles
The key thing is, each of the above items (meatSause, groundBeef, cheese, etc) are individual builds that have their individual set of dependencies.
By contrast, the only section of your pom that pertains to aggregation is the modules section:
<modules>
<module>meatSauce</module>
<module>groundBeef</module>
<module>tomatoPaste</module>
<module>cheese</module>
<module>noodles</module>
</modules>
Aggregation simply tells your build engine that it should run these 5 builds in rapid succession:
groundBeef -> tomatoPaste -> cheese -> noodles -> meatSauce
The main benefit of aggregation is the convenience (just click build once) and ensuring the builds are in the correct order (e.g. you wouldn't want to build meatSauce before tomatoPaste).
Here's the thing though: even if you organize the libs as standalone projects without module aggregation, your build will still come out the same provided you build in the correct order.
Moreover, both Jenkins and Eclipse have mechanisms for triggering builds if a dependent project has changed (e.g. changing groundBeef will automatically trigger meatSauce).
Therefore if you're building out of Jenkins or Eclipse, there is no need for aggregation
I'm having a problem reconciling building a project for use within an application server and for use as a stand-alone application.
To give an overall simplified context, say I have three Projects A, B, C.
Project A depends on Project B which depends on Project C.
Project C has a dependency X which is marked as provided since it was expected that it would be available as a JEE library within say an application server. i.e. jms.jar.
So if I perform an assembly build of Project A, I get all the transitive dependencies save for those marked as provided as expected.
Now I have a new deployment scenario where Project A needs to be used in a standalone environment i.e. outside an application server.
So now I need the jms jar to be a compile dependency. Does this mean that I should explicitly add a compile dependency for X in Project A? Doesn't this violate the Law of Demeter, i.e. don't talk to strangers, in the sense Project A shouldn't explicitly know about Project C but only about Project B?
This is a simple example but in reality I have multiple dependencies which have been marked as provided but are now need to be compile or runtime dependencies so they end up in the artifact produced by the maven assembly plugin.
Is this a fundamental problem with Maven or am I not using the tools correctly?
Thanks in advance for any guidance.
If you need your build to have variations in it for different scenarios, you need to use profiles and keep certain things (such as some of the dependencies) in the various profiles.
http://maven.apache.org/pom.html#Profiles
Different dependencies for different build profiles in maven
answers a similar question - but you can swap in the "release" and "debug" for "Project A" and "Project C"
Provided dependencies are a difficult subject. First of all: Provided dependencies are not transitive in the following sense: If your project C has a provided dependency on X, then A will not get the dependency. It is silently ignored. This fits with the following meaning of "provided" which I propose:
Only the artifacts that are actually deployed should mark dependencies as "provided". Libraries or other jars that are not individually deployed to a specific server should not have provided dependencies. Instead, they should declare their dependencies as compile dependencies. In your example: Project C should have a compile dependency on X. If project A knows that X is provided, it sets X to provided in "dependencyManagement". As project A should know the environment in which it runs it should decide what is provided and what is not. And "dependenyManagement" is the right place to declare this.
If your project A should be able to run within and without a given server, you probably need to make a lot of adjustments, even change the type from ear to jar. So you either use build profiles for this, which then have different dependencyManagement entries, or you split A into two projects which depend on some other project that contains the common elements.
If some given project C already has a provided dependency on X and you cannot change that, this is effectively the same as a missing dependency in C. This has to be repaired at some point, and this could be project A itself.