I have a project that uses gradle and mavenCentral() (plus mavenLocal()). It has enough dependencies that I can't go through them one by one.
Given the name of a .jar file in build/install/x/lib, how do I find out the chain of transitive dependencies that caused it to be included?
update: I discovered gradle dependencies. The output shows:
org.apache.commons:commons-jexl:2.1.1
\---- commons-logging:commons-logging:1.1.1 -> 1.1.3
What does this mean? 1.1.1 is the version I expect, and 1.1.3 is the version I seem to actually end up using. Looking at the pom for commons-jexl it looks like it does indeed list logging:1.1.1 as a requirement. What's going on? Is there a way for me to tell it to avoid certain versions, or force it to use the version it was set to?
The problem in my case is that it's including a -SNAPSHOT version and I'd rather it didn't. In fact I probably want it to just use the version numbers I'm asking for instead of the most recent it can find.
Dependencies of gradle-managed project have their own dependencies (they're called transitive). It may happen (and happens quite often) that two different dependencies has the same (group and module) dependency but in the different version). This is the case with commons-logging:commons-logging. In this case there are two transitive dependencies one versioned with 1.1.1 and the second one with 1.1.3. If both of the libraries will be included in the final artifact it may result in a conflict and exception. To prevent such situation gradle tries to resolve mentioned version resolution problems by picking (by default) the latest version. It's indicated with the right arrow -> see here. You can exclude transitive dependencies from a particular dependency. This chapter of manual might be useful.
Related
I was studying Maven's build system and it adds a lot of transitive dependencies because of its transitive dependency system (maven itself does not add dependencies but transitive dependency system does). I see issues with it like major version conflicts and unknown dependencies coming in.
I was thinking why is the system designed this way and why not take direct dependencies. My library does not need to depend on something which my dependency is using but not my library (I mean I understand why it needs to be included in the build list, my dependencies need to build using those, but why does it needs to cause major version conflict?). Am I missing something fundamental here? One thing that I can think of is that my library's build dependency list can grow to be very big because of all the direct dependencies I will need to take, but that does seem to be as big of a problem as problems with transitive dependency system.
I am new to build systems, so please don't be too harsh. I also tried to google this question but didn't find useful answers but please feel free to comment anything that I might have missed.
Thanks
If you need library A to run, and library A needs library B to run, and this needs C to run, it is very tedious to figure this out and add all the relevant dependencies to your project.
Before Maven and Gradle, many people worked that way and found out, that it is much easier to let a build tool figure out the transitive dependencies.
My library does not need to depend on something which my dependency is using but not my library [...]
This is your major misconception. There are two possibilities:
The direct dependency of your library exposes types from the transitive dependency in its public API. To use this public API you need to access these types, so you need the transitive dependencies during compile time.
The direct dependency of your library only uses its own dependency internally, but not in its public API. In this case, your library does not need to depend on the transitive dependency during compile time. But as soon as your library code runs (even in a test), it may use some functionality of its direct dependency that internally uses functionality of the transitive dependency, causing your library code to fail.
[...] I mean I understand why it needs to be included in the build list, my dependencies need to build using those [...]
There is no actual build list (or order) for external dependencies, because they are used when they are already built (the downloaded .jar files contain compiled .class files). But as I mentioned above, you will need the transitive dependencies either during compile time or during runtime (e.g. tests), so your build system (Maven or Gradle) will fetch them for you.
[...] but why does it needs to cause major version conflict?
#khmarbaise already explained in his comment, why and how version conflict between transitive dependencies may occur:
You are using two libs X and Y. Both of them using another lib (A) So X is using A in version 1.0.0 but Y is using A in version 2.0.0. In the end you can't have both on the classpath there must be done a decision for one version. So depending on how X,Y are implemented either X can fail while using A in V1.0.0 or Y can fail in using A in V1.0.0 or with V2.0.0 the same... This can happen if X or Y are being updated. This is also true for different version combinations like A in 1.0.0 and 1.1.0 (if compatibility is not 100%)
I had a situation in my project where two dependencies relied on a different version of a jar. Just showing the dependency tree only showed the newer version and where it was used, so I ended up going through the POM files (I had an idea where it might be) to find the source.
My question: is it possible to get a dependency tree that includes these dependencies that were removed (showing why they were removed). I seem to recall the m2eclipse plugin can do it, but I'd prefer a command line tool (since I'm not using eclipse).
edit:
Specifically, I already tried the dependency tree, including running with full debug output (-X). As far as I could tell, it doesn't show when it's masking these dependencies.
Just try the maven-dependency-plugin use the tree goal to look at the tree of dependencies. But it might be a good idea to use Eclipse via m2e plugin for such problems.
A problem that relates to basic maven concepts:
Once released I would like to have a guarantee that the project build is fully reproducible. So all project and plugin dependencies, including transitive one, should be always resolved the same way.
Unfortunately it is not the case, if dependencies are expressed in terms of version ranges. It can happen that even though direct dependencies of a project are set (using versions:use-releases), the transitive dependencies can still be resolved in some other way in the future.
How to address the problem? Is there a known solution?
I was thinking (just an idea), about creating a plugin, which on release time would dump all dependencies of the project to a separate file, and then once building in the future, the dependencies read from the file would take precedence over the standard way maven uses to resolve dependencies. But I'm afraid that there is no plugin api for that. So it would require some hacking, which I would like to avoid. Is there another way?
Thanks,
Lukasz
Freeze artifacts versions using <dependencyManagement>. Even if you don't use version ranges (as you said), but rather 3rd party libs (your dependencies) do, your <dependencyManagement> will have higher priority in specifying version of any artifacts.
The simple solution is: Do not use version-ranges. This is bad practice cause it will result in the described problems.
When there is a conflict in the dependency tree (same artifact but different versions) then, AFAIK, Maven will resolve the conflict by selecting the highest version of the dependency and will omit the 'old' ones.
However, when the newer version is a SNAPSHOT then apparently it will choose the older stable version over the SNAPSHOT.
In my case: some-artifact: 0.5.0-SNAPSHOTS (omitted for conflict with 0.4.0) => version 0.4.0 is picked over the wanted 0.5.0-SNAPSHOT.
I assume this functions as designed but I don't understand the reason why. Next to that, any idea if there is a way to tell Maven to take the SNAPSHOT over the stable version?
Your assumption about Maven's always selecting the highest version isn't accurate. Artifacts are chosen based on a number of factors including depth of the dependency in the tree, order in the tree, whether the dependency is a snapshot or a release, and dependency management, which pretty much overrides everything else.
Unfortunately, I don't know of any one, definitive source of information on Maven's dependency resolution algorithms. You'll find bits and pieces of it scattered all over. A few handy references:
Introduction to the Dependency Mechanism gives an overview of the topic with a good, if short, section on Transitive Dependencies and how they're selected from a dependency tree.
The Sonatype Maven book has a more thorough section on Project Dependencies in general that will add a lot to your knowledge about the subject.
An earlier section of that same book discusses Project Versions, which is strongly related to this problem and has a good section on SNAPSHOT versions, though not as much as I could wish on how they play into dependency resolution.
Project Relationships talks about the coordinate system and how project inheritance affects what dependencies get included.
Finally, the POM Reference is a good jumping-off point for almost anything to do with the pom. There's at least a brief description of every pom element that can help you understand enough to be able to begin searching for more info effectively.
As for some practical advice, the output of mvn dependency:tree is highly useful in discovering why a particular version of a dependency was chosen. It'll often even tell you something like "foo:bar:1.2 (was 1.1)". Once you figure out where the errant version is coming from, there are a number of ways to ensure a specific dependency version is used for a project:
Exclude wrongly-versioned dependencies from other dependencies that are causing them to be included in the build.
Add an explicit top-level dependency to your pom instead of relying on a transitive dependency.
List the dependency in the dependencyManagement section of your pom (scroll down a bit from this link) to force the dependency to have the specified characteristics, regardless of what level of transitive dependency it is. Use this option with care, as dependencyManagement is viral, in that other projects depending on your project will be "infected" with your dependency management. There's also a good section on dependency management in the pom reference.
If the 0.4.0 version is being pulled in as a transitive dependency via another dependency in your POM, then you should be able to exclude it. The dependency:tree goal should help you see if this is what's happening.
Maven is designed to favor release versions over snapshot versions. I'm not sure why you would have two dependencies in the same POM and not be able to resolve a conflict by removing one, so I will assume that one of the dependencies is inherited from a parent pom. In this case you can set the inherited dependency as <optional>true</optional> and I THINK it should allow the child POMs to override it, even with a lower version.
bad/hacky solution for if that doesn't work - edit your local repository in such a way that it doesn't realize the 0.5.0 version is a snapshot (or even edit your private nexus repo if you have the ability)
I currently have a project that I have performed an aqua scan on, and it identified the jackson-databind-2.9.8.jar I'm currently using as a critical vulnerability, and has recommended me to replace with version 2.10. To update this, while ensuring all other dependencies/code works fine, I've tried the following code in my build.gradle file, where group_name:microservice-event:0.2.+ shows up on the list of gradle dependencies and apparently brings in the 2.9.8 jar that is causing problems:
implementation 'com.fasterxml.jackson.core:jackson-databind:2.10'
implementation('*group_name*:microservice-event:0.2.+') {
exclude group: 'com.fasterxml.jackson.core', module: 'jackson-databind'
}
I've also removed the implementation '*group_name*:microservice-event:0.2.+' line I previously had in my build.gradle file.
However, now the project fails to build and I have no idea why. Would anyone know of how to write code in the build.gradle file to successfully exclude old jars/dependencies, while allowing for newer jars (as I've tried to do with the line implementation 'com.fasterxml.jackson.core:jackson-databind:2.10'). Note that I do not want to update the spring boot version.
When Gradle encounters two different versions of the same dependency, it will perform a conflict resolution. It defaults to choosing the highest version number.
However, because many libraries like Jackson consists of a number of individual modules like jackson-databind and jackson-core, you may end up in a situation where there is a mismatch between the different versions.
To align them, you can use the Jackson BOM and Gradle's platform dependency mechanism. It looks like this (choose only one of the depencendies below):
dependencies {
// Enforce the specified version
implementation(enforcedPlatform("com.fasterxml.jackson:jackson-bom:2.10.4"))
// Align all modules to the same version, but allow upgrade to a higher version
implementation(platform("com.fasterxml.jackson:jackson-bom:2.10.4"))
}
You don't need to exclude anything from your other dependencies.
If you encounter problems with the use of Jackson after upgrading, you should have a look at the release notes for 2.10 and check if you might be hit by any of the compatibility changes. Of cause, if the problem is in a third-party library, it might be more difficult to fix. But you may try the latest version in the 2.9 line (which is 2.9.10 at this time) and see if the vulnerability is fixed here.