Yarn added the nohoist option to let us exclude individual dependencies from hoisting. But is there a way to exclude an entire package and its dependencies from hoisting?
Example: I have a backend package #mono/server and a frontend package #mono/frontend. I now want to exclude the entire backend package, its dependencies and sub-dependencies from hoisting.
Is this possible?
Yarn 1.x does not have such a feature.
The feature you mention nohoist was removed in yarn 2.0.
There is a new feature in yarn 2.3 called nmHoistingLimits, and it's "dependency" option is getting close to what you are asking for. See more:
https://dev.to/arcanis/yarn-2-3-info-command-detailed-options-nohoist-409h#nohoist
You could accomplish something similar to what you are asking for by eliminating the common ancestor package.json. The structure would be something like:
#mono/ # no package.json in this folder.
#mono/server/package.json
#mono/frontend/package.json
And stepping back a bit, if you want to prevent hoisting of dependencies of server, then it seems server and frontend don't have dependencies in common. In that situation perhaps it would be useful to think of these as two separate packages, giving the common ancestor elimination idea more creedance?
Related
We have a directory structure like so
java
build/build.gradle (This does NOT exist yet, but we want this)
servers
server1/build.gradle
server2/build.gradle
libraries
lib1/build.gradle
lib2/build.gradle
We have 11 servers and 14 libraries with varying uses of dependencies. EACH server is a composite build ONLY depending on libraries (we don’t allow servers to depend on each other). In this way, as our mono-repo grows, opening up server1 does NOT get slower and slower as more and more gradle code is added(ie. gradle only loads server1 and all it’s libraries and none of the other libraries OR servers are loaded keeping things FAST).
Ok, so one problem we are running into is duplication now which is why we need build/build.gradle file AND we want EVERY module in our mono repo to include that somehow for a few goals(each goal may need a different solution)
GOAL 1: To have an ext { … } section containing a Map of Strings to gradle dependencies much like so
deps = [
'web-webserver': "org.webpieces:http-webserver:${webpiecesVersion}",
'web-webserver-test': "org.webpieces:http-webserver-test:${webpiecesVersion}",
'web-devrouter': "org.webpieces:http-router-dev:${webpiecesVersion}"
]
In this way, we want ALL our projects to them import dependencies like so
compile deps['web-webserver']
GOAL 2: We want to 'include' a standard list of plugins so we are versioning all gradle plugins the same across the repo. While the above configures all jars to avoid jar hell in a mono-repo, we would like to do the same with just this section
plugins {
id 'com.github.sherter.google-java-format' version '0.9'
}
Of course, it each project may also want to add a few more plugins OR even not depend on this section(in case of an emergency and trying to just get the job done).
GOAL 3: We want checkstyle configuration (or any plugin config) to be defined the SAME for all projects (eventually!!!). We would like the checkstyle gradle to live in a common area but have all libraries somehow pull it in. Again, it would be nice for it to be optional in that, I can pull the gradle section into my build.gradle OR can create a new one in case of emergencies so I don't have to fix all projects in the monorepo right away.
IDEALLY, perhaps I kind of want configuration injection where when I run server1/build.gradle, it actually runs java/build/build.grade as it’s parent somehow but with overrides (IF I declare 'extends xxx.gradle' maybe) then all libraries it uses also use java/build/build.gradle as their parent. I am not sure this is possible or feasible. I am pretty sure 'extends xxx' doesn't exist in gradle.
Are any of these GOALS possible?
thanks,
Dean
I have been working on a monorepo with the exact same requirement as you, using gradle composite builds as well. The way we have solved this problem is by using pre compiled plugins
You need to do a new gradle project with only the code you want to share. This will create a plugin, that you can just add as a composite build and apply to the other projects.
I'm a bit confused by why you don't just use a "standard" gradle top level build file and compose the others as subprojects.
This solves all 3 of your goals
If you are concerned by build speed, you can target each server individually simply by running
./gradlew :server1:build
But if you are not able to do this for some reason you can use the apply from: syntax as described here
I can't really grasp the purpose of having a vendor folder. Based on what I learned, it seems the vendor folder is only beneficial if you're trying to make your repo compatible with golang versions earlier than 1.11. We are running golang 1.12.14.
When I brought this up to my coworker he said:
Please use vendor with modules - go doesn't have a global artifactory. this is, currently, the best option to make sure you have hermetic builds and your code doesn't break when somebody changes something in their repo.
I thought this is what Go modules does? I asked this question and a commenter is saying I shouldn't use vendor? Does it make sense to add `go mod vendor` to a pre-commit hook?
Go modules bring the guarantee that you will be able to build your packages deterministically by locking down the dependencies into a go.sum. That being said, the promise to deterministically build your project only stands if your dependencies are still accessible in the future. You don't know if this is going to be the case.
Vendoring on the other hand, with or without Go modules, brings stronger guarantees as it enables to commit the dependencies next to the code. Thus even if the remote repository is no longer accessible (deleted, renamed, etc), you will still be able to build your project.
Another alternative is to use Go modules along with a proxy. You can find more information in the official documentation. You can also look at some OSS implementations like gomods/athens or goproxy/goproxy. If you don't feel like setting up and maintaining your own proxy, some commercial offers are available on the market.
So should you go mod vendor each time you commit? Well it's ultimately up to you dependending on the kind of guarantees you want. But yes leveraging a proxy or vendoring your dependencies help getting closer to reproducable builds.
Note: with Go 1.17, go mod vendor (from 1.17 Go commands) might be easier to use:
vendor contents
If the main module specifies go 1.17 or higher, go mod vendor now annotates vendor/modules.txt with the go version indicated by each vendored module in its own go.mod file.
The annotated version is used when building the module's packages from vendored source code.
If the main module specifies go 1.17 or higher, go mod vendor now omits go.mod and go.sum files for vendored dependencies, which can otherwise interfere with the ability of the go command to identify the correct module root when invoked within the vendor tree.
Vendor Folder is a great way to organize and manage third-party dependencies in your project. It is especially useful when your code relies on external libraries or frameworks.
Benefits of having a Vendor Folder:
It helps to reduce dependencies conflicts.
It allows you to keep a separate version of each library / framework installed in your project.
It helps to keep the project structure clean and organized.
It makes it easy to update, install, and remove any dependencies with minimal effort.
It makes it easier to switch between different versions of a library or framework.
In the pyproject.toml configuration file of Poetry you can specify both dependencies and dev-dependencies, but the documentation does not state clearly what the difference is.
I guess from the name the dev-dependencies will not be installed by a release build, but I didn't notice any difference. How do you use these fields correctly, for example exclude the dev-dependencies in a build?
Your assumption is right. The best use case for dev-dependencies is when you creating a library with optional dependencies. For instance, you are developing ORM which should work with MySQL, PostgreSQL, etc. You have to test that your code is working with all of these RDBMS. You put it into dev-dependencies. But for one, who installs your library, these dependencies are optional and they wouldn't be installed automatically.
Commonly, all libraries that are used for testing or building your application are presented at dev-dependencies.
How do you use these fields correctly, for example exclude the dev-dependencies in a build?
poetry install has a no-dev flag for exactly that scenario.
I am trying to modularize a JHipster 5 (Spring Boot 2) application and I ran into a split package problem.
In module-info.java I have the following conflicting automatic modules:
requires problem.spring.web;
requires problem;
requires jackson.datatype.problem;
When I build the project with Maven, I get several errors due to conflicting package name org.zalando.problem like this:
error: the unnamed module reads package org.zalando.problem from both problem and jackson.datatype.problem
error: module problem.spring.web reads package org.zalando.problem from both jackson.datatype.problem and problem
I would like to know how can I solve this issue. Would I have to wait for the third party library to be modularized too? What would be a nice way to solve this conflict?
This article explains a bit on how to solve Split Packages problems. I applied it to solve the split package between jsr305 and java.xml.ws.annotation by using --patch-module argument when building, as explained here. However the project did not compile when I tried the same for these packages.
The source for this project is available on GitHub
If you want to use JARs that split a package as modules, --patch-module is the only way, but it's an arduous one. Beyond patching, you also need to craft the rest of the module graph. Say you're patching module megacorp with the content of start.up, then:
you have to make megacorp read all of start.up's dependencies with --add-reads
you have to make all modules that use start.up read megacorp with --add-reads
you have to ensure that start.up is not on the module path
This can be quite complicated, particularly if you're fighting Maven along the way. Are you sure, there is no way to simply merge the two artifacts?
If not, I'd say this project might not be ready for modularization.
I am currently testing to migrate an existing application to Jigsaw Modules. One of my modules uses ElasticSearch along with its Groovy Plugin.
org.elasticsearch:elasticsearch
org.elasticsearch.module:lang-groovy
Unfortunately, they share a split package, so mvn install gives me:
x reads package org.elasticsearch.script.groovy from both lang.groovy and elasticsearch
once for each required module in the descriptor, where x is the name of each module.
I assume that a newer elasticsearch version will have eliminated the split package by the time Java 9 is final, but is there generally a way to handle split packages in legacy dependencies?
I was hoping to be able to have those on the the classpath instead of the module path, but after reading this conversation on the mailing list it seems that there is no way to tell the Maven compiler to do so.
maven 3.3.9 -
maven-compiler-plugin 3.6.0 -
jdk9-ea+149 -
elasticsearch 2.3.3
After some more testing, I think there are a few options which should tackle many (but definitely not all) 3rd party split package situations.
Clean up dependencies - maybe a dependency isn't actually needed or can be replaced by a newer (or more distinct) JAR
Restructure your own module into two modules, each reading the package from one of both 3rd party modules (if possible/reasonable)
wrap one of the 3rd party modules (or both) in a simple module, which does nothing but explicitly export only the package(s) that are actually needed by your module.
Depending on the situation, one of these options might be a good fit to resolve the split package problem. But none of them can handle situations in which a coherent piece of code actually needs to access classes from both parts of the split package.