I am utilizing Yarn 3 with the Plug'n'Play (PNP) feature for zero-installation of the dependencies in my development environment, which works well. However, when I run a production build, the .yarn/cache folder also includes dev-dependencies, resulting in an increase in the overall size of the Docker image. Is there a way to only include the necessary dependencies in the .yarn/cache folder for the production build?
Related
I am using Yarn's Workspaces feature to manage a monorepo. Some of the individual workspaces have their own dependencies defined in their own package.json, however I also have some shared dependencies (some are for testing the whole repo, some are used by all the workspaces). How do I declare these the correct way?
If I try to add them to the root package.json, Yarn gives me a warning. But adding the same dependency to all n workspaces (there are a lot and it's liable to grow) seems tedious and hard to maintain when upgrading dependencies.
Is the right thing to do to add the shared dependency (say, typescript, or jest or serverless) to every single individual workspace?
Using Yarn 1/classic.
I found also Yarn Workspaces: How and where should I install my dependencies? but it's unanswered
Yarn workspaces shared dependencies talks about using peer dependencies but the user is experiencing trouble
After a half-day search and trying, I found multiple ways to manage shared dependencies
1. Sync dependency version with syncpack
https://github.com/JamieMason/syncpack/
Given the fact that yarn workspaces will share the dependencies if their versions are the same (see https://classic.yarnpkg.com/lang/en/docs/workspaces/), all we have to do is to keep the dependency versions in sync to make dependencies sharing, which is exactly what syncpack does.
I found this is the least harmful way to make sharing dependencies works, while other methods require some degree of twisting package.json.
2. Use peer dependencies
For each package, set sharing dependencies as peerDependencies with version “*”, and set root workspace set the real dependencies. For example:
# ./package.json
{
...
"workspaces": ["packages/*"],
"dependencies": [
"some-lib": "^1.0.0",
]
}
# ./packages/pkg-a/package.json
{
...
"peerDependencies": [
"some-lib": "*",
]
}
and yarn install at workspace root.
3. Share script (require yarn v2)
See https://yarnpkg.com/getting-started/qa#how-to-share-scripts-between-workspaces
Miscellaneous
yarn provides workspace cli to add/remove package, see https://classic.yarnpkg.com/lang/en/docs/cli/workspace/
yarn workspace awesome-package add react react-dom --dev
yarn workspace web-project remove some-package
I have .terraform/modules folder generated by terraform itself.
It's where terraform keeps modules by default and I'm fine with that.
when running terraform init command and if .terraform folder is gone it will try to pull modules again I would like to avoid that step by saying to use pre-populated modules folder from different location - it's like building shared cache folder for terraform for our CI/CD pipelines, pull only if new version of a modules specified otherwise use cache.
NOTE:
We don't run anything on Jenkins locally, every `Stage` in Jenkins uses Ephemeral Docker
container agents to run all the `Steps` and to keep Jenkins clean,
otherwise I would use local workspace cache for all that.
is there a way to do that?
Thank you
I have configured and working following setup
gitlab-ci, which uses docker-machine runner and uploads cache to S3
maven build with configured caching
caching correctly loads and uploads on each job
But the problem is, that every time I run mvn install, something in the local maven repository changes (I assume it updates pom metadata) and gitlab runner keeps uploading new versions of the cache, on every single build.
It is still faster and more reliable to use this "busted" cache, than to download the deps from internet every time, but the upload can take a long time and I would like to shave off this extra time.
How can I modify my build to force maven, to generate cacheable local repository?
Simplified version of my .gitlab-ci.yml:
variables:
# we have a custom java+maven image, that uses this ENV variable,
# to auto-configure path where to put the local maven repository
MAVEN_LOCAL_REPOSITORY: $CI_PROJECT_DIR/.cache/maven
job-build:
stage: build
image: internal-gitlab/java/maven:3.6-jdk8-alpine
script:
- mvn -B clean package
cache:
key: backend-dependencies
paths:
- .cache/
You have a constant as a cache key. Maybe a more fine grained cache would help.
See the link here
In short - prepare your own maven image with required dependencies and use it instead of internal-gitlab/java/maven:3.6-jdk8-alpine.
Some details:
First of all, you need to create a maven docker image where all (or most of) required for your project dependencies are presented. Publish it to your registry (gitlab has one) and use it instead of internal-gitlab/java/maven:3.6-jdk8-alpine.
To create such an image I usually create an additional job in CI triggered manually. You need to trigger it at initial stage and when project dependencies are heavily modified.
Working sample can be found here:
https://gitlab.com/alexej.vlasov/syncer/blob/master/.gitlab-ci.yml
- this project is using the prepared image and also it has a job to prepare this image.
https://gitlab.com/alexej.vlasov/maven/blob/master/Dockerfile
- dockerfile to run maven and download dependencies once.
The pros:
don't need to download dependencies each time - they are inside a
docker image (and docker layers are cached on the runners)
don't need to upload artifacts when job is finished
I have a teamcity installation running on a windows server
I have a build process, that is building a .net application. As part of the build process it is running a gulp task, and is installing various node modules to build all the sass and js files.
Within my teamcity solution i have my clean up rules for "Everything" to be older that the 4th-th successful build
However teamcity does not appear to be cleaning everything up, because my artifacts folder is massive ( and caused the server to run out of disk space )
Now I believe that the issue is because one of the gulp task and all the node_modules being installed. Because of how nested node installs everything.
On windows machines, in most instances you cannot just delete a node_modules folder as you get the path longer than 256 char warning, and you have to use the robocopy trick of copying across an empty folder to delete the node_modules folder
What have others done to resolve this problem of artifacts not being correctly cleaned up ?
I was thinking about another step that would delete the node_modules folder from the artifact folder after the step to process the gulp tasks, but I cannot see in the list of teamcity parameters once that points to the artifact folder for the current build.
If the builds should be deleted by the server clean-up, but you still see the builds in TeamCity UI and they are not pinned, the most common reason for that is having other builds snapshot- or artifact-depending on the builds in question. Check build's Dependencies tab and related TeamCity settings.
I'm building a Docker image from this Dockerfile:
FROM maven:3.3.3-jdk-8
MAINTAINER Mickael BARON
ADD pom.xml /work/pom.xml
WORKDIR /work
RUN mvn dependency:go-offline --fail-never
ADD ["src", "/work/src"]
RUN ["mvn", "package"]
With this Dockerfile, I force to download the dependencies before packaging my Java project. Thus, I don't have to redownload the dependencies every time I changed a file from my src directory.
But, there is a problem and this problem is depending on the version of Maven (base image). In fact, the dependencies are downloaded but they are not persisted into the ~/.m2 directory of the container. It's empty. Thus, when I change some source file all the dependencies are redownloaded.
However, I noticed that if I change the version of Maven from the base image (for example FROM maven:3.2.5-jdk-8), it works.
Very strange, isn't it?
There is a new instruction regarding this topic:
https://github.com/carlossg/docker-maven#packaging-a-local-repository-with-the-image
The $MAVEN_CONFIG dir (default to /root/.m2) is configured as a volume so anything copied there in a Dockerfile at build time is lost. For that the dir /usr/share/maven/ref/ is created, and anything in there will be copied on container startup to $MAVEN_CONFIG.
To create a pre-packaged repository, create a pom.xml with the dependencies you need and use this in your Dockerfile. /usr/share/maven/ref/settings-docker.xml is a settings file that changes the local repository to /usr/share/maven/ref/repository, but you can use your own settings file as long as it uses /usr/share/maven/ref/repository as local repo.
I'm afraid it's because of this VOLUME instruction they've added:
https://github.com/carlossg/docker-maven/blame/8ab542b907e69c5269942bcc0915d8dffcc7e9fa/jdk-8/Dockerfile#L11
It makes /root/.m2 a volume and thus any changes to that folder made by build steps are not brought on to the following build containers.