Is Maven safe for concurrent execution? i.e. can I have several
mvn compile
command running in separate directories at the same time, each of which may be updating $HOME/.m2?
The local repository is NOT safe for concurrent maven instances (see link: https://issues.apache.org/jira/plugins/servlet/mobile#issue/MNG-2802 )
This is a common problem on Continuous Build/Integration systems.
Although at first glace this doesn't affect the 'compile' phase, as the compile phase depends on dependency resolution and the updating of the local repository with downloaded artifacts it is still an issue.
Related
Looking for a way to download into cache all the dependencies enumerated in verification-metadata.xml of a Gradle project without actual executing of the tasks requiring those dependencies.
For instance, if the dependencies cache is empty and we run gradle assembly, it would cause downloading artifacts required for "assembly" and all previous tasks. Since some tasks could take several minutes to end, it would be good to avoid the execution itself and just stop after downloading the dependencies.
I would like to know if there any workaround about having multiple local repositories used by order.
Example :
Default local repository : ~/.m2/repository
Work repository : /var/tmp/m2LocalRepo
When running Maven, to resolve dependencies I would like that at first it looks for the artifact in /var/tmp/m2LocalRepo if it does not find it it looks in the default one.
There is the issue MNG-3655 – Allow multiple local repositories which talks about the same problem but has not been resolved yet.
UPDATE
My use case is as follows:
An application is represented by 2 maven projects projectA and projectB which have the same version.
projectB is a dependency of projectA
when developing a new feature, we create a development branch on each project with the same name. However, we do not modify the version number which remains the same as the main branch.
During a build, on Jenkins for example, I would like to build projectB first but install its artifacts in another local repository (for example, /var/tmp/m2LocalRepo) and not on the default repository so as not to disrupt other builds that also depend on projectB. Then, during the build of projectA, I would like it to get the projectB dependency from /var/tmp/m2LocalRepo and the other dependencies from the local repository.
Re "During a build, on Jenkins for example":
Jenkins Maven projects have an option Build → Advanced... → ☑ Use private Maven repository with its inline help:
Normally, Jenkins uses the local Maven repository as determined by Maven — the exact process seems to be undocumented, but it's ~/.m2/repository and can be overridden by in ~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same node shares a single Maven repository. The upside of this is that you can save the disk space, but the downside of this is that sometimes those builds could interfere with each other. For example, you might end up having builds incorrectly succeed, just because your have all the dependencies in your local repository, despite that fact that none of the repositories in POM might have them.
There are also some reported problems regarding having concurrent Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use $WORKSPACE/.repository as the local Maven repository. This means each job will get its own isolated Maven repository just for itself. It fixes the above problems, at the expense of additional disk space consumption.
When using this option, consider setting up a Maven artifact manager so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed on Jenkins, refer to the technique described here.
and the options:
Default ...
Local to the executor
Local to the workspace
You can copy projectB's artifacts using the Maven Resources Plugin (as described in this answer to Maven, how to copy files?) to projectA's local repo on demand.
We have a Jenkins build which in it's initial stage tries to determine the latest deployed version of some dependencies via version ranges, which are also built by that same Jenkins instance. Each build is for several artifacts, all with a shared version, so we want to select the latest version which has been completed. We do this by using a special pom which has dependencies on specific poms - these are deployed last by the other builds.
The issue is that if one of the other builds is running, but has not yet deployed the pom, maven's version range resolution winds up selecting the version for the build in progress and then fails as some of the dependencies haven't been built yet.
This isn't a downstream build kicked off by the dependency. In any case, we've tried the "Enable isolated resolution for downstream builds" and "Resolve artifacts from Artifactory" but neither have helped.
If the project is a normal UI "maven project" then there is a Use private Maven repository checkbox that uses the workspace specific maven repository cache
Normally, Jenkins uses the local Maven repository as determined by Maven — the exact process seems to be undocumented, but it's ~/.m2/repository and can be overridden by in ~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same node shares a single Maven repository. The upside of this is that you can save the disk space, but the downside of this is that sometimes those builds could interfere with each other. For example, you might end up having builds incorrectly succeed, just because your have all the dependencies in your local repository, despite that fact that none of the repositories in POM might have them.
There are also some reported problems regarding having concurrent
Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use
$WORKSPACE/.repository as the local Maven repository. This means each
job will get its own isolated Maven repository just for itself. It
fixes the above problems, at the expense of additional disk space
consumption.
When using this option, consider setting up a Maven artifact manager
so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed
on Jenkins, refer to the technique described here.
If it's a pipeline job they a mavenLocalRepo setting see: https://plugins.jenkins.io/pipeline-maven/
If it was a freestyle job, your only choice to do what that option does by hand
By using this option, Jenkins will tell Maven to use a custom path for the build as the local Maven repository by using -Dmaven.repo.local
If specified as a relative path then this value will be resolved against the workspace root and not the current working directory.
ie. $WORKSPACE/.repository if .repository value is specified.
The issue is due to the Jenkins maven local repository - it looks like in the current configuration this is shared between the builds. Thus, the in-flight build manages to pick up installed poms that haven't yet been deployed.
We solved this by adding -Dmaven.local.repo=${WORKSPACE}/.m2/repository for when we resolve the version ranges - this way the build is isolated from the other concurrent builds for this stage.
In order to prevent that one build influences another it is possible to configure a Jenkins project to use a own private Maven repository. However, because we have actually a huge list of dependencies, this leads to a lot of wasted disk space and to slow builds. We use a Maven repository proxy, but still the time to download artifacts over the local network is significant.
I could set up another repository proxy directly on the Jenkins machine. Is there an easier solution?
I still want that any "maven install" goes to a project-specific repo, while reading of artifacts that have not been deployed to that project-specific repo should come from a central place on the local file-system. Those artifacts should not be copied for performance and disk space reasons.
To explain the background I append the help text of the "Use private Maven repository" option:
"Normally, Jenkins uses the local Maven repository as determined by
Maven — the exact process seems to be undocumented, but it's
~/.m2/repository and can be overridden by in
~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same
node shares a single Maven repository. The upside of this is that you
can save the disk space, but the downside of this is that sometimes
those builds could interfere with each other. For example, you might
end up having builds incorrectly succeed, just because your have all
the dependencies in your local repository, despite that fact that none
of the repositories in POM might have them.
There are also some reported problems regarding having concurrent
Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use
$WORKSPACE/.repository as the local Maven repository. This means each
job will get its own isolated Maven repository just for itself. It
fixes the above problems, at the expense of additional disk space
consumption.
When using this option, consider setting up a Maven artifact manager
so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed
on Jenkins, refer to the technique described here."
You could have:
one settings.xml that points to a common local repository, used for every mvn clean package command
and one settings.xml per projet that uses a specific local repository for every mvn install command
In order you would:
mvn clean package -s settings-common.xml # using common-repo
mvn install -s settings-jobX.xml # using jobX-repo
The only issue is that the artifact installed by your job wouldn't be available to other jobs if they need it. You'd have to either deploy, or copy manually the artifact to the common-repo.
Please note that I do not understand fully what you mean by "one build influences another". You should clarify that in order to have a better answer (because what you want to do might not be what's best to do).
I'm making a case for moving our builds from Maven to Gradle. Below are a few of the Maven command-line options my team finds useful. What are the Gradle equivalent choices?
-am,--also-makeIf project list is specified, also build projects required by the list
-amd,--also-make-dependentsIf project list is specified, also build projects that depend on projects on the list
-o,--offline Work offline
-pl,--projects Build specified reactor projects
instead of all projects
-rf,--resume-from Resume reactor from specified project
Maven Examples:
I only want to build the sub-project I'm working on and its dependencies.
mvn install --also-makeIf --projects :my-sub-project
After fixing an build issue, I want to start the build from the point of failure.
mvn install --resume-from :my-sub-project
I don't want to download external dependencies from an central repo.
mvn install --offline
Here are some rough analogues:
-am: buildNeeded (This triggers a full build of all upstream projects; building those parts of upstream projects that are required to fulfill the command at hand is automatic in Gradle.)
-amd: buildDependents
-o: --offline
-pl: :subproject1:build :subproject2:build
-rf: No direct analogue (not reliable, wouldn't work for parallel builds, etc.), but Gradle's incremental build will get you to the "resume point" quickly.
Note that Gradle's core concepts differ significantly from Maven's. To give one example, in Gradle build order is solely determined by task relationships, and there is no such concept as an execution dependency between projects. Due to these differences, some Maven features aren't necessary or useful in Gradle, some you get for free, and some come in a different form.