I would like to know if there any workaround about having multiple local repositories used by order.
Example :
Default local repository : ~/.m2/repository
Work repository : /var/tmp/m2LocalRepo
When running Maven, to resolve dependencies I would like that at first it looks for the artifact in /var/tmp/m2LocalRepo if it does not find it it looks in the default one.
There is the issue MNG-3655 – Allow multiple local repositories which talks about the same problem but has not been resolved yet.
UPDATE
My use case is as follows:
An application is represented by 2 maven projects projectA and projectB which have the same version.
projectB is a dependency of projectA
when developing a new feature, we create a development branch on each project with the same name. However, we do not modify the version number which remains the same as the main branch.
During a build, on Jenkins for example, I would like to build projectB first but install its artifacts in another local repository (for example, /var/tmp/m2LocalRepo) and not on the default repository so as not to disrupt other builds that also depend on projectB. Then, during the build of projectA, I would like it to get the projectB dependency from /var/tmp/m2LocalRepo and the other dependencies from the local repository.
Re "During a build, on Jenkins for example":
Jenkins Maven projects have an option Build → Advanced... → ☑ Use private Maven repository with its inline help:
Normally, Jenkins uses the local Maven repository as determined by Maven — the exact process seems to be undocumented, but it's ~/.m2/repository and can be overridden by in ~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same node shares a single Maven repository. The upside of this is that you can save the disk space, but the downside of this is that sometimes those builds could interfere with each other. For example, you might end up having builds incorrectly succeed, just because your have all the dependencies in your local repository, despite that fact that none of the repositories in POM might have them.
There are also some reported problems regarding having concurrent Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use $WORKSPACE/.repository as the local Maven repository. This means each job will get its own isolated Maven repository just for itself. It fixes the above problems, at the expense of additional disk space consumption.
When using this option, consider setting up a Maven artifact manager so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed on Jenkins, refer to the technique described here.
and the options:
Default ...
Local to the executor
Local to the workspace
You can copy projectB's artifacts using the Maven Resources Plugin (as described in this answer to Maven, how to copy files?) to projectA's local repo on demand.
Related
In Jenkins, we configured different maven jobs. Each of those jobs installs their result build artefact to one of some local .m2 repositories being different to the default local .m2 repository (e. g. .m2-dev, specified in the settings.xml that are specified by the -s flag in the job's maven goals).
Now, when using the "Build whenever a SNAPSHOT dependency is built" option, it seems that dependent builds are only triggered if content (.jar) changes in the default .m2 repository are detected.
My question: Is it possible to specify an alternative .m2 settings file or repository location for a job that the "Build whenever a SNAPSHOT dependency is built" will be aware of?
Thank you for any hints or alternative suggestions.
We have a Jenkins build which in it's initial stage tries to determine the latest deployed version of some dependencies via version ranges, which are also built by that same Jenkins instance. Each build is for several artifacts, all with a shared version, so we want to select the latest version which has been completed. We do this by using a special pom which has dependencies on specific poms - these are deployed last by the other builds.
The issue is that if one of the other builds is running, but has not yet deployed the pom, maven's version range resolution winds up selecting the version for the build in progress and then fails as some of the dependencies haven't been built yet.
This isn't a downstream build kicked off by the dependency. In any case, we've tried the "Enable isolated resolution for downstream builds" and "Resolve artifacts from Artifactory" but neither have helped.
If the project is a normal UI "maven project" then there is a Use private Maven repository checkbox that uses the workspace specific maven repository cache
Normally, Jenkins uses the local Maven repository as determined by Maven — the exact process seems to be undocumented, but it's ~/.m2/repository and can be overridden by in ~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same node shares a single Maven repository. The upside of this is that you can save the disk space, but the downside of this is that sometimes those builds could interfere with each other. For example, you might end up having builds incorrectly succeed, just because your have all the dependencies in your local repository, despite that fact that none of the repositories in POM might have them.
There are also some reported problems regarding having concurrent
Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use
$WORKSPACE/.repository as the local Maven repository. This means each
job will get its own isolated Maven repository just for itself. It
fixes the above problems, at the expense of additional disk space
consumption.
When using this option, consider setting up a Maven artifact manager
so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed
on Jenkins, refer to the technique described here.
If it's a pipeline job they a mavenLocalRepo setting see: https://plugins.jenkins.io/pipeline-maven/
If it was a freestyle job, your only choice to do what that option does by hand
By using this option, Jenkins will tell Maven to use a custom path for the build as the local Maven repository by using -Dmaven.repo.local
If specified as a relative path then this value will be resolved against the workspace root and not the current working directory.
ie. $WORKSPACE/.repository if .repository value is specified.
The issue is due to the Jenkins maven local repository - it looks like in the current configuration this is shared between the builds. Thus, the in-flight build manages to pick up installed poms that haven't yet been deployed.
We solved this by adding -Dmaven.local.repo=${WORKSPACE}/.m2/repository for when we resolve the version ranges - this way the build is isolated from the other concurrent builds for this stage.
In order to prevent that one build influences another it is possible to configure a Jenkins project to use a own private Maven repository. However, because we have actually a huge list of dependencies, this leads to a lot of wasted disk space and to slow builds. We use a Maven repository proxy, but still the time to download artifacts over the local network is significant.
I could set up another repository proxy directly on the Jenkins machine. Is there an easier solution?
I still want that any "maven install" goes to a project-specific repo, while reading of artifacts that have not been deployed to that project-specific repo should come from a central place on the local file-system. Those artifacts should not be copied for performance and disk space reasons.
To explain the background I append the help text of the "Use private Maven repository" option:
"Normally, Jenkins uses the local Maven repository as determined by
Maven — the exact process seems to be undocumented, but it's
~/.m2/repository and can be overridden by in
~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same
node shares a single Maven repository. The upside of this is that you
can save the disk space, but the downside of this is that sometimes
those builds could interfere with each other. For example, you might
end up having builds incorrectly succeed, just because your have all
the dependencies in your local repository, despite that fact that none
of the repositories in POM might have them.
There are also some reported problems regarding having concurrent
Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use
$WORKSPACE/.repository as the local Maven repository. This means each
job will get its own isolated Maven repository just for itself. It
fixes the above problems, at the expense of additional disk space
consumption.
When using this option, consider setting up a Maven artifact manager
so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed
on Jenkins, refer to the technique described here."
You could have:
one settings.xml that points to a common local repository, used for every mvn clean package command
and one settings.xml per projet that uses a specific local repository for every mvn install command
In order you would:
mvn clean package -s settings-common.xml # using common-repo
mvn install -s settings-jobX.xml # using jobX-repo
The only issue is that the artifact installed by your job wouldn't be available to other jobs if they need it. You'd have to either deploy, or copy manually the artifact to the common-repo.
Please note that I do not understand fully what you mean by "one build influences another". You should clarify that in order to have a better answer (because what you want to do might not be what's best to do).
Forgive me if this is remedial, but I am still new to Maven and it's functionality.
In my project, when it "builds" and gets to the compile phase, it will create a target directory with just compiled libraries and update (or create if not there) the local .m2 directory.
When I get to the "test" phase, I want it to build against the target directory's library files, and not the local .m2 directory.
Any hints, recommendations, or suggests would be greatly appreciated. Thanks!
Maven has this concept of “the reactor”, which is just a fancy term for the list of projects being built. At the start of a Maven build, and at the end, Maven prints out this list of projects (using /project/name if defined or groupId:artifactId otherwise).
For each project in the reactor, Maven maintains a list of artifacts that have been attached. By default, each module's pom.xml is attached, and as each plugin runs, they have the option of attaching additional artifacts. Most plugins do not attach artifacts, here are some plugins that do:
jar:jar creates a .jar and attaches it
war:war creates a .war and attaches it
source:jar creates a .jar of the source Java code and attaches it with a classifier of source
java doc:jar creates a .jar of the JavaDocs ad attaches it with a classifier of javadoc
There is also a default primary artifact (this is the one that gets replaced by jar:jar) which is actually a directory and not a file, as such it will not get installed or deployed to the local repository cache or a remote repository.
So when in the reactor, and a plugin that attaches the primary artifact has not run yet, and another plugin asks for the primary artifact, it will be given the directory ${project.build.outputDirectory}. If after the primary artifact as been attached, then that primary artifact will be provided.
The test phase happens before the package phase, so will use the directory and not the .jar. The integation-test phase happens after, so will always use the .jar.
Things get more complex in a multi-module project (which is where my long intro should help you out)
Maven has to build the test classpath. If one of the dependencies is within the reactor, Maven will use the artifact attached to the reactor. Otherwise it will use the local cache (populating from the remote repositories if necessary).
When you run
mvn test
In a multimdule project from the root, there is no replacement of the default (directory-based) artifact, so intra-module classpath will be to the target/classes directories.
When you run
mvn package
In the same project, however, because each module completes its life cycle sequentially, all the dependent modules will have swapped in their .jar files as their attached artifact.
All of this should show you that Maven is doing the sensible thing. Hope this has helped.
The test phase is going to execute tests on your project. The project won't reference itself via the dependency mechanism. Only dependencies will be referenced via your local repository, i.e. .m2/repository
Also, it's not the compile phase that installs the artifact to the local repository, it's the install phase. And, then, there's a later phase, called deploy, that will deploy the artifact to a remote repository, provided you have a remote repository configured as the deploy target. Note, install and deploy are nearly identical phases except install is a local only thing; thus, it's the common build phase to hit when doing dev environment work. Normally the build server will do the deploy stuff.
We have three artifacts:
common.jar : with common classes.
public.war : depending on the common.jar, contains only public site resources.
internal.war : depends on both common.jar and public.war, adding authentication
information and security context resource files. Also contains
few administration site classes.
Currently I have structured these in such way, that internal.war overlays itself with public.war.
Building the project locally, installing the artifacts to local repo, works perfectly.
Problems start when trying to get the Hudson builds working with following sequence:
Build all projects in dependency order.
Modify common.jar (say, add a new class method)
Modify internal.war classes in such way that they are compile-time dependent on changes done in 2. step.
Commit both changes, triggering the Hudson builds.
Internal.war build fails because it can not find the symbols added in step 2.
Somehow the build in step 5. is using an old version of the common.jar, and failing because of it.
The common.jar version number does not change, let's say it's 1.0.0-SNAPSHOT for the purposes of this example.
If I DO change the common.jar version number, the build works. (Supposedly because there is only one release by a release version number).
Now, what could cause this using of old artifacts in Hudson builds?
We are running maven builds on Hudson with command "clean package -e -X -U"
"Deploy artifacts to maven repository" has been checked.
It's hard to definitively answer this without access to the real poms, but here is what I would do:
1) Make sure Hudson is using the exact same version of Maven as you are on your local machine
2) Examine the effective pom.xml of internal.war on the Hudson machine in a terminal via mvn help:effective-pom making sure you are running the same mvn executable as your Hudson job does. You need to verify the version of the common.jar in the effective pom.xml of internal.war. It could be different than what you expect due to profiles or settings.xml differences.
3) Check the settings.xml file for your Hudson install of Maven. In particular you need to verify all is well in your distributionManagement, servers, and repositories stanzas. Another good way to check this is to go to your internal.war project and run mvn help:effective-settings and see if what is there matches what is on your local machine.
Something is awry and it won't take long to find with the right analysis.