How to separate write access to Maven repositories in Jenkins? - maven

In order to prevent that one build influences another it is possible to configure a Jenkins project to use a own private Maven repository. However, because we have actually a huge list of dependencies, this leads to a lot of wasted disk space and to slow builds. We use a Maven repository proxy, but still the time to download artifacts over the local network is significant.
I could set up another repository proxy directly on the Jenkins machine. Is there an easier solution?
I still want that any "maven install" goes to a project-specific repo, while reading of artifacts that have not been deployed to that project-specific repo should come from a central place on the local file-system. Those artifacts should not be copied for performance and disk space reasons.
To explain the background I append the help text of the "Use private Maven repository" option:
"Normally, Jenkins uses the local Maven repository as determined by
Maven — the exact process seems to be undocumented, but it's
~/.m2/repository and can be overridden by in
~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same
node shares a single Maven repository. The upside of this is that you
can save the disk space, but the downside of this is that sometimes
those builds could interfere with each other. For example, you might
end up having builds incorrectly succeed, just because your have all
the dependencies in your local repository, despite that fact that none
of the repositories in POM might have them.
There are also some reported problems regarding having concurrent
Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use
$WORKSPACE/.repository as the local Maven repository. This means each
job will get its own isolated Maven repository just for itself. It
fixes the above problems, at the expense of additional disk space
consumption.
When using this option, consider setting up a Maven artifact manager
so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed
on Jenkins, refer to the technique described here."

You could have:
one settings.xml that points to a common local repository, used for every mvn clean package command
and one settings.xml per projet that uses a specific local repository for every mvn install command
In order you would:
mvn clean package -s settings-common.xml # using common-repo
mvn install -s settings-jobX.xml # using jobX-repo
The only issue is that the artifact installed by your job wouldn't be available to other jobs if they need it. You'd have to either deploy, or copy manually the artifact to the common-repo.
Please note that I do not understand fully what you mean by "one build influences another". You should clarify that in order to have a better answer (because what you want to do might not be what's best to do).

Related

Maven - dependency list and download artifacts

Is it possible to find all dependencies in a maven projects like mvn dependency:tree, save results to file and then use mvn command for download all dependencies from list file in command line?
thanks
In my experience the various plugins do not get a 100% coverage so the builds will fail. What worked for me, was to start with an empty .m2/repositories folder, and then do the maven commands you need to do, and then persist the now populated .m2/repositories folder.
I experimented with converting it to a local file repository in the project itself so we could do with offline builds, but this turned out to be too big for our purposes. It might not be that much of an obstacle today.
Unless you have very strict political requirements, the usual pragmatic solution is to have a local repository on your network like nexus or artifactory mirroring maven central which is then accessed as a mirror.

How can I use multiple Maven local repositories with precedence

I would like to know if there any workaround about having multiple local repositories used by order.
Example :
Default local repository : ~/.m2/repository
Work repository : /var/tmp/m2LocalRepo
When running Maven, to resolve dependencies I would like that at first it looks for the artifact in /var/tmp/m2LocalRepo if it does not find it it looks in the default one.
There is the issue MNG-3655 – Allow multiple local repositories which talks about the same problem but has not been resolved yet.
UPDATE
My use case is as follows:
An application is represented by 2 maven projects projectA and projectB which have the same version.
projectB is a dependency of projectA
when developing a new feature, we create a development branch on each project with the same name. However, we do not modify the version number which remains the same as the main branch.
During a build, on Jenkins for example, I would like to build projectB first but install its artifacts in another local repository (for example, /var/tmp/m2LocalRepo) and not on the default repository so as not to disrupt other builds that also depend on projectB. Then, during the build of projectA, I would like it to get the projectB dependency from /var/tmp/m2LocalRepo and the other dependencies from the local repository.
Re "During a build, on Jenkins for example":
Jenkins Maven projects have an option Build → Advanced... → ☑ Use private Maven repository with its inline help:
Normally, Jenkins uses the local Maven repository as determined by Maven — the exact process seems to be undocumented, but it's ~/.m2/repository and can be overridden by in ~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same node shares a single Maven repository. The upside of this is that you can save the disk space, but the downside of this is that sometimes those builds could interfere with each other. For example, you might end up having builds incorrectly succeed, just because your have all the dependencies in your local repository, despite that fact that none of the repositories in POM might have them.
There are also some reported problems regarding having concurrent Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use $WORKSPACE/.repository as the local Maven repository. This means each job will get its own isolated Maven repository just for itself. It fixes the above problems, at the expense of additional disk space consumption.
When using this option, consider setting up a Maven artifact manager so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed on Jenkins, refer to the technique described here.
and the options:
Default ...
Local to the executor
Local to the workspace
You can copy projectB's artifacts using the Maven Resources Plugin (as described in this answer to Maven, how to copy files?) to projectA's local repo on demand.

Maven build on Jenkins - upstream version range resolves to version being built in a concurrent job

We have a Jenkins build which in it's initial stage tries to determine the latest deployed version of some dependencies via version ranges, which are also built by that same Jenkins instance. Each build is for several artifacts, all with a shared version, so we want to select the latest version which has been completed. We do this by using a special pom which has dependencies on specific poms - these are deployed last by the other builds.
The issue is that if one of the other builds is running, but has not yet deployed the pom, maven's version range resolution winds up selecting the version for the build in progress and then fails as some of the dependencies haven't been built yet.
This isn't a downstream build kicked off by the dependency. In any case, we've tried the "Enable isolated resolution for downstream builds" and "Resolve artifacts from Artifactory" but neither have helped.
If the project is a normal UI "maven project" then there is a Use private Maven repository checkbox that uses the workspace specific maven repository cache
Normally, Jenkins uses the local Maven repository as determined by Maven — the exact process seems to be undocumented, but it's ~/.m2/repository and can be overridden by in ~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same node shares a single Maven repository. The upside of this is that you can save the disk space, but the downside of this is that sometimes those builds could interfere with each other. For example, you might end up having builds incorrectly succeed, just because your have all the dependencies in your local repository, despite that fact that none of the repositories in POM might have them.
There are also some reported problems regarding having concurrent
Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use
$WORKSPACE/.repository as the local Maven repository. This means each
job will get its own isolated Maven repository just for itself. It
fixes the above problems, at the expense of additional disk space
consumption.
When using this option, consider setting up a Maven artifact manager
so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed
on Jenkins, refer to the technique described here.
If it's a pipeline job they a mavenLocalRepo setting see: https://plugins.jenkins.io/pipeline-maven/
If it was a freestyle job, your only choice to do what that option does by hand
By using this option, Jenkins will tell Maven to use a custom path for the build as the local Maven repository by using -Dmaven.repo.local
If specified as a relative path then this value will be resolved against the workspace root and not the current working directory.
ie. $WORKSPACE/.repository if .repository value is specified.
The issue is due to the Jenkins maven local repository - it looks like in the current configuration this is shared between the builds. Thus, the in-flight build manages to pick up installed poms that haven't yet been deployed.
We solved this by adding -Dmaven.local.repo=${WORKSPACE}/.m2/repository for when we resolve the version ranges - this way the build is isolated from the other concurrent builds for this stage.

Jenkins CI server and Nexus Server on the same Box

I am in a situation where I have one Build Server box which is to carry out all continuous integration and manage our maven repository. The box works as follows:
There is one maven repository which is hosted through Apache Server as a URL for developers to use
All Jenkins jobs (including release jobs) run mvn install so that artifacts are kept in this one repository.
I would like to get rid of the Apache server and run Nexus on this same box to manage and host repositories, however I have the following questions/ideas:
With Nexus and Jenkins on the same box, will it mean that I will have to manage two repositories, one where maven installs an artifact to a local repository, and one where maven deploys an artifact to nexus? Would it be possible to have Nexus manage the "mvn install" repository also? How can I make sure we don't run out of disk space on the server very very quickly all the time?
Thanks
Added as response to comments: Thank you both, I am thinking I will just set the Jenkins jobs and release plugin goals to mvn package deploy:deploy in order to skip the install phase, that way, artifacts go directly from the target directory to Nexus. However I guess the Jenkins job will require a local repository from which to use depedencies which will get copied from Nexus to the maven local repository during the build, I am not sure if this can be avoided though.
mvn install installs in the local repository
mvn deploy installs to the remote repository
these semantics are defined in the lifecycle and map to different plugins. Their implementations are different.
You don't have to manage the local repository. Actually for some if not most jobs you might even want to define it localized to the job (with the 'Use private Maven repository' option) instead of to the user who is running the job, especially that you plan to use nexus for repository.
You will have to change your jobs to use mvn deploy instead.
How can I make sure we don't run out of disk space on the server very
very quickly all the time?
Configure Jenkins/Nexus. Discard old builds and disable automatic artifact archiving. Both settings can be found in the Jenkins job-configuration. Also you could delete old artifacts automatically from Nexus using Scheduling Tasks.
There is no need to install the artifacts into the local maven repository when using Jenkins/Nexus on a dedicated server.

Maven WAR overlay problems, while using Hudson + Artifactory

We have three artifacts:
common.jar : with common classes.
public.war : depending on the common.jar, contains only public site resources.
internal.war : depends on both common.jar and public.war, adding authentication
information and security context resource files. Also contains
few administration site classes.
Currently I have structured these in such way, that internal.war overlays itself with public.war.
Building the project locally, installing the artifacts to local repo, works perfectly.
Problems start when trying to get the Hudson builds working with following sequence:
Build all projects in dependency order.
Modify common.jar (say, add a new class method)
Modify internal.war classes in such way that they are compile-time dependent on changes done in 2. step.
Commit both changes, triggering the Hudson builds.
Internal.war build fails because it can not find the symbols added in step 2.
Somehow the build in step 5. is using an old version of the common.jar, and failing because of it.
The common.jar version number does not change, let's say it's 1.0.0-SNAPSHOT for the purposes of this example.
If I DO change the common.jar version number, the build works. (Supposedly because there is only one release by a release version number).
Now, what could cause this using of old artifacts in Hudson builds?
We are running maven builds on Hudson with command "clean package -e -X -U"
"Deploy artifacts to maven repository" has been checked.
It's hard to definitively answer this without access to the real poms, but here is what I would do:
1) Make sure Hudson is using the exact same version of Maven as you are on your local machine
2) Examine the effective pom.xml of internal.war on the Hudson machine in a terminal via mvn help:effective-pom making sure you are running the same mvn executable as your Hudson job does. You need to verify the version of the common.jar in the effective pom.xml of internal.war. It could be different than what you expect due to profiles or settings.xml differences.
3) Check the settings.xml file for your Hudson install of Maven. In particular you need to verify all is well in your distributionManagement, servers, and repositories stanzas. Another good way to check this is to go to your internal.war project and run mvn help:effective-settings and see if what is there matches what is on your local machine.
Something is awry and it won't take long to find with the right analysis.

Resources