While setting up a Jenkins CI system for the first time, I wonder how to deal with snapshot dependencies and the local Maven repository.
We have an artifact X with dependency Y. Jenkins is configured to build X everytime Y has a commit.
Strategy A (workspace Maven repository):
The local Maven repository is configured to be "local to the workspace". This means that every built job has its own repository. A commit of Y does trigger the build (a "mvn clean install"), but since the repositories of build X and Y are isolated, X uses the last deployed version of Y, not the last commit.
Using this strategy seems useful to detect if all dependencies are also deployed, but it doesn't detect integration issues (broken Y commit breaks X).
Strategy B (common Maven repository):
Jenkins uses a common local Maven repository. A commit of a (broken) Y snapshot triggers the build of X which uses the current Y snapshot, and integration issues are detected. The drawback is, that this doesn't detect if all build dependencies are deployed to the central snapshot repository (because the CI system does have them all installed in the local repository).
Is there a strategy that combines both advantages of A and B?
We ended up using a Strategy C for this:
Strategy C (let Jenkins deploy):
Every job has its local repository local to the workspace like in strategy A. But instead of the default "clean install" goal, Jenkins is configured to do a "-U clean deploy" on every job. So if Y is broken and commited, Jenkins will build it and deploy it to the central snapshot repository. Jenkins detects the build of the dependency Y and builds X with the latest Y snapshot.
Useful for this strategy is - at least with Maven 3 - to run a automatic cleanup job on the central snapshot repository, since there will be a lot of timestamped snapshots.
Related
We have a Jenkins build which in it's initial stage tries to determine the latest deployed version of some dependencies via version ranges, which are also built by that same Jenkins instance. Each build is for several artifacts, all with a shared version, so we want to select the latest version which has been completed. We do this by using a special pom which has dependencies on specific poms - these are deployed last by the other builds.
The issue is that if one of the other builds is running, but has not yet deployed the pom, maven's version range resolution winds up selecting the version for the build in progress and then fails as some of the dependencies haven't been built yet.
This isn't a downstream build kicked off by the dependency. In any case, we've tried the "Enable isolated resolution for downstream builds" and "Resolve artifacts from Artifactory" but neither have helped.
If the project is a normal UI "maven project" then there is a Use private Maven repository checkbox that uses the workspace specific maven repository cache
Normally, Jenkins uses the local Maven repository as determined by Maven — the exact process seems to be undocumented, but it's ~/.m2/repository and can be overridden by in ~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same node shares a single Maven repository. The upside of this is that you can save the disk space, but the downside of this is that sometimes those builds could interfere with each other. For example, you might end up having builds incorrectly succeed, just because your have all the dependencies in your local repository, despite that fact that none of the repositories in POM might have them.
There are also some reported problems regarding having concurrent
Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use
$WORKSPACE/.repository as the local Maven repository. This means each
job will get its own isolated Maven repository just for itself. It
fixes the above problems, at the expense of additional disk space
consumption.
When using this option, consider setting up a Maven artifact manager
so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed
on Jenkins, refer to the technique described here.
If it's a pipeline job they a mavenLocalRepo setting see: https://plugins.jenkins.io/pipeline-maven/
If it was a freestyle job, your only choice to do what that option does by hand
By using this option, Jenkins will tell Maven to use a custom path for the build as the local Maven repository by using -Dmaven.repo.local
If specified as a relative path then this value will be resolved against the workspace root and not the current working directory.
ie. $WORKSPACE/.repository if .repository value is specified.
The issue is due to the Jenkins maven local repository - it looks like in the current configuration this is shared between the builds. Thus, the in-flight build manages to pick up installed poms that haven't yet been deployed.
We solved this by adding -Dmaven.local.repo=${WORKSPACE}/.m2/repository for when we resolve the version ranges - this way the build is isolated from the other concurrent builds for this stage.
We are using Jenkins with multi-branch pipelines and the Maven Integration plugin. I configured the multi-branch pipeline to "Build whenever a SNAPSHOT dependency is built". When our dev branch was built, it seemed to build every branch of all dependent repos. How may we limit it to only build branch X when branch X is built?
I have a Java project. Some parts are jar files, some are war files. I also have Jenkins and Nexus Pro, whereby when a developer on the team commits to SVN, the Jenkins build automatically kicks off.
Using the Maven versions plugin, I am able to mvn versions:set -DnewVersion=1.0.$SVN_REVISION as a pre-build step, and then mvn clean test deploy. At the end of this process, I've got a my-artifact-1.0.1234.jar uploaded to my Nexus Pro Staging Repository.
Since we're working in a CI-type of environment, we might have a hundred (or more) staging builds. When the time is right, the QA team wants to promote a certain build to a "later" environment (think QA, or SIT, or whatever. Environments are more of a locked-down state here.)
The workflow that I want to have happen is this:
Someone decides that Build 1.0.1357 should be Promoted to QA
They go into Jenkins, go to the "Promote to QA" job
They're presented with a list of all possible builds in the Nexus Staging Repository in a drop-down. They select one, and click the "Run" button.
That artifact is "released" from Nexus Staging to Nexus Releases, and further deployed to the QA environment. (I'm not as concerned about the "and deployed to QA" part -- I know how to do that already. It's included here for completeness-of-my-story sake.)
I already know that I can do this from the command-line, and it's working:
mvn nexus-staging:rc-list -DserverId=nexus -DnexusUrl=http://my.nexus.ip:8081/nexus
mvn nexus-staging:rc-release -DserverId=nexus -DnexusUrl=http://my.nexus.ip:8081/nexus -DstagingRepositoryId=abcd-1000 -Ddescription="Release from CLI."
The problem I'm having is that you have to specify the stagingRepositoryId on the command-line. How might I go about accomplishing this?
What I was doing is parsing the output of
mvn nexus-staging:rc-list -DserverId=nexus -DnexusUrl=http://my.nexus.ip:8081/nexus
and then just match the needed repository with your specifique logic. Using python for me was the best solution (but you can do it on your own with any language):
output = subprocess.check_output("mvn nexus-staging:rc-list -DserverId=nexus -DnexusUrl=http://my.nexus.ip:8081/nexus")
for line in output.split('\n'):
if "repo" in line:
stagingRepositoryId = "repo-" + line[8:23]
Considering output as
[INFO] repo_qa-3514 OPEN Implicitly created (auto staging).
[INFO] repo_qa-3518 Implicitly created (auto staging).
[INFO] repo_qa-3521 OPEN Implicitly created (auto staging).
[INFO] repo-2011 OPEN Implicitly created (auto staging).
You will run the second command after parsing as:
mvn nexus-staging:rc-release -DserverId=nexus -DnexusUrl=http://my.nexus.ip:8081/nexus -DstagingRepositoryId=repo-2011 -Ddescription="Release from CLI."
I think what you are trying to do can be achieved readily by using SNAPSHOT and release repository.
So you do your normal development in a SNAPSHOT build and once you are ready for testing, you can create a tag build which removes SNAPSHOT from version in POM. All this can be achieved using jenkins and Nexus.
Also you dont need SVN revision number in your version, instead an incremental build number will be sufficient which can be managed via release plugin.
So to summarise:
Lets say you are working on release 1.0.
So you take a initial branch with pom version as 1.0-0-SNAPSHOT . Here 1.0 represents release number, '-0' represents next tag build number which we plan to deploy.
Now once you are ready for deployment or want your QA team to test. You run a job or a script with maven release plugin to create a tag. A tag build will be created with version 1.0-0 [Snapshot is removed so it will go to release repository] and uploaded to repository also version in branch will be incremented to 1.0-1-SNAPSHOT [So now changes in branch will be made to release and deploy 1.0-1 if any changes are needed]
All the above steps are automated using Maven release plugin and run via jenkins job.
I have the above setup for my work.
OP stated that 'SNAPSHOT' is not in picture.
In that case this post answers the query : automate deployment to sonatype's oss maven repository [Look at the second answer]
I have a web project split in multiple maven artefacts.
Let say it is :
A, a jar
B, a jar
C, a jar
D, a war
So, D has a dependency on B and C. B has a dependency on A.
Those modules versions are not always synchroneous.
A can be on version 3-SNAPSHOT while B is still in version 5 with a dependency on A version 2.
I configured jenkins to cascade build B when A is built, D when B or C is built.
Those modules also get rebuilt when a change is detected in the git repository.
Artefacts are automatically deployed in nexus repository by a post-build action.
So, if I push in git a new version 3-SNAPSHOT of A, a 3-SNAPSHOT jar is built and pushed in nexus. But, because of Jenkins dependency, a new build of B version 2 (release version) is triggered. The build itself goes ok, but the deploy to nexus fails, as I do not allow redeploy of release artefacts.
How can I avoid this situation ? Not trying to upload to nexus when the artefact has a release version and this version already exists in nexus would be acceptable.
I am using Jenkins 1.480 and maven 3.0.4.
You can trigger parameterized build with https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin and add a parameter to job B.
Conditional build plugin might also help https://wiki.jenkins-ci.org/display/JENKINS/Conditional+BuildStep+Plugin
My maven module A has a dependency on another maven module B provided by other people. When I run "mvn install" under A for the first time, maven downloads B-1.0.jar from a remote repository to my local maven repository. My module A builds fine.
In the mean time, other people are deploying newer B-1.0.jar to the remote repository. When I run "mvn install" under A again, maven does not download the newer B-1.0.jar from the remote repository to my local repository. As a result, my module A build fails due to API changes in B-1.0.jar.
I could manually delete B-1.0.jar from my local repository. Then maven would download the latest B-1.0.jar from the remote repository the next time when I run "mvn install".
My question is how I can automatically let maven download the latest artifacts from a remote repository. I tried to set updatePolicy to "always". But that did not do the trick.
Maven never re-downloads releases - 1.0 is considered final and new releases should use a new version.
If the module B is still under development, you should use the version 1.0-SNAPSHOT - snapshots are a special Maven version that will check for updates, and when deployed is stored with the timestamp and build number for tracking.
I agree with Brett, above: new releases should use new versions. For your case, snapshots are probably the best solution but something else that might also be helpful is to use dependency version ranges.
Thereby you can specify a version of
(1.0,)
stating that you accept any version greater than 1.0.
or
[1.1.1,1.1.7]
accepting anything between (including) versions 1.1.1 and 1.1.7.
The notation follows standard math interval syntax where
[ = inclusion in the interval
( = exclusion from the interval
(in school, I always thought of the square brackets as "holding" that element in, while the softer parenthesis "let it go")
This can be helpful in cases where your dependencies are still under frequent development and you don't want to rely on new snapshots that might be less stable and more likely to break your code. You can specify the safe ranges and adjust the boundaries up or down, as appropriate, over time