We have several jobs on jenkins that are running on hierarchical dependencies on each other.
Lets say, Job2 has a dependencies on the Job1 for the pom.xml version.
so whenever we have changes at the end of sprint for Job1, we are in need to change the versions of the pom on the dependencies inside the Job2 before releasing the artifacts at the end of sprint, so this might not handy if we have Job3, Job4, Job5 that are depending on the version of Job1. we need to change all the version of Job1 on all the pom.xml of all the Jobs that is dependent on it.
So the question, are there any ways to solve this, probably using jenkins env variables? need a helping hand how to perform it based on the issues above.
Thanks.
You can use jenkins Parameterized Trigger Plugin for passing variables to the downstream projects. Plugin wiki got detailed explanation.
So you have dependencies amongst your internally developed artifacts. During development you want to depend on the latest snapshot of those artifacts, but a release build shall depend on the latest release version of those same artifacts.
This can be achieved with Jenkins and Maven. Set up your release build job like so:
Add a "Pre Step" section of type "Invoke top level Maven targets". In this step you run the targets versions:update-properties scm:checkin to update your internal dependencies to point to the latest release version.
In the main build step do the release: release:prepare release:perform -B. This builds the release version, increments the version number to the next snapshot version, builds that next snapshot version and checks this back into scm.
In a "Post Step" run the targets of the Pre Step again (with allowSnapshots=true) to update your dependencies to reference the latest snapshot version.
Examples:
NOTE: For this to work all jobs have to hit the same Maven repository. Locally on the Jenkins server or your corporate Nexus.
Since you will have the required version at the end of execution of Job1, export it as environment variable or store it in some of the file on build server like
pom_version=1.1
Now when triggering all the downstream jobs, set the parameter of jobs as "pom_version" and either pass this file to pick the required key/value or set the value in pre-defined parameter.
After that, make sure all of your downstream jobs are configured as parameterized with parameter "pom_version"
Related
We have a Jenkins build which in it's initial stage tries to determine the latest deployed version of some dependencies via version ranges, which are also built by that same Jenkins instance. Each build is for several artifacts, all with a shared version, so we want to select the latest version which has been completed. We do this by using a special pom which has dependencies on specific poms - these are deployed last by the other builds.
The issue is that if one of the other builds is running, but has not yet deployed the pom, maven's version range resolution winds up selecting the version for the build in progress and then fails as some of the dependencies haven't been built yet.
This isn't a downstream build kicked off by the dependency. In any case, we've tried the "Enable isolated resolution for downstream builds" and "Resolve artifacts from Artifactory" but neither have helped.
If the project is a normal UI "maven project" then there is a Use private Maven repository checkbox that uses the workspace specific maven repository cache
Normally, Jenkins uses the local Maven repository as determined by Maven — the exact process seems to be undocumented, but it's ~/.m2/repository and can be overridden by in ~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same node shares a single Maven repository. The upside of this is that you can save the disk space, but the downside of this is that sometimes those builds could interfere with each other. For example, you might end up having builds incorrectly succeed, just because your have all the dependencies in your local repository, despite that fact that none of the repositories in POM might have them.
There are also some reported problems regarding having concurrent
Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use
$WORKSPACE/.repository as the local Maven repository. This means each
job will get its own isolated Maven repository just for itself. It
fixes the above problems, at the expense of additional disk space
consumption.
When using this option, consider setting up a Maven artifact manager
so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed
on Jenkins, refer to the technique described here.
If it's a pipeline job they a mavenLocalRepo setting see: https://plugins.jenkins.io/pipeline-maven/
If it was a freestyle job, your only choice to do what that option does by hand
By using this option, Jenkins will tell Maven to use a custom path for the build as the local Maven repository by using -Dmaven.repo.local
If specified as a relative path then this value will be resolved against the workspace root and not the current working directory.
ie. $WORKSPACE/.repository if .repository value is specified.
The issue is due to the Jenkins maven local repository - it looks like in the current configuration this is shared between the builds. Thus, the in-flight build manages to pick up installed poms that haven't yet been deployed.
We solved this by adding -Dmaven.local.repo=${WORKSPACE}/.m2/repository for when we resolve the version ranges - this way the build is isolated from the other concurrent builds for this stage.
When we do a release we want to fail the release if any of the dependent apps are still in SNAPSHOT version in the pom. For normal builds this should be allowed.
I guess there might be 2 options:
Is there a maven switch for the jenkins maven plugin to specify such and option?
Run a bash script to check for "SNAPSHOT" string in pom.xml, but then how can I detect "if this is a release" inside a jenkins job?
Thanks.
You can use Maven Release Plugin to perform a release, it will fail the release in case of SNAPSHOT dependencies.
You also can specify checking for timestamped SNAPSHOT dependencies, by default it is false.
How can I restrict automatically running dependent build configurations?
I have a pipeline:
Build
Stage
Release
Those are different configurations chained using snapshot dependencies. However, by default when a Teamcity configuration is run, it checks all the snapshot dependencies and then re-builds those which are not suitable.
Instead, for example, I want the Stage configuration to fail when no suitable Build snapshot dependency is found. (i.e. it should be impossible to run the Stage build without there first being a Build ready).
All I can find in Teamcity is configuration about how to handle situations where the dependency build fails, which is not what I need.
Thanks
When you declare a new Snapshot dependency, inside the options you have a dropdownlist for "On failed dependency"
Default value is: Run build, but add problem
Instead, you can set: Make build failed to start, or Cancel build
We're building feature branches into a version 'feature_'. Each feature build will produce the same version. Since these are no releases, I want to deploy the artefacts into the Snapshots repo, but Nexus does not allow versions without 'SNAPSHOT' into the Snapshot repo.
How to configure Nexus to allow any version in a repo?
Solved it by appending '-SNAPSHOT' to the branch version.
It's quite tricky to get Maven in Jenkins to use the right version. The way I solved it now is like this. In the build job configure Git to build the branches origin/feature/*. Then:
In the 'build' section, first thing to do is execute a shell command to construct a file 'env.properties' containing the feature branch version to be used by the Maven build command.
echo BRANCH_VERSION="feature_${GIT_BRANCH##*/}-SNAPSHOT" > env.properties
This uses the GIT_BRANCH environment property of Jenkins. The '##*/' is a Bash Shell Parameter Expansion which strips everything from the parameter value except the part after the last '/' character.
Then use the Environment Injector Plugin to 'inject environment variables' to the build job using the 'env.properties' created in the previous step.
Put 'env.properties' in the 'Properties File Path' field.
Use Maven to build a versioned pom with the correct version using 'Invoke top-level Maven targets':
help:effective-pom -Dbuild.number=${BRANCH_VERSION} -Doutput=versioned-pom.xml. This step is necessary because otherwise the pom in the jar artefact does not contain the correct version causing other problems.
Use another 'Invoke top-level Maven targets' step to do the actual build and deploy using the pom version created in the previous step: -f versioned-pom.xml clean source:jar deploy
That's all folks. If anyone knows a simpler solution, let me know.
This is actually a Maven restriction. You can still use version like feature_x-1.2.3-SNAPSHOT though.
What are you actually trying to achieve though?
I have a Java project. Some parts are jar files, some are war files. I also have Jenkins and Nexus Pro, whereby when a developer on the team commits to SVN, the Jenkins build automatically kicks off.
Using the Maven versions plugin, I am able to mvn versions:set -DnewVersion=1.0.$SVN_REVISION as a pre-build step, and then mvn clean test deploy. At the end of this process, I've got a my-artifact-1.0.1234.jar uploaded to my Nexus Pro Staging Repository.
Since we're working in a CI-type of environment, we might have a hundred (or more) staging builds. When the time is right, the QA team wants to promote a certain build to a "later" environment (think QA, or SIT, or whatever. Environments are more of a locked-down state here.)
The workflow that I want to have happen is this:
Someone decides that Build 1.0.1357 should be Promoted to QA
They go into Jenkins, go to the "Promote to QA" job
They're presented with a list of all possible builds in the Nexus Staging Repository in a drop-down. They select one, and click the "Run" button.
That artifact is "released" from Nexus Staging to Nexus Releases, and further deployed to the QA environment. (I'm not as concerned about the "and deployed to QA" part -- I know how to do that already. It's included here for completeness-of-my-story sake.)
I already know that I can do this from the command-line, and it's working:
mvn nexus-staging:rc-list -DserverId=nexus -DnexusUrl=http://my.nexus.ip:8081/nexus
mvn nexus-staging:rc-release -DserverId=nexus -DnexusUrl=http://my.nexus.ip:8081/nexus -DstagingRepositoryId=abcd-1000 -Ddescription="Release from CLI."
The problem I'm having is that you have to specify the stagingRepositoryId on the command-line. How might I go about accomplishing this?
What I was doing is parsing the output of
mvn nexus-staging:rc-list -DserverId=nexus -DnexusUrl=http://my.nexus.ip:8081/nexus
and then just match the needed repository with your specifique logic. Using python for me was the best solution (but you can do it on your own with any language):
output = subprocess.check_output("mvn nexus-staging:rc-list -DserverId=nexus -DnexusUrl=http://my.nexus.ip:8081/nexus")
for line in output.split('\n'):
if "repo" in line:
stagingRepositoryId = "repo-" + line[8:23]
Considering output as
[INFO] repo_qa-3514 OPEN Implicitly created (auto staging).
[INFO] repo_qa-3518 Implicitly created (auto staging).
[INFO] repo_qa-3521 OPEN Implicitly created (auto staging).
[INFO] repo-2011 OPEN Implicitly created (auto staging).
You will run the second command after parsing as:
mvn nexus-staging:rc-release -DserverId=nexus -DnexusUrl=http://my.nexus.ip:8081/nexus -DstagingRepositoryId=repo-2011 -Ddescription="Release from CLI."
I think what you are trying to do can be achieved readily by using SNAPSHOT and release repository.
So you do your normal development in a SNAPSHOT build and once you are ready for testing, you can create a tag build which removes SNAPSHOT from version in POM. All this can be achieved using jenkins and Nexus.
Also you dont need SVN revision number in your version, instead an incremental build number will be sufficient which can be managed via release plugin.
So to summarise:
Lets say you are working on release 1.0.
So you take a initial branch with pom version as 1.0-0-SNAPSHOT . Here 1.0 represents release number, '-0' represents next tag build number which we plan to deploy.
Now once you are ready for deployment or want your QA team to test. You run a job or a script with maven release plugin to create a tag. A tag build will be created with version 1.0-0 [Snapshot is removed so it will go to release repository] and uploaded to repository also version in branch will be incremented to 1.0-1-SNAPSHOT [So now changes in branch will be made to release and deploy 1.0-1 if any changes are needed]
All the above steps are automated using Maven release plugin and run via jenkins job.
I have the above setup for my work.
OP stated that 'SNAPSHOT' is not in picture.
In that case this post answers the query : automate deployment to sonatype's oss maven repository [Look at the second answer]