Is there a way to determine that a new SNAPSHOT version of a dependency is available in a Nexus? Ideally it would be a command line that would take as input a group/artifact/version and would return true if it is the case.
Say I have mygroup.b1:1.0.0-SNAPSHOT which has as dependency mygroup.a1:1.0.0-SNAPSHOT, both version controlled and managed by my continuous deployment environment.
I would like for a build of mygroup.b1:1.0.0-SNAPSHOT to be triggered when a new SNAPSHOT of mygroup.a1 has been deployed in Nexus.
In my case, the continuous environment is GOCD Thoughtworks and I would like to trigger a pipeline based on the dependencies defined in a pom.xml file (that of mygroup.b1), some of which are SNAPSHOTs (such as mygroup.a1) that other pipelines could deploy.
I have looked at https://github.com/jenkinsci/maven-dependency-update-trigger-plugin which seems to do this by checking a dependency is downloaded when it is resolved. Since this is a Jenkins plugin it is not exactly what I am looking for.
I have also looked at https://github.com/aresok/go-maven-poller but the process seems tedious: is it required to define all dependent packages so that GOCD can poll Nexus?
Another thing is the usage of the versions:display-dependency-updates goal of the versions-maven-plugin http://www.mojohaus.org/versions-maven-plugin/display-dependency-updates-mojo.html but it does not seem to allow filtering on gav.
EDIT: added an example
EDIT: added information on the usage of the maven-versions-plugin
Related
I'm probably approaching the problem all wrong, but here goes...
We want to implement a development -> releases/x.y.z -> master workflow using Gitlab and Nexus. The artifacts should be labelled x.y.z-SNAPSHOT, x.y.z-rc and x.y.z respectively.
In the ci-build.yml, I have tried to modify the POM depending on the current branch so as to create the correctly labelled artifacts. The problem is obviously that the change will be one commit ahead of the current build commit and the POM used in the build has the original version.
e.g.
a release branch is created from development
the POM has 2.3.4-SNAPSHOT and, during the ci-build, is modified to 2.3.4-rc and is committed and pushed to git
the subsequent build and deploy actions are executed but this deploys a 2.3.4-SNAPSHOT package to Nexus and not 2.3.4-rc as wanted
My question here is the following:
Can I modify the POM and use the updated POM in all following stages of the ci-build?
(I don't feel that this is the correct procedure, but it is what my team-leader has requested so I'm trying to implement it...)
My solution was to avoid commits altogether and use the strategy described in https://maven.apache.org/maven-ci-friendly.html.
This defines a new property, revision, which can be set using a maven parameter, -Drevision=x.y.z-SUFFIX. The POM version is set from this variable.
The build-ci reads project/properties/revision to extract the POM version and alters the version as appropriate depending on the current branch.
Using this approach I can achieve the required results without adding any extra commits as Maven can execute its goals using the version provided externally.
I've spent years trying to deploy libraries that use native code to Maven Central. I've run into the following problems:
There weren't any good plugins for building native code using Maven. native-maven-plugin was a very rigid build system that, among other things, made it difficult to debug the resulting binaries. You'd have to manually synchronize the native-maven-plugin build system with the native IDE you use for debugging.
Maven did not replace variables in deployed pom.xml files: MNG-2971, MNG-4223. This meant that libraries had to declare platform-specific dependencies once per Maven profile (as opposed to declaring the dependency once and setting a different classifier per profile); otherwise, anyone who depended on your library had to re-define those same properties in their project file in order to resolve transitive dependencies. See Maven: Using inherited property in dependency classifier causes build failure.
Jenkins had abysmal support for running similar logic across different platforms (e.g. "shell" vs "batch" tasks, and coordinating a build across multiple machines)
Running Windows, Linux and Mac in virtual machines was way too slow and fragile. Even if you got it working, attempting to configure the VMs as Jenkins slaves was a lesson in frustration (you'd get frequent intermittent build errors).
Maven Central requires a main jar for artifacts that are platform-specific: OSSRH-975
Sonatype OSS Repository Hosting and maven-release-plugin assumed that it would be possible to release a project in an atomic manner from a single machine but I need to build the OS-specific bits on separate machines.
I'm going to use this Stackoverflow question to document how I've managed to overcome these limitations.
Here is how I overcame the aforementioned problems:
I used CMake for building native code. The beauty of this system is that it generates project files for your favorite (native) IDE. You use the same project files to compile and debug the code. You no longer need to synchronize the two systems manually.
Maven didn't support CMake, so I built my own plugin: https://github.com/cmake-maven-project/cmake-maven-project
I manually hard-coded platform-specific dependencies into each Maven profile, instead of defining the dependency once with a different classifier per profile. This was more work, but it doesn't look like they will be fixing this bug in Maven anytime soon.
I plan to investigate http://www.mojohaus.org/flatten-maven-plugin/ and https://github.com/mjiderhamn/promote-maven-plugin as alternatives in the near future.
Jenkins pipeline does a good job of orchestrating a build across multiple machines.
Running Jenkins slaves on virtual machines is still very error-prone but I've managed to workaround most of the problems. I've uploaded my VMWare configuration steps and Jenkins job configuration to help others get started.
I now create an empty JAR file for platform-specific artifacts in order to suppress the Sonatype error. This was actually recommended by Sonatype's support staff.
It turns out that maven-release-plugin delegates to other plugins under the hood. Instead of invoking it, I do the following:
Use mvn versions:set to change the version number from SNAPSHOT to a release and back.
Tag and commit the release myself.
Use nexus-staging:rc-open, nexus-staging:deploy -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}, and nexus-staging:rc-close to upload artifacts from different platforms into the same repository. This is called a Staging Workflow (referenced below).
Upon review, release the repository to Maven Central.
Important: do not enable <autoReleaseAfterClose> in the nexus-staging plugin because it closes the staging repository after each deploy instead of waiting for all deploys to complete.
Per https://issues.sonatype.org/browse/NEXUS-18753 it isn't possible to release SNAPSHOT artifacts atomically (there is no workaround). When releasing SNAPSHOTs, you need to skip rc-open, rc-close and invoke nexus-staging:deploy without -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}. Each artifact will be uploaded into a separate repository.
See my Requirements API for a real-life example that works.
Other quirks to watch out for:
skipNexusStagingDeployMojo must be false in last reactor module (otherwise no artifacts will be deployed): https://issues.sonatype.org/browse/NEXUS-12365. The best workaround is to use Maven profiles to omit whatever modules you want when deploying (don't use skipNexusStagingDeployMojo at all)
skipLocalStaging prevents deploying multiple artifacts into the same repository: https://issues.sonatype.org/browse/NEXUS-12351
I have a number of builds that creates a package that is published to a package manager (for example NPM, Nuget and Maven).
I have subsequent builds that trigger on the completion of this build, they get the artifact from this repository. The problem is they show a warning:
I'm considering adding a Snapshot dependency, however, Teamcity's UI describes snapshot dependencies as builds using the same sources:
There is no source dependency between these projects and in fact, they may have completely different VCS roots.
What is the appropriate way to link these projects? Reading the documentation on Snapshot Dependencies, it sounds like things might not work as expected if I add a dependency without shared sources.
There is no requirement to link these projects, other than by the trigger you already have I don't think.
If the triggered build always gets the latest version from the package manager, then you'll get the behaviour that you want.
Snapshot dependency simply ensures that a build which depends on another build gets the same version of the source code when it builds, and doesn't end up being built using some changes that someone else checked in between the first build starting and the second build starting. This doesn't look like its going to be an issue in your situation (and indeed the builds may use completely different repositories), so I think your finished build trigger is an appropriate solution.
In development, I can reference the latest version of an artifact as 1.2.3-SNAPSHOT. Now I need the same behaviour for release candidates, i.e. I would like to be able to depend on the latest release candidate (there should also be a procedure for the developer to declare development versions as release candidates).
I am not sure how to implement this behaviour properly. Should I use an additional repository for release candidates and move development versions to this repository if the developer requests it? Or can I somehow define a "second snapshot list", like 1.2.3-RC?
You can get there be re-configure a few things:
use a version range for the dependency
change the updatePolicy for the repository you store the release candidates. see https://maven.apache.org/settings.html (updatePolicy). If you store the artifacts in a maven proxy usually you need to allow to overwrite releases.
Remember in a multi module build that they might upload modules before detecting a failed build (due to mvn deploy being a phase not a goal). You need to verify the complete build is ok before starting to upload artifacts in the maven repository. Or stage them somehow.
Remember this will most certainly prevent your builds being reproducible. Since an RC dependency might change between builds. You would need to change the version range - which is not always an issue. Ranges may work for you.
I've better experience to let developers stay on snapshots but have the CI server set an explicit version (for example using the versions plugin) prior to deployment / releasing for the dependency the artifact uses.
I am attempting to create a way in which hermetic builds can be achieved while still relying on SNAPSHOT dependencies in your project.
For the purposes of example, say I have a project which has a dependency structure like this:
┌ other-1.2-SNAPSHOT
mine-1.2.3 ──┤
└ thing-3.1-SNAPSHOT ── gizmo-6.1.3-SNAPSHOT
What I would like to do is resolve all the SNAPSHOT dependencies locally to something which is related to my current version and then deploy those as releases to my Nexus' release repository. Not all of these dependencies are internal so I cannot simply just make a release on each.
So, in this example, other-1.2-SNAPSHOT would become something like other-1.2-mine-1.2.3 and thing-3.1-SNAPSHOT would become thing-3.1-mine-1.2.3. This is relatively trivial in about 60 lines of python.
The problem, however, is in resolving transitive SNAPSHOTs to concrete versions. So I also need to convert gizmo-6.1.3-SNAPSHOT to gizmo-6.1.3-mine.1.2.3 and have thing-3.1-mine-1.2.3 depend on it.
This is only an example of one way in which to achieve what I want. The goal is that in a year or two down the road I can checkout my release branch for version 1.2.3 and be able to run mvn clean package or the like without having to worry about resolving long-since-gone SNAPSHOT dependencies.
It's important that this branch be compilable and not just retain all dependencies using something like the jar-and-dependencies functionality of the assembly plugin. I'd like to potentially be able to modify the source files and make another release build (e.g., applying a hotfix).
So,
Is there anything like this available that will be able to convert SNAPSHOT dependencies in a recursive fashion to be concrete?
Are there any plugins which manage this kind of thing for you? The release plugin had promise with some configuration options on its branch goal but it doesn't resolve external deps to the degree that I want.
Are other techniques available for creating hermetic Maven builds?
This is not a widely used technique, but you can always check your specific SNAPSHOT dependencies into your project as a "project" repository, as described in this blog post: Maven is to Ant as a Nail Gun is to a Hammer
In short, use the Dependencies Plugin to create repository located in your project directory. The below is copied from the linked blog post (which you should read):
1) Run mvn -Dmdep.useRepositoryLayout=true -Dmdep.copyPom=true dependency:copy-dependencies
"This creates /target/dependencies with a repo-like layout of all your projects dependencies"
2) Copy target/dependencies/ to something like libs/
3) Add a repository declaration like the following to your POM:
<repositories>
<repository>
<releases />
<id>snapshots-I-need-forever</id>
<name>snapshots-I-need-forever</name>
<url>file:///${basedir}/libs</url>
</repository>
</repositories>
You make this an automated part of your build/release process: step 1 by configuring the Dependencies plugin to a lifecycle phasephase, and step 2 using AntRun Plugin to move the downloaded dependencies to the right place..
Hope this works for you. I have to go take a shower now...
The maven versions plugin will do most of what you want.
http://mojo.codehaus.org/versions-maven-plugin/
However you will almost certianly need to run it in a pre-build step in which you resolve all the dependencies and update the pom file accordingly. Then re-run maven (which re-reads the pom) to run the real build. You might be able to configure everything within the pom itself triggered with a separate goal thus avoiding a separate script.
This works better if you use particular versions instead of SNAPSHOT dependencies and let the pre-build step upgrade them if necessary. The only real difference for dependency resolution is that maven will always re-download -SNAPSHOT dependencies whereas it will only download normal dependencies if there is a new version available. However many plugins (including the versions plugin) treat -SNAPSHOT dependencies differently causing problems. Since every CI build has a new version number I never use -SNAPSHOT, prefering a different tag like -DEV with more predictable behaviour for things like developer local builds etc.
I've spent a lot of time getting maven to do things similar to this. Most maven projects I know have some kind of pre-build step in order to set version numbers or get around other limitations such as this. Trying to do all this in one step usually fails because maven only reads the pom once, string substitution doesn't work in a few places and the deployed/installed pom doesn't generally doesn't contain the results of string substituion or changes made during the build.