I have an artifact that should be built for several target platforms:
Linux x86
Windows x86
ARM11
Unfortunately due to the lack of crosscompilers, it is not possible to create all versions of the artifact in one go.
Using other words, the goal is to have in the repository something like this
artifact-1.0.0-linux.zip
artifact-1.0.0-windows.zip
artifact-1.0.0-arm11.zip
artifact-1.0.1-linux.zip
artifact-1.0.1-windows.zip
artifact-1.0.1-arm11.zip
...
Note that the versions are in sync. How to accomplish this?
The thing is that the release process upgrades version of the pom.xml after every build. So by building consecutively on various platforms I can achieve having
artifact-1.0.0-linux.zip
artifact-1.0.1-windows.zip
artifact-1.0.2-arm11.zip
artifact-1.0.3-linux.zip
artifact-1.0.4-windows.zip
artifact-1.0.5-arm11.zip
...
but this is not what I am looking for.
I could
run on Linux
mvn release:prepare release:perform -DpushChanges=false
(with pushChanges set to false release won't increase version number in SCM)
and then run on Windows
mvn release:prepare release:perform
(this will increase the version number)
But then the responsibility to trigger the release processes on various platforms in the proper order lies with me. Is there a way maven can help me with this?
Do you have any suggestions?
Thanks
PS. Note that this is not a question about how to organize into modules. It is about how to synchronize release processes of a single artifact on multiple platforms.
Have you found a solution for this yet?
It's good to know I'm not the only one fighting with Maven :-)
Anyway,
Are you allowed to deploy a released version to nexus?
I'm thinking, you could do this:
1 - Do a "mvn release:prepare release:perform" from a windows machine - that should get artifact-1.0.1-windows.zip into nexus.
2 - Checkout the artifact-1.0.1 tag from source control
3 - Do a "mvn deploy" from linux and arm11 (whatever that is :P) - that should get -linux.zip and -arm11.zip into nexus as well.
Though, I believe that depending on how nexus is configured it won't let you redeploy anything with the same GAV (even if the classifier is different)
As I see, you use classifiers (artifact's file name suffix) like linux, windows or arm11 to distinguish various artifact's releases, intended for specific platforms. So, if you create multi-module project managed by Maven, where modules would be artifacts with same groupId, same artifactId, same version (probably inherited from their common parent), but different classifier, you'll get exactly what you want. In such case, you always release your multi-module POM (usually it is common parent for its modules as well) to have all modules released at once. Assuming same-version-for-all-modules-policy (which seems to fit pretty well here), you can basically execute:
mvn release:prepare release:perform -DautoVersionSubmodules
and that's it. You will get artifact-1.0.0-linux.zip, artifact-1.0.0-windows.zip, artifact-1.0.0-arm11.zip artifacts released. Next development version will be set to 1.0.1-SNAPSHOT for all modules (via inheritance from parent).
You helped me out with your question...I didn't know about the pushChanges=false option.
Using Jenkins you can set up a job on each platform that performs the maven release in series using the "Trigger builds remotely (e.g., from scripts)" feature. If you desire, you can use a Parameterized Build to choose which version to build (If you use that, then you need the Parameterized Trigger Plugin to trigger the build on the other platforms).
Related
I've spent years trying to deploy libraries that use native code to Maven Central. I've run into the following problems:
There weren't any good plugins for building native code using Maven. native-maven-plugin was a very rigid build system that, among other things, made it difficult to debug the resulting binaries. You'd have to manually synchronize the native-maven-plugin build system with the native IDE you use for debugging.
Maven did not replace variables in deployed pom.xml files: MNG-2971, MNG-4223. This meant that libraries had to declare platform-specific dependencies once per Maven profile (as opposed to declaring the dependency once and setting a different classifier per profile); otherwise, anyone who depended on your library had to re-define those same properties in their project file in order to resolve transitive dependencies. See Maven: Using inherited property in dependency classifier causes build failure.
Jenkins had abysmal support for running similar logic across different platforms (e.g. "shell" vs "batch" tasks, and coordinating a build across multiple machines)
Running Windows, Linux and Mac in virtual machines was way too slow and fragile. Even if you got it working, attempting to configure the VMs as Jenkins slaves was a lesson in frustration (you'd get frequent intermittent build errors).
Maven Central requires a main jar for artifacts that are platform-specific: OSSRH-975
Sonatype OSS Repository Hosting and maven-release-plugin assumed that it would be possible to release a project in an atomic manner from a single machine but I need to build the OS-specific bits on separate machines.
I'm going to use this Stackoverflow question to document how I've managed to overcome these limitations.
Here is how I overcame the aforementioned problems:
I used CMake for building native code. The beauty of this system is that it generates project files for your favorite (native) IDE. You use the same project files to compile and debug the code. You no longer need to synchronize the two systems manually.
Maven didn't support CMake, so I built my own plugin: https://github.com/cmake-maven-project/cmake-maven-project
I manually hard-coded platform-specific dependencies into each Maven profile, instead of defining the dependency once with a different classifier per profile. This was more work, but it doesn't look like they will be fixing this bug in Maven anytime soon.
I plan to investigate http://www.mojohaus.org/flatten-maven-plugin/ and https://github.com/mjiderhamn/promote-maven-plugin as alternatives in the near future.
Jenkins pipeline does a good job of orchestrating a build across multiple machines.
Running Jenkins slaves on virtual machines is still very error-prone but I've managed to workaround most of the problems. I've uploaded my VMWare configuration steps and Jenkins job configuration to help others get started.
I now create an empty JAR file for platform-specific artifacts in order to suppress the Sonatype error. This was actually recommended by Sonatype's support staff.
It turns out that maven-release-plugin delegates to other plugins under the hood. Instead of invoking it, I do the following:
Use mvn versions:set to change the version number from SNAPSHOT to a release and back.
Tag and commit the release myself.
Use nexus-staging:rc-open, nexus-staging:deploy -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}, and nexus-staging:rc-close to upload artifacts from different platforms into the same repository. This is called a Staging Workflow (referenced below).
Upon review, release the repository to Maven Central.
Important: do not enable <autoReleaseAfterClose> in the nexus-staging plugin because it closes the staging repository after each deploy instead of waiting for all deploys to complete.
Per https://issues.sonatype.org/browse/NEXUS-18753 it isn't possible to release SNAPSHOT artifacts atomically (there is no workaround). When releasing SNAPSHOTs, you need to skip rc-open, rc-close and invoke nexus-staging:deploy without -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}. Each artifact will be uploaded into a separate repository.
See my Requirements API for a real-life example that works.
Other quirks to watch out for:
skipNexusStagingDeployMojo must be false in last reactor module (otherwise no artifacts will be deployed): https://issues.sonatype.org/browse/NEXUS-12365. The best workaround is to use Maven profiles to omit whatever modules you want when deploying (don't use skipNexusStagingDeployMojo at all)
skipLocalStaging prevents deploying multiple artifacts into the same repository: https://issues.sonatype.org/browse/NEXUS-12351
In development, I can reference the latest version of an artifact as 1.2.3-SNAPSHOT. Now I need the same behaviour for release candidates, i.e. I would like to be able to depend on the latest release candidate (there should also be a procedure for the developer to declare development versions as release candidates).
I am not sure how to implement this behaviour properly. Should I use an additional repository for release candidates and move development versions to this repository if the developer requests it? Or can I somehow define a "second snapshot list", like 1.2.3-RC?
You can get there be re-configure a few things:
use a version range for the dependency
change the updatePolicy for the repository you store the release candidates. see https://maven.apache.org/settings.html (updatePolicy). If you store the artifacts in a maven proxy usually you need to allow to overwrite releases.
Remember in a multi module build that they might upload modules before detecting a failed build (due to mvn deploy being a phase not a goal). You need to verify the complete build is ok before starting to upload artifacts in the maven repository. Or stage them somehow.
Remember this will most certainly prevent your builds being reproducible. Since an RC dependency might change between builds. You would need to change the version range - which is not always an issue. Ranges may work for you.
I've better experience to let developers stay on snapshots but have the CI server set an explicit version (for example using the versions plugin) prior to deployment / releasing for the dependency the artifact uses.
I have an application that uses maven 2.2.1 for building. I want to use a plugin that needs maven3 to run during deploy phase of build. Is this possible. Just want to know if someone solved this scenario in a unique way.
You'll either need to use different Maven installations for different phases or convert everything to Maven3.
Having just converted my build process to Maven3, there is one huge gotcha in Maven3: version ranges on snapshots are totally mucked up. There was an attempt to rectify version weirdness in Maven2 and instead broke Maven3. Version ranges with release artifacts work fine, just not snapshot, so something to beware of.
Maven noob here. We have tens of repos with POMs that build rpms and recently moved our CI system to CentOS 6. As a result rpm builds are no longer backwards compatible with RHEL 5. Apparently I need to state within the spec or the maven POM:
<defineStatement>_source_filedigest_algorithm md5</defineStatement>
<defineStatement>_binary_filedigest_algorithm md5</defineStatement>
Can I add this to our parent POM somehow? Otherwise I need to script the addition of these lines to a slew of repositories and that gets ugly and error-prone quickly.
Is that a property? Is that a configuration item for a plugin? If it's a property, you could define a active profile on each machine building it. This would be appropriate if it changes based on the machine running the build. The profile would be defined in the conf/settings.xml of the maven installation, and defined as active if it should be applied to all builds. However, not knowing where that value is meant to be sent, I can only guess right now.
I am attempting to create a way in which hermetic builds can be achieved while still relying on SNAPSHOT dependencies in your project.
For the purposes of example, say I have a project which has a dependency structure like this:
┌ other-1.2-SNAPSHOT
mine-1.2.3 ──┤
└ thing-3.1-SNAPSHOT ── gizmo-6.1.3-SNAPSHOT
What I would like to do is resolve all the SNAPSHOT dependencies locally to something which is related to my current version and then deploy those as releases to my Nexus' release repository. Not all of these dependencies are internal so I cannot simply just make a release on each.
So, in this example, other-1.2-SNAPSHOT would become something like other-1.2-mine-1.2.3 and thing-3.1-SNAPSHOT would become thing-3.1-mine-1.2.3. This is relatively trivial in about 60 lines of python.
The problem, however, is in resolving transitive SNAPSHOTs to concrete versions. So I also need to convert gizmo-6.1.3-SNAPSHOT to gizmo-6.1.3-mine.1.2.3 and have thing-3.1-mine-1.2.3 depend on it.
This is only an example of one way in which to achieve what I want. The goal is that in a year or two down the road I can checkout my release branch for version 1.2.3 and be able to run mvn clean package or the like without having to worry about resolving long-since-gone SNAPSHOT dependencies.
It's important that this branch be compilable and not just retain all dependencies using something like the jar-and-dependencies functionality of the assembly plugin. I'd like to potentially be able to modify the source files and make another release build (e.g., applying a hotfix).
So,
Is there anything like this available that will be able to convert SNAPSHOT dependencies in a recursive fashion to be concrete?
Are there any plugins which manage this kind of thing for you? The release plugin had promise with some configuration options on its branch goal but it doesn't resolve external deps to the degree that I want.
Are other techniques available for creating hermetic Maven builds?
This is not a widely used technique, but you can always check your specific SNAPSHOT dependencies into your project as a "project" repository, as described in this blog post: Maven is to Ant as a Nail Gun is to a Hammer
In short, use the Dependencies Plugin to create repository located in your project directory. The below is copied from the linked blog post (which you should read):
1) Run mvn -Dmdep.useRepositoryLayout=true -Dmdep.copyPom=true dependency:copy-dependencies
"This creates /target/dependencies with a repo-like layout of all your projects dependencies"
2) Copy target/dependencies/ to something like libs/
3) Add a repository declaration like the following to your POM:
<repositories>
<repository>
<releases />
<id>snapshots-I-need-forever</id>
<name>snapshots-I-need-forever</name>
<url>file:///${basedir}/libs</url>
</repository>
</repositories>
You make this an automated part of your build/release process: step 1 by configuring the Dependencies plugin to a lifecycle phasephase, and step 2 using AntRun Plugin to move the downloaded dependencies to the right place..
Hope this works for you. I have to go take a shower now...
The maven versions plugin will do most of what you want.
http://mojo.codehaus.org/versions-maven-plugin/
However you will almost certianly need to run it in a pre-build step in which you resolve all the dependencies and update the pom file accordingly. Then re-run maven (which re-reads the pom) to run the real build. You might be able to configure everything within the pom itself triggered with a separate goal thus avoiding a separate script.
This works better if you use particular versions instead of SNAPSHOT dependencies and let the pre-build step upgrade them if necessary. The only real difference for dependency resolution is that maven will always re-download -SNAPSHOT dependencies whereas it will only download normal dependencies if there is a new version available. However many plugins (including the versions plugin) treat -SNAPSHOT dependencies differently causing problems. Since every CI build has a new version number I never use -SNAPSHOT, prefering a different tag like -DEV with more predictable behaviour for things like developer local builds etc.
I've spent a lot of time getting maven to do things similar to this. Most maven projects I know have some kind of pre-build step in order to set version numbers or get around other limitations such as this. Trying to do all this in one step usually fails because maven only reads the pom once, string substitution doesn't work in a few places and the deployed/installed pom doesn't generally doesn't contain the results of string substituion or changes made during the build.