In my company we have teams working on services which are built using maven pom's and gradle build scripts. The problem I have is that when the team's build their web applications, the jar files that get's created by one team member needs to be available for other team members in their pom files.
What we were thinking was to have a local nexus repo and then push the built jar files to nexus so that when any other team member builds they also can refer the same jar file.
However this could lead to versioning problems as two team members could be generating the same jar file if they change different files in the same project.
What I would like to know is are their any best practices in doing these types of builds and versioning.
There are many different opinions and strategies on how to manage this process. Some aspects are relatively common, however.
I'd say there are two key elements:
* Proper use of version definitions and references
* Automated builds and nexus deployments
If you have work ongoing for a specific artifact, for a specific release, then those changes should all go into a specific numbered version of the artifact. While work is ongoing, that version should end with "-SNAPSHOT". When the work for a release is completed, that version number should remove the '-SNAPSHOT". You also likely want to have separate repositories in the Nexus server for "snapshot" and "release" artifacts.
Concerning pushing artifacts to Nexus, this should always be done through automation. Manually pushing artifacts should be very rare. When a regular build is done for ongoing work, that should automatically deploy the "-SNAPSHOT" artifact to the snapshot repo. When your build automation is running a "release" build, those artifacts will deploy the release artifact to the release repo.
There are many other options and details you'll want to examine. Only implement features in this process that provide clear value in your situation. It's very easy to set up a process that is more complicated than you need.
Related
I have a Gradle build set up in Azure DevOps, which compiles the code in an Azure DevOps git repository, then publishes the generated JARs (as Maven artifacts) to Azure Artifacts, as explained here. Code in other Azure DevOps git repositories can then reference these components as dependencies. This is fine for formal releases of these components (with unique version numbers), but I also need a way to get this working for in-progress snapshot releases. The problem is that I cannot publish an artifact with the same version number (e.g. 1.2.3-SNAPSHOT) more than once. This seems to be because packages in Azure are immutable.
From my understanding, that would mean that Azure Artifacts cannot be used to store in-progress snapshot artifacts. Is that correct?
If it is, is there any alternative that still uses Azure DevOps? I can see that I can publish artifacts to Azure Blob Storage, but presumably this is something you have to pay for on top of existing use of Azure Artifacts. I can also see that there's a number of GitHub Maven plugins for treating a GitHub repo as a Maven repo, but I can't find anything similar for using an Azure DevOps repo as a place to publish Maven artifacts.
In case it makes a difference, I'm talking about the cloud-based Azure stuff, nothing on-premise.
Specifically for maven, there's a difference in how a jar is packaged in a SNAPSHOT as compared to other versions.
Eg. let's say I version my jar as 1.0.0-PRERELEASE, now this version is immutable. Once I publish this to any artifact repository it cannot be overridden. Usually package management systems store the artifact in this kind of logical path - <group_id>/<artifact_id>/<version>/<artifact_id>_<version>.jar.
But for SNAPSHOTs this is slightly different. In maven, SNAPSHOTs are mutable. This means you can override a SNAPSHOT. And this makes sense as by definition SNAPSHOTs are like an image of your code in active development. How SNAPSHOTs are stored in a package management feed is something like <group_id>/<artifact_id>/<version(SNAPSHOT)>/<artifact_id>_<version>_<jar_timestamp>.jar.
This naming convention of the jar in this case makes it mutable. In a single SNAPSHOT version, there can be multiple jars present with different timestamps. And when maven looks for a SNAPSHOT dependency to download, it always picks the most recent one (again logical as that's the most recent snapshot).
At present azure artifacts implements these concepts exactly how they are supposed to. Perhaps this wasn't there before and has been pushed in an update.
http://maven.apache.org/guides/getting-started/index.html#What_is_a_SNAPSHOT_version
https://xebia.com/blog/continuous-releasing-of-maven-artifacts/
The premise of Package Management is that a package is immutable. This enables a whole bunch of caching options that would otherwise not exist. Packages are stored in your local package cache, possibly on a proxy feed package cache and all of these elements assume that packages with the same name+version are unchanged and will serve the cached version instead of the latest version you've pushed. Most package systems are built on this premise, including Nuget and NPM.
The trick to creating development snapshots is to use semantic versioning and adding a unique suffix to your version. For example 1.2.3-SNAPSHOT.1 followed by 1.2.3-SNAPSHOT.2, there are tools available for Azure Pipelines, like GitVersion that can automatically generate a unique version + suffix that you can pass into the version for your artifact.
If you don't want to "mess up" your main package feed, you can setup a second feed for development purposes which holds all your intermediate packages, you can then either promote one of these packages to your main feed or you can run a specific pipeline (configuration) to push the final package to the feed used for your stable packages.
It looks like this is mostly a feature and not a bug as to preserve the immutability of the build results - meaning, that no matter when that build is ran, it will always return the same result. See: How to update a maven dependency with a same version number in Azure Artifacts & Azure Artifact Publishing Fails for Artifact Version Containing '+'
is it possible in Maven to configure different local repositories for SNAPSHOT and RELEASE artifacts?
The reason I am asking, we are using Jenkins for continuous build for our project. To ensure the consistency (if same artifact is built from different Jenkins jobs because of race condition, we can experience chaotic behavior) before build start, we create a fresh local repository for Jenkins.
Now the problem is, our project is huge, so for every build we have to download lots of dependencies from our Nexus but when you think about it, there is no reason to download every time new the RELEASE artifacts. The RELEASE artifacts don't change from build to build, for ex, Spring 4.5, httpclient 4.0, aspectj 1.8.1 is same for one build to another.
So actually to ensure the consistency, we only should not have the SNAPSHOT dependencies in the repository. If we could have two local repositories one for RELEASE artifacts and the other for SNAPSHOT's, then before every build start, we could delete the SNAPSHOT repository but re-use the local RELEASE repository, which would save me gigabytes of download from Nexus.
I know we can do RELEASE, SNAPSHOT configurations for remote repositories, is it possible to do same sort of configuration for local repositories?
If this is not possible, how would you solve this problem.
There is currently no way to achieve this, and yes, I agree with the sentiment.
A reasonably recent versions of Jenkins' Maven plugin allow you to specify a custom local repository without having to edit a settings.xml file — the option is right there at the job definition screen (in the Advanced section, select Use private Maven repository).
So, what I would do is use this option, and precede the Maven build step with a script that deletes all directories, in the local private repository, which end in -SNAPSHOT.
It's repulsive, but I can't think of any other way.
I've spent years trying to deploy libraries that use native code to Maven Central. I've run into the following problems:
There weren't any good plugins for building native code using Maven. native-maven-plugin was a very rigid build system that, among other things, made it difficult to debug the resulting binaries. You'd have to manually synchronize the native-maven-plugin build system with the native IDE you use for debugging.
Maven did not replace variables in deployed pom.xml files: MNG-2971, MNG-4223. This meant that libraries had to declare platform-specific dependencies once per Maven profile (as opposed to declaring the dependency once and setting a different classifier per profile); otherwise, anyone who depended on your library had to re-define those same properties in their project file in order to resolve transitive dependencies. See Maven: Using inherited property in dependency classifier causes build failure.
Jenkins had abysmal support for running similar logic across different platforms (e.g. "shell" vs "batch" tasks, and coordinating a build across multiple machines)
Running Windows, Linux and Mac in virtual machines was way too slow and fragile. Even if you got it working, attempting to configure the VMs as Jenkins slaves was a lesson in frustration (you'd get frequent intermittent build errors).
Maven Central requires a main jar for artifacts that are platform-specific: OSSRH-975
Sonatype OSS Repository Hosting and maven-release-plugin assumed that it would be possible to release a project in an atomic manner from a single machine but I need to build the OS-specific bits on separate machines.
I'm going to use this Stackoverflow question to document how I've managed to overcome these limitations.
Here is how I overcame the aforementioned problems:
I used CMake for building native code. The beauty of this system is that it generates project files for your favorite (native) IDE. You use the same project files to compile and debug the code. You no longer need to synchronize the two systems manually.
Maven didn't support CMake, so I built my own plugin: https://github.com/cmake-maven-project/cmake-maven-project
I manually hard-coded platform-specific dependencies into each Maven profile, instead of defining the dependency once with a different classifier per profile. This was more work, but it doesn't look like they will be fixing this bug in Maven anytime soon.
I plan to investigate http://www.mojohaus.org/flatten-maven-plugin/ and https://github.com/mjiderhamn/promote-maven-plugin as alternatives in the near future.
Jenkins pipeline does a good job of orchestrating a build across multiple machines.
Running Jenkins slaves on virtual machines is still very error-prone but I've managed to workaround most of the problems. I've uploaded my VMWare configuration steps and Jenkins job configuration to help others get started.
I now create an empty JAR file for platform-specific artifacts in order to suppress the Sonatype error. This was actually recommended by Sonatype's support staff.
It turns out that maven-release-plugin delegates to other plugins under the hood. Instead of invoking it, I do the following:
Use mvn versions:set to change the version number from SNAPSHOT to a release and back.
Tag and commit the release myself.
Use nexus-staging:rc-open, nexus-staging:deploy -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}, and nexus-staging:rc-close to upload artifacts from different platforms into the same repository. This is called a Staging Workflow (referenced below).
Upon review, release the repository to Maven Central.
Important: do not enable <autoReleaseAfterClose> in the nexus-staging plugin because it closes the staging repository after each deploy instead of waiting for all deploys to complete.
Per https://issues.sonatype.org/browse/NEXUS-18753 it isn't possible to release SNAPSHOT artifacts atomically (there is no workaround). When releasing SNAPSHOTs, you need to skip rc-open, rc-close and invoke nexus-staging:deploy without -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}. Each artifact will be uploaded into a separate repository.
See my Requirements API for a real-life example that works.
Other quirks to watch out for:
skipNexusStagingDeployMojo must be false in last reactor module (otherwise no artifacts will be deployed): https://issues.sonatype.org/browse/NEXUS-12365. The best workaround is to use Maven profiles to omit whatever modules you want when deploying (don't use skipNexusStagingDeployMojo at all)
skipLocalStaging prevents deploying multiple artifacts into the same repository: https://issues.sonatype.org/browse/NEXUS-12351
I'm in the latter stages of setting up a CI environment for my project. I'm using Maven, Jenkins and Artifactory Pro and can successfully build my project and deploy it's artifacts to Artifactory. I have also written a bash script to retrieve the resulting artifacts of a specific build from Artifactory and copy them somewhere.
The main part I'm missing right now is automated versioning. I've looked at enabling Artifactory release management, which is really cool, but involves the rebuilding of the project. I'm really trying to follow the mantra of 'Build Once, Deploy Anywhere', so any rebuilding is a no-no.
My question boils down to: Is there an automated way (either with one of the aforementioned tools, or a plugin) to handle versioning, without rebuilding an artifact?
Artifactory Pro allows you to easily extend Artifactory's behavior with your own plugins written in groovy. (https://www.jfrog.com/confluence/display/RTF/User+Plugins)
You can find here, an example of Promote extension, that will change your artifacts versions without the needs of new build.
You can find more usefully examples in the GitHub "artifactory-user-plugins" repository.
Wherever you read about continuous delivery or continuous integration it's recommended to use an artifact repository to store the artifacts even though Jenkins already stores them for each build.
So why is it recommended to use an artifact repository? Is there a smooth solution to work with the artifacts of the Jenkins builds, ex. to use these artifacts for deployment?
An artifact repository and continuous integration tools serve two different purposes and one cannot be substituted with the other. Check this video from Artifactory, one of the providers of artifact repositories, about why one should use an artifact repository.
Jenkins stores the artifacts as plain files without versioning while artifacts in an artifact repository can be version controlled. So you have a lot more flexibility in retrieving artifacts and governing them. Read this very good article on why we need them. Surely not all of those things are supported by continuous integration tools like Jenkins.
Moreover, you can also look at the Artifactory plugin for Jenkins which integrates the two.
An artifact repository is needed but the artifact repo is a conceptual piece an not always a distinct tool. With Jenkins you should have MD5 signatures and (I think) a way of downloading the files you want (web service call, right?) from your remote server. Certainly, if you're doing something simple like using the Jenkins build pipeline plugin, it should be able to access the right versions of the files smoothly.
Alternatively, if you are using a separate deployment tool, the better ones bundle an artifact repository.
Regardless, you want what the ITIL folks call a Definitive Media/Software Library. Definitive in that the bits are secure, trusted, and official. And a library in that they can be easily looked up and accessed. When working with an artifact repository, you need to make sure its adequately secure. It is backed up. It is accessible for your deployments (including to production). If you look at Jenkins and it meets your criteria in those categories, consider yourself done. If it's lacking, and I wouldn't be surprised if it was, then you need either a dedicated tool like the Maven repos, or something bundled with the deploy tooling.
For more of my rambling on the subject, there's a recorded webcast. The slides for that are up on Slideshare.
I haven't kept up to date with Jenkins, we still use a version of the CI when it was orginally called Hudson.
In your projects your poms you should normally point to your own artifact repository were you can fetch and deploy your own (company) projects.
Using an artifact repository with your CI server, it can then deploy successfully built snapshot and releases which can be available to other developers.