I have a Gradle build set up in Azure DevOps, which compiles the code in an Azure DevOps git repository, then publishes the generated JARs (as Maven artifacts) to Azure Artifacts, as explained here. Code in other Azure DevOps git repositories can then reference these components as dependencies. This is fine for formal releases of these components (with unique version numbers), but I also need a way to get this working for in-progress snapshot releases. The problem is that I cannot publish an artifact with the same version number (e.g. 1.2.3-SNAPSHOT) more than once. This seems to be because packages in Azure are immutable.
From my understanding, that would mean that Azure Artifacts cannot be used to store in-progress snapshot artifacts. Is that correct?
If it is, is there any alternative that still uses Azure DevOps? I can see that I can publish artifacts to Azure Blob Storage, but presumably this is something you have to pay for on top of existing use of Azure Artifacts. I can also see that there's a number of GitHub Maven plugins for treating a GitHub repo as a Maven repo, but I can't find anything similar for using an Azure DevOps repo as a place to publish Maven artifacts.
In case it makes a difference, I'm talking about the cloud-based Azure stuff, nothing on-premise.
Specifically for maven, there's a difference in how a jar is packaged in a SNAPSHOT as compared to other versions.
Eg. let's say I version my jar as 1.0.0-PRERELEASE, now this version is immutable. Once I publish this to any artifact repository it cannot be overridden. Usually package management systems store the artifact in this kind of logical path - <group_id>/<artifact_id>/<version>/<artifact_id>_<version>.jar.
But for SNAPSHOTs this is slightly different. In maven, SNAPSHOTs are mutable. This means you can override a SNAPSHOT. And this makes sense as by definition SNAPSHOTs are like an image of your code in active development. How SNAPSHOTs are stored in a package management feed is something like <group_id>/<artifact_id>/<version(SNAPSHOT)>/<artifact_id>_<version>_<jar_timestamp>.jar.
This naming convention of the jar in this case makes it mutable. In a single SNAPSHOT version, there can be multiple jars present with different timestamps. And when maven looks for a SNAPSHOT dependency to download, it always picks the most recent one (again logical as that's the most recent snapshot).
At present azure artifacts implements these concepts exactly how they are supposed to. Perhaps this wasn't there before and has been pushed in an update.
http://maven.apache.org/guides/getting-started/index.html#What_is_a_SNAPSHOT_version
https://xebia.com/blog/continuous-releasing-of-maven-artifacts/
The premise of Package Management is that a package is immutable. This enables a whole bunch of caching options that would otherwise not exist. Packages are stored in your local package cache, possibly on a proxy feed package cache and all of these elements assume that packages with the same name+version are unchanged and will serve the cached version instead of the latest version you've pushed. Most package systems are built on this premise, including Nuget and NPM.
The trick to creating development snapshots is to use semantic versioning and adding a unique suffix to your version. For example 1.2.3-SNAPSHOT.1 followed by 1.2.3-SNAPSHOT.2, there are tools available for Azure Pipelines, like GitVersion that can automatically generate a unique version + suffix that you can pass into the version for your artifact.
If you don't want to "mess up" your main package feed, you can setup a second feed for development purposes which holds all your intermediate packages, you can then either promote one of these packages to your main feed or you can run a specific pipeline (configuration) to push the final package to the feed used for your stable packages.
It looks like this is mostly a feature and not a bug as to preserve the immutability of the build results - meaning, that no matter when that build is ran, it will always return the same result. See: How to update a maven dependency with a same version number in Azure Artifacts & Azure Artifact Publishing Fails for Artifact Version Containing '+'
Related
At present we have more than 10 libraries consumed and deployed across 4 services. There are some libraries like common utilities and protos which are consumed by other libraries, so there is a dependency hierarchy in the libraries as well, all the way leading to the service.
We are facing multiple challenges in deploying any change to the library.
We use Gradle for dependency management, Code Artifact for maven repository management, Code Pipeline for CD, ECR for service docker image storing and ECS for service deployment.
Issue 1: Auto Version Management/Increment using CodeArtifact, Git and Gradle.
When a library is changed (Minor changes without any backward compatibility issues), we don't increment the version. We push the changes to master, which triggers a CodePipeline and builds the library by taking any dependent libraries from CodeArtifact. We follow a manual process to delete the current Library-1.0.0 CodeArtifact so that new artifacts can be published. Removal of existing artifacts is a manual process as well as risky as this prevents easy rollbacks.
Issue 2: Rebuilding and Republishing all child libraries if a root library is changed
When a parent library is changed, all the libraries that depend on it need to be rebuilt using the latest parent library artifact before releasing the changes to service. If we don't do that, and directly rebuild the service, only the service will have the latest code of the library, but all the other libraries still use the old library code, leading to unexpected behaviour.
Ideal expectation here is -> When a service image is getting created, a single or latest version of the library should be used by the service and by all the dependent libraries. Only one version of the library should be active in a service, as part of dependency conflict resolution.
Issue 3: Rollback the deployments
Right now rollbacks are impossible. We delete all the old version library artifacts, so it's impossible to rollback a deployment, the only way to rollback a deployment is to revert all the libraries in git and rebuild, republish all libraries. Would like to know what is the best practice for rollbacks and CI/CD using CodeArtifact, Gradle and Git.
Please help in finding out what we are doing wrong and the correct way to setup CI/CD using Gradle, CodeArtifact and CodeBuild.
In my company we have teams working on services which are built using maven pom's and gradle build scripts. The problem I have is that when the team's build their web applications, the jar files that get's created by one team member needs to be available for other team members in their pom files.
What we were thinking was to have a local nexus repo and then push the built jar files to nexus so that when any other team member builds they also can refer the same jar file.
However this could lead to versioning problems as two team members could be generating the same jar file if they change different files in the same project.
What I would like to know is are their any best practices in doing these types of builds and versioning.
There are many different opinions and strategies on how to manage this process. Some aspects are relatively common, however.
I'd say there are two key elements:
* Proper use of version definitions and references
* Automated builds and nexus deployments
If you have work ongoing for a specific artifact, for a specific release, then those changes should all go into a specific numbered version of the artifact. While work is ongoing, that version should end with "-SNAPSHOT". When the work for a release is completed, that version number should remove the '-SNAPSHOT". You also likely want to have separate repositories in the Nexus server for "snapshot" and "release" artifacts.
Concerning pushing artifacts to Nexus, this should always be done through automation. Manually pushing artifacts should be very rare. When a regular build is done for ongoing work, that should automatically deploy the "-SNAPSHOT" artifact to the snapshot repo. When your build automation is running a "release" build, those artifacts will deploy the release artifact to the release repo.
There are many other options and details you'll want to examine. Only implement features in this process that provide clear value in your situation. It's very easy to set up a process that is more complicated than you need.
Maven is a dependency manager in the Java world. It supports snapshot versioning. In Maven-speak a dependency is called an artifact. An artifact with a fixed version number (e.g. 1.0.1) will be downloaded only once because it will never change. On the contrary a snapshot version (e.g. 1.0.1-Snapshot) will be considered as a moving target. It is a current development copy and will likely change in the near future. Therefore it has to be updated on a regular basis. With snapshot versioning you are able to provide the current state of an ongoing development as an artifact through your artifact delivery mechanism (for example with Nexus or Artifactory). In combination with a CI build which creates the snapshot artifacts you are able to setup a development infrastructure with can handle fairly complex projects.
In essence I think snapshot versioning means the dependency manager needs to check and download artifacts with a version tag which it has already downloaded before.
In the .NET world Nuget is the prefered package manager. As a dependency manager it does a very lousy job. In particlur it does not support snapshot versioning:
NuGet Cache And Versioning Issues
Paket is an alternative package manager. It is clearly better suited to do the dependency management in real life projects but I could not find something in the documentation about snapshot versioning.
https://fsprojects.github.io/Paket/
My question: Is it possible to implement snapshot versioning with Paket?
Futher explanations about snapshot versioning:
What exactly is a Maven Snapshot and why do we need it?
http://www.tutorialspoint.com/maven/maven_snapshots.htm
https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN8855
Olaf, I don't think this is supported in paket.dependencies but consider this alternative:
1 put a flexible version restriction in paket.dependencies
nuget GreatDependency ~> 1.0.1
2 invoke custom paket update on build
.paket\paket update nuget GreatDependency
In development, I can reference the latest version of an artifact as 1.2.3-SNAPSHOT. Now I need the same behaviour for release candidates, i.e. I would like to be able to depend on the latest release candidate (there should also be a procedure for the developer to declare development versions as release candidates).
I am not sure how to implement this behaviour properly. Should I use an additional repository for release candidates and move development versions to this repository if the developer requests it? Or can I somehow define a "second snapshot list", like 1.2.3-RC?
You can get there be re-configure a few things:
use a version range for the dependency
change the updatePolicy for the repository you store the release candidates. see https://maven.apache.org/settings.html (updatePolicy). If you store the artifacts in a maven proxy usually you need to allow to overwrite releases.
Remember in a multi module build that they might upload modules before detecting a failed build (due to mvn deploy being a phase not a goal). You need to verify the complete build is ok before starting to upload artifacts in the maven repository. Or stage them somehow.
Remember this will most certainly prevent your builds being reproducible. Since an RC dependency might change between builds. You would need to change the version range - which is not always an issue. Ranges may work for you.
I've better experience to let developers stay on snapshots but have the CI server set an explicit version (for example using the versions plugin) prior to deployment / releasing for the dependency the artifact uses.
Wherever you read about continuous delivery or continuous integration it's recommended to use an artifact repository to store the artifacts even though Jenkins already stores them for each build.
So why is it recommended to use an artifact repository? Is there a smooth solution to work with the artifacts of the Jenkins builds, ex. to use these artifacts for deployment?
An artifact repository and continuous integration tools serve two different purposes and one cannot be substituted with the other. Check this video from Artifactory, one of the providers of artifact repositories, about why one should use an artifact repository.
Jenkins stores the artifacts as plain files without versioning while artifacts in an artifact repository can be version controlled. So you have a lot more flexibility in retrieving artifacts and governing them. Read this very good article on why we need them. Surely not all of those things are supported by continuous integration tools like Jenkins.
Moreover, you can also look at the Artifactory plugin for Jenkins which integrates the two.
An artifact repository is needed but the artifact repo is a conceptual piece an not always a distinct tool. With Jenkins you should have MD5 signatures and (I think) a way of downloading the files you want (web service call, right?) from your remote server. Certainly, if you're doing something simple like using the Jenkins build pipeline plugin, it should be able to access the right versions of the files smoothly.
Alternatively, if you are using a separate deployment tool, the better ones bundle an artifact repository.
Regardless, you want what the ITIL folks call a Definitive Media/Software Library. Definitive in that the bits are secure, trusted, and official. And a library in that they can be easily looked up and accessed. When working with an artifact repository, you need to make sure its adequately secure. It is backed up. It is accessible for your deployments (including to production). If you look at Jenkins and it meets your criteria in those categories, consider yourself done. If it's lacking, and I wouldn't be surprised if it was, then you need either a dedicated tool like the Maven repos, or something bundled with the deploy tooling.
For more of my rambling on the subject, there's a recorded webcast. The slides for that are up on Slideshare.
I haven't kept up to date with Jenkins, we still use a version of the CI when it was orginally called Hudson.
In your projects your poms you should normally point to your own artifact repository were you can fetch and deploy your own (company) projects.
Using an artifact repository with your CI server, it can then deploy successfully built snapshot and releases which can be available to other developers.