At present we have more than 10 libraries consumed and deployed across 4 services. There are some libraries like common utilities and protos which are consumed by other libraries, so there is a dependency hierarchy in the libraries as well, all the way leading to the service.
We are facing multiple challenges in deploying any change to the library.
We use Gradle for dependency management, Code Artifact for maven repository management, Code Pipeline for CD, ECR for service docker image storing and ECS for service deployment.
Issue 1: Auto Version Management/Increment using CodeArtifact, Git and Gradle.
When a library is changed (Minor changes without any backward compatibility issues), we don't increment the version. We push the changes to master, which triggers a CodePipeline and builds the library by taking any dependent libraries from CodeArtifact. We follow a manual process to delete the current Library-1.0.0 CodeArtifact so that new artifacts can be published. Removal of existing artifacts is a manual process as well as risky as this prevents easy rollbacks.
Issue 2: Rebuilding and Republishing all child libraries if a root library is changed
When a parent library is changed, all the libraries that depend on it need to be rebuilt using the latest parent library artifact before releasing the changes to service. If we don't do that, and directly rebuild the service, only the service will have the latest code of the library, but all the other libraries still use the old library code, leading to unexpected behaviour.
Ideal expectation here is -> When a service image is getting created, a single or latest version of the library should be used by the service and by all the dependent libraries. Only one version of the library should be active in a service, as part of dependency conflict resolution.
Issue 3: Rollback the deployments
Right now rollbacks are impossible. We delete all the old version library artifacts, so it's impossible to rollback a deployment, the only way to rollback a deployment is to revert all the libraries in git and rebuild, republish all libraries. Would like to know what is the best practice for rollbacks and CI/CD using CodeArtifact, Gradle and Git.
Please help in finding out what we are doing wrong and the correct way to setup CI/CD using Gradle, CodeArtifact and CodeBuild.
Related
I have a set of applications, all use Maven and the local repository. The applications form a dependency tree using <dependency> in their pom.xml. All of these projects have -SNAPSHOT in their version.
Is it possible for Maven (or some compatible dependency manager) to build an application together with all of its local dependencies whose source changed?
I do not want to create a multi-module project, because:
the projects are exactly libraries, not modules;
I do not want an additional complexity just to have a form of build which is already precisely defined;
I want the process to be dynamic: if a library is mature enough to be put into a remote repository, it would be no more rebuilt with the main project and that's ok.
For now, there is a lot of refactoring, moving code from one library to another etc. and it happens often that substantial parts of the dependency tree need to be rebuilt. I thus need to manually write mvn install in several projects in order to assure that there is no stale code.
No, it doesn't work. Even with a multi-module project, maven does not detect which modules have changed sources in it and which do not.
There was a (flaky) implementation in Maven 2, but it was not continued in 3.x, see How to get maven 3.0 to only build modules with local scm changes
I hoped they would include it again in maven 4, but I didn't see it yet: https://maarten.mulders.it/2020/11/whats-new-in-maven-4/
I once did a similar setup, but had to use shell scripts with some git magic to get it working.
You can also decide to put your libraries in separate repo's from the start, and use the repo tool that google uses for android development: https://github.com/GerritCodeReview/git-repo/blob/main/README.md
Once you run mvn install on the particular Maven project, it will be accessible for all other Maven projects, which are on the same workstation, during dependency collection (before the compile phase).
Official Maven Build Lifecycle description:
install - install the package into the local repository, for use as a dependency in other projects locally
It's not necessary to keep libraries as part of the same project(or have it as a multi-module project). But once you want to share those libraries with your teammates, you would need either to force them installing libraries locally (as you did), or store those libraries at some external repo, like Artifactory or Nexus
I have a Gradle build set up in Azure DevOps, which compiles the code in an Azure DevOps git repository, then publishes the generated JARs (as Maven artifacts) to Azure Artifacts, as explained here. Code in other Azure DevOps git repositories can then reference these components as dependencies. This is fine for formal releases of these components (with unique version numbers), but I also need a way to get this working for in-progress snapshot releases. The problem is that I cannot publish an artifact with the same version number (e.g. 1.2.3-SNAPSHOT) more than once. This seems to be because packages in Azure are immutable.
From my understanding, that would mean that Azure Artifacts cannot be used to store in-progress snapshot artifacts. Is that correct?
If it is, is there any alternative that still uses Azure DevOps? I can see that I can publish artifacts to Azure Blob Storage, but presumably this is something you have to pay for on top of existing use of Azure Artifacts. I can also see that there's a number of GitHub Maven plugins for treating a GitHub repo as a Maven repo, but I can't find anything similar for using an Azure DevOps repo as a place to publish Maven artifacts.
In case it makes a difference, I'm talking about the cloud-based Azure stuff, nothing on-premise.
Specifically for maven, there's a difference in how a jar is packaged in a SNAPSHOT as compared to other versions.
Eg. let's say I version my jar as 1.0.0-PRERELEASE, now this version is immutable. Once I publish this to any artifact repository it cannot be overridden. Usually package management systems store the artifact in this kind of logical path - <group_id>/<artifact_id>/<version>/<artifact_id>_<version>.jar.
But for SNAPSHOTs this is slightly different. In maven, SNAPSHOTs are mutable. This means you can override a SNAPSHOT. And this makes sense as by definition SNAPSHOTs are like an image of your code in active development. How SNAPSHOTs are stored in a package management feed is something like <group_id>/<artifact_id>/<version(SNAPSHOT)>/<artifact_id>_<version>_<jar_timestamp>.jar.
This naming convention of the jar in this case makes it mutable. In a single SNAPSHOT version, there can be multiple jars present with different timestamps. And when maven looks for a SNAPSHOT dependency to download, it always picks the most recent one (again logical as that's the most recent snapshot).
At present azure artifacts implements these concepts exactly how they are supposed to. Perhaps this wasn't there before and has been pushed in an update.
http://maven.apache.org/guides/getting-started/index.html#What_is_a_SNAPSHOT_version
https://xebia.com/blog/continuous-releasing-of-maven-artifacts/
The premise of Package Management is that a package is immutable. This enables a whole bunch of caching options that would otherwise not exist. Packages are stored in your local package cache, possibly on a proxy feed package cache and all of these elements assume that packages with the same name+version are unchanged and will serve the cached version instead of the latest version you've pushed. Most package systems are built on this premise, including Nuget and NPM.
The trick to creating development snapshots is to use semantic versioning and adding a unique suffix to your version. For example 1.2.3-SNAPSHOT.1 followed by 1.2.3-SNAPSHOT.2, there are tools available for Azure Pipelines, like GitVersion that can automatically generate a unique version + suffix that you can pass into the version for your artifact.
If you don't want to "mess up" your main package feed, you can setup a second feed for development purposes which holds all your intermediate packages, you can then either promote one of these packages to your main feed or you can run a specific pipeline (configuration) to push the final package to the feed used for your stable packages.
It looks like this is mostly a feature and not a bug as to preserve the immutability of the build results - meaning, that no matter when that build is ran, it will always return the same result. See: How to update a maven dependency with a same version number in Azure Artifacts & Azure Artifact Publishing Fails for Artifact Version Containing '+'
I've spent years trying to deploy libraries that use native code to Maven Central. I've run into the following problems:
There weren't any good plugins for building native code using Maven. native-maven-plugin was a very rigid build system that, among other things, made it difficult to debug the resulting binaries. You'd have to manually synchronize the native-maven-plugin build system with the native IDE you use for debugging.
Maven did not replace variables in deployed pom.xml files: MNG-2971, MNG-4223. This meant that libraries had to declare platform-specific dependencies once per Maven profile (as opposed to declaring the dependency once and setting a different classifier per profile); otherwise, anyone who depended on your library had to re-define those same properties in their project file in order to resolve transitive dependencies. See Maven: Using inherited property in dependency classifier causes build failure.
Jenkins had abysmal support for running similar logic across different platforms (e.g. "shell" vs "batch" tasks, and coordinating a build across multiple machines)
Running Windows, Linux and Mac in virtual machines was way too slow and fragile. Even if you got it working, attempting to configure the VMs as Jenkins slaves was a lesson in frustration (you'd get frequent intermittent build errors).
Maven Central requires a main jar for artifacts that are platform-specific: OSSRH-975
Sonatype OSS Repository Hosting and maven-release-plugin assumed that it would be possible to release a project in an atomic manner from a single machine but I need to build the OS-specific bits on separate machines.
I'm going to use this Stackoverflow question to document how I've managed to overcome these limitations.
Here is how I overcame the aforementioned problems:
I used CMake for building native code. The beauty of this system is that it generates project files for your favorite (native) IDE. You use the same project files to compile and debug the code. You no longer need to synchronize the two systems manually.
Maven didn't support CMake, so I built my own plugin: https://github.com/cmake-maven-project/cmake-maven-project
I manually hard-coded platform-specific dependencies into each Maven profile, instead of defining the dependency once with a different classifier per profile. This was more work, but it doesn't look like they will be fixing this bug in Maven anytime soon.
I plan to investigate http://www.mojohaus.org/flatten-maven-plugin/ and https://github.com/mjiderhamn/promote-maven-plugin as alternatives in the near future.
Jenkins pipeline does a good job of orchestrating a build across multiple machines.
Running Jenkins slaves on virtual machines is still very error-prone but I've managed to workaround most of the problems. I've uploaded my VMWare configuration steps and Jenkins job configuration to help others get started.
I now create an empty JAR file for platform-specific artifacts in order to suppress the Sonatype error. This was actually recommended by Sonatype's support staff.
It turns out that maven-release-plugin delegates to other plugins under the hood. Instead of invoking it, I do the following:
Use mvn versions:set to change the version number from SNAPSHOT to a release and back.
Tag and commit the release myself.
Use nexus-staging:rc-open, nexus-staging:deploy -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}, and nexus-staging:rc-close to upload artifacts from different platforms into the same repository. This is called a Staging Workflow (referenced below).
Upon review, release the repository to Maven Central.
Important: do not enable <autoReleaseAfterClose> in the nexus-staging plugin because it closes the staging repository after each deploy instead of waiting for all deploys to complete.
Per https://issues.sonatype.org/browse/NEXUS-18753 it isn't possible to release SNAPSHOT artifacts atomically (there is no workaround). When releasing SNAPSHOTs, you need to skip rc-open, rc-close and invoke nexus-staging:deploy without -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}. Each artifact will be uploaded into a separate repository.
See my Requirements API for a real-life example that works.
Other quirks to watch out for:
skipNexusStagingDeployMojo must be false in last reactor module (otherwise no artifacts will be deployed): https://issues.sonatype.org/browse/NEXUS-12365. The best workaround is to use Maven profiles to omit whatever modules you want when deploying (don't use skipNexusStagingDeployMojo at all)
skipLocalStaging prevents deploying multiple artifacts into the same repository: https://issues.sonatype.org/browse/NEXUS-12351
I have a number of builds that creates a package that is published to a package manager (for example NPM, Nuget and Maven).
I have subsequent builds that trigger on the completion of this build, they get the artifact from this repository. The problem is they show a warning:
I'm considering adding a Snapshot dependency, however, Teamcity's UI describes snapshot dependencies as builds using the same sources:
There is no source dependency between these projects and in fact, they may have completely different VCS roots.
What is the appropriate way to link these projects? Reading the documentation on Snapshot Dependencies, it sounds like things might not work as expected if I add a dependency without shared sources.
There is no requirement to link these projects, other than by the trigger you already have I don't think.
If the triggered build always gets the latest version from the package manager, then you'll get the behaviour that you want.
Snapshot dependency simply ensures that a build which depends on another build gets the same version of the source code when it builds, and doesn't end up being built using some changes that someone else checked in between the first build starting and the second build starting. This doesn't look like its going to be an issue in your situation (and indeed the builds may use completely different repositories), so I think your finished build trigger is an appropriate solution.
Maven is a dependency manager in the Java world. It supports snapshot versioning. In Maven-speak a dependency is called an artifact. An artifact with a fixed version number (e.g. 1.0.1) will be downloaded only once because it will never change. On the contrary a snapshot version (e.g. 1.0.1-Snapshot) will be considered as a moving target. It is a current development copy and will likely change in the near future. Therefore it has to be updated on a regular basis. With snapshot versioning you are able to provide the current state of an ongoing development as an artifact through your artifact delivery mechanism (for example with Nexus or Artifactory). In combination with a CI build which creates the snapshot artifacts you are able to setup a development infrastructure with can handle fairly complex projects.
In essence I think snapshot versioning means the dependency manager needs to check and download artifacts with a version tag which it has already downloaded before.
In the .NET world Nuget is the prefered package manager. As a dependency manager it does a very lousy job. In particlur it does not support snapshot versioning:
NuGet Cache And Versioning Issues
Paket is an alternative package manager. It is clearly better suited to do the dependency management in real life projects but I could not find something in the documentation about snapshot versioning.
https://fsprojects.github.io/Paket/
My question: Is it possible to implement snapshot versioning with Paket?
Futher explanations about snapshot versioning:
What exactly is a Maven Snapshot and why do we need it?
http://www.tutorialspoint.com/maven/maven_snapshots.htm
https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN8855
Olaf, I don't think this is supported in paket.dependencies but consider this alternative:
1 put a flexible version restriction in paket.dependencies
nuget GreatDependency ~> 1.0.1
2 invoke custom paket update on build
.paket\paket update nuget GreatDependency