I have yet to find clear guidance on semantically versioning software releases using Azure DevOps (Server) CI/CD.
My basic understanding is that I would set up a CI pipeline for our team so that we get the benefits of build failure notifications, test execution, code coverage, static code analysis and even more goodies.
The CD pipeline picks up on the CI pipeline, rolling out the artifacts on the different environments upon their completion.
Using this approach seems to not make sense to me. What about builds that fail because a dev didn't pay attention or that the team wants to discard? Such builds won't find their way into production but might use up version numbers, leading to gaps in our versioning scheme.
What is your experience or approach for semantically versioning software releases using CI/CD pipelines? Do you cherry-pick builds? Do you have a separate build pipeline for building releases?
Builds that fail don't get deployed. For builds that succeed but fail QA or integration testing when deployed to lower environments, you can put approval gates so that someone has to approve the release before it proceeds to higher environments.
If you build version 1.0.1 of your application and deploy it to dev and it's no good, that doesn't mean that version 1.0.1 doesn't exist. It exists. It was comprised of specific code assets. It was bad, and your users will never see it, but that's fine. If the users see version 1.0.1 jump to 1.0.94, why does it matter?
Related
I would like to integrate an automated versioning system in my ASP.NET project. For each release, it should have a version number based on the previous release. I am planning to integrate Gitversion https://gitversion.net/. Does anyone use it in your projects? For the CI/CD pipeline we have teamcity and octopus deploy.
What is the best practice for automated software release versions?
Thanks in advance
As one of the maintainers of GitVersion, I'm obviously biased, but since you're asking how to use GitVersion to implement a "best practice for automated software release", I'm going to unashamedly give you a textual description of a talk I've done on how I prefer to version and release software with GitVersion, TeamCity and Octopus Deploy.
Developer Workflow
The first thing you should figure out is what kind of developer workflow you want for your software. GitVersion supports Git Flow and many simplified variants of it, and GitHub Flow (as well as many other trunk based development flows). Which workflow you should choose depends on what type of software you are developing, your team and most importantly; your personal preference.
Once you have chosen your workflow, you can configure which mode GitVersion should operate under.
Version Source
GitVersion works by calculating a version number from your Git repository. This means that you should not commit a version number to Git in any shape or form. Not within a package.json, pom.xml, .csproj, or any other build- or project-related file that often mandates the existence of a version number.
Versioning
Instead, you should allow GitVersion to produce a version number based on the Git history, using the currently checked out commit as its starting point, and searching through the parents and their tags to calculate an appropriate version number for the current commit. This version number can then be used in any way you want in your build pipeline. You can, for instance, write GitVersion's FullSemVer variable to package.json by executing the following command:
npm version $GitVersion_FullSemVer
If you are developing on the .NET platform, it's also possible to use GitVersion to patch your AssemblyInfo.cs files so the calculated version number is compiled into your assemblies. With InformationalVersion containing the SHA of the Git commit being built and versioned, you'll be able to identify the exact origin of a compiled assembly.
Build
Once you have your workflow in order and GitVersion has a good source of information to use for its versioning, you can go ahead and create a build pipeline for your software. A typical build pipeline will look something like this:
git clone. (Ensure that the clone is complete and not shallow or a detached HEAD. See requirements for more information.)
GitVersion. Execute GitVersion by whichever means make most sense to your environment.
Patch. Patch the version number created by GitVersion in step 2 into every file in your repository that makes sense, such as AssemblyInfo.cs, package.json, etc.
Build. Perform the build of your software.
Test. Run tests that ensure the quality of the software.
Package. Create a package of your software, using the version number created by GitVersion in step 2.
Release. Release the package using the package management software of your choice, such as Octopus Deploy, npm, nuget, composer, or similar.
Test. Perform automatic tests of the released software, if possible. If successful, it's possible to automatically promote the released software to other environments, if applicable.
Through GitVersion's built-in support for build servers, the calculated version number will also be promoted to the build server to version the build itself. This will be done automatically on TeamCity, for instance. In TeamCity, I recommend that you run GitVersion as its own build configuration exposing the required variables which can then be used in dependent build configurations later on.
Release
Once you have a built artifact containing the version number generated by GitVersion, you can use the same version number to create a package, create a release and deploy the release in Octopus Deploy.
You told you want to integrate an automated versioning system? I would like to throw my hat in the ring.
I am author of Vernuntii, a simple semantic versioning library with git integration.
The answer of #Asbjørn told you already about the best practices for example choosing a workflow that is suited for your project.
The main part a versioning tool like GitVesion or Vernuntii is to generate a suitable NEXT VERSION based on (non-)existing git tags.
So at the end of the day it is a matter of taste what kind of complexity you want HOW the NEXT VERSION is calculated.
So when you want cross-branch versioning, then you are good to go with GitVersion, but if you don't need that kind of complexity then you can try a single-branch versioning like it is implemented in Vernuntii.
For more informations take a look at the README.md of Vernuntii.
Here, for giving you my impression of versioning tools and their complexity, let me give you a list (sorted from least complex to most complex):
MinVer
Verlite
Vernuntii
Nerdbank.GitVersioning
GitVersion
A fun fact beside: all libraries from top to Vernuntii also allows to calculate the next version from detached HEAD.
We are using codepipeline and codebuild with bitbucket cloud for source repositories.
We won't be releasing frequently and our releases are heavily regulated (Reviews by customers in a staging environment first) before releases etc.
We have separate branches for develop, staging, and production environment as the code pipeline doesn't support tag-based releases and feature branches will be frequently merged to the develop branch. The develop branch will be infrequently merged to staging the staging will be merged to master.
We also have a code pipeline per branch that gets triggered when there are updates in those branches.
We currently create pull requests manually from develop to staging and staging to master to trigger the respective pipelines.
I would like to understand how should the current set up be modified (minimally) to make it a true CI-CD but without a complete overhaul of the CI-CD pipeline.
How should I be making sure that the staging/production pipeline should not go through unless the develop pipeline had been triggered for the same code ( I was thinking of updating the list of versions in SSM when develop pipeline deploys so that if the version in staging/master is not found in SSM, the pipeline won't go through. Is there a better way of doing it? should I be doing this or taking a different approach?
P.S: Feature flags don't sound like a good option because we will end up with a lot of code with just feature flags since infrequent deployments. Yes, I know how trunk-based branching is what everyone keeps talking about but I don't see how is it sitting well for us with code pipeline (No tag-based triggers) and infrequent deployments to staging/prod?
It seems to me that your build and deployment process accurately reflects your release process.
I'm assuming in your company something like the following applies:
You need the production branch for hot-fixes to production. These changes need merged back to staging.
Upcoming releases (prod +1) sit in staging for a long time. You
continue to make changes on the staging branch that need merged back
to development and forward to the production branch.
Future releases (prod +2) are done on the development branch and this
code needs merged to the staging branch when ready.
You spend a lot of time managing code merges (and likely firefighting
the problems caused by code regressions)
The C in CI\CD stands for 'Continuous'. CI/CD is designed to accelerate frequent releases to production yet you explicitly say you release infrequently.
Your questions:
Closer to true CI would be to use trunk based development with short lived feature branches. Your pipeline would build a single deployment artefact and deploy this same artefact through develop, staging and production.
Feature flags or equivalent would be used to keep your code always release-able - there are no tag based triggers needed as every commit is potentially release-able.
You have indicated you understand this to some extent but don't feel it applies to your operating environment.
If you must have different pipelines running off different branches for different environments, to me this is a question of your code management process and not for code-pipeline. You should mainly handle this at source, which is in your code management tools and processes.
Your question 2 indicates to me you are not confident in this process internally at your company and are looking inside AWS to fix problems outside of AWS.
This is all a reflection of the unfortunate reality of having long lived branches for different environments.
This is exactly the scenario that that CI was developed as an antidote to but you are discounting two of the main techniques needed for the benefits of CI/CD ( Trunk based development and feature flags) and your release cycle does not support one of the main benefits, i.e frequent releases to production.
Every organisation has their own constraints and particular trade-offs and I of course don't have the full picture.
In my opinion the underlying issues in your question are a direct result of how you release and manage your code and you can't expect the benefits of CI/CD without the practices, which does mean a fundamental overhaul of how you work.
Doing trunk based development to achieve continuous deployment. And this is our branching strategy.
master > whats live in production
release > test passed and release point created by CI server
dev > daily merges from the development team.
If we consider doing the pull request from release to master stage.
What are the pros and cons to that approach? How can we communicate this with the development team where they want to do PR at dev branch?
I don't really have an answer, but think the context of the question is worth further discussion.
If you're doing Continuous Deployment, then I'm not sure of the purpose of the Release branch. It seems to be duplicating the purpose of either:
'master': code being deployed/released
'develop' integrated feature
branches, but not ready release
Or, are you using it to group milestones or planned major releases (i.e. Release/1.0, Release/2.0), like a mini-master branch.
I would not consider this Continuous Delivery (deployment, maybe), but it is certainly a valid pattern if your project requires staged releases rather than Continuous Delivery.
It's also important to also consider your CI setup and how it integrates with your branches. It isn't source code that is deployed to Production, but the build artifacts from your CI system. Thinking about this might simplify your branching model.
If you want to rollback, you don't want to rebuild the application from source, you want to re-deploy the pre-built artifacts of the previous version. Likewise, if you have a build that has passed all your tests and is ready to ship - hopefully is already running on your pre-production environments - you don't want to merge it to a different branch, rebuild it and then deploy the new build - you use the build that was tested.
The next consideration is that every additional branch adds time & complexity for the developers. Merging, pull requests, waiting for CI to run, etc. are not free so reducing the complexity of the branching strategy to the minimum you require is a good goal.
To answer your question about where to PR to/from, do you consider "Develop" as the mainline and attempt to keep it stable and working at all times?
If so, then the PRs from feature-to-develop are the key integration. Develop is then built, tested, and deployed to your test environments.
Branching from develop at that point (i.e. to create a Release branch) is then known to be good.
Promoting that artifact from your test environment to production, without re-building, might remove the need for one of your branches.
I'm looking for advice on the best approach to setup TeamCity/Octopus.
Currently I have multiple branches in TFS2015 - dev, main and release (currently we create a release branch for each release).
Our procedure is to develop in dev and deploy to dev environment. When we are ready for testing we merge from dev to main and deploy to test from main. When happy we create a release branch and deploy to live from the release branch. This is a manual process.
Hotfixes are done on the release branch and deployed to live. We then merge back to main/dev.
I'm totally new to this and so far in a VM playground I've setup TFS2015, TeamCity and Octopus and can check-in to TFS, build/create package on TeamCity and deploy this pack from Octopus. But...
I'm unsure how I should setup TeamCity and Octopus to work with multiple branches? Multiple projects for each branch and generate different artifacts?
When I do this for real I have a TFS VM, I plan on installing TeamCity and Octopus on this along with the build agent. Is this a bad idea? Should I create a new VM just for TM and Octopus?
Any advice or best practice would be appreciated.
Although your question is good in scope, but a good answer must cover many details to be complete.
Let me try to point out the main areas that you will need to further investigate and configure.
TeamCity
A VCS root can be configured to listen to multiple branches via a branch specification
A VCS root can contain multiple projects/solutions and these can be built in multiple steps within TeamCity.
Given that Team City does not support conditional build steps, you will need a different strategy to allow you to vary build steps (and parameters) per release channels / environments.
My recommend approach is to split up the builds into a build definition per release channel (target environment).
Dev and Feature branches could share a single build definition.
Master and Hotfix branches can share a single build definition since they both publish to staging/production environments.
Release branches will need a separate build definition and publish to QA/Testing environment.
This gives you fine grained control over parameter and configurations of each release channel. build a debug version of your app from Dev branch for example at major version 3, while build a release version from Master branch with major version 2.
Every build definition will have a step to publish its artefacts/packages to Octopus Deploy, and specify the channel of which the artifacts belong to.
Octopus Deploy
In Octopus Deploy, define the channels to reflect the release channels, that also reflect your branching model.
Develop, Test, Release are my standard goto channels
Each channel can enforce a different Lifecycle to limit the environments that a channel can deploy to and how an application progresses through your overall ALM cycle.
I know this is not a complete answer. but it is a good start that I hope can help you refine your question to more specific technical details.
We're having somewhat similar CI setup requirements except TFS. In our case workflow for most projects is: GitHub -> TeamCity -> Octopus Deploy.
So I'm not sure about multi-branch setup with TFS, but in case with GitHub repos it's pretty easy to configure in TeamCity. You just have to specify branch-related settings in your VCS root (see Branch configuration). When you have configured that, TeamCity will let you run build's for every specified branch separately and will display build statuses for every branch nicely.
In Octopus we use Channels feature to split workflows of releases coming from different branches. That means we have channel-per-branch convention for the projects, so that TeamCity is pushing packaged releases from particular branch (in our case it's develop and master) into it's respective channel in Octopus (see Channels in Octopus).
Probably you can setup all the services on single machine but imho it's not the best practice to do performance-wise and scalability-wise.
Off course I don't know you code etc but I think you should step away from merging from dev to test and then creating a version from test. That way you essentially are building a different application compared to the one you were having on dev. Once you merge from test to production and build your application from there, you are releasing a build you haven't been testing.
You should strive for a flow in which you build once and deploy multiple times. So, build one package which you promote from dev to test to production.
Off course you can have a production branch on which you could fix bugs etc. The Channels feature in Octopus works great for scenario's like that.
So answering my own question (sorry), the approach I ended up taking was to simplify my branches and configure TeamCity/Octopus like so...
Branching Strategy
I've moved from
--dev
--main
--release
----release1
----release2
to
--master
--release
----release1
----release2
Master is where most of the devs do their work, when we are ready for a release we have a cut-off point and merge master into a new release branch.
The release branch is deployed to test and any fixes from testing are made on the release branch.
When testing is complete we deploy to live/production from this branch.
This means that the binaries we have tested are exactly the same as the ones we deploy to live/production.
Teamcity
In TeamCity master is automatically built each time a check-in occurs. Then the package is pushed to Octopus. Octopus acts as a repository in this case. TeamCity also creates the release on Octopus. So it should be checkin->build->create release->deploy.
To do this, I have one VCS Root and have a build configuration called Build-Master. This uses the checkout rules to ensure I'm only using the master branch. I use the Ocotpus packaging to build the package then use the OctopusDeploy runner in TeamCity to create a release automatically and deploy to the dev servers.
Release is different. I want to deploy to the test servers manually rather than each time a check-in occurs. When happy promote this to the live production servers.
Any fixes from test will be made to the release branch and deployed to test.
When testing is complete we promote to live and any hotfixes are made on the release branch. Obviously all fixes/hotfixes are merged down to master.
So, in TeamCity to achieve this I have a build configuration called Build-Release. Again, I use the checkout rules to ensure I'm dealing with the correct release branch.
The build creates a package using OctoPack, however this time it's not pushed to Octopus.
Octopus
Octopus has a project specifically for deploying master to our dev servers, for example projectnamehere-dev.
In Octopus, I have a separate project for Test/Prod. I've setup an external feed which points at TeamCity so I can pickup the package created in TeamCity. This is setup in Library->External Feeds.
So, to deploy to test. I create the release branch in TFS and give it a version number, 1,2,3 etc. I then change the Build-Release build config to point at this new branch. Change the version number.
Then in Octopus, I create a release, select this package and deploy to test. Any fixes from testing are made on this release branch. I just build the package again and create a new release and choose the new package.
When testing is complete, in Octopus I just promote the last release to the live production servers.
Channels in Octopus are used on the two projects because they have different life cycles. I also created two new life cycles, the default is dev/test/prod. I created just a dev and then test/prod.
Hope this helps.
In the version control settings -> vcs -> Branch Specifications: "*" ("This will do all branch, filter as needs be" e.g. +:refs/heads/master +:refs/heads/develop)
enter image description here
Octopus doesn't handle branches, it only deploys, you can however use their rest api, so for example, if test pass in develop then call the octopus rest api to create a new release and deploy.
I am referred to Hudson today.
I have heard about continuous integration before, but I have no idea what the heck is a ci-server.
Hudson is really easy to install in Ubuntu and in several minutes I managed to set up an instance of it.
But I don't quite understand the workflow of a ci-server, or how am I supposed to use it?
Please tell me if you have experience about ci, thanks in advance.
Edit:
I am currently using Mercurial as my SCM, and I wonder what is the right way to use it with Hudson.
I have installed the Mercurial Plugin of Hudson, and I create a new job with a local repository. When I commit in the repository the Hudson job is built with the latest version of my source code.
If what I used is a remote repository, what's the workflow like?
Is it something like the following?
Set up a Hudson job with the repository
Developer makes a local clone of the repository
Developer commit and push changes
The remote repository update with the incoming changeset
Run a Hudson build
There may be something I misunderstanded at all, please help me point it out.
Continuous Integration is the process of "integrating software" continuously i.e. as frequently as possible (ultimately after each set of changes) to avoid any big-bang integration and all subsequent problems by getting immediate feedback.
To implement Continuous Integration, you first need to automate the build of your software (where build means of course compiling sources, packaging them, but also compiling tests, running the tests, running quality checks, etc, anything that will help to get feedback on the health of your code). Then you need to trigger the build on the latest version of the sources on a particular event (a change in the repository, a temporal event), to generate reports and to send notifications upon failure (by mail, twitter, etc).
And this is precisely the responsibility of a CI engine: offering trigger mechanisms, being able to get the latest version of the sources, running the build, generating and publishing reports, sending notifications. CI engines do implement this.
And because running a build is CPU and Disk intensive, CI engines usually run on a dedicated machine (or even a farm of machines if you want to build lots of projects).
Back to your question now. Once you've got Hudson running, configure it (Manage Hudson > Configure System): setup the JDK, build tools, etc. Then setup an Hudson Job and follow the steps: configure the location of the source repository, the build tool, the trigger, a notification channel and you're done (you can do more complex things but that's a start).
For more details on the setup, check:
The official Use Hudson guide for more details. << START HERE
Continuous Integration with Hudson - Tutorial.
Spot defects early with Continuous Integration.
Martin Fowler's overview of continuous integration is one of the canonical references. In my opinion, using automation to make sure your code base is healthy is one of the most useful things that you can set up.
Update Sorry that I didn't have much time earlier to expand on my reply. #Pascal_Thivent is right that in order to effectively use CI, you need to be able to automate your builds, tests, etc. CI is actually a good forcing function for this. For me, it's one of those little warning flags if I start to think that it would be too painful to put a build into Hudson. It means that something is not quite right.
What I like about Hudson is that it's flexible enough to accommodate different workflows. We use it for both builds / unit tests and releases. And it eliminates a lot of the worry about certain release procedures only working in one person's environment.
What I don't like about Hudson is that it is occasionally unstable when new builds break plugins. I've had a couple of upgrades (2 out of 10 or so) go bad because of incompatibilities. I do two things now:
I never upgrade my team's Hudson server to the latest and greatest right away. I generally only upgrade when there are significant new features, or bug fixes.
I now have a basic Hudson instance set up with all my plugins on a virtual machine with some dummy builds that I fire up to test out any new upgrades before doing it on the public server.