Doing trunk based development to achieve continuous deployment. And this is our branching strategy.
master > whats live in production
release > test passed and release point created by CI server
dev > daily merges from the development team.
If we consider doing the pull request from release to master stage.
What are the pros and cons to that approach? How can we communicate this with the development team where they want to do PR at dev branch?
I don't really have an answer, but think the context of the question is worth further discussion.
If you're doing Continuous Deployment, then I'm not sure of the purpose of the Release branch. It seems to be duplicating the purpose of either:
'master': code being deployed/released
'develop' integrated feature
branches, but not ready release
Or, are you using it to group milestones or planned major releases (i.e. Release/1.0, Release/2.0), like a mini-master branch.
I would not consider this Continuous Delivery (deployment, maybe), but it is certainly a valid pattern if your project requires staged releases rather than Continuous Delivery.
It's also important to also consider your CI setup and how it integrates with your branches. It isn't source code that is deployed to Production, but the build artifacts from your CI system. Thinking about this might simplify your branching model.
If you want to rollback, you don't want to rebuild the application from source, you want to re-deploy the pre-built artifacts of the previous version. Likewise, if you have a build that has passed all your tests and is ready to ship - hopefully is already running on your pre-production environments - you don't want to merge it to a different branch, rebuild it and then deploy the new build - you use the build that was tested.
The next consideration is that every additional branch adds time & complexity for the developers. Merging, pull requests, waiting for CI to run, etc. are not free so reducing the complexity of the branching strategy to the minimum you require is a good goal.
To answer your question about where to PR to/from, do you consider "Develop" as the mainline and attempt to keep it stable and working at all times?
If so, then the PRs from feature-to-develop are the key integration. Develop is then built, tested, and deployed to your test environments.
Branching from develop at that point (i.e. to create a Release branch) is then known to be good.
Promoting that artifact from your test environment to production, without re-building, might remove the need for one of your branches.
Related
We are using codepipeline and codebuild with bitbucket cloud for source repositories.
We won't be releasing frequently and our releases are heavily regulated (Reviews by customers in a staging environment first) before releases etc.
We have separate branches for develop, staging, and production environment as the code pipeline doesn't support tag-based releases and feature branches will be frequently merged to the develop branch. The develop branch will be infrequently merged to staging the staging will be merged to master.
We also have a code pipeline per branch that gets triggered when there are updates in those branches.
We currently create pull requests manually from develop to staging and staging to master to trigger the respective pipelines.
I would like to understand how should the current set up be modified (minimally) to make it a true CI-CD but without a complete overhaul of the CI-CD pipeline.
How should I be making sure that the staging/production pipeline should not go through unless the develop pipeline had been triggered for the same code ( I was thinking of updating the list of versions in SSM when develop pipeline deploys so that if the version in staging/master is not found in SSM, the pipeline won't go through. Is there a better way of doing it? should I be doing this or taking a different approach?
P.S: Feature flags don't sound like a good option because we will end up with a lot of code with just feature flags since infrequent deployments. Yes, I know how trunk-based branching is what everyone keeps talking about but I don't see how is it sitting well for us with code pipeline (No tag-based triggers) and infrequent deployments to staging/prod?
It seems to me that your build and deployment process accurately reflects your release process.
I'm assuming in your company something like the following applies:
You need the production branch for hot-fixes to production. These changes need merged back to staging.
Upcoming releases (prod +1) sit in staging for a long time. You
continue to make changes on the staging branch that need merged back
to development and forward to the production branch.
Future releases (prod +2) are done on the development branch and this
code needs merged to the staging branch when ready.
You spend a lot of time managing code merges (and likely firefighting
the problems caused by code regressions)
The C in CI\CD stands for 'Continuous'. CI/CD is designed to accelerate frequent releases to production yet you explicitly say you release infrequently.
Your questions:
Closer to true CI would be to use trunk based development with short lived feature branches. Your pipeline would build a single deployment artefact and deploy this same artefact through develop, staging and production.
Feature flags or equivalent would be used to keep your code always release-able - there are no tag based triggers needed as every commit is potentially release-able.
You have indicated you understand this to some extent but don't feel it applies to your operating environment.
If you must have different pipelines running off different branches for different environments, to me this is a question of your code management process and not for code-pipeline. You should mainly handle this at source, which is in your code management tools and processes.
Your question 2 indicates to me you are not confident in this process internally at your company and are looking inside AWS to fix problems outside of AWS.
This is all a reflection of the unfortunate reality of having long lived branches for different environments.
This is exactly the scenario that that CI was developed as an antidote to but you are discounting two of the main techniques needed for the benefits of CI/CD ( Trunk based development and feature flags) and your release cycle does not support one of the main benefits, i.e frequent releases to production.
Every organisation has their own constraints and particular trade-offs and I of course don't have the full picture.
In my opinion the underlying issues in your question are a direct result of how you release and manage your code and you can't expect the benefits of CI/CD without the practices, which does mean a fundamental overhaul of how you work.
I recently got my team to switch to CI. We're still in the baby steps, but so far we've started using this branching strategy and it's been working great. We don't have any automated tests running yet and do all our testing manually. We are also currently doing weekly releases so we create a release branch from our Dev branch Monday afternoon, run further tests Tuesday and push it live to our Master branch on Wednesday morning and monitor through out the day. The problem is that sometimes our QA team doesn't get to test all our features in time or the feature fails QA, but is already in the Dev branch and could be going live prior to the issues being fixed. Any ways to mitigate this? This is all fairly new to us so I am open to changing everything if necessary.
Hopefully this diagram is a good explanation of how things work currently:
You're not doing Continuous Integration (CI).
In CI you only have one branch (master/main/etc) - on which the CI tool operates - and maybe very short lived child branches that get merged into that branch, ideally in < 1 day. Pretty much synonymous with trunk-based development.
In traditional CI breakages are normal, part of the process - bad changes are identified by the CI tool (one of the reasons you're using it to start with) after they are merged into the branch. Breakages should be fixed ASAP as they block the project's critical path. Unfortunately this requires human intervention, which can be a serious speed bump for larger projects.
There is also gating CI, which performs automated orchestrated verifications of the changes prior to merging them into the branch, rejecting those failing the verifications and automatically merging into the branch the successful ones, thus ensuring that the branch maintain a minimum quality level at all times. IMHO the only comfy and deterministic way to do integration, especially in larger scale projects. AFAIK only 2 such tools out there: Zuul and ApartCI (which is my baby).
The QA/release stages are just subsequent steps in your CI/CD pipeline which are automatically reached if the previous steps pass, see the picture in this answer. Optionally you can pull release branches from the respective refpoints, which would allow complex multi-releases stories, hot-fixes, etc. But these branches are never merged back into their parent, the hot-fixes are instead double-committed/cherry-picked into the master branch, potentially with additional changes as required by the different branch context.
GitFlow is not CI:
you have multiple branches (silos) which you only merge once in a while.
branch merges can rapidly become a nightmare, especially in larger projects where you can have more than 1 feature branch active simultaneously. Try to add 1 then 2 more parallel feature branches in your picture, as well as the additional upstream branch merges that need to be performed before you can do the merges you already show. And the additional QA executions you may need because of these extra merges. Just to get an idea about how badly such strategy scales.
tests done in these isolated branches are out of context - the branches do not reflect the overall project integration. Even if these tests pass the master branch may fail them or be plain broken after merge. See an example of how just 2 simple changes can cause such problems, let alone branches with multiple changes.
As you probably guessed by now, I'm not a big fan of GitFlow, it's awfully similar to the pre-CI integration practices I had to endure a long time ago ;)
We have separate TFS 2012 branches setup for each environment (DEV/QA/PROD).
Changes are checked-in to DEV branch which triggers a release via RM for Visual Studio 2013 Update 4 to the DEV server.
Current release template has Build Definition from DEV branch selected, but we need to switch to QA/PROD branches when moving to the next stage.
Do we need to create separate templates for each stage, instead of using a single template with all stages included?
RM was built on the idea of binary promotion instead of branching per stage. The idea is to have one set of binaries that you promote from one stage to the next. This makes the release process faster (no extraneous builds are happening) and decreases your QA time -- if you test the functionality of the code in QA, then rebuild for production, you're releasing untested binaries into production. It also helps with repeatability. If a release fails between QA and Prod with a branch-per-environment model, you potentially have extra answers to the question "Why did this work in one environment and not the other?". The only answer to that question should be "because there's a problem with the environment". It should never be "because we botched a merge".
You should reevaluate your branching strategy such that you build and release out of one branch, instead of relying on multiple builds out of multiple branches.
That said, your approach of creating separate release paths and templates for each branch would be the best way to tackle it if you can't mess with your branching strategy at the moment.
I usually do something like this, assuming DEV -> QA -> PROD:
Dev branch goes to a dev environment.
QA branch goes to a QA environment.
Prod branch goes to a QA environment, then to prod.
This lets developers continue working on new features while the previous release is being stabilized. If you take this approach, you'll pretty quickly realize that the QA branch is extraneous -- you build for release, then test the thing that you're intending to release.
Once you hit that point, it's a short path to realizing that it's best to keep dev branches short-lived, relying on frequent merges of new features from dev to trunk. Longer-running changes that aren't ready for public consumption can be isolated behind feature flags, so you can continue to roll out smaller changes that are feature-complete and tested while longer-running development work is still taking place.
I have been tasked with updating our build process right now to be more efficient and I have spent a week reading best practices and strategies but I still have not found a solution for our current problem.
Background
Currently we have one monolithic build/application that really needs to be split apart into at least 4 applications with some shared libraries. We currently do not branch unless we absolutely have to. We have a teamcity build that builds on each check-in to TFS. When we are ready for a release we have a code freeze and only check-in fixes for bugs found in QA. Obviously this is a terrible practice and we have finally gotten approval to change it.
Proposed Solutions
The proposed solution will be to split up the application and have different release cycles for each application, move from ant to maven and branch per release.
Branching - Right now we just have a main trunk in source control. I think we want to branch off the trunk when we are ready for a release, and update the branch for bugs found in QA. When the build is ready to be released, merge the branch changes back into the trunk.
Here is how I was planning on setting up TFS.
+Apps
+App1
+Components
+Core
+Web
+Branches
+App2
+Components
+Core
+Web
+Branches
+Libraries
+Lib1
+Lib2
+Branches
Thinking about managing all of the POMs and versions in the POMs seems WAY too difficult right now. I've read up on the maven release plugin, but I'm not sure if it can branch in the way I'm thinking we want to.
Next problem is getting teamcity working. I was thinking of having 3 teamcity projects for each app. A dev project that always points at the trunks, a QA project for testing the QA build and a production project to build changes for hotfixes. Each time a new release comes to QA I would have to update the QA teamcity project to point at the new release branch and update the release build number in teamcity. When that release passes QA I would have to update the production teamcity project to point that the branch that just passed QA and update the build number to the build number that just passed QA.
Surely there is a better strategy that this.
Questions
Where should I be putting these branch folders?
Should QA builds be snapshots still until the build goes to pre-production?
How do you configure teamcity to pick up these branches without changing the sources path for every release?
Should there be parent POMs for each app that the developers use to make sure all of their dependencies are compiled and up to date?
I just want to question your thinking that your applications should be on different release cycles. Modularization is a good thing for code quality but if your modules are on separate release cycles you introduce a lot of overhead. In particular, version management becomes quite a burden and if you get it wrong you can introduce runtime bugs.
How do these separate applications relate to each other? Is there any dependency between them (maybe via a shared library)? Do they communicate with each other? Are they deployed together?
If it is not necessary that they be on separate release cycles then you're probably better off keeping them together.
I've been doing some reading about continuous integration recently and there is a scenario which could occur which I don't understand how to deal with appropriately.
We have a stable mainline/trunk branch and create branches for features. Each developer will keep their own feature branches up to date by merging from trunk into their branch on a regular basis. However it is entirely possible that two or more feature branches could be created and worked on over a period of several weeks or months. In this time many releases of the software could be deployed. This where my confusion arises.
It is very likely that changes for one feature branch will cause merge conflicts with other feature branches. CI suggests you should merge into trunk at least daily which would resolve the conflicts quickly. However, you may not want to merge the feature code into trunk because it may not be finished or you may not want that feature available in the next release. So, how do you deal with this scenario and still follow CI principles of daily code integration?
There are no feature branches in proper CI. Use feature toggles instead.
The idea explained more fully in this article is to merge from the trunk/release branch to feature branches daily, but only merge back in the other direction once a feature meets your definition of 'done'.
Code written by one feature team will be pushed into the trunk once it's complete, and will be 'distributed' to the other teams, where conflicts can be dealt with, as part of the daily merge process.
This doesn't go as far as satisfying Nick's desire for a version control system that can be used a backup tool, unless the changes being made are small enough that they can be committed to the feature branch within a timeframe where the the risk of losing your work is acceptable.
I personally don't try to reintegrate code into the release branch before it's done, and although I've never really tried, I'm sure building feature toggles in for unfinished work has its own issues.
I think they mean merging mainline into the feature branch, not the other way 'round. This way, the feature branch will not deviate from mainline too much, and be kept in an easily mergeable state.
The git folks do the same thing by rebasing feature branches on top of the master branch before submitting a feature.
In my experience with CI, the way that you should keep your feature branches up to date with the main line changes as others have suggested. This has been working me for several releases. If you are using subversion make sure you to merge with the merge history enable. This way when you are trying to merge your changes back to line it will only like you are merging the feature changes to line, not trying resolve conflicts which your feature might have with the main line. If you are using more advance VCS like git the first merge will be a rebase where the second will be a merge.
There are tools that can support you to get thins done more smoothly like this Feature branches with Bamboo
Feature branches committing back into the mainline, and OFTEN is an essential feature of Continuous Integration. For a thorough breakdown, see This Article
There's now some good resources showing how to combine both CI and feature branches. Bamboo or Feature Branch Notifier are some ways to look.
And this is another quite long article showing pros of so called distributed CI. Hereunder, one excerpt explaining the benefits:
Distributed CI has the advantage for Continuous Deployment because it keeps a clean and stable Mainline branch that can always be deployed to Production. During a Centralized CI process, an unstable Mainline will exist if code does not integrate properly (broken build) or if there is unfinished work integrated. This works quite well with iteration release planning, but creates a bottleneck for Continuous Deployment. The direct line from developer branch to Production must be kept clean in CD, Distributed CI does this by only allowing Production ready code to be put into the Mainline.
One thing that still can be challenging is keeping the branch build isolated so that it doesn't pollute your repository of binaries by pushing its branch builds to it. Bamboo seems to address that, but not sure it's as easy with Jenkins.