We have separate TFS 2012 branches setup for each environment (DEV/QA/PROD).
Changes are checked-in to DEV branch which triggers a release via RM for Visual Studio 2013 Update 4 to the DEV server.
Current release template has Build Definition from DEV branch selected, but we need to switch to QA/PROD branches when moving to the next stage.
Do we need to create separate templates for each stage, instead of using a single template with all stages included?
RM was built on the idea of binary promotion instead of branching per stage. The idea is to have one set of binaries that you promote from one stage to the next. This makes the release process faster (no extraneous builds are happening) and decreases your QA time -- if you test the functionality of the code in QA, then rebuild for production, you're releasing untested binaries into production. It also helps with repeatability. If a release fails between QA and Prod with a branch-per-environment model, you potentially have extra answers to the question "Why did this work in one environment and not the other?". The only answer to that question should be "because there's a problem with the environment". It should never be "because we botched a merge".
You should reevaluate your branching strategy such that you build and release out of one branch, instead of relying on multiple builds out of multiple branches.
That said, your approach of creating separate release paths and templates for each branch would be the best way to tackle it if you can't mess with your branching strategy at the moment.
I usually do something like this, assuming DEV -> QA -> PROD:
Dev branch goes to a dev environment.
QA branch goes to a QA environment.
Prod branch goes to a QA environment, then to prod.
This lets developers continue working on new features while the previous release is being stabilized. If you take this approach, you'll pretty quickly realize that the QA branch is extraneous -- you build for release, then test the thing that you're intending to release.
Once you hit that point, it's a short path to realizing that it's best to keep dev branches short-lived, relying on frequent merges of new features from dev to trunk. Longer-running changes that aren't ready for public consumption can be isolated behind feature flags, so you can continue to roll out smaller changes that are feature-complete and tested while longer-running development work is still taking place.
Related
We are using codepipeline and codebuild with bitbucket cloud for source repositories.
We won't be releasing frequently and our releases are heavily regulated (Reviews by customers in a staging environment first) before releases etc.
We have separate branches for develop, staging, and production environment as the code pipeline doesn't support tag-based releases and feature branches will be frequently merged to the develop branch. The develop branch will be infrequently merged to staging the staging will be merged to master.
We also have a code pipeline per branch that gets triggered when there are updates in those branches.
We currently create pull requests manually from develop to staging and staging to master to trigger the respective pipelines.
I would like to understand how should the current set up be modified (minimally) to make it a true CI-CD but without a complete overhaul of the CI-CD pipeline.
How should I be making sure that the staging/production pipeline should not go through unless the develop pipeline had been triggered for the same code ( I was thinking of updating the list of versions in SSM when develop pipeline deploys so that if the version in staging/master is not found in SSM, the pipeline won't go through. Is there a better way of doing it? should I be doing this or taking a different approach?
P.S: Feature flags don't sound like a good option because we will end up with a lot of code with just feature flags since infrequent deployments. Yes, I know how trunk-based branching is what everyone keeps talking about but I don't see how is it sitting well for us with code pipeline (No tag-based triggers) and infrequent deployments to staging/prod?
It seems to me that your build and deployment process accurately reflects your release process.
I'm assuming in your company something like the following applies:
You need the production branch for hot-fixes to production. These changes need merged back to staging.
Upcoming releases (prod +1) sit in staging for a long time. You
continue to make changes on the staging branch that need merged back
to development and forward to the production branch.
Future releases (prod +2) are done on the development branch and this
code needs merged to the staging branch when ready.
You spend a lot of time managing code merges (and likely firefighting
the problems caused by code regressions)
The C in CI\CD stands for 'Continuous'. CI/CD is designed to accelerate frequent releases to production yet you explicitly say you release infrequently.
Your questions:
Closer to true CI would be to use trunk based development with short lived feature branches. Your pipeline would build a single deployment artefact and deploy this same artefact through develop, staging and production.
Feature flags or equivalent would be used to keep your code always release-able - there are no tag based triggers needed as every commit is potentially release-able.
You have indicated you understand this to some extent but don't feel it applies to your operating environment.
If you must have different pipelines running off different branches for different environments, to me this is a question of your code management process and not for code-pipeline. You should mainly handle this at source, which is in your code management tools and processes.
Your question 2 indicates to me you are not confident in this process internally at your company and are looking inside AWS to fix problems outside of AWS.
This is all a reflection of the unfortunate reality of having long lived branches for different environments.
This is exactly the scenario that that CI was developed as an antidote to but you are discounting two of the main techniques needed for the benefits of CI/CD ( Trunk based development and feature flags) and your release cycle does not support one of the main benefits, i.e frequent releases to production.
Every organisation has their own constraints and particular trade-offs and I of course don't have the full picture.
In my opinion the underlying issues in your question are a direct result of how you release and manage your code and you can't expect the benefits of CI/CD without the practices, which does mean a fundamental overhaul of how you work.
I recently got my team to switch to CI. We're still in the baby steps, but so far we've started using this branching strategy and it's been working great. We don't have any automated tests running yet and do all our testing manually. We are also currently doing weekly releases so we create a release branch from our Dev branch Monday afternoon, run further tests Tuesday and push it live to our Master branch on Wednesday morning and monitor through out the day. The problem is that sometimes our QA team doesn't get to test all our features in time or the feature fails QA, but is already in the Dev branch and could be going live prior to the issues being fixed. Any ways to mitigate this? This is all fairly new to us so I am open to changing everything if necessary.
Hopefully this diagram is a good explanation of how things work currently:
You're not doing Continuous Integration (CI).
In CI you only have one branch (master/main/etc) - on which the CI tool operates - and maybe very short lived child branches that get merged into that branch, ideally in < 1 day. Pretty much synonymous with trunk-based development.
In traditional CI breakages are normal, part of the process - bad changes are identified by the CI tool (one of the reasons you're using it to start with) after they are merged into the branch. Breakages should be fixed ASAP as they block the project's critical path. Unfortunately this requires human intervention, which can be a serious speed bump for larger projects.
There is also gating CI, which performs automated orchestrated verifications of the changes prior to merging them into the branch, rejecting those failing the verifications and automatically merging into the branch the successful ones, thus ensuring that the branch maintain a minimum quality level at all times. IMHO the only comfy and deterministic way to do integration, especially in larger scale projects. AFAIK only 2 such tools out there: Zuul and ApartCI (which is my baby).
The QA/release stages are just subsequent steps in your CI/CD pipeline which are automatically reached if the previous steps pass, see the picture in this answer. Optionally you can pull release branches from the respective refpoints, which would allow complex multi-releases stories, hot-fixes, etc. But these branches are never merged back into their parent, the hot-fixes are instead double-committed/cherry-picked into the master branch, potentially with additional changes as required by the different branch context.
GitFlow is not CI:
you have multiple branches (silos) which you only merge once in a while.
branch merges can rapidly become a nightmare, especially in larger projects where you can have more than 1 feature branch active simultaneously. Try to add 1 then 2 more parallel feature branches in your picture, as well as the additional upstream branch merges that need to be performed before you can do the merges you already show. And the additional QA executions you may need because of these extra merges. Just to get an idea about how badly such strategy scales.
tests done in these isolated branches are out of context - the branches do not reflect the overall project integration. Even if these tests pass the master branch may fail them or be plain broken after merge. See an example of how just 2 simple changes can cause such problems, let alone branches with multiple changes.
As you probably guessed by now, I'm not a big fan of GitFlow, it's awfully similar to the pre-CI integration practices I had to endure a long time ago ;)
Doing trunk based development to achieve continuous deployment. And this is our branching strategy.
master > whats live in production
release > test passed and release point created by CI server
dev > daily merges from the development team.
If we consider doing the pull request from release to master stage.
What are the pros and cons to that approach? How can we communicate this with the development team where they want to do PR at dev branch?
I don't really have an answer, but think the context of the question is worth further discussion.
If you're doing Continuous Deployment, then I'm not sure of the purpose of the Release branch. It seems to be duplicating the purpose of either:
'master': code being deployed/released
'develop' integrated feature
branches, but not ready release
Or, are you using it to group milestones or planned major releases (i.e. Release/1.0, Release/2.0), like a mini-master branch.
I would not consider this Continuous Delivery (deployment, maybe), but it is certainly a valid pattern if your project requires staged releases rather than Continuous Delivery.
It's also important to also consider your CI setup and how it integrates with your branches. It isn't source code that is deployed to Production, but the build artifacts from your CI system. Thinking about this might simplify your branching model.
If you want to rollback, you don't want to rebuild the application from source, you want to re-deploy the pre-built artifacts of the previous version. Likewise, if you have a build that has passed all your tests and is ready to ship - hopefully is already running on your pre-production environments - you don't want to merge it to a different branch, rebuild it and then deploy the new build - you use the build that was tested.
The next consideration is that every additional branch adds time & complexity for the developers. Merging, pull requests, waiting for CI to run, etc. are not free so reducing the complexity of the branching strategy to the minimum you require is a good goal.
To answer your question about where to PR to/from, do you consider "Develop" as the mainline and attempt to keep it stable and working at all times?
If so, then the PRs from feature-to-develop are the key integration. Develop is then built, tested, and deployed to your test environments.
Branching from develop at that point (i.e. to create a Release branch) is then known to be good.
Promoting that artifact from your test environment to production, without re-building, might remove the need for one of your branches.
Right now I have an Azure PaaS solution with a single repo in TFS - we right-click publish from VS to an App Service and then swap slots to get code to production. Small team, disciplined check-ins (or so I thought), etc.
I made a decision to check code in that wasn’t production ready, thinking if needed, I could roll back and publish a hotfix should the need arise.
Well, the need has arisen and I've rolled things back to apply the fix. This was a bit of a headache though.
I’m a little unclear on what the right thing to do is moving forward. What I want to try is:
Create a branch from our MAIN repo and stick all ongoing development there, call it DEV. We'll create two workspaces on our machines - one for each branch.
When we're ready to push a feature, merge down to MAIN and then QA before right-click publishing > Staging > Prod.
At a high level, does this seem like a step in the right direction?
What I’m trying to do is keep this project/alm lean and simple. I don't want to go as far as introducing a build server with RM and other expensive (time, materials, process) components - I just want a sensible, incremental upgrade in the maturity of our current setup to avoid the above headache and this is all I could come up with.
That's two way for your reference, one base on the working flow , one base on the publish (releaseing)
A. Just using mainline and tagging for release
Pros:
Avoid merge hell.
Keeping to the mainline encourages some best practices like proper
release planning, not introducing a lot of WIP, using branching by
abstraction to deal with out-of-band long term work, and using the
open closed system and configurable features for dealing with
managing works in progress that may; or may not; need to be disabled
now or in the future in order to release or to avoid a full rollback.
Cons:
Dealing with works in progress becomes an issue and adds to potential
surface attack area when it comes time to release. However, if your
developers are disciplined then new features should be configurable
and modular and therefore easily disabled/enabled, or there is no WIP
and at each release point all work is either completed or has not yet
been started (i.e. Scrum).
Large scale/out-of-band changes require more thinking ahead of time
to implement.
B. Branch by release
Pros:
You can begin working on the next iteration while the current
iteration finishes its round of acceptance testing.
Cons:
Tons of branches.
Still need to tag branches at release points.
Still need to deal with WIP and merge WIP from previous release
branch into next release branch if it's not going to make it and
still need to disable or yank it of release branch and re-run
acceptance tests.
Hot fixes need to be applied to more branches (release branch +
hotfix + new tag merge hotfix into vnext branch and possibly
vnextnext depending on where the hotfix fails.)
With respect to your point 1. I would not recommend using two workspaces, since you already are running "two workspaces" internally with two branches. The approach is not that bad, just a litte hard to do in TFVC, meaning the old server based version control inside TFS. I do hope your planning to merge everything from dev to main at point in time.
In general your guide is more matching with Git as source-control, and especially gitflow http://nvie.com/posts/a-successful-git-branching-model/ as a branching model. We are running that with success within my team.
You can migrate from TFVC to git using git-tf http://git-tfs.com/
If you are looking for a cheap model that scales out with buildservers and such I would reccommend looking at Visual studio team services https://www.visualstudio.com/en-us/products/visual-studio-team-services-vs.aspx as well to host and build your code. There you also have Release management integrated without cost(up to 5 people/free for all visual studio subscribers)
I have been tasked with updating our build process right now to be more efficient and I have spent a week reading best practices and strategies but I still have not found a solution for our current problem.
Background
Currently we have one monolithic build/application that really needs to be split apart into at least 4 applications with some shared libraries. We currently do not branch unless we absolutely have to. We have a teamcity build that builds on each check-in to TFS. When we are ready for a release we have a code freeze and only check-in fixes for bugs found in QA. Obviously this is a terrible practice and we have finally gotten approval to change it.
Proposed Solutions
The proposed solution will be to split up the application and have different release cycles for each application, move from ant to maven and branch per release.
Branching - Right now we just have a main trunk in source control. I think we want to branch off the trunk when we are ready for a release, and update the branch for bugs found in QA. When the build is ready to be released, merge the branch changes back into the trunk.
Here is how I was planning on setting up TFS.
+Apps
+App1
+Components
+Core
+Web
+Branches
+App2
+Components
+Core
+Web
+Branches
+Libraries
+Lib1
+Lib2
+Branches
Thinking about managing all of the POMs and versions in the POMs seems WAY too difficult right now. I've read up on the maven release plugin, but I'm not sure if it can branch in the way I'm thinking we want to.
Next problem is getting teamcity working. I was thinking of having 3 teamcity projects for each app. A dev project that always points at the trunks, a QA project for testing the QA build and a production project to build changes for hotfixes. Each time a new release comes to QA I would have to update the QA teamcity project to point at the new release branch and update the release build number in teamcity. When that release passes QA I would have to update the production teamcity project to point that the branch that just passed QA and update the build number to the build number that just passed QA.
Surely there is a better strategy that this.
Questions
Where should I be putting these branch folders?
Should QA builds be snapshots still until the build goes to pre-production?
How do you configure teamcity to pick up these branches without changing the sources path for every release?
Should there be parent POMs for each app that the developers use to make sure all of their dependencies are compiled and up to date?
I just want to question your thinking that your applications should be on different release cycles. Modularization is a good thing for code quality but if your modules are on separate release cycles you introduce a lot of overhead. In particular, version management becomes quite a burden and if you get it wrong you can introduce runtime bugs.
How do these separate applications relate to each other? Is there any dependency between them (maybe via a shared library)? Do they communicate with each other? Are they deployed together?
If it is not necessary that they be on separate release cycles then you're probably better off keeping them together.