I have been tasked with updating our build process right now to be more efficient and I have spent a week reading best practices and strategies but I still have not found a solution for our current problem.
Background
Currently we have one monolithic build/application that really needs to be split apart into at least 4 applications with some shared libraries. We currently do not branch unless we absolutely have to. We have a teamcity build that builds on each check-in to TFS. When we are ready for a release we have a code freeze and only check-in fixes for bugs found in QA. Obviously this is a terrible practice and we have finally gotten approval to change it.
Proposed Solutions
The proposed solution will be to split up the application and have different release cycles for each application, move from ant to maven and branch per release.
Branching - Right now we just have a main trunk in source control. I think we want to branch off the trunk when we are ready for a release, and update the branch for bugs found in QA. When the build is ready to be released, merge the branch changes back into the trunk.
Here is how I was planning on setting up TFS.
+Apps
+App1
+Components
+Core
+Web
+Branches
+App2
+Components
+Core
+Web
+Branches
+Libraries
+Lib1
+Lib2
+Branches
Thinking about managing all of the POMs and versions in the POMs seems WAY too difficult right now. I've read up on the maven release plugin, but I'm not sure if it can branch in the way I'm thinking we want to.
Next problem is getting teamcity working. I was thinking of having 3 teamcity projects for each app. A dev project that always points at the trunks, a QA project for testing the QA build and a production project to build changes for hotfixes. Each time a new release comes to QA I would have to update the QA teamcity project to point at the new release branch and update the release build number in teamcity. When that release passes QA I would have to update the production teamcity project to point that the branch that just passed QA and update the build number to the build number that just passed QA.
Surely there is a better strategy that this.
Questions
Where should I be putting these branch folders?
Should QA builds be snapshots still until the build goes to pre-production?
How do you configure teamcity to pick up these branches without changing the sources path for every release?
Should there be parent POMs for each app that the developers use to make sure all of their dependencies are compiled and up to date?
I just want to question your thinking that your applications should be on different release cycles. Modularization is a good thing for code quality but if your modules are on separate release cycles you introduce a lot of overhead. In particular, version management becomes quite a burden and if you get it wrong you can introduce runtime bugs.
How do these separate applications relate to each other? Is there any dependency between them (maybe via a shared library)? Do they communicate with each other? Are they deployed together?
If it is not necessary that they be on separate release cycles then you're probably better off keeping them together.
Related
Doing trunk based development to achieve continuous deployment. And this is our branching strategy.
master > whats live in production
release > test passed and release point created by CI server
dev > daily merges from the development team.
If we consider doing the pull request from release to master stage.
What are the pros and cons to that approach? How can we communicate this with the development team where they want to do PR at dev branch?
I don't really have an answer, but think the context of the question is worth further discussion.
If you're doing Continuous Deployment, then I'm not sure of the purpose of the Release branch. It seems to be duplicating the purpose of either:
'master': code being deployed/released
'develop' integrated feature
branches, but not ready release
Or, are you using it to group milestones or planned major releases (i.e. Release/1.0, Release/2.0), like a mini-master branch.
I would not consider this Continuous Delivery (deployment, maybe), but it is certainly a valid pattern if your project requires staged releases rather than Continuous Delivery.
It's also important to also consider your CI setup and how it integrates with your branches. It isn't source code that is deployed to Production, but the build artifacts from your CI system. Thinking about this might simplify your branching model.
If you want to rollback, you don't want to rebuild the application from source, you want to re-deploy the pre-built artifacts of the previous version. Likewise, if you have a build that has passed all your tests and is ready to ship - hopefully is already running on your pre-production environments - you don't want to merge it to a different branch, rebuild it and then deploy the new build - you use the build that was tested.
The next consideration is that every additional branch adds time & complexity for the developers. Merging, pull requests, waiting for CI to run, etc. are not free so reducing the complexity of the branching strategy to the minimum you require is a good goal.
To answer your question about where to PR to/from, do you consider "Develop" as the mainline and attempt to keep it stable and working at all times?
If so, then the PRs from feature-to-develop are the key integration. Develop is then built, tested, and deployed to your test environments.
Branching from develop at that point (i.e. to create a Release branch) is then known to be good.
Promoting that artifact from your test environment to production, without re-building, might remove the need for one of your branches.
I have been working with TeamCity, Jenkins and Bamboo my last 8years. Latest 2years very involved in ContinuousIntegration factory setup and mainteinance on my team with very good results, giving me a lot of habits about how to deal with builds, artifacts and pipelines.
Now, i'm on a new company, new team, new CI, TFS2015, first time for me.
Just one month before I arrive to this new team they were on TFS2012, with XAMLs, so I took the migration to vNext builds.
At first look, I found on vNext builds the classic build definition, i mean adding steps as a single task to the build, instead of monolithic XAML file.
But with the time, i was trying to create more complex builds, like TeamCity build chains, but this is not possible, strike one...
Then, I was trying to deal with multiple branches, one continuous build for each branch (we are on TFSVC), create packages from each branch, and I found I was duplicating my builds just to change repository paths and a few details, so I moved builds to templates trying to reuse build definitions, introducing variables to generates paths(for repos and branches) and versions and expecting to change only in one place the build and having this changes reflected in all builds derived from templates... but that's was not the case... :
variables are not accepted everywhere, like in repo paths.
can't change templates after created them, just replace them, and builds created from templates are not affected after template is changed.
strike two ?.....
I'm wondering if maybe I'm doing things wrong with TFS, maybe this is a different system and I can't do things like in other CIs.
Any advice in how to approach TFS to have a good, dynamic and reusable set of builds ??
here isn’t the feature that the changes of a build definition or template affects other existing build definitions.
If the build steps of build definitions are the same for each branch, you just need a build definition and add filters for each branch (Triggers > Continuous integrations (CI), after that it will uses corresponding source to build for CI build, for example, develop branch changes > Trigger a build automatically with develop branch source.
On the other hand, you can change branch and source version when queue build manually or through REST API.
If the build steps are different for each branch, you need to modify the definitions independently for detail requirements.
We have separate TFS 2012 branches setup for each environment (DEV/QA/PROD).
Changes are checked-in to DEV branch which triggers a release via RM for Visual Studio 2013 Update 4 to the DEV server.
Current release template has Build Definition from DEV branch selected, but we need to switch to QA/PROD branches when moving to the next stage.
Do we need to create separate templates for each stage, instead of using a single template with all stages included?
RM was built on the idea of binary promotion instead of branching per stage. The idea is to have one set of binaries that you promote from one stage to the next. This makes the release process faster (no extraneous builds are happening) and decreases your QA time -- if you test the functionality of the code in QA, then rebuild for production, you're releasing untested binaries into production. It also helps with repeatability. If a release fails between QA and Prod with a branch-per-environment model, you potentially have extra answers to the question "Why did this work in one environment and not the other?". The only answer to that question should be "because there's a problem with the environment". It should never be "because we botched a merge".
You should reevaluate your branching strategy such that you build and release out of one branch, instead of relying on multiple builds out of multiple branches.
That said, your approach of creating separate release paths and templates for each branch would be the best way to tackle it if you can't mess with your branching strategy at the moment.
I usually do something like this, assuming DEV -> QA -> PROD:
Dev branch goes to a dev environment.
QA branch goes to a QA environment.
Prod branch goes to a QA environment, then to prod.
This lets developers continue working on new features while the previous release is being stabilized. If you take this approach, you'll pretty quickly realize that the QA branch is extraneous -- you build for release, then test the thing that you're intending to release.
Once you hit that point, it's a short path to realizing that it's best to keep dev branches short-lived, relying on frequent merges of new features from dev to trunk. Longer-running changes that aren't ready for public consumption can be isolated behind feature flags, so you can continue to roll out smaller changes that are feature-complete and tested while longer-running development work is still taking place.
We are using Perforce and Maven and we are in a situation where we have one branch A that hosts a mature project and another branch B, that we are starting to dev on that uses the code hosted in the first branch.
I anticipate that going forward, there could be cases where a check-in into Branch A will result in failed compilation in Branch B because the developers working on them are not aware of specific usages and cross-dependencies. And then devs will waste time updating B code and figure that its not compiling.
Branch/project A needs to remain as its own separate entity because it's code could be used in another future project.
Can someone advice me around handling and alleviating such problem? Any best-practices that you guys can suggest? Much appreciated.
You need to set up a build environment:
1) Set up a continuous integration (CI) server, such as Jenkins. When a developer commits to Branch A, it should detect the commit and trigger a build.
2) When Branch A completes its build, it deploys its artifacts to a shared repository. Highly recommended (buy not strictly necessary) is that the build for Branch A deploys to a Repository Server, such as Nexus (but a repository in source control or a FTP server does the job too).
3) Set up Branch B in your CI server, but with a build "trigger" that runs whenever a build for Branch A succeeds.
4) Configure both build configurations to email relevant team members (or everybody?) whenever a build fails.
Also, both branches should depend on your repository from #2 above. If you need the entire environment to be repeatable and self-contained, go the repository-in-scm route.
I've been doing some reading about continuous integration recently and there is a scenario which could occur which I don't understand how to deal with appropriately.
We have a stable mainline/trunk branch and create branches for features. Each developer will keep their own feature branches up to date by merging from trunk into their branch on a regular basis. However it is entirely possible that two or more feature branches could be created and worked on over a period of several weeks or months. In this time many releases of the software could be deployed. This where my confusion arises.
It is very likely that changes for one feature branch will cause merge conflicts with other feature branches. CI suggests you should merge into trunk at least daily which would resolve the conflicts quickly. However, you may not want to merge the feature code into trunk because it may not be finished or you may not want that feature available in the next release. So, how do you deal with this scenario and still follow CI principles of daily code integration?
There are no feature branches in proper CI. Use feature toggles instead.
The idea explained more fully in this article is to merge from the trunk/release branch to feature branches daily, but only merge back in the other direction once a feature meets your definition of 'done'.
Code written by one feature team will be pushed into the trunk once it's complete, and will be 'distributed' to the other teams, where conflicts can be dealt with, as part of the daily merge process.
This doesn't go as far as satisfying Nick's desire for a version control system that can be used a backup tool, unless the changes being made are small enough that they can be committed to the feature branch within a timeframe where the the risk of losing your work is acceptable.
I personally don't try to reintegrate code into the release branch before it's done, and although I've never really tried, I'm sure building feature toggles in for unfinished work has its own issues.
I think they mean merging mainline into the feature branch, not the other way 'round. This way, the feature branch will not deviate from mainline too much, and be kept in an easily mergeable state.
The git folks do the same thing by rebasing feature branches on top of the master branch before submitting a feature.
In my experience with CI, the way that you should keep your feature branches up to date with the main line changes as others have suggested. This has been working me for several releases. If you are using subversion make sure you to merge with the merge history enable. This way when you are trying to merge your changes back to line it will only like you are merging the feature changes to line, not trying resolve conflicts which your feature might have with the main line. If you are using more advance VCS like git the first merge will be a rebase where the second will be a merge.
There are tools that can support you to get thins done more smoothly like this Feature branches with Bamboo
Feature branches committing back into the mainline, and OFTEN is an essential feature of Continuous Integration. For a thorough breakdown, see This Article
There's now some good resources showing how to combine both CI and feature branches. Bamboo or Feature Branch Notifier are some ways to look.
And this is another quite long article showing pros of so called distributed CI. Hereunder, one excerpt explaining the benefits:
Distributed CI has the advantage for Continuous Deployment because it keeps a clean and stable Mainline branch that can always be deployed to Production. During a Centralized CI process, an unstable Mainline will exist if code does not integrate properly (broken build) or if there is unfinished work integrated. This works quite well with iteration release planning, but creates a bottleneck for Continuous Deployment. The direct line from developer branch to Production must be kept clean in CD, Distributed CI does this by only allowing Production ready code to be put into the Mainline.
One thing that still can be challenging is keeping the branch build isolated so that it doesn't pollute your repository of binaries by pushing its branch builds to it. Bamboo seems to address that, but not sure it's as easy with Jenkins.