What are the pros/cons of having a centralized continuous integration/build setup (in our case it will be cruise control) as opposed to project-specific setup?
So far, we had a project specific setup of cruisecontrol, but now many other groups want to move to CI as well, and are asking for a central cruiseserver. I'd like to know if anyone has any experience (good/bad) to share that can help me on this.
Here are some initial thoughts:
Pros: A central system would be easy to administer avoiding duplication of efforts all across the organization in setting up and maintaining cruise instances for various projects.
Cons: The central system may have to be a more controlled setup w.r.t access etc., and hence may need to be owned by a dedicated person/group, while the project cruise instances can be under the control of the project group, making changes/enhancements faster and project-specific.
I'd appreciate any inputs on this.
One issue we have with our centralized systems is queue length. The more projects you have on the system the more time you have to wait. Or the more you have to split your CI into distributed tasks.
Related
I have scoured the internet to find out what I can on this, but have come away short. I need to know two things.
Firstly, is there a best practice for how TFS & Team Build should be used in a Development > Test > Production environment? I currently have my local VS get the latest files. Then I work on them & check them in. This creates a build that then pushes the published files into a location on the test server which IIS references. This creates my test environment. I wonder then what is the best practice for deploying this to a Live environment once testing is complete?
Secondly, off the back of the previous - my web application is connected to a database. So, the test version will point to a test database. But when this is then tested and put live, I will need that process to also make sure that any data connections are changed to the live database.
I am pretty much doing all this from scratch and am learning as I go along.
I'd suggest you to look at Microsoft Release Management since it's the tool that can help you to do exactly the things you mentioned. It can also be integrated with TFS.
In general, release management is:
the process of managing, planning, scheduling and controlling a
software build through different stages and environments; including
testing and deploying software releases.
Specifically, the tool that Microsoft offers would enable you to automate the release process, from development to production, keeping track of what and how everything is done when a particular stage is reached.
There's an MSDN article, Automate deployments with Release Management, that gives a good overview:
Basically, for each release path, you can define your own stages, each one made of a workflow (the so-called deployment sequence) containing the activities you want to perform using pre-defined machines from a pool.
It's possible to insert manual interventions/approvals if necessary, and the whole thing can be triggered automatically once your build is done.
Since you are pretty much in control of the actions performed on each machine in each stage (through the use of built-in or custom actions/components) it is also certainly possible to change configuration files, for example to test different scenarios, etc..
Another image to give you and idea of how it can be done:
Our team has a full licence for the TeamCity server, as well as 7 additional agents. Another unrelated team has reached the limits of their free TeamCity licence and is eyeing our licences up.
The powers that be think it's a good idea to run both teams using the same enterprise licence, which means that we'd be hosting the TeamCity configurations on the same server, and either sharing agents or somehow assigning some agents to one team, some to another.
One concern I have is that configuring an agent to only accept certain builds is difficult - our team has hundreds of build configurations, and we create new ones all the time. To limit an agent to certain builds, you have to fully specify the whitelist. So maintaining the agents such that we have full use of some agents, and the other team has full use of theirs will be a pain. On the other hand, just using one pool of agents means now you have arguments over priority and starvation, etc.
Does anyone have any experience of this? Is it a workable solution? How do you configure agents to reserve them for a particular team? How do you configure the server so that each team only sees their own projects, build configurations and agents? Basically what we'd want is complete separation of the projects, just using the same TeamCity server and agents.
As a gut feeling it doesn't look like a good idea...
edit: As an aside, does Hudson do this better? The ivory tower architects want us to change from TeamCity to Hudson because other people are using Hudson. If I tell them this sharing TeamCity won't work, the Hudson camp will probably use it as a stick to beat us with. Joy.
Not sure what version of TeamCity you're using but the newly released TeamCity v7.0 now has a new Agent Pool feature that provides a much easier way to distribute agents. It may be of interested to you, check out the What's New section or the Agent Pools docs for more info.
I had a similar issue with our two departments starting to share the same TeamCity instance to save the expenses of additional licenses. I must admit we didn't really have any issues apart from our agents were now twice as busy.
I enabled Per-project permissions on the Global Settings page and created 2 user groups, one for 'us', and the other for 'them'. You can then configure each group's roles accordingly. If a group does not have the Project Viewer role for a project then it does not appear for them - a great way to only display necessary projects to the group; but there are plenty of other role options to use.
I have never used Hudson so can't compare unfortunately. I should really try it out but as I've always got on so well with TC I've never had a reason too.
You can make builds on run on certain agent, from the build configuration of each build in the agent requirements section, thereby limiting any build configuration to certain agents.
For example if your agent for one team is teamcity1 you can specify:
system.agent.name does not equal teamcity1
So it will never run on that agent.
That way you can at least copy build configurations and they will run on seperate agents without the fiddle agent configuration.
The other team can create a new Teamcity server, and it will have its own new set of free build configurations and agents.
We don't do this any more, but we used to split our agents into pseudo-pools so we could reserve some for compilations and others for automated tests (because automated test jobs can swamp the grid). We added a "can_run_tests" property to the test agents, and made those builds require that property as an agent condition. It worked great, and it's the sort of thing you can bake into the AMI for a set of cloud agents.
What we do now is to make the compilation and test builds require on different AMIs, which does essentially the same thing.
I've been dealing with the problem of scaling CI at my company and at the same time trying to figure out which approach to take when it comes to CI and multiple branches. There is a similar question at stackoverflow, Multiple feature branches and continuous integration. I've started a new one because I'd like to get more of discussion and provide some analysis in the question.
So far I've found that there are 2 main approaches that I can take (or maybe some others???).
Multiple set of jobs (talking about Jenkins/Hudson here) per branch
Write tooling to manage the extra jobs
Create/modify/delete Jobs in bulk
Custom settings for each job per branch (SCM url, dep management repos duplications)
Some examples of people tackling this problem with shell tools, ant scripts and Jenkins CLI. See:
http://jenkins.361315.n4.nabble.com/Multiple-branches-best-practice-td2306578.html
http://jenkins.361315.n4.nabble.com/Is-it-possible-to-handle-multiple-branches-where-some-jobs-should-run-on-each-one-without-duplicatin-td954729.html
http://jenkins.361315.n4.nabble.com/Parallel-development-with-branches-td1013013.html
Configure or Create hudson job automatically
Will cause more load on your CI cluster
Feedback cycle for devs slows down (if the infrastructure cannot handle the new load)
Multiple set of jobs per 2 branches (dev & stable)
Manage the two sets manually (if you change the conf of a job then be sure to change in the other branch)
PITA but at least so few to manage
Other extra branches won't get a full test suite before they get pushed to dev
Unsatisfied devs. Why should a dev care about CI scaling problems. He has a simple request, when I branch I would like to test my code. Simple.
So it seems if I want to provide devs with CI for their own custom branches I need special tooling for Jenkins (API or shellscripts or something?) and handle scaling. Or I can tell them to merge more often to DEV and live without CI on custom branches. Which one would you take or are there other options?
When you talk about scaling CI you're really talking about scaling the use of your CI server to handle all your feature branches along with your mainline. Initially this looks like a good approach as the developers in a branch get all the advantages of the automated testing that the CI jobs include. However, you run into problems managing the CI server jobs (like you have discovered) and more importantly, you aren't really doing CI. Yes, you are using a CI server, but you aren't continuously integrating the code from all of your developers.
Performing real CI means that all of your developers are committing regularly to the mainline. Easy to say, but the hard part is doing it without breaking your application. I highly recommend you look at Continuous Delivery, especially the Keeping Your Application Releasable section in Chapter 13: Managing Components and Dependencies. The main points are:
Hide new functionality until it's finished (A.K.A Feature Toggles).
Make all changes incrementally as a series of small changes, each of which is releasable.
Use branch by abstraction to make large-scale changes to the codebase.
Use components to decouple parts of your application that change at different rates.
They are pretty self explanatory except branch by abstraction. This is just a fancy term for:
Create an abstraction over the part of the system that you need to change.
Refactor the rest of the system to use the abstraction layer.
Create a new implementation, which is not part of the production code path until complete.
Update your abstraction layer to delegate to your new implementation.
Remove the old implementation.
Remove the abstraction layer if it is no longer appropriate.
The following paragraph from the Branches, Streams, and Continuous Integration section in Chapter 14: Advanced Version Control summarises the impacts.
The incremental approach certainly requires more discipline and care - and indeed more creativity - than creating a branch and diving gung-ho into re-architecting and developing new functionality. But it significantly reduces the risk of your changes breaking the application, and will save your and your team a great deal of time merging, fixing breakages, and getting your application into a deployable state.
It takes quite a mind shift to give up feature branches and you will always get resistance. In my experience this resistance is based on developers not feeling safe committing code the the mainline and this is a reasonable concern. This in turn usually stems from a lack of knowledge, confidence or experience with the techniques listed above and possibly with the lack of confidence with your automated tests. The former can be solved with training and developer support. The latter is a far more difficult problem to deal with, however branching doesn't provide any extra real safety, it just defers the problem until the developers feel confident enough with their code.
I would set up separate jobs for each branch. I've done this before and it isn't hard to manage and set up if you've set up Hudson/Jenkins correctly. A quick way to create multiple jobs is to copy from an existing job that has similar requirements and modify them as needed. I'm not sure if you want to allow each developer to setup their own jobs for their own branches, but it isn't much work for one person (i.e. a build manager) to manage. Once the custom branches have been merged into stable branches, corresponding jobs can be removed when they are no longer necessary.
If you're worried about the load on the CI server, you could set up separate instances of the CI or even separate slaves to help balance the load across multiple servers. Make sure that the server you are running Hudson/Jenkins on is adequate. I've used Apache Tomcat and just had to ensure that it had enough memory and processing power to process the build queue.
It's important to be clear on what you want to achieve using CI and then figure out a way to implement it without much manual effort or duplication. There's nothing wrong with using other external tools or scripts that are executed by your CI server that help simplify your overall build management process.
I would choose dev+stable branches. And if you still want custom branches and afraid of the load, then why not move these custom ones to the cloud and let developers manage it themselves, e.g. http://cloudbees.com/dev.cb
This is the company where Kohsuke is now.
There is an Eclipse Tooling also, so if you are on Eclipse, you will have it tightly integrated right into dev env.
Actually what is really problematic is build isolation with feature branches. In our company we have a set of separate maven projects all be part of a larger distribution. These projects are maintained by different teams but for each distribution all projects need to be released. A featurebranch may now overlap from one project to another and thats when build isolation gets painfully. There are several solutions we've tried:
create separate snapshot repositories in nexus for each feature branch
share local repositories on dedicated slaves
use the repository-server-plugin with upstream repositories
build all within one job with one private repository
As a matter of fact, the last solution is the most promising. All other solutions lack in one or another way. Together with the job-dsl plugin it is easy to setup a new feature branch. simply copy and paste the groovy script, adapt branches and let the seed job create the new jobs. Make sure that the seed job removes nonmanaged jobs. Then you can easily scale with feature branches over different maven projects.
But as tom said well above, it would be nicer to overcome the necessity of feature branches and teach devs to integrate cleanly, but that is a longer process and the outcome is not clear with many legacy system parts you won't touch any more.
my 2 cents
Is it possible to use CruiseControl.Net to set up a build farm? We currently have 4 different build machines building different things at different times and have a bit of a headache to manually balance the load somehow. I would prefer to designate one of them to be the master build machine, which would delegate work to the other ones when they are free.
As far as I can determine, there is no support in CruiseControl.Net for build farms - at least not operating the way you describe. CCNet's interpretation of "farm" seems to assume that projects are assigned manually to a machine and a given project will always be built on the same machine.
If you wanted to dynamically select which machine actually performs the build, you would need to create your own mechanism to select that machine and trigger the build on it. There is likely to be quite a bit of complexity associated with this. For instance you would probably need to ensure that the same project does not get built simultaneously on two different machines if a second commit occurs while the previous commit is still being processed.
If there is a shared location that all the build machines can access, it may be possible to use the Filesystem source control block or CCNet's ForceBuild mechanism to start the build on the designated machine, but have all the build machines publish their output for a given project to the same final location.
See load-Balancing the Build Farm with CruiseControl.NET blogpost for a possible solution
I am wondering what is the best way to use workspaces in visual studio. We have 150+ webservices that are all part of the same team project $/MyTeamProject without being related at all. The reason for them all being part of the same team project as they are all maintained by the same group of people within our organization.
Each webservice has the following structure
$/MyTeamProject/WebService1
$/MyTeamProject/WebService1/Main
$/MyTeamProject/WebService1/Release/1.0
$/MyTeamProject/WebService1/Release/1.1
$/MyTeamProject/WebService2
$/MyTeamProject/WebService2/Main
... etc.
I am wondering what is the proper way to use workspaces with this setup as currently there is only mapping for the teamproject so all my webservices are part of the same workspace. Should I create mapping for the main branch and for each release branch as a separate workspace or should I map each webservice to its own workspace such as $/MyTeamProject/WebService1?
Any help would be appreciated.
I can't put each of the webservices as its own teamproject as the tfs is being shared between several groups with the same magnitude of projects and therefore we are limited by our own teamproject for all of the webservices.
You said that the webservices was not related at all. I then assume that they are released separately.
I can't see any reason to use multiple workspaces in your scenario.
I suggest that you remove the mapping from the top level project "MyTeamProject". Then you have flexibility to map the subfolders to whatever local folder you want.
If you have one workspace per webservice then you will get 150*#releases workspaces. This will get really hard to manage for build versions, labelling etc. Fewer workspaces will mean less builds to manage.
I would recommend having a workspace per release. Generally you would probably only be working on one release at a time and this will allow you to manage all code for a single release effectively. Also you can then map your workspaces on the client into a more logical structure as you allude to in the question.
For example
\Projects\MyTeamProject\Main\WebService1
\Projects\MyTeamProject\Main\WebService2 ...
\Projects\MyTeamProject\Release1.0\WebService1
\Projects\MyTeamProject\Release1.0\WebService2 ...
\Projects\MyTeamProject\Release1.1\WebService1
\Projects\MyTeamProject\Release1.1\WebService2 ...
etc