Our busy enterprise server has 550 active build configurations running on 30 agents. We need a way to prevent some builds from running while other builds are running. I already understand artefact dependencies and this does not solve our problem as the builds do not depend on each other they just share some resources like server port numbers and database connections. Some build configurations conflict with each other when run simultaneously and we need to prevent this by having a way to queue a build (rather than run it) when one of a number of other builds are running.
We already use dumb tricks like restricting builds to specific agents, etc... I'm thinking about adding a first build step that checks for a flag in a db table or something but this would add lots of failed builds when what we really need is proper build queueing.
Am I missing something? Does this functionality already exist in TeamCity?
It looks like what you need is on its way in the form of TeamCity 8 Shared Resources. You can already get access to the EAP here.
My huge respect for 550 configurations and 30 agents.
I think you already found the solution:
Have one environment per agent
Assign configuration for specific agents
I could imagine how much it is work. But I think it's much cleaner and will work much faster than restriction to run several configurations at the same time.
Related
I'm migrating our build system over to TeamCity and, because we have quite long build times, I'm trying to make good use of parallelism in build configurations.
If two configs can run in parallel they are obviously not dependent on each other. However there are some cases where, if two parallel builds are serialised (due to lack of available agents) then I would prefer one to run ahead of another (for example one is a set of regression tests that I'd like to see the result of before a packaging step is run - but if resources are available I'd like them both to run concurrently).
I can't find an explicit way to specify ordering of logically independent builds. However I've observed that the build order tends to be lexicographical - although I'm not sure if that's on the config name or ID.
I could experiment but would prefer a more definite answer, if possible.
This used to be available as a plugin, but has since been bundled into the product.
Go to the build queue and click on Configure Build Priorities
If you add a class with a high number, you can then associate that with the build you'd like to be built first
Managing Build Priorities - TeamCity documentation
Hope this helps
I would like a module to be run by all agents when a VCS trigger condition is met.
Is this possible?
One way you can do this is by adding a Schedule Trigger which has an option to run on all agents.
Having looked into it, as far as I can see, not directly,
the behaviour could potentially be achieved by using the command line remote runner plugin,(http://confluence.jetbrains.net/display/TW/Command+Line+Remote+Run+Tool) through a seperate build configuration linked to the VCS to detect the changes, calling the Remote Run tool from a command line build step to build the project on each required agent.
Further research into the Command Line Remote Run tool would be required to confirm this is possible.
There may also be some functionality allowing this in the REST API, although my look through the documentation didn't show anything up.
Have you had much luck working on alternative solutions?
I've created a build configuration to update our source managed third party referenced assemblies directory and this is a snapshot dependency on most if not all build configurations. When I update this directory with a new or more recent assembly, I'll too would like this configuration to be run on all build agents.
At the moment, I've simply duplicated the configuration and bound each to a specific agent. It adds management overhead, but has temporarily resolved the issue.
You could install this plugin and specify the list of agent names and it will run once per "value" in the matrix.
https://github.com/presidentio/teamcity-matrix-build-plugin
I’d like to be able to specify that one build project should pause while another specified project is running.
In my case, there is a project #1 that builds and deploys compiled code to our servers, and a project #2 that performs Selenium tests on that deployed code. The tests can take several minutes, and I’d like to make sure that the build-and-deploy doesn’t happen in the middle of testing.
So, I’d like to pause #1 (build) while #2 (test) is running.
Dependencies or Triggers do not offer this option, as far as I can tell. Can this be configured?
There is a request on TeamCity issue tracker TW-3798. You are welcome to vote and watch.
This functionality is currently being developed. For now you can use one of the features provided by GroovyPlugin called StartBuildPrecondition. To ensure that only one build is executed add the same write lock to both of the builds
The way we do this is by restricting them to one build agent. They you are insured that only one runs at a time.
Our team has a full licence for the TeamCity server, as well as 7 additional agents. Another unrelated team has reached the limits of their free TeamCity licence and is eyeing our licences up.
The powers that be think it's a good idea to run both teams using the same enterprise licence, which means that we'd be hosting the TeamCity configurations on the same server, and either sharing agents or somehow assigning some agents to one team, some to another.
One concern I have is that configuring an agent to only accept certain builds is difficult - our team has hundreds of build configurations, and we create new ones all the time. To limit an agent to certain builds, you have to fully specify the whitelist. So maintaining the agents such that we have full use of some agents, and the other team has full use of theirs will be a pain. On the other hand, just using one pool of agents means now you have arguments over priority and starvation, etc.
Does anyone have any experience of this? Is it a workable solution? How do you configure agents to reserve them for a particular team? How do you configure the server so that each team only sees their own projects, build configurations and agents? Basically what we'd want is complete separation of the projects, just using the same TeamCity server and agents.
As a gut feeling it doesn't look like a good idea...
edit: As an aside, does Hudson do this better? The ivory tower architects want us to change from TeamCity to Hudson because other people are using Hudson. If I tell them this sharing TeamCity won't work, the Hudson camp will probably use it as a stick to beat us with. Joy.
Not sure what version of TeamCity you're using but the newly released TeamCity v7.0 now has a new Agent Pool feature that provides a much easier way to distribute agents. It may be of interested to you, check out the What's New section or the Agent Pools docs for more info.
I had a similar issue with our two departments starting to share the same TeamCity instance to save the expenses of additional licenses. I must admit we didn't really have any issues apart from our agents were now twice as busy.
I enabled Per-project permissions on the Global Settings page and created 2 user groups, one for 'us', and the other for 'them'. You can then configure each group's roles accordingly. If a group does not have the Project Viewer role for a project then it does not appear for them - a great way to only display necessary projects to the group; but there are plenty of other role options to use.
I have never used Hudson so can't compare unfortunately. I should really try it out but as I've always got on so well with TC I've never had a reason too.
You can make builds on run on certain agent, from the build configuration of each build in the agent requirements section, thereby limiting any build configuration to certain agents.
For example if your agent for one team is teamcity1 you can specify:
system.agent.name does not equal teamcity1
So it will never run on that agent.
That way you can at least copy build configurations and they will run on seperate agents without the fiddle agent configuration.
The other team can create a new Teamcity server, and it will have its own new set of free build configurations and agents.
We don't do this any more, but we used to split our agents into pseudo-pools so we could reserve some for compilations and others for automated tests (because automated test jobs can swamp the grid). We added a "can_run_tests" property to the test agents, and made those builds require that property as an agent condition. It worked great, and it's the sort of thing you can bake into the AMI for a set of cloud agents.
What we do now is to make the compilation and test builds require on different AMIs, which does essentially the same thing.
Is it possible to use CruiseControl.Net to set up a build farm? We currently have 4 different build machines building different things at different times and have a bit of a headache to manually balance the load somehow. I would prefer to designate one of them to be the master build machine, which would delegate work to the other ones when they are free.
As far as I can determine, there is no support in CruiseControl.Net for build farms - at least not operating the way you describe. CCNet's interpretation of "farm" seems to assume that projects are assigned manually to a machine and a given project will always be built on the same machine.
If you wanted to dynamically select which machine actually performs the build, you would need to create your own mechanism to select that machine and trigger the build on it. There is likely to be quite a bit of complexity associated with this. For instance you would probably need to ensure that the same project does not get built simultaneously on two different machines if a second commit occurs while the previous commit is still being processed.
If there is a shared location that all the build machines can access, it may be possible to use the Filesystem source control block or CCNet's ForceBuild mechanism to start the build on the designated machine, but have all the build machines publish their output for a given project to the same final location.
See load-Balancing the Build Farm with CruiseControl.NET blogpost for a possible solution