We have a teamcity set-up on with multiple agents. There is one particular build which uses some underlying resources and services of a particular environment. ( say uat)
We want that this build should not run n parallel to avoid resource contention. i.e. only a single running build at a time. How can I achieve this ?
Thanks
Under the project, setup a shared resource:
Then under the build configurations you don't wish to be able to run in parallel add a build feature and select the resource you just created and "Write Lock":
This will mean any extra builds triggered will stay in the build queue and will not be allowed to run concurrently.
Related
This is my first question so please presume ignorance and positive intent.
As part of our build pipeline I need to run what our dev team calls "unit tests." These unit tests are run via an ant target. Before running that ant target we must spin up, configure and partially populate (the ant target does some of the population) several containers including:
application server
ldap instance
postgres instance
It looks as if each task only supports a single container. Is there a straightforward way to get all of these running together? Ideally I could create a task that would allow me to specify a pod template with the commands running in one of the containers of that pod.
I realize that I could hack this together by using the openshift client or kubernetes actions but I was hoping for something a bit more elegant. I feel like using either of those tasks would require that I build out status awareness, error checking, retry logic, etc that is likely already part of the pipeline logic and then parse the output of the ant run to determine if all of the tests were successful. This is my first foray into tekton so accepted patterns or idioms would be greatly appreciated.
For greater context I see these tasks making up my pipeline:
clone git repo
build application into intermediate image
launch pod with all necessary containers
wait for all containers to become "ready"
execute ant target to run unit tests
if all tests pass build runtime image
copy artifacts from runtime image to external storage (for deployment outside of openshift)
Thank you for your time
Have a look at sidecars. The database must be up and running for the tests to execute, so start the database in a sidecar. The steps in the task will be started as soon as all sidecars are running.
VSTS allows you to select which branches automatically trigger a CI build by specifying a branch pattern.
However, my unit tests are using a real database which causes a problem when more than one build triggers e.g. master and feature-123 as they will clash on the database tests.
Is there a way of specifying that only one such build should be run at at time; I don't want to go away from executing tests against a real database as there are significant differences between an in-memory database and SQL Azure.
VSTS already serialize builds which are triggered by the same CI build.
Even CI build can be triggered by multiple branches, but for a certain time, only one build is running by default (unless you use pipelines to run builds concurrently).
Such as if both master branch and feature-123 branch are pushed to remote repo at the time time, the CI build definition will trriger two builds serially (not concurrently).
If you are using pipeline and need to run the triggered builds serially, you should make sure only one agent is used for the CI builds. You can use the way below:
In your CI build definition -> Options Tab -> add demands to specify which agent you want to use for the CI build.
Assume in default agent pool, there are three agents with the agent name: default1, default2 and default3.
If you need to specify default2 agent to run the CI build, then you can add the demands as below:
Now even multiple branches have been pushed at the same time, they will be triggered one by one since only one agent is available for the CI build.
If you use a YAML pipeline, you can use a deployment job in stead of a regular job.
With a deployment job you select a named Environment you want to deploy to.
You can configure the Environment in azure devops under Pipelines->Environments and you can choose to add an Exclusive Lock.
Then only one run can use the environment at a time and this serializes your runs.
Unfortunately, if you have multiple runs waiting for the environment (because one run currently has it locked), when the environment becomes unlocked only the latest run will continue. All the others waiting for the lock will be cancelled.
If you want to do it via .yml or .yaml file you can do following
- phase: Build
queue:
name: <Agent pool name>
demands:
- agent.name -equals <agent name from agent pool>
steps:
- task: <taskname>
displayName: 'some display name'
inputs:
value: '<input variable based on type of task'
variableName: '<input variable name>'
I have a sub project in which I would like to ensure the build configurations in this sub project will never run at the same time.
All the build configurations are based on the same template and I have tried setting "limit the number of concurrent build to" 1 but this has no effect. I assume since this binds to the concrete build configuration and not the template.
I assumed I could set an agent requirement to limit the name to the build agent in a snapshot dependence but apparently agent requirements will not evaluate variables.
I also have a build configuration called All which has snapshot dependencies on all the build configurations in my sub project. Under snapshot dependencies settings I tried setting "Run on same build agent" but this could result in the build queue could run on "No agents". I assume because I started one build configuration manually. Then invoked all which confused Teamcity.
The only thing that works is hardcoding the agent name as a requirement but I'm not a fan of this solution. How can I limit simultanious builds across a sub project without hardcoding an agent name.
I think what you need is Shared Resources.
From official documentation:
The Shared Resources build feature allows limiting concurrently
running builds using a shared resource, such as an external (to the CI
server) resource, e.g. test database, or a server with a limited
number of connections, etc.
In Teamcity 9.0.1 I have several builds, which ends up on similar steps, but with different parameters.
In fact, those are servicing steps, not needed by a build to conclude, that build is ok, but they are needed for service, so, currently, they are just running and consuming time.
What I am trying to reach - is move those steps to completely separate build, and invoke it on successful build end trigger. However, I could not figure out - how to pass changes parameters to these newly created build?
Snapshot dependency does not give me level of control I would like to have, as it requires me to define dependency parameters from specific build id, instead of reading them from build, which just ends.
Using artifact dependency by downloading .teamcity/properties/build.finish.gz!** fails with error:
Unknown archive type
and, actually, will oblige me to create additional xml parser
There's nothing built in to trigger builds with custom parameters on a finish-build trigger.
You will have to use the TeamCity REST API / HTTP endpoint to trigger the servicing build with custom parameters - you can pass them as key/value pairs using the following syntax (preferably use HTTPS of course):
http://testuser:testpassword#teamcity.jetbrains.com:8111/httpAuth/action.html?add2Queue=MyBuildConf&name=<full property name1>&value=<value1>&name=<full property name2>&value=<value2>
For further information see the docs.
Implementation-wise you should be able to do what you want in a few lines of powershell as a last step in the triggering build configs - they would have to pass the build parameters you want. If you use this in multiple build configurations I'd move this step into a common template or use a meta-runner.
I don't want Build Config A and Build Config B to run at the same time. This is because they share the same resource which cannot be accessed simultaneously. However each build config is run by a separate agent so it is possible for them to run simultaneously.
Instead I would like one build config, when triggered, to wait for the other to finish if it is running. For example if Build Config B begins to run but Build Config A is already running, then B would wait until A finishes and then B would run.
I don't think a snapshot dependency will work because that assumes one config has a dependency on the other which is not true in my case.
Keith, there are two plugins that can help you:
The first one is Groovy plugin. It has functionality of creating name locks over all projects.
The second one is TeamCity.SharedResources. It has functionality of definig shared resources and locking them with read and write locks. However, resources defined in this plugin, are are defined per-project. We are actively developing this plugin, so you are welcome to watch its page in our tracker
For a quick and dirty, just make a rule on the compatible agents tab for the project. That they have to run on the agent name containing xxx ( whatever that is ) then those builds can only run on that agent and will never run at the same time.
In the newest version of TeamCity, you can configure this parameter. Go to the Edit Configuration Settings -> Edit Configuration Settings and set parametr:
Limit the number of simultaneously running builds (0 — unlimited)