I have a sub project in which I would like to ensure the build configurations in this sub project will never run at the same time.
All the build configurations are based on the same template and I have tried setting "limit the number of concurrent build to" 1 but this has no effect. I assume since this binds to the concrete build configuration and not the template.
I assumed I could set an agent requirement to limit the name to the build agent in a snapshot dependence but apparently agent requirements will not evaluate variables.
I also have a build configuration called All which has snapshot dependencies on all the build configurations in my sub project. Under snapshot dependencies settings I tried setting "Run on same build agent" but this could result in the build queue could run on "No agents". I assume because I started one build configuration manually. Then invoked all which confused Teamcity.
The only thing that works is hardcoding the agent name as a requirement but I'm not a fan of this solution. How can I limit simultanious builds across a sub project without hardcoding an agent name.
I think what you need is Shared Resources.
From official documentation:
The Shared Resources build feature allows limiting concurrently
running builds using a shared resource, such as an external (to the CI
server) resource, e.g. test database, or a server with a limited
number of connections, etc.
Related
We have a teamcity set-up on with multiple agents. There is one particular build which uses some underlying resources and services of a particular environment. ( say uat)
We want that this build should not run n parallel to avoid resource contention. i.e. only a single running build at a time. How can I achieve this ?
Thanks
Under the project, setup a shared resource:
Then under the build configurations you don't wish to be able to run in parallel add a build feature and select the resource you just created and "Write Lock":
This will mean any extra builds triggered will stay in the build queue and will not be allowed to run concurrently.
I wrote a pipline to build my Java application with Maven. I have feature branches and master branch in my Git repository, so I have to separate Maven goal package and deploy. Therefore I created two jobs in my pipeline. Last job needs job results from first job.
I know that I have to cache the job results, but I don't want to
expose the job results to GitLab UI
expose it to the next run of the pipeline
I tried following solutions without success.
Using cache
I followed How to deploy Maven projects to Artifactory with GitLab CI/CD:
Caching the .m2/repository folder (where all the Maven files are stored), and the target folder (where our application will be created), is useful for speeding up the process by running all Maven phases in a sequential order, therefore, executing mvn test will automatically run mvn compile if necessary.
but this solution shares job results between piplines, see Cache dependencies in GitLab CI/CD:
If caching is enabled, it’s shared between pipelines and jobs at the project level by default, starting from GitLab 9.0. Caches are not shared across projects.
and also it should not be used for caching in the same pipeline, see Cache vs artifacts:
Don’t use caching for passing artifacts between stages, as it is designed to store runtime dependencies needed to compile the project:
cache: For storing project dependencies
Caches are used to speed up runs of a given job in subsequent pipelines, by storing downloaded dependencies so that they don’t have to be fetched from the internet again (like npm packages, Go vendor packages, etc.) While the cache could be configured to pass intermediate build results between stages, this should be done with artifacts instead.
artifacts: Use for stage results that will be passed between stages.
Artifacts are files generated by a job which are stored and uploaded, and can then be fetched and used by jobs in later stages of the same pipeline. This data will not be available in different pipelines, but is available to be downloaded from the UI.
Using artifacts
This solution is exposing the job results to the GitLab UI, see artifacts:
The artifacts will be sent to GitLab after the job finishes and will be available for download in the GitLab UI.
and there is no way to expire the cache after finishing the pipeline, see artifacts:expire_in:
The value of expire_in is an elapsed time in seconds, unless a unit is provided.
Is there any way to cache job results only for the running pipline?
There is no way to send build artifacts between jobs in GitLab that only keeps them as long as the pipeline is running. This is how GitLab has designed their CI solution.
The recommended way to send build artifacts between jobs in GitLab is to use artifacts. This feature always upload the files to the GitLab instance, that they call the coordinator in this case. These files are available through the GitLab UI, as you write. For most cases this is a complete waste of space, but in rare cases it is very useful as you can download the artifacts and check why your pipeline broke.
The artifacts are available for download by project members that are at least Reporters, but can be viewed by everybody if public pipelines is enabled. You can read more about permissions here.
To not fill up your hard disk or quotas, you should use an expire_in. You could set it to just a few hours if you really don't want to waste space. I would not recommend this though, as if a job that depend on these artifacts fails and you retry it, if the artifacts have expired, you will have to restart the whole pipeline. I usually put this to one week for intermediate build artifacts as that often fits my needs.
If you want to use caches for keeping build artifacts, maybe because your build artifacts are huge and you need to optimize it, it should be possible to use CI_PIPELINE_ID as the key of the cache (I haven't tested this):
cache:
key: ${CI_PIPELINE_ID}
The files in the cache should be stored where your runner is installed. If you make sure that all jobs that need these build artifacts are executed by runners that have access to this cache, it should work.
You could also try some of the other predefined environment variables as key our your cache.
In Teamcity 9.0.1 I have several builds, which ends up on similar steps, but with different parameters.
In fact, those are servicing steps, not needed by a build to conclude, that build is ok, but they are needed for service, so, currently, they are just running and consuming time.
What I am trying to reach - is move those steps to completely separate build, and invoke it on successful build end trigger. However, I could not figure out - how to pass changes parameters to these newly created build?
Snapshot dependency does not give me level of control I would like to have, as it requires me to define dependency parameters from specific build id, instead of reading them from build, which just ends.
Using artifact dependency by downloading .teamcity/properties/build.finish.gz!** fails with error:
Unknown archive type
and, actually, will oblige me to create additional xml parser
There's nothing built in to trigger builds with custom parameters on a finish-build trigger.
You will have to use the TeamCity REST API / HTTP endpoint to trigger the servicing build with custom parameters - you can pass them as key/value pairs using the following syntax (preferably use HTTPS of course):
http://testuser:testpassword#teamcity.jetbrains.com:8111/httpAuth/action.html?add2Queue=MyBuildConf&name=<full property name1>&value=<value1>&name=<full property name2>&value=<value2>
For further information see the docs.
Implementation-wise you should be able to do what you want in a few lines of powershell as a last step in the triggering build configs - they would have to pass the build parameters you want. If you use this in multiple build configurations I'd move this step into a common template or use a meta-runner.
For testing purposes I sometimes copy some Jenkins jobs from a produtive instance into our testing instance.
Repositories, branches etc. are controlled by environment variables but I want to block the "Deploy artifacts to Maven repository" post build action for all jobs in this Jenkins instance.
Any idea how to do this?
There is a Jenkins plugin for conditional build steps: Conditional Build Step Plugin. While plugin supports a variety of condition types, for reference here simple use of toggling deployment via global Jenkins property used in a boolean condition.
Property can be defined as Global Property in Jenkins configuration (Jenkins home > Manage > Configure: Global Properties)
As example lets say you have defined a property with name doMavenDeploy:
production environment: property value is true
test environment: property value is false
job configuration would need to be modified to add a Conditional build step with :
Run? : Boolean condition
Token : $doMavenDeploy
Step to run if condition is met : add maven deployment there.
In that way you would need to modify your jobs once, after modified configuration is active you can move them between both servers without impacts.
There are other alternatives possible depending on specifics and constraints in your projects:
introduce different maven profiles for 'normal' and test builds and control activation via environment variable, file present or similar
if you use artifactory/nexus or similar: configure test server without deployer credentials
I don't want Build Config A and Build Config B to run at the same time. This is because they share the same resource which cannot be accessed simultaneously. However each build config is run by a separate agent so it is possible for them to run simultaneously.
Instead I would like one build config, when triggered, to wait for the other to finish if it is running. For example if Build Config B begins to run but Build Config A is already running, then B would wait until A finishes and then B would run.
I don't think a snapshot dependency will work because that assumes one config has a dependency on the other which is not true in my case.
Keith, there are two plugins that can help you:
The first one is Groovy plugin. It has functionality of creating name locks over all projects.
The second one is TeamCity.SharedResources. It has functionality of definig shared resources and locking them with read and write locks. However, resources defined in this plugin, are are defined per-project. We are actively developing this plugin, so you are welcome to watch its page in our tracker
For a quick and dirty, just make a rule on the compatible agents tab for the project. That they have to run on the agent name containing xxx ( whatever that is ) then those builds can only run on that agent and will never run at the same time.
In the newest version of TeamCity, you can configure this parameter. Go to the Edit Configuration Settings -> Edit Configuration Settings and set parametr:
Limit the number of simultaneously running builds (0 — unlimited)