Platform: Teamcity 8.1.4
We have a pipeline with stages that have several snapshot dependencies on other smaller builds. Our problem is that if any of these smaller builds fail then the pipeline that depends on these smaller builds chokes up and doesn't run until it is resolved.
This is certainly expected behaviour: we don't want breaking changes in the smaller pipelines to reach further down our main pipeline. However in our main pipeline we would rather use the Last Successful Build from each of the smaller pipelines instead of the Most Recent Build so the main pipeline isn't blocked and we can allow new successful changes to filter through from the other smaller pipes.
I've noticed that we can depend on the Last Successful Build if it was an Artifact Dependency instead of a Snapshot Dependency and they look on the surface to be very similar.
Questions
Are there downsides to using Artifact Dependencies instead of Snapshot Dependencies?
We need to pass through things like the Build Number and certain environment variables from the smaller pipelines to the main one - can we do this using Artifact Dependencies?
There are no downsides to using artefact dependencies. You woudl just need to fetch some artefacts. I ussualy fetch the build.finish.properties of the parent build
You can easily pass around all parameters (like build number , env variables etc) to artefact dependencies
You can use this syntax in the targets to get teamcity build finish properties
".teamcity/properties/build.finish.properties.gz"
Related
This question was migrated from Super User because it can be answered on Stack Overflow.
Migrated last month.
Context:
We're running the free version of Teamcity to manage our projects. Some of those projects have dependencies between each others.
The problem
Some projects have chained Snapshot Dependencies, and those dependencies are always being built instead of the latest artifacts from those dependencies being used.
Example: A depends on B, B depends on C. Push A triggers a build of C, followed by a build of B and finally a build of A.
Ideally: A would be built based on the latest built versions of B and C
Where I think the problem lies (but I might be wrong)
Each of our projects has a number of Snapshot dependencies, and each snapshot dependency is configured with the following parameters turned on:
[x] Do not run new build if there is a suitable one
[x] Only use successful builds from suitable ones
For the first option, the documentation says:
If this option is set, TeamCity will not run a new dependency build, if another dependency build in progress or completed with the appropriate source revision already exists. See also Suitable builds: (https://www.jetbrains.com/help/teamcity/2022.10/snapshot-dependencies.html#Suitable+Builds).
If we look in the Suitable Builds doc, it shows a list of requirements for a build to be considered suitable. The one I think is relevant is here:
It must not have any custom settings, including those defined via reverse.dep. (related feature request: TW-23700: (http://youtrack.jetbrains.com/issue/TW-23700)).
However, we currently have reverse.dep.*.env.SOME_PARAMETER as a Configuration Parameter in every single one of our builds (it's inherited through a template).
Based on that, it seems to me that the "Do not run new build if there is a suitable one" option is doing nothing, and therefore that's why all our dependencies are built every time (or am I wrong?)
We also have, in every one of our builds, an environment variable called env.SOME_PARAMETER which has the same value as the reverse.dep configuration parameter.
My question
Is there a way to avoid using reverse.dep in my situation so that the Do not run new build if there is a suitable one option works? Perhaps by using the environment variable instead?
I asked the senior developer at the company I work in, and they said that in theory it should work, but in practice it doesn't, but he seems recitent to explain further. I'm just a beginner in Teamcity, so detailed explanations are welcome
First things first: what is a Snapshot Dependency in a nutshell?
A Snapshot Dependency in a nutshell is a dependency between two build configurations which are linked by shared VCS Roots. VCS Roots are sort of like time lines in a book: they represent a chain of events (e.g. git commit history) and let you build from a givent point in time (e.g. commit).
Now, TeamCity excels at doing what it is intended to do: Continuous Integration and Deployment. It does so by being tied up closely to VCS Roots and effectively changes in these (optionally narrowed down scopes of the) VCS roots. A Snapshot Dependency is a dependency which links together the dependency based on VCS Roots and their changes.
An example
Build A has two VCS Roots, Foo and Bar. Foo and Bar are, say, different Git repositories which Build A needs to fetch before it is able to build the "A" artifact. Build B only needs Foo, and thus only has Foo attached as a VCS Root. Build A has a Snapshot Dependency on Build B, which we configure like yours: "Do not run new build if there is a suitable one" and "Only use successful builds from suitable ones".
So far so good. Now let's push a new commit to the "Foo" repository. TeamCity notices this and potentially triggers a new build of Build A, because the latest build of A is at that point outdated (it did not have our latest Foo commit included). The Snapshot Dependency of B in A links these two build configurations together so that - with our above configuration of the Dep. - we can require a build of B which includes the same revision of Foo, that build A was kicked off with (e.g. the latest commit). Because this does not (yet) exist, a build of B is started and put above build A in the queue.
Simply put: the VCS Root is a timeline, the Snapshot Dependency is a link between two build configurations based on the timeline(s) they have in common, and the configuration of the dependency dictates what should happen when a dependency is needed (e.g. "Do not run new build if there is a suitable one").
If we had manually started a build B with the latest Foo revision included, this would have been a suitable candidate for reuse, and TeamCity would simply remove the queued B build once it discovered that a build of B already exists, which shares the same changes that build A is started with (the latest push to Foo).
If you want just the latest artifacts of B and C...
...use Artifact Dependencies and only these. Removing the Snapshot Dependency of the build removes your need of having the dependency build every time Build A is triggered by a new change in its VCS Root. It however also means that there is no timeline linkage between the two builds and that you yourself need to ensure or be sure that the artifacts produced by B and C are not tightly linked to A. E.g. Build C could produce a driver as an artifact, B could produce a user manual of the driver and build A could just be using the driver, only expecting that it is in a working condition (but otherwise does not depend on changes in it).
Your question about reverse.dep.*...
I've not heard about this causing trouble before. I would however expect that a Snapshot Dependency (and not just an artifact dependency) is required by TeamCity for you to be allowed to use it.
Question is: do you need it? It sounds like you've got the value elsewhere already, and honestly fetching values from previous builds is likely going to cause you trouble in the long run, especially if you don't have a specifically good reason to do so.
I'm trying to setup a pipeline for running all kinds of tests against pull requests into my repository. Repository is git repo hosted with Bitbucket Server and TeamCity is Enterprise 2019.1.5 (build 66605). There are a few key aspects to the task:
There's a lot of tests. One way or the other tests should run in parallel to achieve reasonable execution time. The tests are already partitioned as separate TeamCity build configurations, each having good enough execution time.
There's much less build agents available for the task, so it's not impossible for a particular build to spend quite some time (up to 1-2 hours) in a build queue.
The result of the testing should be reported to Bitbucket as a single aggregate value. I.e. if there are, say, 3 individual builds from p.1 with two passing and one failing then Bitbucket should receive single "failed" build status.
Ideally, pipeline should be triggered by a feature branch change (refs/pull-requests/123/from in Bitbucket lingo), but checkout merges of the feature branch into target branch (refs/pull-requests/123/merge in Bitbucket lingo).
Given above requirements, I experimented with Composite Build Configuration (https://www.jetbrains.com/help/teamcity/composite-build-configuration.html) as it seemed perfect fit for the job. So I set up single composite build with proper builds from p.1 as snapshot dependencies, "Pull Requests" and "Commit status publisher" build features. It works perfectly, except for one thing...
The only thing I cannot seem to be able to work around is the fact that VCS roots in the dependencies collect changes when the build chain is added to the build queue. This means that because of p.2 (non-negligible max time spent in build queue) some builds end up running against a little bit dated sources. Ideally, I would like to be able to run the builds against the latest sources.
So my question is if there's any way to disable revision synchronization for dependencies? Or maybe I could approach the whole problem in some completely different way without using snapshot dependencies?
Cross-posted at community forum: https://teamcity-support.jetbrains.com/hc/en-us/community/posts/360006745840
I am trying to find a way to assure that the build number for the same components are unique and it seems that the only place that is really appropriate to get next build number it would be the artifact repository (Artifactory in my case).
In case someone wonders why I am not using the build number from the build system, that's because I have components that are build, in parallel, by different build systems and each one of them may be allowed to upload the artefacts.
Obviously I do not want to introduce a new build-number-server dependency I do indent to use the destination to assure that I have a proper (unique) build number.
Using SCM for build numbers is not good because of several things like forks and branches.
Update: It seems that if I try to PUT a the buildInfo.json with the same build number on Artifactory the server returns a 500 error code with a message Could not insert build Build{...}.
It seems that this is not possible at this moment because you can only publish build info once, which means that even if you can figure out which build number should be next, you cannot reserve it till you finish the build.
https://www.jfrog.com/jira/browse/RTFACT-7676 is asking to allow POST in addition to PUT for builds, so we can modify them (publish it as incomplete at the first place, and updating it later when the build process is finished).
We have a lot of tests. I can break these up so that they run on seperate agents after an initial compile build happens, but is there a way I can recombine these results? Having 8 build configurations that all need to be green makes it hard to see if you've got one ubergreen build.
Is there a way in TeamCity to recombine / join builds once we've split them out? TW-9990 might help - allowing ANDs in the dependencies.
We found the answer which certainly works from TeamCity 5:
One compile build,
N test only builds that take compile.zip!** and copy to where the compile output would normally be. (via a template)
Consolidated finish:
Finish Build Trigger: Wait for a successful build in: ...
Snapshot Dependencies: Do not run new build if there is a suitable one
Only use successful builds from suitable ones
This all seems to work nicely and the whole shbang is easily copied for branches etc. Am very happy - this has worked well for us for many months now.
No idea how to do that natively. Here's my first thoughts on how I would try and tackle such a thing though:
Saving test results to files
Publishing the test result files as build artifacts
Creating a 'Merge build'
Adding artifact dependency onto the individual test projects
Writing a custom 'build' script using something like (N)Ant. This would parse the individual test results and publish the results as per the TC KB
Good luck!
Thinking outside the box you could have an overall build which doesn't really do anything (or use one of your test build configs as your 'master'), with snapshot dependencies on each of your split test builds. That way if any of them fail, the 'master' will fail because one the dependent build failed.
TW-9990 looks to be concerned with build triggering rather than dependencies.
My TeamCity's nightly build produces more than 130Mb java doc as Build Configuration artifact.
I have to share these artifact for other teams(developers), but javadoc every time has another one URL(I know that it's possible to get it like .lastFinished), and get's new place on Build Machine.
Is it possible on each build replace old artifact with new one because I don't need need previous versions? And have independent from build version URL for accessing.
You can configure TeamCity to delete artifacts after the n-th subsequent build - set this to 1, and it won't keep much for you. Look under Administration at Build History Clean-up Policy.
And the independent from build version URL should then probably be the .lastFinished one you mention.