Maybe I’m approaching this incorrectly but say we have a module A which is built on by B (and others). We will make some changes to each of these two modules on dev branches before merging back to trunk. Sometimes we will make a change to both A and B at the same time. In this case, as they are built as independent modules, we have to publish a snapshot of the A branch and change the A-Snapshot dependency in B to point to the branch snapshot of A.
We currently use the branch information to determine the A dependency, when building the trunk the version ends up as a trunk-snapshot, when on a release branch it defaults to a snapshot of that branch. What would be nice would be to determine we are on a task branch of B, which we could do, and then try to resolve a matching A dependency and if that doesn’t exist revert back to the current approach which would give us a trunk snapshot for example.
So in B we would have something like
ifExists(‘myOrg:A:dev-snapshot’).orElse(‘myOrg:A:trunk-snapshot’)
I’ve seen there are some substitution mechanisms but I can’t see that any deal with missing dependencies so I’m not sure it’s possible.
We obviously can and do deal with this manually when it occurs it would just be nice to incorporate into the grade script somehow to avoid any accidental merge of the dev snapshot dependency onto the B trunk.
Thanks for any suggestions.
Related
This question was migrated from Super User because it can be answered on Stack Overflow.
Migrated last month.
Context:
We're running the free version of Teamcity to manage our projects. Some of those projects have dependencies between each others.
The problem
Some projects have chained Snapshot Dependencies, and those dependencies are always being built instead of the latest artifacts from those dependencies being used.
Example: A depends on B, B depends on C. Push A triggers a build of C, followed by a build of B and finally a build of A.
Ideally: A would be built based on the latest built versions of B and C
Where I think the problem lies (but I might be wrong)
Each of our projects has a number of Snapshot dependencies, and each snapshot dependency is configured with the following parameters turned on:
[x] Do not run new build if there is a suitable one
[x] Only use successful builds from suitable ones
For the first option, the documentation says:
If this option is set, TeamCity will not run a new dependency build, if another dependency build in progress or completed with the appropriate source revision already exists. See also Suitable builds: (https://www.jetbrains.com/help/teamcity/2022.10/snapshot-dependencies.html#Suitable+Builds).
If we look in the Suitable Builds doc, it shows a list of requirements for a build to be considered suitable. The one I think is relevant is here:
It must not have any custom settings, including those defined via reverse.dep. (related feature request: TW-23700: (http://youtrack.jetbrains.com/issue/TW-23700)).
However, we currently have reverse.dep.*.env.SOME_PARAMETER as a Configuration Parameter in every single one of our builds (it's inherited through a template).
Based on that, it seems to me that the "Do not run new build if there is a suitable one" option is doing nothing, and therefore that's why all our dependencies are built every time (or am I wrong?)
We also have, in every one of our builds, an environment variable called env.SOME_PARAMETER which has the same value as the reverse.dep configuration parameter.
My question
Is there a way to avoid using reverse.dep in my situation so that the Do not run new build if there is a suitable one option works? Perhaps by using the environment variable instead?
I asked the senior developer at the company I work in, and they said that in theory it should work, but in practice it doesn't, but he seems recitent to explain further. I'm just a beginner in Teamcity, so detailed explanations are welcome
First things first: what is a Snapshot Dependency in a nutshell?
A Snapshot Dependency in a nutshell is a dependency between two build configurations which are linked by shared VCS Roots. VCS Roots are sort of like time lines in a book: they represent a chain of events (e.g. git commit history) and let you build from a givent point in time (e.g. commit).
Now, TeamCity excels at doing what it is intended to do: Continuous Integration and Deployment. It does so by being tied up closely to VCS Roots and effectively changes in these (optionally narrowed down scopes of the) VCS roots. A Snapshot Dependency is a dependency which links together the dependency based on VCS Roots and their changes.
An example
Build A has two VCS Roots, Foo and Bar. Foo and Bar are, say, different Git repositories which Build A needs to fetch before it is able to build the "A" artifact. Build B only needs Foo, and thus only has Foo attached as a VCS Root. Build A has a Snapshot Dependency on Build B, which we configure like yours: "Do not run new build if there is a suitable one" and "Only use successful builds from suitable ones".
So far so good. Now let's push a new commit to the "Foo" repository. TeamCity notices this and potentially triggers a new build of Build A, because the latest build of A is at that point outdated (it did not have our latest Foo commit included). The Snapshot Dependency of B in A links these two build configurations together so that - with our above configuration of the Dep. - we can require a build of B which includes the same revision of Foo, that build A was kicked off with (e.g. the latest commit). Because this does not (yet) exist, a build of B is started and put above build A in the queue.
Simply put: the VCS Root is a timeline, the Snapshot Dependency is a link between two build configurations based on the timeline(s) they have in common, and the configuration of the dependency dictates what should happen when a dependency is needed (e.g. "Do not run new build if there is a suitable one").
If we had manually started a build B with the latest Foo revision included, this would have been a suitable candidate for reuse, and TeamCity would simply remove the queued B build once it discovered that a build of B already exists, which shares the same changes that build A is started with (the latest push to Foo).
If you want just the latest artifacts of B and C...
...use Artifact Dependencies and only these. Removing the Snapshot Dependency of the build removes your need of having the dependency build every time Build A is triggered by a new change in its VCS Root. It however also means that there is no timeline linkage between the two builds and that you yourself need to ensure or be sure that the artifacts produced by B and C are not tightly linked to A. E.g. Build C could produce a driver as an artifact, B could produce a user manual of the driver and build A could just be using the driver, only expecting that it is in a working condition (but otherwise does not depend on changes in it).
Your question about reverse.dep.*...
I've not heard about this causing trouble before. I would however expect that a Snapshot Dependency (and not just an artifact dependency) is required by TeamCity for you to be allowed to use it.
Question is: do you need it? It sounds like you've got the value elsewhere already, and honestly fetching values from previous builds is likely going to cause you trouble in the long run, especially if you don't have a specifically good reason to do so.
I would like to have more information about Maven best practices when releasing projects.
Say I have a parent pom with N modules. All are in SNAPSHOT versions including the parent. If I want to release one but not all of the modules, should I pass version for the parent pom project in RELEASE, or leave it in SNAPSHOT?
What are the pros and cons?
A bit more generally speaking, there are huge advantages in terms of simplicity, and very little disadvantages (a bit more hard drive consumption on the Nexus box), to keeping all version numbers uniform across a parent-child structure project, including keeping their RELEASE / SNAPSHOT status in sync. By always doing simultaneous releases of all modules, you do not have to keep in mind and manage the component inter-dependencies. You do not need to announce the releases to those clients, who only depend on a component that has not really changed.
That being said, if you do decide to "micro manage" a parent-child structure project, and every time to release only a certain subset of components, I would follow the following rules:
1.keep a component in its old RELEASE version until it needs to change. Promote it to the next SNAPSHOT only once it has been modified.
1a.That applies to parent poms too.
2.if a module depends on another module that has been modified, that counts as a modification of the former module as well
(if you want to modify a dependency, but to keep the item that depends on it in its old version, it helps a lot in terms of consistency and being able to reason about what is going on to simply branch out, and modify and release the dependency in a separate branch, in which the item that depends on it is also considered as being modified, but from where we do not release it. This helps you keep in mind that the dependency and the item that depends on it are being released from different branches, and are therefore deliberately inconsistent with each other at present.)
2a.this extends to cases where instead of "depends on a module that" we have "has a parent that". In a lot of ways, parents are like dependencies, in maven.
3.when you want to release (promote to the next RELEASE version) a module, you need to also release all modified modules it depends on (including its modified parents)
Respecting these will keep your project in a consistent state and make it easy to reason about its status, in my experience.
Since your parent is in SNAPSHOT, I assume it has been modified. Therefore you need to bump it to the next RELEASE when releasing the child you want to release.
If the parent has not changed since its last release, you could have kept it in its last RELEASE version, and this way you would not need to release it now together with the child that "depends on" (in terms of being its child) it.
You are writing 'All are in SNAPSHOT versions including the parent' you should not set in your parent SNAPSHOT in this way you can't release all the project.
If you want to release only a subpart of the project then make an other parent and include between your parent and all the projects or use as an other parent. In fact you have your reason to release only subset of your project and in this way you have a better documentation of your work.
There are multiple modules in our applications and each of them have their separate version and depend on other modules(external to our organization). They all have a parent POM which has it's own version, independent from the children's version.
When one of those modules change, they're converted to snapshots.
For the following example:
Parent v14.0
- module1 v1.5.0
- dependency1(module2 v15.0.0)
- dependency2(external-jar v12.0.1)
- module2 v15.0.0
- module3 v3.1.0
If there would be a change in module2, then module2's version would become v15.0-SNAPSHOT, then module1 becomes v1.5-SNAPSHOT. The parent remains the same.
The purpose for not having the same version on parent+modules, is that we want to localize the updates made to some modules and not affect the others' versions.
This has been designed like this a long time ago and there are several bash scripts to support the updates, although they're not handling all the cases. In any case, we don't have a one-click release process and we feel we are quite far from it with this approach.
We don't know how to convince the management towards a single version approach on all modules. How do you feel about the above? Did you ever encountered a project using the above structure and how well did it go?
Thank you!
I've had to deal with such situations before. There is an actual benefit from having decentralized versions, especially in cases where your product is made out of a large number of modules and this is because of the following facts:
You don't have to release all of them as a whole, if only a handful have been changed (which, from my observation is almost always the case).
You don't have to create unnecessary tags in your version control for code which hasn't changed since the previous release.
You don't have to waste an excessive amount of time releasing modules which don't need to be released.
You know with certainty which modules have changed in a release, which helps a lot when you need to investigate a complex bug, which seems to be dating back a while.
You can actually release certain modules/aggregators before the actual release date of the complete product, allowing for more testing time and a feeling of completeness for a given part of the product.
You can make feature branch releases much easier and implement a continuous delivery in a better way.
You can re-use the same code across multiple development branches without wondering if that branched version matches the one for your branch (or at least with less confusion).
What we ended up doing was:
Extract a parent or set of parents (with no sub-modules).
Try to use fixed versions for parents as much as possible. This is a bit of a caveat, as you must change all modules that inherit it, but in the end it improves the stability.
Extract each of the modules whose versions are independent of the rest to separate modules.
Extract sets of modules whose versions must always move along together into aggregators.
Create jobs in your CI server that can do releases or manually release these modules.
Use the versions-maven-plugin.
I think it's a lot more mature of a project and company's development principles to use decentralized versions and I must admit that in the beginning I was very reluctant to this approach. You might not realize or understand the benefits immediately, but with some practice and a proper setup, you will start seeing the upsides. I'm not saying there aren't caveats like... for example bumping the version of a parent, or having to know in which modules to bump the version of one of your modules.
From my experience, this module actually works better in the end, once you've become used to working with it.
From my experience: Everywhere we tried this, it eventually failed.
Although Maven supports this approach it is not advisable because of the additional effort.
I try to use the following criteria when choosing whether to use distinct projects or a multimodule structure:
If all projects have the same release cycle, I put them in a common multi module structure. In that case, I give them all the same version and release them together.
If a part of the project is used by different other projects (organizational projects), I always split them of and give them a separate lifecycle and a separate version.
If one part of my project stabilizes, I split it off, and give it a separate lifecycle (Maven refactoring)
To do it different alway results in homebrew solutions that neither scale well, nor are easy to maintain.
Hope that helps.
I am thinking about a deployment pipeline using SVN, Jenkins and Maven. At the moment I'm stuck at the point where I usually would call mvn release:perform on a working copy.
When thinking in deployment pipelines, I want to create a pipeline where every commit could be used to release a software to test/production. Let's say I have 5 builds, and I decide to release build 3 (with revision 3) to production. There will already be 2 new commits to trunk (which is now at revision 5).
Is it possible to use the maven-release-plugin to checkout/build/tag/commit a release at revision 3? When the maven-release-plugin finishes the release it usually commits the modified POMs to trunk.
I'm happy about any kind of information or advice here, so feel free to point me to books (like http://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912), blog posts, Jenkins documentation... Maybe I'm completely on the wrong track.
By default, the release plugin creates the release based on the contents of your working copy, it just ensures that you don't have any uncommitted content before doing so. AFAIK it doesn't force an update of the sources, as that's usually the job of the Continuous Integration system (Jenkins in your case). So whatever is checked out by Jenkins will be released.
What you're trying to do sounds more like a configuration change on the Jenkins side, pointing it to the right revision.
On the other hand, if the POM files are modified as part of the release, but have been changed in SVN in the meantime, you will run into a conflict when Maven wants to check in the modified POM files. That's a situation that might happen, depending on how for back you want to go with the release.
Based on this, it might make more sense to always create a branch before doing a release. So you would create a branch based on revision 3 and then create your release in that branch. This way, you wouldn't run into issues with committing resources that have changed in more recent revisions.
Creating the branch and checking it out could probably be automated through Jenkins and Maven as well.
As far as I tested it, it is not possible.
More explicitely, as nwinler said, when you release, maven try to commit the modified pom. But, if it's an older revision than the current one, SVN will complain that your sources are not up to date. So it won't work. ... as far as I know.
You may read docs about promotion build. I don't find any one clear enough to be pointed out (in th few minutes of the writing of this message).
I have been tasked with updating our build process right now to be more efficient and I have spent a week reading best practices and strategies but I still have not found a solution for our current problem.
Background
Currently we have one monolithic build/application that really needs to be split apart into at least 4 applications with some shared libraries. We currently do not branch unless we absolutely have to. We have a teamcity build that builds on each check-in to TFS. When we are ready for a release we have a code freeze and only check-in fixes for bugs found in QA. Obviously this is a terrible practice and we have finally gotten approval to change it.
Proposed Solutions
The proposed solution will be to split up the application and have different release cycles for each application, move from ant to maven and branch per release.
Branching - Right now we just have a main trunk in source control. I think we want to branch off the trunk when we are ready for a release, and update the branch for bugs found in QA. When the build is ready to be released, merge the branch changes back into the trunk.
Here is how I was planning on setting up TFS.
+Apps
+App1
+Components
+Core
+Web
+Branches
+App2
+Components
+Core
+Web
+Branches
+Libraries
+Lib1
+Lib2
+Branches
Thinking about managing all of the POMs and versions in the POMs seems WAY too difficult right now. I've read up on the maven release plugin, but I'm not sure if it can branch in the way I'm thinking we want to.
Next problem is getting teamcity working. I was thinking of having 3 teamcity projects for each app. A dev project that always points at the trunks, a QA project for testing the QA build and a production project to build changes for hotfixes. Each time a new release comes to QA I would have to update the QA teamcity project to point at the new release branch and update the release build number in teamcity. When that release passes QA I would have to update the production teamcity project to point that the branch that just passed QA and update the build number to the build number that just passed QA.
Surely there is a better strategy that this.
Questions
Where should I be putting these branch folders?
Should QA builds be snapshots still until the build goes to pre-production?
How do you configure teamcity to pick up these branches without changing the sources path for every release?
Should there be parent POMs for each app that the developers use to make sure all of their dependencies are compiled and up to date?
I just want to question your thinking that your applications should be on different release cycles. Modularization is a good thing for code quality but if your modules are on separate release cycles you introduce a lot of overhead. In particular, version management becomes quite a burden and if you get it wrong you can introduce runtime bugs.
How do these separate applications relate to each other? Is there any dependency between them (maybe via a shared library)? Do they communicate with each other? Are they deployed together?
If it is not necessary that they be on separate release cycles then you're probably better off keeping them together.