I have a repo with three rather independent subpackages:
- packages
- backend
- app
- app-deploy-ops
Cloud Build samples show only a single cloudbuild.yaml.
Would the strategy to make a repo like this CI/CD-able (run tests on PRs, for example, but relevant tests based on what changed), involve making three build definitions? That's the only approach I can think of. Then pick those in the Cloud Build triggers, as appropriate.
My question likely is: am I missing something?
You can have only one trigger with all your code in Cloud Build and customize only one big cloudbuild.yaml file to build only the relevant part.
The other option is to create a trigger per part of your project. the cloudbuild.yaml file is in each subdir of your code (you can precise that in the trigger) and thus the file is smaller and dedicated to the part to build.
I recommend you the 2nd approach, but yes, you need 3 files and to define 3 triggers.
Related
In my organization, we are in a transition phase. Big projects get split up into micro services. While this is nice to bring complexity down, the downside is that some parts which should be the same everywhere are more work.
For example, I would like every project to have some tools in the CI pipeline:
Software Composition Analysis (SCA)
Static Application Security Testing (SAST)
Unit Tests
What the tools are might differ from project to project (essentially by programming language). It might also be that this changes - for example, one might want to add the type checker later. Once the type checker is there, one might enforce some of the values (while keeping others flexible, to be changed by the microservices).
Is it possible to have a shared template for a CI pipeline in GitLab? I'm not looking something people can copy-and-paste. I'm looking for a solution that allows me to adjust the CI pipeline of multiple projects at once, without causing more work for me when more microservices are added (the changes don't have to be applied instantly)
Yes you can.
You may develop one or several templates (let's say for e.g.: a Java template (build&test), a Python one (build&test), a SonarQube (SAST), a Kubernetes (deploy), an AWS (deploy)) and then let developers/projects include the ones they need to assemble their pipeline.
We have set of different web application projects, which is delivered by Teamcity to different environments. At this moment, we are doing all-in-one builds: compile, package and deploy at once; all based on a templates.
Now I am investigating a way to separate concenrs: one build tests and produces package, another - delivers. Naturally, both builds are having own templates. Is there a way to template this build chain - so, when I choose some meta-template - both builds will be created with present Artifact dependencies?
Sadly not. What we've had to do is clone the build chains. We've put in place a mechanism which makes sure the set-ups of the different chains do not diverge.
Another option, admittedly ugly, is to set up a single build chain and let each of your projects pretend that it's a separate VCS branch. In this case there's a single set-up (for the single build chain) and to view the history of a given project you filter by that project "branch". Needless to say, that's not how branches should be used and you may run into some issue down the line.
Our company currently uses TFS for source control and build server. Most of our projects are written in C/C++, but we also have some .NET projects and wouldn't want to be limited if we need to use other languages in the future.
We'd like to use Git for our source control and we're trying to understand what would be the best choice for a build server. We have started looking into TeamCity, but there are some issues we're having trouble with which will probably be relevant regardless of our choice of build server:
Build dependencies - We'd like to be able to control the build dependencies for each <project, branch>. For example, have <MyProj, feature_branch> depend on <InfraProj1, feature_branch> and <InfraProj2, master>.
From what we’ve seen, to do that we might need to use Gradle or something similar to build our projects instead of plain MSBuild. Is this correct? Are there simpler ways of achieving this?
Local builds - Obviously we'd like to be able to build projects locally as well. This becomes somewhat of a problem when project dependencies are introduced, as we need a way to reference these resources or copy them locally for the build to succeed. How is this usually solved?
I'd appreciate any input, but a sample setup which covers these issues will also be a great help.
IMHO both issues you mention fall really in the config management category, thus, as you say, unrelated to the build server choice.
A workspace for a project build (doesn't matter if centralized or local) should really contain all necessary resources for the build.
How can you achieve that? Have a project "metadata" git repo with a "content" file containing all your project components and their dependencies (each with its own git/other repo) and their exact versions - effectively tying them together coherently (you may find it useful to store other metadata in this component down the road as well, like component specific SCM info if using a mix of SCMs across the workspace).
A workspace pull wrapper script would first pull this metadata git repo, parse the content file and then pull all the other project components and their dependencies according with the content file info. Any build in such workspace would have all the parts it needs.
When time comes to modify either the code in a project component or the version of one of the dependencies you'll need to also update this content file in the metadata git repo to reflect the update and commit it - this is how your project makes progress coherently, as a whole.
Of course, actually managing dependencies is another matter. Tons of opinions out there, some even conflicting.
I would like to use some build parameters from Project 1 in Project 2. I know that I can make Project 1 a dependency of Project 2 and then access its build parameters as described in Dependencies Properties, but I do not want Project 1 to be built in response to a build of Project 2. For example, suppose I want Project 2 to be built nightly, while I only want Project 1 built monthly.
Is there any way Project 2 can access Project 1's build parameters under these conditions?
I would use a build configuration template that is shared between the two projects.
This means you can share properties between the projects, but also override certain ones in each individual project.
We use this for hourly builds that are not tagged and nightly ones that are tagged.
Then use a different build trigger to set one off nightly and the other monthly.
EDIT
I'll just expand slightly as a result of your comment.
In TeamCity we have 2 build configuration for the same project. One that builds on every check-in to give developers quick feedback on their contribution (build within 15 minutes). It does the following:
Builds the project in Debug
Runs all unit tests
Checks results of build into Subversion
The other configuration runs every night at midnight; it build everything and as a result takes a long time (around 45 minutes). It does the following:
Build the project in Debug and Release
Runs all unit tests
Builds Sandcastle documentaion
Checks results of build into Subversion
Grabs the Sandcastle output at an artefact so developers can easily download it.
As you pointed out this isn't as straightforward as one would like; however you can use the following to achieve it:
We use the Autoincrementer to share build numbers between the two configurations (they both increment the same build number when built).
We have a property on template that defines what artefacts to collect and is referenced from the artefacts field. The property is overridden on the second build config to define the sandcastle output to grab.
Sharing VCS Roots is mentioned on the documentation. Both our builds get the source from the same place, and tag the results to the same place. One VCS is most definitely all we need.
Bit of a long edit but I think it goes exactly on the lines of what you're trying to achieve. I appreciate I should have included this in the original answer.
HTH
Dependency is different from Build Triggering in TeamCity. If you make one project dependent on another ( artifact dependency ), it does not mean that the the latter will trigger the former.
Even when one project has been defined as dependent on another ( and also, even if not ) you have to specify explicitly the build trigger ( in this case a Finish build trigger ) for the dependent project to be triggered.
I am tasked to improve quality and implement TeamCity for continuous integration. My experience with TeamCity is very limited - I use mostly TFS myself and have some experience with CC.NET.
A lot should happen within a build process... actually the build is already pushed into three different configurations that will run one after the next.
My main problem is that in each of those I actually would need to start multiple runners. For example, the first build step shall consist of:
The generation of new AssemblyInfo.cs files for consistent in assembly numbering
The actual compilation
A partial unit test run (all tests that run fast and check core functionality)
An FxCop run
A StyleCop run
The current version of TeamCity only allows to configure one runner ... which leaves me stuck with a lot of things.
How you would approach this? My current idea is going towards using the MsBuild runner for everything and basically start my own MsBuild based script which then does all the things, pretty much the way that TFS handles it (and the same way i did things back in the cc.net way with my own Nant build script).
On a further problem the question is how to present statistical information, for example from unit tests running in different stages (build configurations). We have some further down that take some time to run and want that to run in a 2nd or 3rd step (the latest for example testing database generation code which, including loading base data, takes about 15+ minutes to run). OTOH we would really like test results to be somehow consolidated.
Anyone any ideas?
Thanks.
TeamCity 6.0 allows multiple build steps for a single build configuration. Isn't it what you're looking for?
You'll need to script this out, at least parts of it. TeamCity provides some nice UI based config for some of your needs, but not all. Here's my suggestion:
Create an msbuild script to handle your first two bullet points, AssemblyInfo generation and compilation. Configure the msbuild runner to run your script, and to run your tests. Collect your assemblies as artifacts.
Create a second build configuration for FxCop. Trigger it from the first build. Give it an 'artifact dependency' on the first build, which is how it gets a hold of your dlls.
For StyleCop, TC doesn't support it out of the box like it does FxCop. Add it to your msbuild script manually, and have it produce an html report (which TeamCity can then display).
You need to take a look at the Dependencies functionality in the TeamCity. This feature allows you to create a sequence of build configurations. In other words, you need to create a build configuration for each step and then link all them as dependencies.
For consolidating test results please take a loot at the Artifact Dependencies. It might help.