TeamCity appears to store the definitions for builds, projects, templates etc as XML internally.
This is exposed in the "Administration > Audit" view where you can see diffs that people made to individual configurations, at URLs like http://teamcityserver/admin/settingsDiffView.html?id=project:project10&versionBefore=8&versionAfter=9&actionId=3151
I'd like to manage a TeamCity setup partially from outside the web interface - e.g. for example keep the build definitions in version control and perhaps programmatically generate them.
Is there any way I can directly upload definitions in this format (or any similar alternative)? I'm aware that there are various APIs and extension points to TeamCity but haven't managed to find any that gives direct access to anything like this.
I can live with the format changing with TeamCity versions if necessary - it would be a reasonable price to pay for the other benefits.
For TeamCity 9.x and newer
As reported by Ganesh in the comments to this answer, an option was added in 9.x that supports changes and versioning through Source Code Management (SCM) tools. Please see his answer for 9.x and beyond.
For TeamCity 8.x and older
It might not be the "approved" way, but you can edit the project files on disk, and those changes will appear in your build configs. I have successfully edited them outside of the Web UI after they were created.
So, you could probably open that folder up as a restricted network share or set up ssh.
You'll find it at $TeamCityData/config/projects/ and then they are stored in subfolders such as $projectName/buildTypes/$buildFile.xml
An example is:
E:\TeamCityData\config\projects\CSandbox\buildTypes\CSandbox_Project1TrunkBuildUnitTest.xml
TeamCity 9 adds a new "Versioned Settings" feature which keeps these XML files under version control and allows changes to be made via the VCS.
In TeamCity 9.0 this can be git or mercurial, and the upcoming TeamCity 9.1 will add support for Perforce and Subversion.
I've been using it with git for a few months and it works quite nicely in practice.
I sometimes have trouble persuading TeamCity to notice changes coming in from the VCS - particularly when deleting projects - but otherwise it's been really useful for standardising configuration and spinning up new job chains quickly.
Another slight annoyance is that you can't configure the location within the repository that the settings come from - it's always .teamcity in the root - so I've had to use multiple branches or repositories to manage multiple TeamCity servers.
Related
I am looking for a tool to manage the collection of binary files (input components) that make up a software release. This is a software product and we have released multiple versions each year for the last 20 years. The details and types of files may vary, but this is something many software teams need to manage.
What's a Software Release made of?
A mixture of files go into our software releases, including:
Windows executables/binaries (40 DLLs and 30+ EXE files).
Scripts used by the installer to create a database
API assemblies for various platforms (.NET, ActiveX, and Java)
Documentation files (HTML, PDF, CHM)
Source code for example applications
The full collected files for a single version of the release are about 90MB. Most are built from source code, but some are 3rd party.
Manual Process
Long ago we managed this manually.
When starting each new release the files used to build the last release would be copied to a new folder on a shared drive.
The developers would manually add or update files in this folder (hoping nothing was lost or deleted accidentally).
The software installer script would be compiled using the files in this folder to produce a SETUP.EXE (output).
Iterate steps 2 and 3 during validation & testing until release.
Automatic Process
Some years ago we adopted CI (building our binaries nightly or on-demand).
We resorted to putting 3rd party binaries under version control since they usually don't change as often.
Then we automated the process of collecting & updating files for a release based on the CI build outputs. Finally we were able to automate the construction of our SETUP.EXE.
Remaining Gaps
Great so far, but this leaves us with two problems:
Rebuilding Assemblies The CI mostly builds projects when something has changed, but when forced it will re-compile a binary that doesn't have any code change. The output is a fresh build of a binary we've previously tested (hint: should we always trust these are equivalent?).
Latest vs Stable Mostly our CI machine builds the latest versions of each project. In some cases this is ok, but often we want to release an older tested or stable version. To do this we have separate CI projects for the latest and stable builds - this works but is clumsy.
Thanks for your patience if you've got this far :-)
I Still Haven't Found What I'm Looking For
After some time searching for solutions it seems it might be easier to build our own solution, but surely someone else has solved these problems before!?
What we want is a way to store and manage binary files (either outputs from CI, or 3rd party files) such that each is tagged with a version (v1.2.3.4) that allows:
The CI to publish new versions of each binary (but reject rebuilt versions that already exist).
The development team to make a recipe for a software release (kinda like NuGet packages.config) that specifies components to include:
package name
version
path/destination in the release folder
The Automatic package script to use the recipe collect the required files, and compile the install package (e.g. SETUP.EXE).
I am aware of past debates about storing binaries in a VCS. For now I am looking for a better solution. That approach does not appear ideal for long-term ongoing use (e.g. how to prune old binaries)... amongst other issues.
I have tried some artifact repositories currently available. From my investigation these provide a solution for component/artifact storage and version control. However they do not provide tools for managing a list of components/artifacts to include in a software release.
Does anybody out there know of tools for this?
Have you found a way to get your CI infrastructure to address these remaining issues?
If you're using an artifact repository to solve this problem, how do you manage and automate the process?
This is a very broad topic, but it sounds like you want a release management tool (e.g. BuildMaster, developed by my company Inedo), possibly in conjunction with a package management server like ProGet (which you tagged, and is how I discovered this question).
To address some specific questions you have, I'll associate it with a feature that would solve the problem:
A mixture of files go into our software releases, including...
This is handled in BuildMaster with artifacts. This video gives a basic overview of how they are manually added to releases and deployed to a file system: https://inedo.com/support/tutorials/buildmaster/deployments/deploying-a-simple-web-app-to-iis
Of course, once that works to satisfaction, you can automate the import of artifacts from your existing CI tool, create them from a BuildMaster deployment plan itself, pull them from your package server, whatever. Down the line you can also have your CI tool call the BuildMaster release management API to create a release and automatically have it include all the artifacts and components you want (this is what most of our customers do now, i.e. have a build step in TeamCity create a release from a template).
Rebuilding Assemblies ... The output is a fresh build of a binary we've previously tested (hint: should we always trust these are equivalent?)
You can mostly assume they are equivalent functionally, but it's only the times that they are not that problems arise. This is especially true with package managers that do not lock dependencies to specific version numbers (i.e. NuGet, npm). You should be releasing exactly the same binary that was tested in previous environments.
[we want] the development team to make a recipe for a software release (kinda like NuGet packages.config) that specifies components to include:
This is handled with releases. A developer can choose its name, dates, etc., and associate it with a pipeline (i.e. a set of testing stages that the artifacts are deployed to), then can "click the deploy button" and have the automation do all the work.
Releases are grouped by "application", similar to a project in TeamCity. As a more advanced use case, you can use deployables. Deployables are essentially individual components of an application you include in a release; in your case the "Documentation" could be a deployable, and maybe contain an artifact of the .pdf and .docx files. Deployables from other applications (maybe a different team is responsible for them, or whatever) can then be referenced and "included" in a release, or you can reference ones from a past release.
Hopefully that provides some overview and fits your needs. Getting into this space is a bit overwhelming because there are so many terms, technologies, and methodologies, but my advice is to start simple and then slowly build upon it, e.g.:
deploy a single, manually uploaded component through BuildMaster to a share drive, then manually deploy it from there
add a deployment plan that imports the component
add a second plan and associate it with the 2nd stage that takes the uploaded artifact and deploys it to the target, bypassing the need for the share drive
add more deployment plans and associate them with pipeline stages and promote through them all to "close out" a release
add an agent and deploy to that instead of the default localhost server
add more components and segregate their deployment with deployables
add event listeners to email team members at points in the process
start adding approvals if you require gated "sign-offs"
and so on.
Our company currently uses TFS for source control and build server. Most of our projects are written in C/C++, but we also have some .NET projects and wouldn't want to be limited if we need to use other languages in the future.
We'd like to use Git for our source control and we're trying to understand what would be the best choice for a build server. We have started looking into TeamCity, but there are some issues we're having trouble with which will probably be relevant regardless of our choice of build server:
Build dependencies - We'd like to be able to control the build dependencies for each <project, branch>. For example, have <MyProj, feature_branch> depend on <InfraProj1, feature_branch> and <InfraProj2, master>.
From what we’ve seen, to do that we might need to use Gradle or something similar to build our projects instead of plain MSBuild. Is this correct? Are there simpler ways of achieving this?
Local builds - Obviously we'd like to be able to build projects locally as well. This becomes somewhat of a problem when project dependencies are introduced, as we need a way to reference these resources or copy them locally for the build to succeed. How is this usually solved?
I'd appreciate any input, but a sample setup which covers these issues will also be a great help.
IMHO both issues you mention fall really in the config management category, thus, as you say, unrelated to the build server choice.
A workspace for a project build (doesn't matter if centralized or local) should really contain all necessary resources for the build.
How can you achieve that? Have a project "metadata" git repo with a "content" file containing all your project components and their dependencies (each with its own git/other repo) and their exact versions - effectively tying them together coherently (you may find it useful to store other metadata in this component down the road as well, like component specific SCM info if using a mix of SCMs across the workspace).
A workspace pull wrapper script would first pull this metadata git repo, parse the content file and then pull all the other project components and their dependencies according with the content file info. Any build in such workspace would have all the parts it needs.
When time comes to modify either the code in a project component or the version of one of the dependencies you'll need to also update this content file in the metadata git repo to reflect the update and commit it - this is how your project makes progress coherently, as a whole.
Of course, actually managing dependencies is another matter. Tons of opinions out there, some even conflicting.
I am setting up a CruiseControl.Net server. So far, it only builds a project (.Net website), and I kind-of know how to set up unit testing, code coverage, etc in the future.
What I will need to have soon is this:
The developers commit changes to SVN continually, thus CCNet builds often.
CCNet will publish the latest version to the development server, as soon as a commit is validated (with unit tests etc).
The project manager validates a specific version, in order to publish it to the pre-production server, and create a SVN tag from this revision.
The last point is where my problem lies: how exactly can I set up things so the project manager can, for instance, browse to the CCNet web dashboard, select a previous specific build, and says "this is the build I want to publish" ?
I believe that my thinking is flawed somewhere, but I can't put my finger on it. Maybe CCNet is not the right place to do these manipulations ?
In my mind, I can create a SVN tag using CCNet, and mostly work from the trunk, but maybe I can't ? Maybe it's the other way around, and I should add a CCNet project every time a tag is created under SVN ?
The final goal is that I want to automate the publication process: zip creation (for archiving), web.config modification (using Nant for instance), and website publication (using FTP).
In all these steps, I want to limit the manual intervention to the maximum. If I can avoid to add a new project to CCNet every time a tag or branch is created in SVN, that would be awesome.
Thanks for your help, and sorry if it's not very easy to read, but it's not very clear in my head either...
Since you can create any task, you should be able to achieve the goal, though unfortunately not out-of-the-box.
Since you use SVN, it all depends actually on revision. I think I'd create a separate project for your third scenario and added a parameter where PM would provide revision number. Then based on that I'd tag sources etc. in my own task.
Regarding the other points, I think this is similar. Recently for web projects we started using MSDeploy, and in each stage build the MSDeploy package was created. Then there was a separate build called Deploy, that when forced allows us to select which package we want to deploy using MSDeploy.
Having several environments, however, started a little bit like overkill for managing with CCNet, and I'll be looking into kwakee at some time.
Overview
I am using VisualSVN in Visual Studio, VisualSVN Server on Windows, and of course, TortoiseSVN. I wanted to know what the best method of sharing multiple projects over multiple solutions was, and if there was a better method.
Layout
My Repository kind of looks like this (not their real names):
Library.Common
Library.Web
Library.DB
Library.CMS
Customer1.Site
Customer2.Site
Process
To create a new site that contains common projects:
Create Repository in SVN-Server, e.g. "Customer3.Site"
Create Web site using Visual Studio 2008, named "Customer3.Site", VisualSVN used to commit to the repository created in step (1).
Edit properties of Customer3.Site and specify the necessary projects as svn:externals, e.g. "Library.Common", "Library.DB", etc.
Perform an update, to get these external projects, and add them to my solution in Visual Studio, add the necessary references to the Customer3.Site web project and hit build.
So far so good.
The Problem
All this works fine, I am happy that if I have to modify any of the core Library projects I can do so right in the same environment and commit them to the repository. As more and more customer sites are built, I will then have to keep track of what I've done and remember to SVN Update and rebuild those sites which seems quite a long-winded task.
Is there a better way of doing this, a more best-practice solution? Am I breaking any fundamental SVN laws by doing it this way? I want to find a good solution that doesn't cost too much time and isn't overly complex either.
I've been facing a similar issue ... I am setting up a base install package for WordPress, something we would use to quickly get a site setup, it contains the core of wordpress + a set of baseline plugins, both third party and custom ones we've created. Everything pretty much comes from SVN.
Different plugins have different versions/tags and to setup an svn external pointing to a specific tagged version per project would be a nightmare ... only to then have to go into each and every project and do a property adjustment and then an update.
What going to be implementing is a vendor branch with specific versions as needed. All I should then have to do is update the client sites, since they will always be pointing to the latest versions (under my control in the vendor branches).
As to your problem, and also in my case: I would probably write a commit script to update all projects automatically when something in the vendor branch is updated.
In my CruiseControl instances, I have version controlled the ccnet.config file.
When I want to update CruiseControl, I run an "update config" job which fetches the config from version control.
In this manner, the very build process of a release is configuration managed.
I am wondering how to achieve these goals effectively under TeamCity.
I try to keep what ever CI I am using as light as possible and put as much of the running of the build into an msbuild or nant script including running tests, code coverage, etc.
The benefit of this is:
The build file is version controlled.
You can run the script in any environment.
Easier to move between CI environments.
Everyone becomes responsible for the build.
This has been introduced in TeamCity 9. Also answered in another post:
Version control (e.g. in TFS) build configuration for TeamCity - is it possible?
I've been wanting a way to source control TeamCity config for a long time. I ended up writing a Windows Service which monitors the configuration directory and commits changes to git.
The project is on GitHub: https://github.com/grenade/teamcity-config-monitor
You might try looking at the folders that are backed up prior to upgrade (or when restoring team city) as those represent the configurations and changes you've made since initial installation.
http://confluence.jetbrains.net/display/TCD4/TeamCity+Data+Backup
Some of the relevant data is actually a database, (and in fact the documentation advises you to point team city to a real database like mysql instead of the default embedded database it uses)
You could try checking those into SVN, but you'll want to stop team city for any check-in actions.