I am in the process of setting up a continuous integration build for a Spring Roo application using the Rational Team Concert (RTC) IDE and Jazz build engine. When setting up the build definition, the Build Workspace field on the Jazz Source Control tab allows the selection of either a user's repository workspace or a stream.
The RTC Continuous Integration Best Practices and other Jazz build resources consistently refer to using a dedicated repository workspace associated with a build user, leading me to believe that this is the preferred approach. I have not been able to find any information on building from the stream directly. Our project's stream contains all of the artifacts required to build, and I have tested and confirmed that the continuous integration build works from the stream. I am unable to think of any reason why I would need to create and manage a specific workspace for this purpose.
My question is, am I playing with fire by building directly off of the stream? Are there potential downstream complications with this approach that I am not aware of?
Answering my own question in case another SO user has the same question in the future.
After some experimentation, I discovered that a drawback to building directly from the stream was that it ignores the "Build only if there are changes accepted" property on the Jazz Source Control tab. As a result, builds from a stream may only be done at predefined intervals - it is not possible to configure the build to only happen when new changes have been committed to the stream.
A dedicated workspace is required for the build to accept new changes from the stream and use them to trigger a build request.
There is another BIG difference here. It has to do with HOW the build gets done. Let me highlight the difference here.
If you build from a dedicated build repository workspace, then your build workspace already has a copy of all of the code. When your changes are delivered, and the build is kicked off, then only the changed files (your change set) need to be updated and physically copied from the repository to the build repository workspace. Since most changes are small, this involves the copying of anywhere from 0.1% to 2% of your codebase from the repository.
If you build from "the stream", then your build workspace needs to be created (you have to compile somewhere!). So when this is created, your ENTIRE codebase needs to be updated and physically copied from the repository to the build repository workspace. This means retrieving 100% of your codebase from the repository.
Each file operation involves a call to discover the needed resource, fetching this resource from the database hosting the repository, and then having the Jazz application provide this source file over the network. It results in a load on the database server, the web server, and the application server. The more you download like this, the more of a load that you put on these components.
There are some things you can use to minimize this load on the Jazz infrastructure. Using content caching proxies (using a simple Squid proxy server) can help.
For more detail on your options here, and the relative merits of those options, go and read my blog post and whitepaper on Jazz Performance concerns (http://dtoczala.wordpress.com/2013/02/11/jazz-performance-a-guide-to-better-performance/). That article is almost a year old now, but still remains valid. You can also look at the Jazz Deployment Wiki (https://jazz.net/wiki/bin/view/Deployment/WebHome), and check out the sections on performance troubleshooting and performance concerns.
Related
Our company currently uses TFS for source control and build server. Most of our projects are written in C/C++, but we also have some .NET projects and wouldn't want to be limited if we need to use other languages in the future.
We'd like to use Git for our source control and we're trying to understand what would be the best choice for a build server. We have started looking into TeamCity, but there are some issues we're having trouble with which will probably be relevant regardless of our choice of build server:
Build dependencies - We'd like to be able to control the build dependencies for each <project, branch>. For example, have <MyProj, feature_branch> depend on <InfraProj1, feature_branch> and <InfraProj2, master>.
From what we’ve seen, to do that we might need to use Gradle or something similar to build our projects instead of plain MSBuild. Is this correct? Are there simpler ways of achieving this?
Local builds - Obviously we'd like to be able to build projects locally as well. This becomes somewhat of a problem when project dependencies are introduced, as we need a way to reference these resources or copy them locally for the build to succeed. How is this usually solved?
I'd appreciate any input, but a sample setup which covers these issues will also be a great help.
IMHO both issues you mention fall really in the config management category, thus, as you say, unrelated to the build server choice.
A workspace for a project build (doesn't matter if centralized or local) should really contain all necessary resources for the build.
How can you achieve that? Have a project "metadata" git repo with a "content" file containing all your project components and their dependencies (each with its own git/other repo) and their exact versions - effectively tying them together coherently (you may find it useful to store other metadata in this component down the road as well, like component specific SCM info if using a mix of SCMs across the workspace).
A workspace pull wrapper script would first pull this metadata git repo, parse the content file and then pull all the other project components and their dependencies according with the content file info. Any build in such workspace would have all the parts it needs.
When time comes to modify either the code in a project component or the version of one of the dependencies you'll need to also update this content file in the metadata git repo to reflect the update and commit it - this is how your project makes progress coherently, as a whole.
Of course, actually managing dependencies is another matter. Tons of opinions out there, some even conflicting.
Our project currently have more than 500 SNAPSHOT jars to be downloaded daily. This is becoming a burden for development especially when Artifactory is in place remotely, hence the connection is not ideal.
I would like to know the best way to reduce redundant SNAPSHOTs jars that are deployed to Artifactory. We have a large multi-module project that is deploying almost 200 newly SNAPSHOT jars although there is actually only 1 module that contains changes.
I found a similar question raised in the forum but no definite answer there too. Incremental build is not viable for us due to this issue.
Any suggestion is appreciated!
The "Managing Disk Space Usage" page in the Artifactory wiki describes various methods for cleaning up old snapshots:
Limiting the Number of Snapshots - you can specify the maximum number of snapshots that may be stored. To specify the maximum number of snapshots that may be stored, in the Edit Repository dialog, select the Basic Settings tab. You first need to check the Handle Snapshots checkbox which then enables you to set the Max Unique Snapshots field. This value is zero by default, which means that all snapshots are saved.
Using a user plugin for custom cleanup logic - you may write scripts to implement virtually any custom cleanup logic. This provides you with an extensive and flexible set of customization capabilities. See examples of such scripts on Github
If only one module changes, then why do you rebuild the entire project? Or in other words, your often changing module appears to have a different lifecycle than the rest of your project, so you should consider moving it to a separate project with its own release cycle.
We currently use SilkCentral Test Manager (SCTM) integrated with our source control system via SCTM source control profiles. However, we would like to explore integrating with build artifacts checked into Maven's remote Nexus repository instead.
The idea being that the application-under-test is built and checked into Nexus along with the automated tests only if the build and the tests pass. Therefore, when QA is ready to run tests from SCTM (manual or automated), there is a well-defined combination of application build artifacts and test build artifacts in Nexus that present a more reliable target for SCTM as compared to getting the latest code from the source control system.
All of this is more relevant during active development when the code and the tests and changing daily and the builds are snapshot builds rather than formal builds with tags in the source control system that SCTM could use.
SCTM apparently has support for both universal naming convention (UNC) and Apache virtual file system (VFS) and either of these should potentially be utilizable to point the SCTM source control profiles to Nexus artifacts rather than raw source code. However, I wanted to check with the community to see if there's a simpler approach. (For example, I noted the existence of a Hudson SCTM plugin.) Also, I welcome alternative thoughts and ideas.
There are probably many solutions for solving this, I'd try the following:
Manage the build/first test/publishing steps in Hudson/Jenkins.
For example by modelling it with dependent jobs, the publish job is only triggered if the tests pass. There are also more advanced gatekeeper plugins available (for example a Downstream Ext plugin) which might solve this even more comfortable.
Once the publishing is done, use the Hudson/Jenkins-Silk Central plugin to trigger the executions on Silk Central. There, instead of using UNC or VFS, I'd rather use a setup script which pulls the artifacts from the repository and prepares everything for the tests. This would allow you to use something Maven/Nexus aware to pull the correct artifacts from the repository, instead of somehow trying to make it accessible via UNC or VFS.
I have recently been charged with building out our "software infrastructure" and so I am putting together a continuous integration server.
After a build completes would it be considered bad form for the CI system to check in some of the artifacts it creates into a tag so that it can be fetched easily later (or if the build breaks you can more easily recreate the problem.)
For the record we use SVN and BuildMaster (free edition) here.
This is more of a best practices question rather than a how-to question. (It is pretty easy to do with BuildMaster)
Seth
If you believe this approach would be beneficial to you, go ahead and do it. As long as you maintain a clear trace of what source code was used to build each artifact, you'll be fine.
You should keep this artifact repository separated from the source code repository.
It is however a little odd to use a source code repository for this - these are typically used for things that will change, something your artifacts most definitely should not.
Source code repositories are also often used in a context where you want to check out "everything", for example the entire trunk. With artifacts you are typically looking for a specific version, and checking out all of the would only be done if exporting them to some other medium.
There are several artifact repositories specialized for this, for example Artifactory or Apache Archiva, but a properly backed up file server will thought-through access settings might be a simple and good-enough solution.
I would say it's a smell to check in binaries as a tag. Your build artifacts should be associated with a particular build version in your build system, and that build should be associated with a particular checkin. You should be able to recreate the exact source code from that information. If what you're looking for is a one-stop-function to open the precise source-code revision that generated the broken build, I'd suggest that you invest some time into building a Powershell module that will do that for you.
Something with a signature like:
OpenBuild -projectName "some project name" -buildNumber "some build number"
I currently have a CI environment setup using the following tools:
VCS - ClearCase (UCM enabled)
CI Server - Jenkins
Build Engine - MSBuild
Basically Jenkins is polling my UCM Project's Integration stream every 2 minutes and building via a msbuild script I wrote.
While in ClearCase it is not a best practice having a individual stream for each developer, good CI demands private builds to be run before commiting the code. Added to that, ideally i would have atomic commits, which ClearCase provides just on the form of Deliver to stream.
Currently we are working directly on integration stream, and sometimes our builds fail because Jenkins starts building before the developer finishes her check-ins.
My question is, how can I have a private work area (Sandbox) and atomic commits on ClearCase without creating a stream for each developer? Am I missing something?
Currently we are working directly on integration stream, and sometimes our builds fail because Jenkins starts building before the developer finishes her check-ins
You can write your build script in order to detect if a deliver is in progress.
A deliver is characterized by an activity named deliver.xxx: you can list its content and see if any version in it is in checkout. If yes, the deliver is in progress.
If the most recent deliver has only checked-in versions, you can safely start your build.
Or:
How can I have a private work area (Sandbox) and atomic commits on ClearCase without creating a stream for each developer
A private area for Jenkins to use would be a snapshot view on each developer stream.
As the name suggests, a snapshot view would take a snapshot of the code, but you need to define a criteria suggesting that Jenkins can build what the snapshot view has updated.
What I have seen used is a 'BUILD' shifting label (a label you re-apply on the newly updated code, and used by Jenkins in his snapshot view with a selection rule based on that label):
The developer move his/her label when he/she thinks the current code is ready to be build, and a Jenkins jobs update its snapshot view on the developer stream, based on the versions referenced by said shifting label 'BUILD'.