What is the difference between nightly builds and continuous integration? Is it just the frequency of checkin/build/test? For example, if a nightly build process were run every hour or every minute would it be indistinguishable from CI?
This question is similar but not identical: Continuous Integration vs. Nightly Builds
Continuous Integration focuses on building (and running tests) every time a checkin is done. Nightly builds do it once a day and may emphasize testing less.
In practice the continuous integration checkins that trigger building and running tests might be separate from checkins that a developer does throughout the day. They might occur, for instance, only when checking in to the trunk while most development for the day is checked in to a branch. It is encouraged that these occur relatively frequently, for instance once a day.
Building more frequently would produce similar results but would probably result in some wasted effort, that is running tests when nothing has changed. Another possible difference is that a significant level of testing is done with each build in continuous integration. Sometimes requiring developer intervention to fix errors that result. This would be more difficult with a nightly build process for several reasons. Including being more difficult to tie the errors to a particular set of changes and responsible party.
It may depends on the number of tests and how long they take. If a full build and full tests takes a few minutes, you can run it in continuous integration after each push to the sources.
For my part, our nightly build is about 2 hours, building all sources, and running all tests. In daily builds, we build only some librairies and run some tests, depending on the branch and the thing that the developers are likely to modify.
Related
According to my understanding, continuous integration means whenever a developer checkin the code to a branch, the code is automatically built, unit test (or other basic test) and then merged to master branch. one tool to do that is Jenkins.
continuous delivery means the code is always READY to be or CAN be deployed, though it may not be deployed.
so what else should be done to move the step from continuous integration to continuous delivery? package the code after more detailed tests like integration/performance/stress tests, tests in difference OS, in different stages (test, production),etc?
There is a long and a short answer. The short one is: automate all the steps of packaging and deploying to production and create safety net that automatically checks that the software is ready for release.
The first includes automation database migration taking into consideration zero time deployment (if needed), packaging the binaries, updating configuration files, gradually deploying to different data centers.
The second includes creating test suites for functional and non functional tests. Such as performance, load testing, security penetration, licensing etc.
We usually do smoke tests to check critical functionalities whenever we receive a new build. After executing the smoke tests, we are sure to go to next stage (next level of testing). I heard from my colleagues that smoke tests are really useful when your team employs Continuous Integration and DevOps. Smoke tests are always beneficial, but how it will be more beneficial with the combination of CI and DevOps?
Testing is interesting and every time a new challenge for QA which requires higher level of efforts in the final deployment of product. This consist of continuous delivery in continuous integration environment. In this continuous deployment process, requires testing to be followed in parallel in order to keep the process moving.
I've usually heard smoke testing used to refer to manual testing that you run to sanity-check builds. This article defines smoke testing as follows:
Smoke Testing, also known as “Build Verification Testing”, is a type
of software testing that comprises of a non-exhaustive set of tests
that aim at ensuring that the most important functions work. The
results of this testing is used to decide if a build is stable enough
to proceed with further testing.
First, I would certainly hope that people are doing this whenever they check code into the main branch to ensure that their changes didn't break the software in some obvious way. That holds whether you're doing continuous integration or not. (One of my personal pet peeves has always been people who check in code and then leave for the day without checking to make sure that it worked).
Also, keep in mind that in a typical CI cycle nowadays a build will often occur with every checkin to the main branch (or, at a minimum, there will be a nightly automated build; at my current company we have both), so you don't really have time to manually run your entire test suite for every build. One of the main purposes of CI is to have integration (and, as an extension, builds) occur much more frequently than is typical in other kinds of development cycles.
As one final comment: if you're doing continuous integration, I'd strongly encourage you to have some kind of automated testing (e.g. coded UI tests, unit tests, etc.) as part of that. Those can provide basic smoke/sanity testing and regression testing and reduce the burden of having to do all of it manually for every build.
I've been wondering how you go about doing CI-style testing when you're dealing with physical devices.
I imagine you have a suite of tests, and a pool of devices against which they can be run.
Additionally:
Some tests may require specific device models.
Some tests may require the use of more than one device.
What CI servers have support for this?
I'm still interested in those which have partial support, either natively or through plugins, as I'm interested in how it's done.
Continuous Integration enables a team integrate & test their work frequently. Automated builds are meant to compile, link and run unit-tests. You want your CI to run fast, especially if you run it at every check-in. That is why you would want to restrict CI activities to simple confirmation of the build and unit tests alone. What you're asking seems more-along the lines of quality assurance (QA) testing...and having QA failures mixed into your CI efforts would detract development efforts from progressing.
As such, I'm more under the impression activities associated with CI are not dependent upon the final physical machine said work may eventually be migrated to.
Now...this doesn't mean you CAN'T take the CI-compiled package and run it against some final target-machine....but again...that is really considered a seperate activity.
This seems to be re-enforced in the following article by Martin Fowler.
Notice he doesn't talk about the final targeted devices...only the build machine.
I can suggest Test Manager that is part of the TFS suite of Microsoft. I have not tried it with many different environments apart from windows based though I know there are many connectors. For windows based environments I believe it will satisfy most needs.
I use it for nightly builds to perform smoke tests (Turn it on, see if any smoke comes out) but you have to be careful to keep tests small in order to have them finished in a matter of hours and not days, if you want it to be part of your CI.
Then when you have a good enough quality you can proceed onto regression tests and integration tests if needed.
I wouldn't get too caught up in what a CI system is supposed to do or not do. Instead I would focus on the problem you are trying to solve. It sounds like that problem is to facilitate development on multiple platforms. You can use the concept of Continuous Integration and add to it successfully address the issue. I know, because I've done it in the past.
I implemented a build system for code that needed to compile and test successfully on 4 different platforms (nt, wince, linux-arm, linux-x86). The CI server would:
Used a linux and winnt build server compilation (and cross compilation)
The compiled tests and supporting libs would then be copied to the appropriate devices and an automated test run executed.
After the test suite was completed the log would be copied back, (or it was written to a network mounted fs)
If the test suite was successful we would tag the source, and package the libs and executables.
This same platform was reused for developer verification before commits. Developers would run a partial build and test (only updated source would be recompiled and those tests rerun). The CI would execute a full build (from scratch).
Our build were pretty fast because we had a proper DAG for build dependencies. This allowed for concurrent compilation within a platform build. Each platform build was also concurrent. As a result partial builds took a few seconds, full builds took ~30 minutes. Our build servers were quite beefy (optimized for fast compiles) and the codebase was of moderate size (I don't remember the stats).
I've been exploring different strategies for running integration tests within some Nant build scripts. Typically a number of different scripts are chained in one monolithic build that has separate targets: staging (build a staging version, like build), build (just build the stuff), integration (build the stuff and run the integration tests). This works reasonably well, the build target takes about a third of the time to run as the integration target and it's not painfully long so I don't find myself disinclined to run it frequently.
The integration target on the other hand takes long enough that I don't want to do it very often - ideally just before I'm ready to do a deploy. Does this seem like a reasonable strategy? IOW, am I doing it right?
The plan is to eventually move this project to Continuous Integration. I'm new to the whole Continuous Integration thing but I think I understand the concept of "breaking the build" so I'm wondering what are some good practices to pick up in order to make the most of it?
Any good sources of reading on this subject would be appreciated as well. Thanks!
Yes, you are on the right track. What you need to do now is to hook up your nant target to an automated process. I recommend using either Team City or Cruise Control for as your CI tool. Once you have your automated server setup you can run your build and unit tests on each check in (Continuous Integration). Your integration tests could then run at night or over the weekend since they typically take longer to run. If your integration tests are successful, you can then have a job that will deploy to some QA or other server.
Sounds like you're 99% of the way there. My advice is to just dive in and start doing it. You'll learn a lot more by actually taking the plunge and doing it than by thinking about whether you're doing it right.
My company is currently using CruiseControl and I personally think it's great.
See this related thread What is a good CI build process?
You are on the right track. If you're using a decent CI tool, you should be able to set each setup up as a separate project that triggers the next step in the chain... i.e. sucessfull build triggers tests which trigger deployment which triggers integration etc
This way your ealiest "break" stops the line so to speak.
We use CruiseControl to build, unit-test, configure and deploy, run integration tests and code coverage, run acceptance tests, and package for release. This is with a system of 8 or so web services, and a dozen or so databases, all with interralated configuration and deployment dependencies with across multiple environments with different configurations (anythin from single boxes to redundent boxes for each component)
Setting up an integration server, I’m in doubt about the best approach regarding using multiple tasks to complete the build. Is the best way to set all in just one big-job or make small dependent ones?
You definitely want to break up the tasks. Here is a nice example of CruiseControl.NET configuration that has different targets (tasks) for each step. It also uses a common.build file which can be shared among projects with little customization.
http://code.google.com/p/dot-net-reference-app/source/browse/#svn/trunk
I use TeamCity with an nant build script. TeamCity makes it easy to setup the CI server part, and nant build script makes it easy to do a number of tasks as far as report generation is concerned.
Here is an article I wrote about using CI with CruiseControl.NET, it has a nant build script in the comments that can be re-used across projects:
Continuous Integration with CruiseControl
The approach I favour is the following setup (Actually assuming you are in a .NET project):
CruiseControl.NET.
NANT tasks for each individual step. Nant.Contrib for alternative CC templates.
NUnit to run unit tests.
NCover to perform code coverage.
FXCop for static analysis reports.
Subversion for source control.
CCTray or similar on all dev boxes to get notification of builds and failures etc.
On many projects you find that there are different levels of tests and activities which take place when someone does a checkin. Sometimes these can increase in time to the point where it can be a long time after a build before a dev can see if they have broken the build with a checkin.
What I do in these cases is create three builds (or maybe two):
A CI build is triggered by checkin and does a clean SVN Get, Build and runs lightweight tests. Ideally you can keep this down to minutes or less.
A more comprehensive build which could be hourly (if changes) which does the same as the CI but runs more comprehensive and time consuming tests.
An overnight build which does everything and also runs code coverage and static analysis of the assemblies and runs any deployment steps to build daily MSI packages etc.
The key thing about any CI system is that it needs to be organic and constantly being tweaked. There are some great extensions to CruiseControl.NET which log and chart build timings etc for the steps and let you do historical analysis and so allow you to continously tweak the builds to keep them snappy. It's something that managers find hard to accept that a build box will probably keep you busy for a fifth of your working time just to stop it grinding to a halt.
We use buildbot, with the build broken down into discrete steps. There is a balance to be found between having build steps be broken down with enough granularity and being a complete unit.
For example at my current position, we build the sub-pieces for each of our platforms (Mac, Linux, Windows) on their respective platforms. We then have a single step (with a few sub steps) that compiles them into the final version that will end up in the final distributions.
If something goes wrong in any of those steps it is pretty easy to diagnose.
My advice is to write the steps out on a whiteboard in as vague terms as you can and then base your steps on that. In my case that would be:
Build Plugin Pieces
Compile for Mac
Compile for PC
Compile for Linux
Make final Plugins
Run Plugin tests
Build intermediate IDE (We have to bootstrap building)
Build final IDE
Run IDE tests
I would definitely break down the jobs. Chances are you're likely to make changes in the builds, and it'll be easier to track down issues if you have smaller tasks instead of searching through one monolithic build.
You should be able to create one big job from the smaller pieces, anyways.
G'day,
As you're talking about integration testing my big (obvious) tip would be to make the test server built and configured as close as possible to the deployment environment as possible.
</thebloodyobvious> (-:
cheers,
Rob
Break your tasks up into discrete goal/operations, then use a higher-level script to tie them all together appropriately.
This makes your build process easier to understand for other people (you're documenting as you go so anyone on your team can pick it up, right?), as well as increasing the potential for re-use. It's likely you won't reuse the high-level scripts (although this could be possible if you have similar projects), but you can definitely reuse (even if it's copy/paste) the discrete operations rather easily.
Consider the example of getting the latest source from your repository. You'll want to group the tasks/operations for retrieving the code with some logging statements and reference the appropriate account information. This is the sort of thing that's very easy to reuse from one project to the next.
For my team's environment, we use NAnt since it provides a common scripting environment between dev machines (where we write/debug the scripts) and the CI server (since we just execute the same scripts in a clean environment). We use Jenkins to manage our builds, but at their core each project is just calling into the same NAnt scripts and then we manipulate the results (ie, archive the build output, flag failing tests etc).