What is the point of using a Continuous Integration system to test your code if you already have a system like Husky running that allows you to test you code before pre-commit and pre-push?
Pre-commit and pre-push hooks are great for quick operations and tests. Sometimes you can even setup a hook in your IDE that will run quick unit tests every time you save a file. But usually you have multiple suites of tests and unlike unit tests functional, integration and performance tests often take longer time to run, which is not feasible for hooks.
Also, you want to run your tests in the same environment where you build your deliverables, which is usually not your local machine.
Another reason to use CI system is to run post-merge tests to verify that there are no issues introduced by multiple parallel merges.
All-in-all, the more tests you run, the better and a CI system allows you to run both pre-merge tests usually triggered by some sort of pull request hook and post-merge tests. And all of that in a controlled reliable environment.
I'm not really interested about whether it passes in your local environment, where you may have a different version of some dependent library on your environment path. I want to know for sure that anyone's contributions don't break the software when linked against the specific library versions that we ship with.
One reason to test using a Continuous Integration platform like Travis would be to assure developers haven't circumvented their own local development environment's testing git hooks.
CI is not only tests, it's a lot more, but the test stage is of course a very important part of the flow.
As you said in your own answer, local environments could be changed, the tests on the CI could have stricter settings, the environment you test on could be more like the environment that the end-user uses (say, set versions of software or even hardware).
Say for example that you develop a PHP package. The package have support for everything between php 5.6 and 7.2, it should also support multiple types of operating systems and should behave differently if ext/open_ssl is installed or not. A local test suite would rarely have a setup allowing the developer to test each of the possible versions on each of the required platforms, but a test suite set up in a CI pipeline could.
And honestly, it's always a good idea to test one more time, just to be safe! ;)
In certain useful and reasonable workflows, it is acceptable to commit and push broken commits (though not to the master branch). Preventing such workflows with git hooks is annoying.
Rebasing or merging as an example does not run hooks again, even though files are changed.
Hooks are also very difficult to get right. They check a local state which might not be what gets pushed (if certain files are present that are not in git).
CI servers also provide a stable predictable environment. E.g. consider a CI server having Linux and developers using MacOs laptops. The git hooks run on MacOs, which has case insensitive file system, allowing tests to pass even if filenames are wrong.
Hooks also punish diligent developers who run checks manually before committing, because tests are just run again one more time.
Each professional project should have CI. The real question is why any project should maintain annoying slow fragile broken local hooks when you already have CI.
Use hooks only for private toy projects.
Related
I just got to know about Travis CI and went through some of their docs. It seems to be a nice solution for open source projects.
With my reading so far through Travis docs, I am doubtful whether I will be able to connect it to my personal hardware in some manner.
I am working on some IoT related project written in C/C++ hosted on github. Building and publishing images on artifactory, on Travis CI should not be a problem. But when it comes to testing, definitely it cannot be tested on their (Travis's) hardware. The binaries need to be put on my development board (raspberry pi) and then test cases should be executed. Once test suit finishes, Travis CI should be notified of the results.
Is Travis allows such functionality? If not, then it would be a great limitation.
But when it comes to testing, definitely it cannot be tested on their
(Travis's) hardware
By this statement you answered your own question. If you want to use Travis for showing test status publicly, your best chance is running the tests on premises using Jenkins, GitLab CI or any other CI server and then offering an API to Travis (e.g. a file with the test results). All Travis would then do is fetching and showing the test results.
If you want to control things from within Travis, you could trigger builds from Travis using a HTTP call and then wait for the remote build to finish before showing its results. Both Jenkins and GitLab CI offer remote triggering. However, this requires your CI servers to be publicly accessible, which might be a security risk (e.g. people triggering non-stop builds).
TravisCI is primarily useful for testing libraries and projects that can be run and tested on common systems (linux, etc.) I don't believe there is a way to run TravisCI locally however, MinionCI seems to offer a solution for running a CI server locally following a style similar to TravisCI, check it out here.
I've been wondering how you go about doing CI-style testing when you're dealing with physical devices.
I imagine you have a suite of tests, and a pool of devices against which they can be run.
Additionally:
Some tests may require specific device models.
Some tests may require the use of more than one device.
What CI servers have support for this?
I'm still interested in those which have partial support, either natively or through plugins, as I'm interested in how it's done.
Continuous Integration enables a team integrate & test their work frequently. Automated builds are meant to compile, link and run unit-tests. You want your CI to run fast, especially if you run it at every check-in. That is why you would want to restrict CI activities to simple confirmation of the build and unit tests alone. What you're asking seems more-along the lines of quality assurance (QA) testing...and having QA failures mixed into your CI efforts would detract development efforts from progressing.
As such, I'm more under the impression activities associated with CI are not dependent upon the final physical machine said work may eventually be migrated to.
Now...this doesn't mean you CAN'T take the CI-compiled package and run it against some final target-machine....but again...that is really considered a seperate activity.
This seems to be re-enforced in the following article by Martin Fowler.
Notice he doesn't talk about the final targeted devices...only the build machine.
I can suggest Test Manager that is part of the TFS suite of Microsoft. I have not tried it with many different environments apart from windows based though I know there are many connectors. For windows based environments I believe it will satisfy most needs.
I use it for nightly builds to perform smoke tests (Turn it on, see if any smoke comes out) but you have to be careful to keep tests small in order to have them finished in a matter of hours and not days, if you want it to be part of your CI.
Then when you have a good enough quality you can proceed onto regression tests and integration tests if needed.
I wouldn't get too caught up in what a CI system is supposed to do or not do. Instead I would focus on the problem you are trying to solve. It sounds like that problem is to facilitate development on multiple platforms. You can use the concept of Continuous Integration and add to it successfully address the issue. I know, because I've done it in the past.
I implemented a build system for code that needed to compile and test successfully on 4 different platforms (nt, wince, linux-arm, linux-x86). The CI server would:
Used a linux and winnt build server compilation (and cross compilation)
The compiled tests and supporting libs would then be copied to the appropriate devices and an automated test run executed.
After the test suite was completed the log would be copied back, (or it was written to a network mounted fs)
If the test suite was successful we would tag the source, and package the libs and executables.
This same platform was reused for developer verification before commits. Developers would run a partial build and test (only updated source would be recompiled and those tests rerun). The CI would execute a full build (from scratch).
Our build were pretty fast because we had a proper DAG for build dependencies. This allowed for concurrent compilation within a platform build. Each platform build was also concurrent. As a result partial builds took a few seconds, full builds took ~30 minutes. Our build servers were quite beefy (optimized for fast compiles) and the codebase was of moderate size (I don't remember the stats).
my team is currently using buildbot to automate overnight and continuous-integration builds and regression tests. For builds and unit tests, the builder just invokes a script which syncs the sources from p4 and then runs xcodebuild. The regression tests are also launched by a shell script, and are themselves combinations of shell scripts and AppleScripts. The builds are all performed on one system but then buildbot triggers tests to happen on multiple Macs, with different architecture and OS combinations. The things I like about buildbot are the automatic triggers (so the tests run only if and when the build succeeds), and the reporting including the waterfall view to see the overall status.
However, I see a number of problems too. The "master" process which coordinates the builds is either leaky or just has a huge working set which means that it consumes a couple of gigabytes of memory. Occasionally network problems mean that a slave will be lost; rather than retrying later it will just fail the build. In fact if the slave is supposed to be triggered for a dependent build, it will fail the first build after it has otherwise successfully completed.
So what are the rest of you using to automate your Xcode builds and unit tests? How do those solutions work for you? Anything you would recommend?
I use TeamCity in a windows environment but I believe it will work on Mac and has xcode build agents.
Another similar question here.
Our team use Hudson with a script which converts the Unit Test output from XCode to NUnit format.
I know this is an older question, but for those using TFS to hold their Xcode projects, I created a custom build activity to make automating Xcode builds via TFS easier. The code is hosted on Codeplex here: http://tfsxcodebuild.codeplex.com/.
Hope someone finds it useful!
We worked out what was using all the memory - log files - older buildbot keeps all logs forever and keeps them in memory (possibly until a restart).
Newer buildbot can be configured to keep a fixed amount of history.
Buildbot build log files should be limited in size to avoid the problem.
I use the XcodeBuilder that's part of CruiseControl. Of course it helps that I wrote it. :)
But I did use it on a real multiple person project for an iPhone app (Surf) that's for sale in the app store.
Now that it's three years later I have my own answer to provide to this question. I'm using Jenkins, mainly for the plug-ins that it provides. It has a plug-in for building targets in Xcode projects (or schemes in workspaces). There's a plug-in to run the Clang static analyzer. It interfaces with my bug-tracker system, it'll automatically push builds to Testflight too.
Our software is built on linux and windows platforms. Depending on the preference of the developer a contribution is developed and tested on either platform and then committed to our subversion repository. It then turns out that the contribution doesn't build on the other platform, and a fix has to be made. The fix on the other platform may again break the build on the original platform, and so on.
I'd rather see that a contribution is built (and regression tested) on the other platform as well before being committed. We have a continuous build server (CruiseControl), but that server builds from the repository. I am looking for a solution where the continuous build server builds on the other platform as a pre-commit check and then commits the stuff if the build and test succeed.
Any suggestions?
Teamcity handles pre-tested commit, you may be able to do something with the new build chaining features in 4.0( http://www.jetbrains.com/teamcity/features/newfeatures.html ). Agents are cross platform and can be configured to only run particular bits of the build, so could possibly be configured to only run a subset of tests.
Note that I haven't actually done this :)
It might be easier to have two branches, one where people checkin, and another which they merge their changes into after they have passed continuous integration.
Matheiu Godlewski has made a good suggestion at the CruiseControl wiki
If you combine his suggestion with the veto element, I think you should be set.
We used a custom build and test rig that could remotely deploy to multiple OSes (and multiple Database products on multiple OSes). This was done as a nightly build with a rule that you fix your bugs the next morning.
No entirely continuous then, but that's potentially a lot of work to do on a pre-commit hook. Especially if your source control repository locks the affected files for the duration of the pre-commit hook execution.
I think there's a difference between a continuous integration test that runs during the day, per-commit, and a system integration test that runs nightly.
Douglas Leeder suggested an "integration" branch - the nice thing about it is that it's possible to automate. If the tests pass - merge to the 'trunk'.
Some version control systems (e.g. bzr/hg/git) make this easier than others, but it's possible on most.
Setting up an integration server, I’m in doubt about the best approach regarding using multiple tasks to complete the build. Is the best way to set all in just one big-job or make small dependent ones?
You definitely want to break up the tasks. Here is a nice example of CruiseControl.NET configuration that has different targets (tasks) for each step. It also uses a common.build file which can be shared among projects with little customization.
http://code.google.com/p/dot-net-reference-app/source/browse/#svn/trunk
I use TeamCity with an nant build script. TeamCity makes it easy to setup the CI server part, and nant build script makes it easy to do a number of tasks as far as report generation is concerned.
Here is an article I wrote about using CI with CruiseControl.NET, it has a nant build script in the comments that can be re-used across projects:
Continuous Integration with CruiseControl
The approach I favour is the following setup (Actually assuming you are in a .NET project):
CruiseControl.NET.
NANT tasks for each individual step. Nant.Contrib for alternative CC templates.
NUnit to run unit tests.
NCover to perform code coverage.
FXCop for static analysis reports.
Subversion for source control.
CCTray or similar on all dev boxes to get notification of builds and failures etc.
On many projects you find that there are different levels of tests and activities which take place when someone does a checkin. Sometimes these can increase in time to the point where it can be a long time after a build before a dev can see if they have broken the build with a checkin.
What I do in these cases is create three builds (or maybe two):
A CI build is triggered by checkin and does a clean SVN Get, Build and runs lightweight tests. Ideally you can keep this down to minutes or less.
A more comprehensive build which could be hourly (if changes) which does the same as the CI but runs more comprehensive and time consuming tests.
An overnight build which does everything and also runs code coverage and static analysis of the assemblies and runs any deployment steps to build daily MSI packages etc.
The key thing about any CI system is that it needs to be organic and constantly being tweaked. There are some great extensions to CruiseControl.NET which log and chart build timings etc for the steps and let you do historical analysis and so allow you to continously tweak the builds to keep them snappy. It's something that managers find hard to accept that a build box will probably keep you busy for a fifth of your working time just to stop it grinding to a halt.
We use buildbot, with the build broken down into discrete steps. There is a balance to be found between having build steps be broken down with enough granularity and being a complete unit.
For example at my current position, we build the sub-pieces for each of our platforms (Mac, Linux, Windows) on their respective platforms. We then have a single step (with a few sub steps) that compiles them into the final version that will end up in the final distributions.
If something goes wrong in any of those steps it is pretty easy to diagnose.
My advice is to write the steps out on a whiteboard in as vague terms as you can and then base your steps on that. In my case that would be:
Build Plugin Pieces
Compile for Mac
Compile for PC
Compile for Linux
Make final Plugins
Run Plugin tests
Build intermediate IDE (We have to bootstrap building)
Build final IDE
Run IDE tests
I would definitely break down the jobs. Chances are you're likely to make changes in the builds, and it'll be easier to track down issues if you have smaller tasks instead of searching through one monolithic build.
You should be able to create one big job from the smaller pieces, anyways.
G'day,
As you're talking about integration testing my big (obvious) tip would be to make the test server built and configured as close as possible to the deployment environment as possible.
</thebloodyobvious> (-:
cheers,
Rob
Break your tasks up into discrete goal/operations, then use a higher-level script to tie them all together appropriately.
This makes your build process easier to understand for other people (you're documenting as you go so anyone on your team can pick it up, right?), as well as increasing the potential for re-use. It's likely you won't reuse the high-level scripts (although this could be possible if you have similar projects), but you can definitely reuse (even if it's copy/paste) the discrete operations rather easily.
Consider the example of getting the latest source from your repository. You'll want to group the tasks/operations for retrieving the code with some logging statements and reference the appropriate account information. This is the sort of thing that's very easy to reuse from one project to the next.
For my team's environment, we use NAnt since it provides a common scripting environment between dev machines (where we write/debug the scripts) and the CI server (since we just execute the same scripts in a clean environment). We use Jenkins to manage our builds, but at their core each project is just calling into the same NAnt scripts and then we manipulate the results (ie, archive the build output, flag failing tests etc).