I know this may be overkill for a single developer solution (personal project and not yet enterprise software), but I was wondering how to better respond to my needs.
I would be needing to accomplish the following:
Run integration tests (none UI for the moment) at least daily in order to see if any of my commits breaks the build.
Build the entire solution daily to see if any of my commits are incomplete and would cause problems when checked out on another folder.
Be run on my personal computer at least once a day (using another computer to automate the build process is not an option for the moment)
I know that automated build software such as Jenkins are easily capable of doing the previous (even on the same machine as committed?), but I was wondering if lighter solutions are available. Ex: Post-commit actions on the repository?, scripts?, planned tasks etc...
Edit
Forgot to mention that I was using a Windows machine with a c# project running nunit tests. I use visual studio 2012 to compile solution and run tests with nunit. I use tortoise svn and Ank svn as repository browser.
You might make a crontab(5) entry to periodically (e.g. daily) run your build or tests.
I have a crontab entry invoking some shell script to fetch the source tree by svn or git version control in a fresh place and build it daily.
You could consider using inotify(7) facilities, perhaps thru incron, to have a test run as soon as you modify some file (e.g. an executable).
Look also at D.Moreno's garlic project (which I never used).
You could also simply have some Makefile targets for tests, and run them from emacs. I have
(load-library "compile")
(global-set-key [f8] 'recompile)
in my ~/.emacs so I just compile things by pressing the F8 key in my emacs editor.
Use Jenkins - no reason not to, considering its reasonably lightweight itself (despite being a java app). Its very self-contained too, backup involves stopping the Jenkins service and copying the installation directory so it's not going to pollute your OS.
Anything else you come up with is going to be too complex (in terms of maintaining a bundle of scripts, scheduled tasks and so on) or just as 'heavyweight'. You might as well save your time and use the tool that fits from the start.
Related
We have a large collection of nAnt scripts that build our various products. They almost all have the following structure:
Erase old working copy.
Check out complete fresh copy from version control.
Increment build number in appropriate file (custom nAnt task).
Run static analysis (StyleCop, Perl scripts)
Build solution using Visual Studio - ends up with MSI output.
Run unit tests (nUnit, JSUnit)
Run static analysis (FxCop)
Zip up deliverables (MSI, readme, etc) into well-named package.
Put this zip package onto a server share.
Email results to team.
From our research, it seems that CruiseControl(.net?)/Hudson/BuildBot would only add the trigger that causes the build, which at the moment is double-clicking the nAnt script over Remote Desktop and a status dashboard.
Are we missing anything else significant?
The question is subjective, and thus so is my answer.
In the projects I've automated before, CruiseControl was used essentially for that one purpose: so we didn't have to remote into the build machine and trigger builds. The CI part is that CruiseControl will monitor the repository for you, triggering builds at the intervals you define.
It also gave us the dashboard from which could trigger releases, or go back to examine logs and artefacts from past builds.
For us that was enough benefit to implement CruiseControl. Perhaps it doesn't "seem" like much until you've finished it and a month later realized you haven't had to touch your build system because it's off silently and thanklessly doing its thing for you.
A Continuous Integration server such as Hudson would do 1, 2, 3, 9 and 10 for you so that you don't have to implement them yourself. If you've already got it working that's maybe not a huge improvement for your current project but it makes things simpler for subsequent projects. It would also, as you mention, take care of when to trigger the build.
Hudson will also chart various trends over time, such as test coverage, build time, static analysis results. You can also have more sophisticated notifications than just e-mail if you choose.
The most important thing it gives you is visual feedback (the bigger the screen is better). When you have one machine, dedicated to displaying buildresults, visible to all team members, it works like a catalyst to people see that something is wrong and fixes it.
If you have something like that standing in a place where your boss can see it and ask you "Hey Wilkinson, why is this screen red?" will you fix your build faster?
Thay all look the same, you can pick whatever you think fits your needs, just have one setup and running.
my team is currently using buildbot to automate overnight and continuous-integration builds and regression tests. For builds and unit tests, the builder just invokes a script which syncs the sources from p4 and then runs xcodebuild. The regression tests are also launched by a shell script, and are themselves combinations of shell scripts and AppleScripts. The builds are all performed on one system but then buildbot triggers tests to happen on multiple Macs, with different architecture and OS combinations. The things I like about buildbot are the automatic triggers (so the tests run only if and when the build succeeds), and the reporting including the waterfall view to see the overall status.
However, I see a number of problems too. The "master" process which coordinates the builds is either leaky or just has a huge working set which means that it consumes a couple of gigabytes of memory. Occasionally network problems mean that a slave will be lost; rather than retrying later it will just fail the build. In fact if the slave is supposed to be triggered for a dependent build, it will fail the first build after it has otherwise successfully completed.
So what are the rest of you using to automate your Xcode builds and unit tests? How do those solutions work for you? Anything you would recommend?
I use TeamCity in a windows environment but I believe it will work on Mac and has xcode build agents.
Another similar question here.
Our team use Hudson with a script which converts the Unit Test output from XCode to NUnit format.
I know this is an older question, but for those using TFS to hold their Xcode projects, I created a custom build activity to make automating Xcode builds via TFS easier. The code is hosted on Codeplex here: http://tfsxcodebuild.codeplex.com/.
Hope someone finds it useful!
We worked out what was using all the memory - log files - older buildbot keeps all logs forever and keeps them in memory (possibly until a restart).
Newer buildbot can be configured to keep a fixed amount of history.
Buildbot build log files should be limited in size to avoid the problem.
I use the XcodeBuilder that's part of CruiseControl. Of course it helps that I wrote it. :)
But I did use it on a real multiple person project for an iPhone app (Surf) that's for sale in the app store.
Now that it's three years later I have my own answer to provide to this question. I'm using Jenkins, mainly for the plug-ins that it provides. It has a plug-in for building targets in Xcode projects (or schemes in workspaces). There's a plug-in to run the Clang static analyzer. It interfaces with my bug-tracker system, it'll automatically push builds to Testflight too.
I've just installed Hudson and it is running beautifully. It builds, runs JUnit-tests and also CheckStyle analysis.
Next step for us would be to create an installation, install it and then run automated tests on the actual installation. I would then like to fail the build if the tests fail or at least publish the results somehow. I think we would set it up so that part runs periodically or manually triggered.
We use InstallAnywhere for installation and IBM Rational Functional Tester for automated tests.
So questions are: anyone created a similar setup? are there any plugins, tutorials or other resource that could help me along. Or do you have any tips or advice in general.
The command line reference for Rational Functional Tester:
http://publib.boulder.ibm.com/infocenter/rfthelp/v8r0m0/index.jsp?topic=/com.ibm.rational.test.ft.doc/topics/RobotJCommandLine.html
Sample command for running a test:
java -classpath "C:\IBM\RFT\FunctionalTester\bin\rational_ft.jar"
com.rational.test.ft.rational_ft -datastore \\My_project\AUser\RobotJProjects -user admin -project
\\My_project\AUser\TestManagerProjects\Test.rsp -build "Build 1" -logfolder "Default" -log
"Al_SimpleClassicsA#1" -rt.log_format "TestManager" -rt.bring_up_logviewer true -playback
basetests.SimpleClassicsA_01
An additional note, you'll want to configure windows properly on your agent machine which will be running the tests. This is not advice specific to Hudson or RFT, but rather all GUI automation tools on Windows. RFT will require an interactive desktop environment for it to be able to click buttons, etc. If you have your Hudson agent running as a Windows service, there will be no desktop. See the following: Silverlight tests not working unless RDP connection open
We have run a fairly complicated distributed build on Hudson, it is a process that basically follows:
Test on Windows.
Test on OSX, run code coverage & push results to server.
Test on OSX Tiger.
Package for OSX Leopard & push build to server.
Package for Windows & push build to server.
Update product website.
We don't use InstallAnywhere or Rational Functional Tester, but have similar sorts of mechanisms in their place. The key we found to making it all sing in Hudson was being able run our various steps from the command line. Maven and appropriate plugins made short work of this task. So my advice would be just that, using whatever build tool you are using (ant, maven, ?) configure them so that you can run your rational functional tester and install anywhere from the command line with a simple goal passed to your build tool (i.e. mvn test or mvn assembly:assembly).
After that, make sure whatever machine Hudson is running on has everything installed (i.e. Rational Functional Tester) and configured, so that you can open up the command line and type in the goal and have your tests correctly execute.
Hooking it up in Hudson from that point on is fairly simple - just pass in the goal when you configure the build.
I believe the best answer is that integrating RFT with Hudson/Jenkins is a useless endeavor.
As this IBM FAQ says, to make RFT work you must:
be logged in the machine;
the screen can't be locked;
if you are remotely connected, you can't minimize the connection screen.
So you can't run Jenkins/Hudson as a service, making it not very useful. You must run it from your logged account. If you are in a corporate computer (very probable if you are using RFT), you probably must use a hack to prevent the screen saver to start. If the screen is locked, your tests will always fails.
It isn't very difficult to configure your tests to run from the command line, you just have to take care of the return codes when the tests fail and succeed.
Jenkins/Hudson would also give you some advantages, like integrating the tests with your version control, probably automatically running the tests when a commit is made. It would also help sending emails when the tests fail.
But you still would have to integrate the RFT logs with some kind of JUnit plugin to have a nice report. You also would have to have script to run the tests using the command line.
I think it is not worth the trouble to use an continuous integration server with RFT. Better just have your tests running every day in Windows Task Scheduler. It is a simpler solution with less failure points.
Or use my final solution: quit RFT and use the free Selenium with a headless web driver.
I have some general advice on this because I have not yet implemented this myself.
I am assuming you want to have Hudson run the RFT scripts automatically for you via a build or Hudson process?
I want to implement something similar in my organisation as well.
I have not yet been able to implement this because of organisational constraints but here is what I have thought out/done so far:
Downloaded Windows process viewer, got the command for running the tests.
Made shell Script out of it, separated out the variables etc
The future plan is to setup a Windows Slave machine which would have all the tools in it that would be required once the Tests are kicked off, for eg. the correct versions of browsers, and environment variables, and other tools that are required.
Hudson would kick off a process which runs the shell scripts created which runs all the RFT Scripts and performs necessary operations on the slave machine.
I am trying to set up a Continuous Integration process. For my various build tasks(compiling, testing, documentation etc.)I need to have tools that perform these tasks(csc, NUnit, NDoc etc.). My question is should these tools too go into my source control repository?
Why I think that they should is because I read in some online article that the developer environment should be as much similar to the build server environment. To fulfill this requirement, the article suggested that you put everything that is required for your build in the repository and when you check out the code(or the build server checks out the code) you are ready to build the project right away without first installing any other tools. But on the other hand if I put these tools with my source code in the repository then the build server will have to install them whenever a build is run.
Is it OK to install these tools? Won't it increase the time for each build unnecessarily?
It's often more trouble than it's worth to try to check in tools to source control. Rather, write a list of software requirements that must be installed before the source can be checked out and built (one thing that would need to be on this list in any case is the source control system itself). If you rely on software being in source control, some tools might need to be installed in certain paths or be otherwise configured (registry entries come to mind).
I would certainly not check in the compiler itself to source control, and I probably wouldn't check in NUnit or NDoc either. Just install these beforehand, as they are not likely to change too much over the lifetime of your project. Your build script might want to check that the expected version(s) of the required software packages are installed before the build may proceed.
Unless you're customizing the tools there's probably no reason to put their source code in your repository. However there are excellent reasons for putting your config files in the repository.
Re-installing the tools for every single build is overkill and will slow you down.
However it's by far better to have a server dedicated to the continuous integration so that you know its state ; you sure nobody installed anything that may have an impact on the outcome of the build.
If you want to be able to re-generate today's build next year, you need to be able to re-create your environment first. Make sure you'll be able to re-install your tools (exact same version), either by keeping them on your server (installing the newer versions in different directories), or storing the whole package in your configuration management tool.
Think about how you would create another continuous integration server, either to have two of them, or for a second site, or to recover after a disaster. Document how the continuous integration server was set up.
What really needs to be version controlled, is the build scripts, that access the right versions of the tools, especially if you opt for installing several versions of the tools.
Setting up an integration server, I’m in doubt about the best approach regarding using multiple tasks to complete the build. Is the best way to set all in just one big-job or make small dependent ones?
You definitely want to break up the tasks. Here is a nice example of CruiseControl.NET configuration that has different targets (tasks) for each step. It also uses a common.build file which can be shared among projects with little customization.
http://code.google.com/p/dot-net-reference-app/source/browse/#svn/trunk
I use TeamCity with an nant build script. TeamCity makes it easy to setup the CI server part, and nant build script makes it easy to do a number of tasks as far as report generation is concerned.
Here is an article I wrote about using CI with CruiseControl.NET, it has a nant build script in the comments that can be re-used across projects:
Continuous Integration with CruiseControl
The approach I favour is the following setup (Actually assuming you are in a .NET project):
CruiseControl.NET.
NANT tasks for each individual step. Nant.Contrib for alternative CC templates.
NUnit to run unit tests.
NCover to perform code coverage.
FXCop for static analysis reports.
Subversion for source control.
CCTray or similar on all dev boxes to get notification of builds and failures etc.
On many projects you find that there are different levels of tests and activities which take place when someone does a checkin. Sometimes these can increase in time to the point where it can be a long time after a build before a dev can see if they have broken the build with a checkin.
What I do in these cases is create three builds (or maybe two):
A CI build is triggered by checkin and does a clean SVN Get, Build and runs lightweight tests. Ideally you can keep this down to minutes or less.
A more comprehensive build which could be hourly (if changes) which does the same as the CI but runs more comprehensive and time consuming tests.
An overnight build which does everything and also runs code coverage and static analysis of the assemblies and runs any deployment steps to build daily MSI packages etc.
The key thing about any CI system is that it needs to be organic and constantly being tweaked. There are some great extensions to CruiseControl.NET which log and chart build timings etc for the steps and let you do historical analysis and so allow you to continously tweak the builds to keep them snappy. It's something that managers find hard to accept that a build box will probably keep you busy for a fifth of your working time just to stop it grinding to a halt.
We use buildbot, with the build broken down into discrete steps. There is a balance to be found between having build steps be broken down with enough granularity and being a complete unit.
For example at my current position, we build the sub-pieces for each of our platforms (Mac, Linux, Windows) on their respective platforms. We then have a single step (with a few sub steps) that compiles them into the final version that will end up in the final distributions.
If something goes wrong in any of those steps it is pretty easy to diagnose.
My advice is to write the steps out on a whiteboard in as vague terms as you can and then base your steps on that. In my case that would be:
Build Plugin Pieces
Compile for Mac
Compile for PC
Compile for Linux
Make final Plugins
Run Plugin tests
Build intermediate IDE (We have to bootstrap building)
Build final IDE
Run IDE tests
I would definitely break down the jobs. Chances are you're likely to make changes in the builds, and it'll be easier to track down issues if you have smaller tasks instead of searching through one monolithic build.
You should be able to create one big job from the smaller pieces, anyways.
G'day,
As you're talking about integration testing my big (obvious) tip would be to make the test server built and configured as close as possible to the deployment environment as possible.
</thebloodyobvious> (-:
cheers,
Rob
Break your tasks up into discrete goal/operations, then use a higher-level script to tie them all together appropriately.
This makes your build process easier to understand for other people (you're documenting as you go so anyone on your team can pick it up, right?), as well as increasing the potential for re-use. It's likely you won't reuse the high-level scripts (although this could be possible if you have similar projects), but you can definitely reuse (even if it's copy/paste) the discrete operations rather easily.
Consider the example of getting the latest source from your repository. You'll want to group the tasks/operations for retrieving the code with some logging statements and reference the appropriate account information. This is the sort of thing that's very easy to reuse from one project to the next.
For my team's environment, we use NAnt since it provides a common scripting environment between dev machines (where we write/debug the scripts) and the CI server (since we just execute the same scripts in a clean environment). We use Jenkins to manage our builds, but at their core each project is just calling into the same NAnt scripts and then we manipulate the results (ie, archive the build output, flag failing tests etc).