My team is developing a desktop application (mixed C++/Tcl) that is used in a client-server setup. Currently it is Windows-only, but soon we will need to port it to Linux. CruiseControl.NET builds it every night from the source code in SVN and packages it into NSIS installer, but we have no automated tests to run.
It is nearly impossible to add any unit tests, but integration testing of the application is easy, because it is already heavily script-based.
The main task is to install the app into 3 PCs, configure it (that involves copying some files around), run it, monitor a possible crash, wait till integration testing is done, collect a summary, send emails. It could be done with a bunch of custom PowerShell scripts, but
In future we will want to add more features and more testing, and
what used to be a simple script soon blows up (as usual), so I want
to minimize custom scripting, and if I need to script something, I
prefer bash/cygwin (I am not familiar with Python or Ruby).
I want a web dashboard that will report current progress, and if
something failed - show logs
I need some supervisor that will monitor the app under test and
report if it hangs or crashes
we will need to test it also on Linux
ideally I would like to orchestrate some test steps between the PCs
(e.g. run test X on PC1 and test Y on PC2 in parallel, wait till they
both finish, then run test Z on PC1, while monitoring that nothing
crashes on PC2 etc)
So, I am looking for a COTS tool/set of tools that will help me to do it and don't have a steep learning curve. Ideally, for free, but if it is really good and has fair pricing, my company may purchase a license.
The process should be triggered from CruiseControl.NET when the NSIS installer is ready, and then perform everything described above. Basically, it should allow at least remote installation of software, running custom scripts and have a web dashboard.
Apparently, SCCM tools like Chef could be used, but so far neither of them supports a Windows server, only nodes. I would like to avoid setting up a Linux VM just for that, although I can do it, if I have no other choice. Also, Chef seems to be a bit overkill - good for 10k machines, but I have only 3... maybe 5 in future. And I am particularly curious about chances to orchestrate a distributed test.
Most of the similar questions here on StackOverflow and in internets are about web apps, Java containers, Maven etc, and there are just so many tools and plugins for these tools to evaluate.
Thanks in advance.
Install ccnet on your test machines. Have those ccnet projects listen to a file that gets edited when a new installer is ready. Have the test machines install that new installer and run tests. There you go. ccnet sends emails so there's your basic reporting.
Have the test results get reported into a database via web services using gSOAP(that's what we did). For linux you can run java cruisecontrol if you must. Write a gSOAP enabled test controller program to report the test results from the test machines. A little c++ app will do. Then write a website(we use ASP.NET) to query the database(Postgresql) and show results. Have the test machines auto update themselves via SVN to get the latest changes to the configuration. Use Nant. Nant is far superior to just using ccnet to run tasks. Nant works through ccnet. Use XML, XSL and CSS with ccnet to make test emails have the information you want(new passes, new failures, SVN differences to code bases, etc...)
Our latest development is putting a big TV in the kitchen with a summary of test results so people can know more readily what they broke!
The first thing I'd get working is a test machine listening for the new installer, installing it, running some basic tests and emailing the results back. Put the ccnet and nant configuration in version control and get that auto updating on the test machine so you don't have to log into every test machine and do an update every time you make a change.
This is hugely broad and pretty close to opinion based. Chef can handle steps like deploying the application to the test machines but it isn't a GUI test framework so you would need something else to handle that. Jenkins supports distributing tests to windows hosts so that seems like a good choice on that side of things but it isn't that great at multi-node tests or orchestration between them. I suspect you'll need to write most of this yourself given the requirements.
Related
We have a testing framework made in AutoIt for our Windows apps (older legacy apps that we will continue supporting). These have never been run on a schedule or part of CI (always been run manually). I tried to get some kind of auto run (or even just status reporting) out of the tests, with minimal luck.
I have a VM where the tests can run. I experimented with my own web app, which works okay locally for running and status reporting. But when set up on the server, AutoIt reports it cannot open the application. Same thing happens if I try to run the tests from a .bat file.
My current solution is to have AutoIt call my web app to report status (working okay locally, untested on the server), or to see if I can get AutoIt to report results back to TeamCity. I have the agent installed but when I run the build from TeamCity, AutoIt reports it can't launch the application. I tried this while logged into the VM, logged out, with RDS open; no luck.
Is it possible to run the tests manually from the VM and send results back to TeamCity? When I run them from TeamCity it reads the AutoIt output (which is in the expected format), but I need to let TeamCity know to update the results (so we can use TeamCity rather than my web app to show the results).
I may need to find a way to let TeamCity know a build has been started, which might then let it know to keep an eye on the process' output, but I'm not positive. Any ideas?
I solved this so it could be done more traditionally.
If anyone is confused by what "running the agent from the console" meant, it just means installing the agent without selecting the "as a service" checkbox, and then manually starting the agent by cd'ing to the BuildAgent/bin directory and running the command agent start. I also created a batch file that will do this automatically (but you must run it as admin).
Further, I found AutoIt couldn't do anything if the test doesn't run from the right directory, so I had to devise a solution to this.
The only issue now is that I have to have an old laptop always connected to the virtual machine the tests and app run on (since the AutoIt tests won't work without the VM desktop being interactive).
Currently we are running some of our small ruby scripts through CI machines like teamcity. But the problem is that teamcity is only free to a certain point and we are reaching that cap. The thing I like about teamcity is the fact that I can define how to run the scripts in it and then have the logs shown in each of the "build" processes so if something goes wrong or I want to verify something I don't have log onto the server and inspect individual files.
The problem is that I need to be able to run the same process at about 4x the capacity I am now, which means I need about 4 times the build agents which leaving the free licensing. Now obviously I could just spin up more teamcity servers but that then becomes a pain.
So my question is, what is another way that I could just basically setup cron processes on linux machines (i have a lot "freely") but then give myself the exposure and ease of access of logs similarly that I gain from teamcity. Obviously I know that setting up cron processes aren't hard but I really want to avoid having to log onto the machine to check and see if my automated processes are running correctly or struggling.
Thanks in advance!
p.s. I also have access to windows machines if there is an easier way to do it there.
Install an open-source CI server like Jenkins if you want to host it yourself. You can also run it on your own machine, though it's usually better to have it always on in the cloud.
I know that I can code on one machine and have it build on a different machine (ie. a build server). Now I have also heard that you can have visual studio run a build on a virtual machine (i think it requires Virtual PC). Now my question is if anyone has been able to code on machine A, have it compile on machine B and run a debugging sesion on machine C?
This is pretty common in enterprise development and just about the de facto standard way of doing things.
Typically, a dev works locally. Once s/he is happy with their changes, they'll check it into a source control system.
From that point there are a couple of options ranging from automated building to having someone push the button to cause the remote build.
Once the build is complete there are a host of options available for deploying the app to one or more other servers. And yet other options for kicking off automated test suites.
Concerning remote debugging, you can do that independently of whether you are using a build/deployment/automated testing. It's just a matter of getting the right stuff installed and configured (see ho1's answer for a link).
All of that said, I highly recommend you never enable remote debugging on a production server. Some people might disagree with me but I personally think it's dangerous for security reasons and can certainly lead to site outages.
Finally, the only reasons you would need a virtual machine is if the servers aren't available or if you just want to sandbox everything.
You can do remote debugging, so if you had an automated process to copy the compiled code from B to C, I suppose you could do what you're asking.
See this MSDN article for more details: How to: Set Up Remote Debugging
can anyone help?
I have a solution in vs 2010 which contains a WCF Service application and a class library which has a number of Nunit test.
The nunit tests the service. Problem is that there is no Asp.net Development server running - is it possible to force it to run under debug mode..
I can press F5 but then i can't run my tests.
Maybe i am doing this all wrong, should i host them under IIS 7 ? rather than the built in development server.
That way the server is ALWAYS available rather then loading on demand
See http://www.codeproject.com/KB/aspnet/test-with-vs-devserver-2.aspx for a very lightweight and, in my biased opinion, elegant way to integrate webdev/cassini into your testing environment.
NOTE: the first example of controlling the .exe is for edutainment purposes only. Read through to the second, very brief, code sample for the money shot.
Update:
I use the technique listed above with nunit and most every unit test runner around, including continuous integration on servers that I do not control, extensively with great results.
When I need to simulate a real server with a host name and non-loopback ip address, which WebDev.WebServer and WebDev.WebHost cannot do,
I add the host name to my hosts file, e.g. localhost.skysanders.net 192.168.0.10
use the CassiniDev .dll in almost exactly the same way
it is so simple and bullet proof it should be illegal. ;-)
CassiniDev is also a perfect replacement for the WebDev.WebServer.exe for VS 2008/2010. It has integrated traffic monitoring and is quick a bit faster, both at startup and at processing.
I haven't tried this but since you haven't gotten any other responses:
http://xneuron.wordpress.com/2009/03/27/launch-cassini-using-command-line/
This looks like a command you could add as a pre-build step to your NUnit tests. Hopefully Resharper honors those commands and ensures they're run before doing its own test-specific builds.
I have a solution with exactly the same layout. I would much rather test them in IIS since that is where they will be deployed, and it's nice to know that the code definitely runs under the restricted IIS account rather than my domain account. I run the unit tests as follows:
Make sure IIS is running.
Right-click on the WCF Service project and Publish to IIS.
Fire up NUnit on the client test project and run the tests.
If all tests pass, then great, Otherwise...
Leaving NUnit open, switch back to Visual Studio, and attach the debugger to the IIS worker process (aspnet_wp.exe on my dev machine).
Switch back to NUnit and re-run the failed tests. Any exceptions and breakpoints will break into the debugger as normal.
Rinse and repeat as many times as you like.
As a guy who frequently switches between QA, build and operations, I keep running into the issue of what to do about operating system updates on the build server. The dichotomy is the same on Windows, Linux, MacOS or any other o/s that can update itself via the internet:
The QA team wants to keep the build server exactly as it is from the beginning of the product release cycle to the end, since installing updates could destabilize the server and means that successive builds aren't made against the same baseline.
The ops team wants the software to be deployed on a system with all the latest security patches; this can mean that the software isn't deployed on exactly the same version of the o/s that it was built on.
I usually mitigate this by taking release candidate builds and installing them on a test server that has a completely up-to-date o/s, repeating the automated tests that are run on the build server and doing some additional system level testing to make sure everything looks good before deployment. However, this seems inefficient to me; does anyone have a better way ?
Personally i don't think you have much of an issue here - just apply the latest updates to the build server. The main reasons i say this are:
it is highly unlikely that your code or any of the dependencies on the build server are so tightly coupled to the OS version that installing regular updates is going to affect anything, let alone break it. There can be minor differences between window messages etc between Windows versions, but those are few and far between, and are usually quite well documented out there on teh interweb. If you are using managed technology stacks like WPF/Silverlight or ASP.Net and even mostly Winforms then you will be isolated from these changes - they should only affect you if you are doing hardcore stuff using the WinAPI directly to create your windows or draw your buttons.
it is a good practice to always engineer your product against the latest version of the OS, because you need to encourage your customer to implement those updates too - IOW you should not be in a position where you have to say to your client to not install update xyz because your application will not run against it - especially if that update is a critical security update
testing for differences between OS versions should be done by the QA team and should independant of what is on the build server
you do not want your build server to get in to such a state that it has been so isolated from the company update process that when you finally do apply them all it barfs and spits molten silicon everywhere. IOW, the longer you wait to update, the higher the risk of something going wrong and doing so catastrophically. Small and frequent/incremental updates are lower risk than mass updates once per decade :)
The build server updates that you do have to be cautious about are third party controls or library updates - they can frequently contain breaking changes or considerably altered behavior. They really should be scheduled, and followed up by a round of testing looking for any changes.
Virtualize!
Using stuff like VMWare Server you can script the launch and suspend of virtual machines. So you can script VM resume, SSH to launch build, copy, VM suspend, repeat. (I say this, but I abandoned my work on this. Still, I was making progress at the time.)
Also, you can trust your OS vendors. Can't you?
They have an interest in compatibility. If you build on Windows XP it is almost certain to work on XP SP3 and Vista and Windows 7.
If you build on RedHat Enterprise 5, it had better work on 5.1, 5.2, 5.3, 5.4, etc.
In my experience this has worked out OK so far for me and I recommend building on your lowest patch OS versions. With the Linux stuff in particular I have found newer releases linking to more recent libraries not available on older versions.
Of course it doesn't hurt to test your code on a copy of the deployment server. It all depends on how certain you want to be.
Take the build server off the network, that way you do not need to worry about installing security updates. Only load the source from CD, thumb drive or whatever other means.
Plug it back in at the end of your release cycle and then let all the updates take place.
Well, for the most stable process, I would have two build servers, "Build with Initial config, Build with update config", and two autotest test servers with similar differences. Use virtualization to do this effectively and scriptably.