I have a Delphi application that has many dependencies, and it would be difficult to refactor it to use DUnit (it's huge), so I was thinking about using something like AutomatedQA's TestComplete to do the testing from the front-end UI.
My main problem is that a bugfix or new feature sometimes breaks old code that was previously tested (manually), and used to work.
I have setup the application to use command-line switches to open-up a specific form that could be tested, and I can create a set of values and clicks needed to be done.
But I have a few questions before I do anything drastic... (and before purchasing anything)
Is it worth it?
Would this be a good way to test?
The result of the test should in my database (Oracle), is there an easy way in testcomplete to check these values (multiple fields in multiple tables)?
I would need to setup a test database to do all the automated testing, would there be an easy way to automate re-setting the test db? Other than drop user cascade, create user,..., impdp.
Is there a way in testcomplete to specify command-line parameters for an exe?
Does anybody have any similar experiences.
I would suggest you plan to use both DUnit and something like TestComplete, as they each serve a different purpose.
DUnit is great for Unit Testing, but is difficult to use for overall application testing and UI testing.
TestComplete is one of the few automated testing products that actually has support for Delphi, and our QA Engineer tells me that their support is very good.
Be aware though that setting up automated testing is a large and time-consuming job. If you rigourously apply unit testing and auomated UI testing, you could easily end up with more test code than production code.
With a large (existing) application you're in a difficult situation with regards to implementing automated testing.
My recommendation is to set up Unit Testing first, in conjunction with an automated build server. Every time someone checks anything in to source control, the Unit Tests get run automatically. DO NOT try to set up unit tests for everything straight up - it's just too big an effort for an existing application. Just remember to create unit tests whenever you are adding new functionality, and whenever you are about to make changes. I also strongly suggest that whenever a bug is reported that you create a unit test that reproduces the bug BEFORE you fix it.
I'm in a similar situation. (Large app with lots of dependencies). There is almost no automated testing. But there is a big wish to fix this issue. And that's why we are going to tackle some of the problems with each new release.
We are about to release the first version of the new product. And the first signs are good. But it was a lot of work. So next release we sure need some way to automate the test process. That's why I'm already introducing unit tests. Although due to the dependencies, these are no real unit tests, but you have to start somewhere.
Things we have done:
Introduced a more OO approach, because a big part the code was still procedural.
Moved stuff between files.
Eliminated dependencies where possible.
But there is far more on the list of requirements, ensuring enough work for the entire team until retirement.
And maybe I'm a bit strange, but cleaning up code can be fun. Refactoring without unit tests is a dangerous undertaking, especially if there are a lot of side effects. We used pair programming to avoid stupid mistakes. And lots of test sessions. But in the end we have cleaner code, and the amount of new bugs introduced was extremely low.
Oh, and be sure that you know this is an expensive process. It takes lots of time. And you have to fight the tendency to tackle more than one problem in a row.
I can't answer everything, as I've never used testcomplete, but I can answer some of those.
1 - Yes. Regression testing is worth it. It's quite embarrassing to you as a developer when the client comes back to you when you've broken something that used to work. Always a good idea to make sure everything that used to work, still does.
4 - Oracle has something called Flashback which lets you create a restore point in the database. After you've done your testing you can just jump back to this restore point. You can write scripts to use it too, FLASHBACK DATABASE TO TIMESTAMP (FEB-12-2009, 00:00:00);, etc
We're looking at using VMWare to isolate some of our testing.
You can start from a saved snapshot, so you always have a consistent environment and local database state.
VMWare actions can be scripted, so you can automatically install your latest build from a network location, launch your tests and shut down afterwards.
Is it worth it?
Probably. Setting up and maintaining tests can be a big job, but when you have them, tests can be executed very easily and consistently. If your project is evolving, some kind of test suite is very helpful.
Would this be a good way to test?
I would say that proper DUnit test suite is better first step. However if you have large codebase which is not engineered for testing, setting up functional tests is even bigger pain than setting up GUI tests.
The result of the test should in my database (Oracle), is there an easy
way in testcomplete to check these values (multiple fields in multiple tables)?
TestComplete has ADO and BDE interface. Or you can use OLE interface in VBScript to access everything that's available.
Is there a way in testcomplete to specify command-line parameters for
an exe?
Yes.
One way to introduse unittesting in an (old) application could be to have a "Start database" (like the "Flashback" feature described by Rich Adams).
The program som unittest using DUnit to control the GUI.
Se the "GUI testing with DUnit" on http://delphixtreme.com/wordpress/?p=181
Every time the test is started by restoring to "Start database", because, then a known set of data, can be used.
I would need to setup a test database to do all the automated
testing, would there be an easy way to
automate re-setting the test db?
Use transactions: perform a rollback when the test completed. This should revert everything to the initial state.
Recommended reading:
http://xunitpatterns.com/
Related
I have an understanding that it might be the DBA's job to push the changes, much like a refresh, but without the DML (Data Manipulation).
Any comments/suggestions will be great!
There is no simple answer for this, but essentially the task of deploying code is similar for any computer language, with the main differences for the database component being
We can't drop and rebuild tables because we need to keep their contents.
Our code compiles in the database, so there are no binaries to deploy.
The first rule for PL/SQL development is that you should always, always work from source-controlled files, for example the code for mypackage would be in a source file named mypackage.pck (or whatever file extension works best with your chosen IDE, as long as it's not .sql). Don't edit database source code directly except for trivial testing when you don't care about keeping the changes.
Some sites only ever do incremental deployments, while others use a full teardown and rebuild from a release branch for major releases, which takes a bit more thought but is ultimately cleaner. Then deployment consists of running the scripts, recompiling the schema and perhaps running some tests and checks to ensure it has worked. You'll need a branching strategy, perhaps some kind of 'run everything in this folder' script, and ideally some tools such as Team City or Jenkins to automate as much as possible, though I don't think there is as much out there ready-made for PL/SQL as there is for more mainstream languages such as Java.
The deployment process is usually performed by an application support team as it does not require any DBA privileges unless you are creating schemas or roles etc, although some sites may organise support roles differently.
Yes. Partially it will involve DBA. But, as the developer, you probably need to provide the entire DDL script by exporting (probably in a file) to the DBA to deploy.
Check 'Importing and exporting SQL scripts' part of this link for exporting db scripts: https://docs.oracle.com/cd/B25329_01/doc/appdev.102/b25309/sql_rep.htm#BABBHEHA
I've been thinking about this a bit. The idea is that something goes wrong in PROD. The data that was captured causes the web app to behave differently than in other environments. So, also data in other environments gets out of sync with prod (as expected). However, a bug comes along and for some reason it only happens in PROD, probably because of the differences in data.
I'm wondering what is a good practice to remedy these kinds of problems? More tests, for sure. But beyond that? One could create new data in dev, but the whole point is that some data point, or some combination of actions causes a data point to be wrong. Perhaps when using some other data source to arrive at the "actual" data point, which is different then the "expected" data point. Apologizes that this isn't a great description, and tries to be both an example and a definition of a general production bug.
I know this isn't a very precise question. Hopefully, there are references that make good suggestions.
This is a very interesting question. One approach I've used before is to deliberately do my final testing in production (TIP).
Before you skewer my effigy with multiple pointy needles, hear me out for a minute while I talk about continuous deployment :-)
The idea is to deploy a new build into production and then use custom routing to direct traffic between the old and new production builds. In principle this is quite simple: you start by routing the old build to your current customers and the new build only to your engineering team. Your customers don't see any change. But your team can start testing your new build, including messy stuff like disaster recovery and stress-testing. You will hopefully discover the type of bugs that you talk about in your question.
If there's a problem, then you simply rollback the new build. If your tests don't find any problem, you roll-out to say 5% of your client base. Then 10% and 20% and so on.
Whilst simple in principle, there are some issues that you need to plan for from the very start. The first is data and data schemas, which need to function correctly across both old and new builds. As long as the services used by your web app are designed to handle at least one rollback after a new build is deployed, and your new build understands both the old and new data, then you should be okay.
The second issue is API/interface changes. Rather than editing or deleting methods or parameters, you need to create a new API/interface that mostly re-directs to the old API/interface, except for the new/changed code.
Other issues including incompatible changes to configuration and settings between builds. These issues aren't fatal, but you do need to do some planning and testing beforehand. And the big reward is that you can safely do final testing of your code in production without affecting your customers.
Some links on testing in production:
There's no place like production
The future of software testing
Production is a mixed blessing
TIP - malpractice?
TIP really happens
Why TIP isn't as stupid as it sounds
Sorry if a similar question has been posed before. There are a lot of deployment questions but none seemed to address my problem.
Anyway. I'm working with asp.net, C# and using Visual Studio.
The Organization I'm working in is changing rapidly. There are a lot of projects coming in the pipeline that will require multiple code changes and iterative deployments over the next few months. While working, these changes are always 'on the forefront', so sometimes I have to code certain parts of the same program multiple times.
Since these projects are all staggered, I can't just make one sweeping change all at once; I have to deploy and redeploy the same program multiple times, using only the changes that are required for that deployment.
If this is confusing, here's a simple example:
Application is being used on an Intranet. This application calls our Database, using Driver A.
There are two environments, test and production.
Certain Stored procedures have to be called with parameters that register 'Test' to allow certain other applications to run even with bad data (for testing purposes).
When deploying applications, these stored procedures have to be modified, removing Test parameters
We have an Operating System upgrade, allowing us to move to a much faster Driver B, but requires changes to be made to the code to use Driver B.
So that's two wholly different deployments where some code must be changed for Deployment 1 and other code must be changed for Deployment 2.
Currently I'm just using notepad for an overall change list, regular debugging break points and a multitude of in-code comments, and then I manually slog through the code to make sure that everything is changed. With hundreds of thousands of lines of code over multiple files, classes, objects, etc. this gets pretty tedious, as well as there being a good chance of missing something (causing it to break) or pushing wrong changes (causing it to either break or allow bad data).
Is there a tool that could be used to help in this situation? Preferably one that I can discern what needs to change for Deployment A and what needs to change for Deployment B? I'm also open to hearing other schools of thought as well (tips are definitely accepted!)
Sure, I understand your problem.
I would suggest a couple of things
Installers : Why don't you think of installers, there are loads of installers i.e Install shield, Wix, MSI installer.
These installers will give you flexibilty to update files which you need to update, i.e. Full Control.
But you need to choose the best of them, I have worked around MSI and Wix a lot, so I know this can sort your problem, however its your call.
Publish : I haven't played around much with this, I have just done website publish. However I know it does wonders, so try it also.
Has anyone seen this very strange behaviour before?
I've got a solution whith 70 unit tests. All of them pass on my dev machine.
Whenever I commit my changes, our continuous integration process kicks in and the build box will eventually run the same 70 unit tests.
There is only ONE test in the build box that fails all the time.
The error is in one line that only gets a record from our unit test db. (I know it sucks having unit test to rely on data but please don't focus on this as it's not relevant now)
The most weird thing is when I logon myself to the build box, open up the same visual studio solution and manually kick off the unit tests. Result: ALL PASS!
Has anyone ever had this weird situation? I'm guessing there is some weird thing going on with Cruise Control.NET and MSTest?
Surely your unit test runner produces a good log that shows the exact exception message or error? It's kinda pointless to guess at it but an "access denied" kind of error would be an obvious candidate. Setup whatever dbase engine you use (you forgot to mention that too) to give the user account that runs the tests on the build grunt access to the tables.
As said in another answer, it doesn't make too much sense to guess about it when there are detailed logs around...
But because I had this situation several times, here's a guess anyway:
The account, which is used by the CI server to run the tests, may not have appropriate permissions in the database. This would also explain, why the same test succeeds when you run it manually (then with your user account)...
HTH!
Thomas
thanks for your inputs but it wasn't anything related to credentials at all.
I've found out that other tests that were running before that particular one were leaving my unit test database in an inconsistent state, therefore causing errors to the test in question.
It's not a good practice to have your unit tests relying on data, so unless you are extremely bound to it like myself, this is what a recommend to everyone: DO NOT RELY ON DATA TO DO YOUR UNIT TESTING !!!! Make sure you have all the good stuff in place, specifically a good IOC/dependency injector container so your classes are loosely coupled and you can mock up any interface you may want to unit test easily!
If you have system tests that you want to run on your build server or in general, want to be able to run correctly on any machine, including your own, then you must make sure that their states are independent.
In your case, you should have each test init prepare the DB it uses (either by copying a file-based DB or by emptying/filling a service-based DB). Each test should also attempt to undo its changes (delete file or empty DB) but not assume that other tests have done so successfully.
As a follow up to one of my previous posts 'Using Version Control for Home Development', I am now asking about opinions as regards using a Build Server for a pet project.
Lately I have been reading about this 'Build Servers' concept, and I have looked at applications such as Maven and CruiseControl.Net.
And thus I ask, how feasible is it to use something like CruiseControl.Net for my home pet projects?
Reason I ask is that I think that these Build Servers are mainly aimed for team projects...but then again, I'm still very new to this Automated Build process.
Keep in mind that most of the time, these pet projects are only handled by one man, not a team.
So should I look more into this concept for the sake of using at home, or should I just get some practice on it for experience's sake?
[EDIT]
Although I thank you all for your answers as regards alternatives to CC.Net and such, no one has yet really tackled the issue of whether it is feasible or not to implement a Build System for Home Development ?
It is completely feasible to implement a build server for your home projects. I've implemented CC.Net for my home projects myself and it is pretty easy to do so, even for the first time. I would say the learning curve (depending on your experience) is less than a day to get your first project up and building, though there is always the longer tail on that curve as you dig into some of the more interesting details.
The question to me is more one of the motivation for continuous integration on these projects. If you are using "Home Project" synonomous with "Throw-away Project", there probably isn't much point in going to the trouble of CI unless you are using it specifically as a CI learning excercise.
However, assuming these are not throw-away projects you are talking about, I've found (in addition to the more obvious benefits of automation) that implementing CI helps reduce the overhead involved in coming back to a project you've walked away from for some period of time. Of course, unit tests are the most valuable asset in this regard, but the combination of unit tests with an automated build/deployment process really allows you to focus on the new and changed requirements when you come back to a project after having set it down for a while.
Additionally, as mghie points out in the comments to this answer, "CI will give even greater benefit for home projects if they build upon each other, so changes in one project could cause the build to break in others."
My advice, just do it once so you have a clearer picture of what is involved and the benefits you might reap and drawbacks you might incur. Then make the decision for yourself as to whether or not it is worth continuing to do. Like I said, the learning curve is reasonably low so the investment you will have to make in just giving it a try shouldn't be the reason not to.
Nutshell: Feasible - Yes, Desirable for home projects - Quite Possibly, Worth further investigation - Definitely, Investment - Relatively low
As an alternative to CC.Net, I recommend you to give a look to TeamCity, is really easy to setup and get it up and running.
Related question:
Best Continuous Integration Setup for a solo developer (.NET)
i installed CC.net months ago it took me a whole night to configure it and create the xml configurations and i have no regrets about it, it smoothly integrates with SVN, Nunit, Nant or Msbuild. you should try it only if it is to gain experience
Take a look at Hudson its very easy. You need to just deploy in Tomcat or any other servlet you use container. Once up every configuration can be done using browser. Hudson supports maven, ant etc and supports all the major SCMs. I have been using hudson for the past one year and not faced any trouble.
CC.NET is very feasible, in fact with the free cost and wide ranging supported actions. Not to mention the fact that since you can get the source code you can modify it to you needs I could not imagine anything better. I read the other compliants about how difficult it is to set up, but to be honest I had my first simple TFS/VS2005 project up within an hour. Just remember if you run into any issues or snags CC.NET has a pretty active google groups for Users and Devs who would be willing to help you through any gotchas.
I love CC.NET and I'm a big fan of CI, but I have to ask: with only one developer on the project, what integration scenarios exist? Wouldn't you just build the entire project in Visual Studio, negating the need for CI?
I would agree that CC.NET is a great option for local/home development. I wanted to add that it does not require an SCM tool in order to make it work. There is a file system watcher plugin that will just monitor for a file change. That way you don't have to have a check-in in order for it to execute. Also you don't have to wait for the CI cycle to complete, it's much like having F6 run everytime except for the IDE doesn't use all it's resources, you can keep coding away. If it breaks, you can choose to investigate or just ignore. There is no one way to do CI.
If you do create unit tests, having that constantly execute against your code on every save certainly has some advantages in early problems. Using CCTray allows you to see it, but not be intruded upon. My 2 cents.
Finally, setting it up the first time can be a little tough, but you can tweak out a Visual Studio C# template or whatever you desire to automatically configure your CI setup with the least amount of information required by the user.