Related
In the old days programming used to involve less guesswork. I would write some lines of code and be 100% certain about what the code does and what it does not at a glance. Errors were mostly typos, but not about the functionality.
The last years I believe there is a trend for this "trial-and-error" programming : write the code (as if in draft), and then debug iteratively until the program's behavior appears to comply with the requirements. Test, and test again, and then again.
Funny thing is, in my Visual Studio the "Run" button has been replaced by a button labelled "Debug" (= I know you have some bugs!). I have to admit that in several apps that I write I cannot guarantee a bug-free code.
What do you think ? Or maybe our systems are now overly complicated (browser/OS/Service Pack compatibilities, etc etc) and this justifies testing on all types of environments.
I've experienced the opposite, actually. Whereas it used to be a case of running until it worked, I now unit test until the tests pass... and this seems to be at least a reasonably common transition, as far as I can see.
I have to say that code which worked first time with only typos has never been the norm in my experience. The difference is that now I can find the problems much more quickly, and also spot if old problems come back. I can sometimes manage pretty short and simple bits of code with no errors (and posting on Stack Overflow has improved that ability) but large, complex systems? Heck no.
To answer the title of your post - the "test, test, test" principle is a good one, in my view... but I don't associate that with running the whole program repeatedly. I associate it with running unit tests frequently. I rarely need to use the debugger for unit tests - usually a failure makes the cause suitably obvious by inspection, because only a small amount of code is being tested.
The one word answer is "Complexity". The real answer is "Unnecessary Complexity"!
The accounting principles has not changed for the past 30 years. Why then is writing an accounting system is so much more difficult today? It is good to have a Graphic User Interface but do we have to go overboard?
Software development has been caught in a vicious circle for many years. The complexity is feeding itself and instead of reducing it we simply hide it under layers and layers of wrappers. Eventually something is going to give.
When we favor form over function, we have to pay the price.
Could it be that in later years developers have come to the realization that the "100% certainty" might not actually be correct? Developing software is very complex, and even though the tools have evolved over the years, so has our realization that writing good code is hard. True, debugging and automated unit tests have made us more productive, but we still produce bugs, just as we did back then, only now we have different tools to catch them with.
You may write code that you think you know 100% what it does and does not do, but there is always that edge case that you haven't thought of or the odd exception thrown that you don't expect. Some times trial-and-error programming can be a helpful tool to narrow down a problem, with the debuggers help.
Its important to know what tools are available to you to help produce code with minimal bugs.
I have found that the Test-Test approach helps me design the code. Sometimes the work that has to be done is too complex to do it all in one. Testing forces me to split it into smaller parts and as I solve these I am able to put them together into a larger whole.
I think the advantage comes in an indirect way: When you embrace tests and unit tests, you have to write your application in such a way that you can actually write tests:
Classes need to be written in such a way that you can instantiate a single object without the whole application and OS around it, but just a few helper objects. This means you need to minimize the dependencies, and make all communication to the surrounding system explicit.
Implementing the test cases means that you have to find a minimum sequence of commands and calls that makes your class do something meaningful. This often points to awkward design decisions, or shows you that classes are very difficult to use for certain purposes.
All in all, when you embrace tests, you end up with a system that has a minimum of interdependencies between its components, and the test cases serve as documentation of how to use your components.
Testing (executing your system) tells you something about "the presence of bugs but NOT about the absence of them" (afaik this term is coinced by dijkstra). It points to the direction that the strength of your test-suite is the key of testing: "You have so many test cases, that you can say, that many bugs do not exist. This implies that big parts of your software work as expected".
Some examples for having a strong/mighty test-suite:
A lot of code is executed by your unit tests (the traditional coverage term)
You have no false-negative tests (test which show green but in fact should be red). False negative tests are evil, because they give you a wrong sense of test-case quality. For details of good test-asserts and false-negatives see also blog-entry#1 and blog-entry#2.
The requirements are well understood (I have seen a lot of cases where an automated test was testing the wrong thing and the developer misunderstood the requirement from business). For the developer is was green, but for business the system was not working as expected (another kind of false-negative example but on a higher level).
In a sense the correctness of a program is only proven, when it is done with mathematical proofs (which only pays off for life-critical and money-intense systems). Still you can achieve a lot with automated testings (apart from unit-testing, automated integration testing always helped a lot).
Regarding debugging: I use debugging to as often as I used to be, but sometimes when adding new functionality to code (my new test-case shows green) I break other test-cases. By the assert I instantly see that something went wrong, but still didn't locate the bug. For locating the bug debugging is still helpful (with the red test-case I execute the problematic code-paths, with the debugger I locate the bug).
If you're interested in test-automation have a look at masterpiece xUnit Test patterns.
I've read one book ("TDD by example" by Kent Beck) which indeed seems to take that "trial and error" approach to an extreme: but it's more like "make the unit tests work". Still, I couldn't get myself to finish this book - a rare occurence, especially since I really hoped to get a better understanding. Still, committing obviously imbecile code to be improved later makes me shiver.
Science: Automated tests have their advantages. However, they are not the silver bullet they are claimed to be. No single test method is sufficient to findenough defects, and other methods have a better detection rate.
Gut feel: Our problems are facets of ever-increasing complexity. Complexity highly correlates with the amount of code we have to manage. In this light, TDD attempts to solve the problems of to much code by writing even more code.
Advantages: We now have an established formalism to make testing repeatable, accountable and immediately documented. It is definitely a way out of the "works on my machine" and "strange, it worked yesterday, I'll give you the latest DLL" trap.
I currently practice Test Driven Development (TDD), or at least write many unit tests to verify that most/all of my code behaves the way I expect it to behave. Taking this approach forces me to look at my program from the perspective of the consumer. Also, as I write tests, I often think of boundary limits, additional scenarios that I didn't originally envision, etc.
I've now come to the point where I'm afraid to make changes to older programs, as I'm afraid that I'll break something. Regression testing is onerous, compared with running a suite of unit tests.
I'm in the process of doing a study on Test-Driven Development and one of the discussion points is the "Barrier to Entry" associated with TDD. Does anyone have any experience around this area, on any projects you've worked on that decided not to use TDD because the barrier to entry was too high?
From what I can tell the only barrier to entry is knowledge (and as such experience) of individual developers, with most not being entirely accustomed to the process and it being slightly alien. Financially it seems to be very appealing given most of the market leading tools are open source, freely available, well documented and well supported.
Thoughts/feelings appreciated.
Thanks,
EDIT - does anyone know of any high profile quotes of people advocating TDD? Would love to see how high it goes up the chain. Cheers.
Some barriers include:
An existing code base which doesn't lend itself to unit testing.
A problem domain that is hard to unit test meaningfully, such as GUI work or integrations with third party systems.
A perception of integration problems over unit problems (in other words, if it doesn't work end to end it doesn't do anything, so what is the point of testing the unit).
A mindset that wants to design ahead of time and have a clear system design rather than have tests drive design
A political culture where design is done by a different person/group than development, and that design is not unit-test friendly.
An inability to get over the fact that TDD is not about testing for conformance (arguments like "the one who writes the tests shouldn't be the one who codes it, they will be too lenient on themselves" and such variants).
It isn't they way they have coded until now, so the shift is harder.
Sometimes a certain test can be hard to set up, so the method will get abandoned because it "feels" slower.
Design requirements that don't lend themselves to evolving design well or at all (think Nuclear Plant control software or other systems were actual lives depend on their functioning correctly).
If everyone isn't running the test before checking in code, tests start to break often for wrong reasons (that is the intended behavior of the code changed, but the test didn't keep up, so the test is wrong, not the code) so they can be perceived as a drag.
In terms of barriers to entry, effectively, because you are explicitly writing tests that must pass before code is considered to be complete, the lead time in the dev cycle involved in getting functional code is longer. Now, when using TDD, you're effectively guaranteeing a certain level of quality on the code (whatever level of quality you choose to test against) and so that is generally more than enough compensation for the lag in lead time, but strictly speaking, there IS a greater lead time to getting functional code using TDD.
Effectively, if you have coders that write bug-free code, TDD will be a drag on your development cycle. The value of TDD, of course, is that there aren't any coders who can always write bug-free code, and so the cost of fixing bugs has to be factored in somewhere; in TDD, the cost of the test infrastructure is front-loaded.
Note that this is not in any way a negative thing about TDD; I'm just saying, that front-loading COULD be considered to be a "barrier to entry". Personally, as a coder, I would say that the Return on Investment is more than worth the effort, and I think most experienced dev managers would as well.
Team and/or management buy-in is the biggest obstacle in some companies. If you're the lone developer trying to use TDD and you can't get others on the project interested, it can be very frustrating.
Of course that's not a financial barrier at all. The biggest perceived financial barrier is probably time. If you have a large code base that you need to write unit tests for, it can seem quite daunting. Your manager (or someone above them) will question why you want to spend time writing code that will not add features/functionality to the code. Many people don't realize that writing the tests up front (as you do in TDD) can actually save you time, both immediately and in the long run when you're maintaining that code.
I think one major barrier is how it requires you to change the way you think.
Before I tried TDD, I would create a class, say Employee, then I would stub in things like FirstName, LastName, Email, etc. Then I would write some logic and forget that I missed a few fields or something else. And before I knew it I had a pretty complex class without knowing if those fields were ever necessary.
Also, it's a complete change from how we are used to writing software. We are used to writing software as we receive features from the guys who sign our checks. We are not used to writing code which doesn't compile, making it compile, then making it work to make our tests pass.
The first time you do this, you feel a bit.. well silly and stupid. Why am I making my code intentionally fail? It seems illogical to the "make it work" philosophy we've all been taught for so long.
A few reasons why it has failed so far where I work:
Most of the project at work on are older apps. Not pre-.NET but, .NET 2.0 and in some cases .NET 1.0.
Some of these projects are not well factored, either because the technology wasn't there in 1.0, or it was built quickly because they needed something NOW..
As Jon pointed out, some things are still a PIA (pain-in-the-***) to unit test, UI, database, etc.
Expensive tools. If you are only allowed to Microsoft tools, it's a high price tag to do this the "right way". We use resharper, so it really isn't a problem.
Time. I'm in a team of three guys supporting a department of 30 people. We are considered overhead, and many of our development consists of interfacing systems together
Yes the main barrier to entry is in your head or in the head of other programmers.
In the beginning you don't know what to srite in your tests.
The trick is to think about how your code will be used instead of focusing on how you re going to write it. Easier said than done ...
When you start to "get it", it's a bit hard to know where to stop writing tests.
You have to remember that tests prove nothing so you just can't write tests to covers all cases, you have to select the most useful ones ... and that's already a lot !
I've certainly seen plenty of resistance. The barriers I've encountered are:
Unit testing user interfaces (web or thick client) is tricky. I know there are lots of attempts to solve the problem, but I don't think any of them have made it really simple - because it's a naturally hard problem.
At the other end, although there are various ways of making it easier to test the code involved with the database, it's still tricky and time-consuming.
While good tests definitely speed up development overall, testing is a skill - and while you suck at it, unit testing may well be more trouble than it's worth, which means you never build up the skill...
Managers often see it as an optional extra to development - a nice to have rather than critical. This means it's the first thing to go when the project inevitably has a resource squeeze.
I wrote a long-ish article about this a few weeks back, "Why I write tests first".
I think the biggest barrier is building the discipline to start with tests first, but I don't believe the TDD (or any practice for that matter) should be approached as an always, absolutely, 100% of the time solution.
TDD is a tool in each developer's arsenal. I tend to think it works well for me most of the time. A developer that isn't as accustomed to writing tests (first or otherwise) may it difficult to get anything done if TDD is forced on them because they can't think in terms of writing the test first.
I consider myself an experienced test-writer, but I can't always think in terms of tests. Some problems don't lend themselves well to it, or at least my head doesn't get wrapped around it some days. And some types of code (such as UI and client-side code) doesn't lend itself well to always writing tests.
If you have a team of developers that do not write tests as a matter of habit, I'd push that first. I have no problem requiring that all new code have accompanying unit tests where possible/practical. Once testing is a discipline, converting developers to TDD individually or as a team is much easier.
One non-obvious barrier (non-obvious to me, at least) is the build infrastructure. If developers don't have control over the build process, or if the infrastructure is too baroque to be manageable, then integrating tests into the build process is going to be shunted to the side in the name of "efficiency". (Of course, in these situations it's the build infrastructure that should be shunted aside in the name of efficiency.)
I am just curious since all TDD examples I have seen is web programming related. And if it's not a normal approach, why is it not?
TDD has become a favored approach by software developers who are serious about their profession. [IEEE:TDD] The benefits of the approach are significant, and the costs are low by comparison. [The Three Laws of TDD]
There are no software domains for which TDD is inappropriate, or ineffective. However, there are domains in which it is challenging. Gaming happens to be one of these.
Actually, the challenge is not so much gaming as it is UI. The reason UI is a challenge is that you often don't know what you want the UI to look like until you've seen it. UI is one of those things that you have to fiddle with. Getting it right is a deeply iterative process that is full of twists and turns and dead ends and back alleys. Writing tests first for UI is likely to be both difficult and wasteful.
Now before everybody roars off and says: "Uncle Bob says: 'Don't do TDD for UI'" let me say this. The fact that it's hard to do pure TDD for UI does not mean you can't do pure TDD for almost everything else. Much of gaming is about algorithm, and you can use TDD with those algorithms to your heart's delight. It's true, especially in gaming, that some of those algorithms are the kind of code you have to fiddle with, just like UI, and so are probably not amenable to being tested first. But there is a lot of other algorithmic code that can and should be written test first.
The trick here is to follow the single responsibility principle (SRP) and separate those kinds of code that you have to fiddle with, from those kinds that are deterministic. Don't put easy-to-test algorithms in with your UI. Don't mix your speculative code with your non-speculative code. Keep the things that change for reason A separate from the things that change for reason B.
Also, keep this in mind: The fact that some code is hard to test first, does not mean that this code is hard to test second. Once you have fiddled and tweaked and gotten the code to work just the way you like, then you can write the tests demonstrate that the code works the way you think. (You'll be surprised at how many times you find bugs while doing this.)
The problem with writing tests "after the fact" is that often the code is so coupled that it is hard to write the kinds of surgical tests that are most helpful. So if you are writing the kind of code that is hard to test first, you should take care to follow the dependency inversion principle (DIP), and the open/closed principle (OCP) in order to keep the code decoupled enough to test after the fact.
The simple answer is "no", TDD is not a normal approach in game development. Some people will point at Highmoon and Neil Llopis as counter-examples, but it's a big industry and they are the only people I know of who have fully embraced TDD. I'm sure there are others, but they are the only ones I know of (and I've been in the industry for 5 years).
I think a lot of us have dabbled in unit testing at some point, but for one reason or another it hasn't taken hold. Speaking from personal experience it is hard for a games studio to switch to TDD. Usually a codebase is kept from project to project, and applying TDD to a large existing codebase is both tedious and largely thankless. I'm sure that eventually it would prove fruitful, but getting games coders to buy into it is difficult.
I have had some success in writing unit tests for low-level game engine code, because this code tends to have very few dependencies and is easily encapsulated. This has always been testing after the fact though and not TDD. The higher-level game code is usually harder to write tests for because it has far more dependencies and often is associated with complex data and state. Taking AI as an example, to test AI require some kind of context, meaning a navigation mesh and other objects in the world. Setting up that kind of test in isolation can be non-trivial, especially if the systems involved weren't designed for it.
What is more common in game development, and I've had more personal success with, is smoke testing. You'll often see smoke testing used in conjunction with continuous integration to provide various kinds of feedback on the behaviour of the code. Smoke testing is easier because it can be done by just feeding data into the game and reading back information, without having to compartmentalize your code into tiny testable pieces. Taking AI as the example again, you can tell the game to load up a level and provide a script that loads an AI agent and gives it commands. Then you simply determine if the agent performs those commands. This is a smoke test rather than a unit test because you are running the game as a whole and not testing the AI system in isolation.
In my opinion it is possible to get decent test coverage by unit testing the low-level code while smoke testing the high level behaviours. I think (hope) that other studios are also taking a similar approach.
If my opinion of TDD sounds somewhat ambiguous that's because it is. I'm still somewhat on the fence about it. While I see some benefits (regression testing, emphasis on design before code), applying it and enforcing it while working with a pre-existing codebase seems like a recipe for headaches.
Games from Within has an article discussing their use of unit testing, the limitations of unit testing with regards to games in particular, and an automated functional testing server that they set up to help with this.
If you are referring to the practice of writing and maintaining unit tests for every bit of code, I'd venture a guess and state that this is not in widespread use in the gaming industry. There are many reasons for this, but I can think of 3 obvious ones:
Cultural. Programmers are conservative, game programmers doubly so.
Practical. TDD does not fit very well to the problem domain (too many moving parts).
Crunchological. There's never enough time.
The TDD paradigm works best in application domains which are not very stateful, or at least where the moving parts are not all moving at the same time, to put it colloquially.
TDD is applicable to parts of the game development process (foundation libraries and such) but "testing" in this line of work usually means running automated fly-through, random key testing, timing io loads, tracking fps spikes, making sure the player can't wriggle his way into causing visual instabilities, stuff like that. The automaton is also very often a humanoid.
TDD can be a useful tool, but its status as a silver bullet that must-be-ubiquitous-when-making-a-system is rather questionable. Development should not be driven by tests, but by reason. RDD is a crappy acronym though - it won't catch on. ;)
Probably the main reason is that TDD is preferred by those with languages more conducive to it. But apart from that, games themselves are a poor match for the paradigm anyway.
Generally speaking (and yes, I do mean generally speaking, so please don't swamp me with counterexamples), test-driven design works best for event-driven systems and not so well for simulation-style systems. You can still use tests on your low-level components in games, whether test-driven or just plain unit testing, but for more higher level tasks there is rarely any sort of discrete event that you can simulate with deterministic results.
For example, a web application typically has very distinct inputs (an HTTP request), changes a very small amount of state (for example, records in the database), and generates a largely deterministic output (for example, HTML page). These can be easily checked for validity, and since generating the input is simple it's trivial to create tests.
However with games the input may be hard to simulate (especially if it needs to occur at a certain point... think of getting past loading screens, menu screens, etc.), the amount of state you change may be large (for example, if you have a physics system, or complex reactive AI) and the output is rarely deterministic (random number use is the main culprit here, though things like floating point precision loss is another, as might be hardware specifications, or available CPU time, or the performance of a background thread, etc.).
To do TDD you need to know exactly what you expect to see in a certain event and to have an accurate way of measuring it, and both of these are difficult problems with simulations that avoid discrete events, deliberately include random factors, act differently on different machines, and have analogue outputs such as graphics and audio.
Additionally, there's one massive practical issue which is process startup time. Many of the things you will want to test require the loading of large quantities of data, and if you mock up the data you're not truly testing the algorithm. With this in mind it quickly becomes impractical to have any sort of test scaffolding that just performs individual tasks. You can run tests against a web server without having to take the webserver down each time - that's rarely possible with games unless you do the testing from an embedded scripting language (which is reasonable, and does indeed take place in the industry).
For example, you want to add volumetric shadow rendering to your in-game buildings. So you'd need to write a test that starts up all the necessary subsystems (for example, renderer, game, resource loaders), load in buildings (incl. mesh, materials/textures), load in a camera, position that camera to point at the building, enable the shadows, render a scene, and then somehow decide whether the shadows actually appear in the frame buffer. It's less than practical. In reality you'd have all this scaffolding already there in the form of your game, and you'd just fire it up to conduct a visual test in addition to any assertions within the code itself.
Most game developers aren't exactly with it in terms of modern development practices. Thankfully.
But a test-driven development model emphasizes concentrating on how something would be used first, then fleshing out what it does. That in general is good to do since it forces you to concentrate on how a particular feature will actually fit into whatever you're doing (say, a game).
So good game developers do this naturally. Just not explicitly.
#Rune Once again, please emphasise the 'D' rather than the 'T'. At a unit level, the tests are a thinking tool to help you understand what you want and to drive the design of the code. Certainly at the unit level, I find I end up with cleaner, more robust code. The better the quality of the pieces I put into the system, the better they fit together with fewer (but not no) bugs.
That's not the same thing at all as the sort of serious testing that games need.
TDD isn't really a 'normal' approach anywhere yet as it's still relatively new and not universally understood or accepted yet. That isn't to say that some shops don't work that way now but I'm still surprised to hear anyone using it at all at this point.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I'm a contract programmer with lots of experience. I'm used to being hired by a client to go in and do a software project of one form or another on my own, usually from nothing. That means a clean slate, almost every time. I can bring in libraries I've developed to get a quick start, but they're always optional. (and depend on getting the right IP clauses in the contract) Many times I can specify or even design the hardware platform... so we're talking serious freedom here.
I can see uses for constructing automated tests for certain code: Libraries with more than trivial functionality, core functionality with a high number of references, etc. Basically, as the value of a piece of code goes up through heavy use, I can see it would be more and more valuable to automatically test that code so that I know I don't break it.
However, in my situation, I find it hard to rationalize anything more than that. I'll adopt things as they prove useful, but I'm not about to blindly follow anything.
I find many of the things I do in 'maintenance' are actually small design changes. In this case, the tests would not have saved me anything and now they'd have to change too. A highly iterative, stub-first design approach works very well for me. I can't see actually saving myself that much time with more extensive tests.
Hobby projects are even harder to justify... they're usually anything from weekenders up to a say month long. Edge-case bugs rarely matter, it's all about playing with something.
Reading questions such as this one, The most voted on response seems to say that in that poster's experience/opinion TDD actually wastes time if you've got less than 5 people (even assuming a certain level of competence/experience with TDD). However, that appears to be covering initial development time, not maintenance. It's not clear how TDD stacks up over the entire life cycle of a project.
I think TDD could be a good step in the worthwhile goal of improving the quality of the products of our industry as a whole. Idealism on it's own is no longer all that effective at motivating me, though.
I do think TDD would be a good approach in large teams, or any size team containing at least one unreliable programmer. That's not my question.
Why would a sole developer with a good track record adopt TDD?
I'd love to hear of any kind of metrics done (formally or not) on TDD... focusing on solo developers or very small teams.
Failing that, anecdotes of your personal experiences would be nice, too. :)
Please avoid stating opinion without experience to back it. Let's not make this an ideology war. Also the skip greater employment options argument. This is simply an efficiency question.
I'm not about to blindly follow anything.
That's the right attitude. I use TDD all the time, but I don't adhere to it as strictly as some.
The best argument (in my mind) in favor of TDD is that you get a set of tests you can run when you finally get to the refactoring and maintenance phases of your project. If this is your only reason for using TDD, then you can write the tests any time you want, instead of blindly following the methodology.
The other reason I use TDD is that writing tests gets me thinking about my API up front. I'm forced to think about how I'm going to use a class before I write it. Getting my head into the project at this high level works for me. There are other ways to do this, and if you've found other methods (there are plenty) to do the same thing, then I'd say keep doing what works for you.
I find it even more useful when flying solo. With nobody around to bounce ideas off of and nobody around to perform peer reviews, you will need some assurance that you're code is solid. TDD/BDD will provide that assurance for you. TDD is a bit contraversial, though. Others may completely disagree with what I'm saying.
EDIT: Might I add that if done right, you can actually generate specifications for your software at the same time you write tests. This is a great side effect of BDD. You can make yourself look like super developer if you're cranking out solid code along with specs, all on your own.
Ok my turn... I'd do TDD even on my own (for non-spike/experimental/prototype code) because
Think before you leap: forces me to think what I want to get done before i start cranking out code. What am I trying to accomplish here.. 'If I assume I already had this piece.. how would I expect it to work?' Encourages interface-in design of objects.
Easier to change: I can make modifications with confidence.. 'I didn't break anything in step1-10 when i changed step5.' Regression testing is instantaneous
Better designs emerge: I've found better designs emerging without me investing effort in a design activity. test-first + Refactoring lead to loosely coupled, minimal classes with minimal methods.. no overengineering.. no YAGNI code. The classes have better public interfaces, small methods and are more readable. This is kind of a zen thing.. you only notice you got it when you 'get it'.
The debugger is not my crutch anymore : I know what my program does.. without having to spend hours stepping thru my own code. Nowadays If I spend more than 10 mins with the debugger.. mental alarms start ringing.
Helps me go home on time I have noticed a marked decrease in the number of bugs in my code since TDD.. even if the assert is like a Console trace and not a xUnit type AT.
Productivity / Flow: it helps me to identify the next discrete baby-step that will take me towards done... keeps the snowball rolling. TDD helps me get into a rhythm (or what XPers call flow) quicker. I get a bigger chunk of quality work done per unit time than before. The red-green-refactor cycle turns into... a kind of perpetual motion machine.
I can prove that my code works at the touch of a button
Practice makes perfect I find myself learning & spotting dragons faster.. with more TDD time under my belt. Maybe dissonance.. but I feel that TDD has made me a better programmer even when I don't go test first. Spotting refactoring opportunities has become second nature...
I'll update if I think of any more.. this is what i came up with in the last 2 mins of reflection.
I'm also a contract programmer. Here are my 12 Reasons Why I Love Unit Tests.
My best experience with TDD is centered around the pyftpdlib project. Most of the development is done by the original author, and I've made a few small contributions, but it's essentially a solo project. The test suite for the project is very thorough, and tests all the major features of the FTPd library. Before checking in changes or releasing a version, all tests are checked, and when a new feature is added, the test suite is always updated as well.
As a result of this approach, this is the only project I've ever worked on that didn't have showstopper bugs appear after a new release, have changes checked in that broke a major feature, etc. The code is very solid and I've been consistently impressed with how few bug reports have been opened during the life of the project. I (and the original author) attribute much of this success to the comprehensive test suite and the ability to test every major code path at will.
From a logical perspective, any code you write has to be tested, and without TDD then you'll be testing it yourself manually. On the flip side to pyftpdlib, the worst code by number of bugs and frequency of major issues, is code that is/was solely being tested by the developers and QA trying out new features manually. Things don't get tested because of time crunch or falling through the cracks. Old code paths are forgotten and even the oldest stable features end up breaking, major releases end up with important features non-functional. etc. Manual testing is critically important for verification and some randomization of testing, but based on my experiences I'd say that it's essential to have both manual testing and a carefully constructed unit test framework. Between the two approaches the gaps in coverage are smaller, and your likelihood of problems can only be reduced.
It does not matter whether you are the sole developer or not. You have to think of it from the application point of view. All the applications needs to work properly, all the applications need to be maintained, all the applications needs to be less buggy. There are of course certain scenarios where a TDD approach might not suit you. This is when the deadline is approaching very fast and no time to perform unit testing.
Anyways, TDD does not depend on a solo or a team environment. It depends on the application as a whole.
I don't have an enormous amount of experience, but I have had the experience of seeing sharply-contrasted approaches to testing.
In one job, there was no automated testing. "Testing" consisted of poking around in the application, trying whatever popped in your head, to see if it broke. Needless to say, it was easy for flat-out-broken code to reach our production server.
In my current job, there is lots of automated testing, and a full CI-system. Now when code gets broken, it is immediately obvious. Not only that, but as I work, the tests really document what features are working in my code, and what haven't yet. It gives me great confidence to be able to add new features, knowing that if I break existing ones, it won't go unnoticed.
So, to me, it depends not so much on the size of the team, but the size of the application. Can you keep track of every part of the application? Every requirement? Every test you need to run to make sure the application is working? What does it even mean to say that the application is "working", if you don't have tests to prove it?
Just my $0.02.
Tests allow you to refactor with confidence that you are not breaking the system. Writing the tests first allows the tests to define what is working behavior for the system. Any behavior that isn't defined by the test is by definition a by-product and allowed to change when refactoring. Writing tests first also drive the design in good directions. To support testability you find that you need to decouple classes, use interfaces, and follow good pattern (Inversion of Control, for instance) to make your code easily testable. If you write tests afterwards, you can't be sure that you've covered all the behavior expected of your system in the tests. You also find that some things are hard to test because of the design -- since it was likely developed without testing in mind -- and are tempted to skimp on or omit tests.
I generally work solo and mostly do TDD -- the cases where I don't are simply where I fail to live up to my practices or haven't yet found a good way that works for me to do TDD, for example with web interfaces.
TDD is not about testing it's about writing code. As such, it provides a lot of benefits to even a single developer. For many developers it is a mindshift to write more robust code. For example, how often do you think "Now how can this code fail?" after writing code without TDD? For many developers, the answer to that question is none. For TDD practioners it shifts the mindset to to doing things like checking if objects or strings are null before doing something with them because you are writing tests to specifically do that (break the code).
Another major reason is change. Anytime you deal with a customer, they can never seem to make up their minds. The only constant is change. TDD helps as a "safety net" to find all the other areas that could break.Even on small projects this can keep you from burning up precious time in the debugger.
I could go and on, but I think saying that TDD is more about writing code than anything should be enough to justify it's use as a sole developer.
I tend to agree with the validity of your point about the overhead of TDD for 'one developer' or 'hobby' projects not justifying the expenses.
You have to consider however that most best practices are relevant and useful if they are consistently applied for a long period of time.
For example TDD is saving you testing/bugfixing time in a long run, not within 5 minutes after you've created the first unit test.
You're a contract programmer which means that you will leave your current project when it will be finished and will switch to something else, most likely in another company. Your current client will have to maintain and support your application. If you do not leave the support team a good framework to work with they will be stuck. TDD will help the project to be sustainable. It will increase the stability of the code base so other people with less experience will not be able not do too much damage trying to change it.
The same applies for the hobby projects. You may be tired of it and will want to pass it to someone. You might become commercially successful (think Craiglist) and will have 5 more people working besides you.
Investment in proper process always pays-off, even if it is just gained experience. But most of the time you will be grateful that when you started a new project you decided to do it properly
You have to consider OTHER people when doing something. You you have to think ahead, plan for growth, plan for sustainability.
If you don't want to do that - stick to the cowboy coding, it's much simpler this way.
P.S. The same thing applies to other practices:
If you don't comment your code and you have ideal memory you'll be fine but someone else reading your code will not.
If you don't document your discussions with the customer somebody else will not know anything about a crucial decision you made
etc ad infinitum
I no longer refactor anything without a reasonable set of unit tests.
I don't do full-on TDD with unit tests first and code second. I do CALTAL -- Code A LIttle, Test A Little -- development. Generally, code goes first, but not always.
When I find that I've got to refactor, I make sure I've got enough tests and then I hack away at the structure with complete confidence that I don't have to keep the entire old-architecture-becomes-new-architecture plan in my head. I just have to get the tests to pass again.
I refactor the important bits. Get the existing suite of tests to pass.
Then I realize I forgot something, and I'm back to CALTAL development on the new stuff.
Then I see things I forgot to delete -- but are they really unused everywhere? Delete 'em and see what fails in the testing.
Just yesterday -- part way through a big refactoring -- I realized that I still didn't have the exact right design. But the tests still had to pass, so I was free to refactor my refactoring before I was even done with the first refactoring. (whew!) And it all worked nicely because I had a set of tests to validate the changes against.
For flying solo TDD is my copilot.
TDD lets me more clearly define the problem in my head. That helps me focus on implementing just the functionality that is required, and nothing more. It also helps me create a better API, because I'm writing a "client" before I write the code itself. I can also refactor without having to worry about breaking anything.
I'm going to answer this question quite quickly, and hopefully you will start to see some of the reasoning, even if you still disagree. :)
If you are lucky enough to be on a long-running project, then there will be times when you want to, for example, write your data tier first, then maybe the business tier, before moving on up the stack. If your client then makes a requirement change that requires re-work on your data layer, a set of unit tests on the data layer will ensure that your methods don't fail in undesirable ways (assuming you update the tests to reflect the new requirements). However, you are likely to be calling the data layer method from the business layer as well, and possibly in several places.
Let's assume you have 3 calls to a method in the business layer, but you only modify 2. In the third method, you may still be getting data back from your data layer that appears to be valid, but may break some of the assumptions you coded months before. Unit tests at this level (and above) should have been designed to spot broken assumptions, and in failing they should highlight to you that there is a section of code that needs to be revisited.
I'm hoping that this very simplistic example will be enough to get you thinking about TDD a little more, and that it might create a spark that makes you consider using it. Of course, if you still don't see the point, and you are confident in your own abilities to keep track of many thousands of lines of code, then I have no place to tell you you should start TDD.
The point about writing the tests first is that it enforces the requirements and design decisions you are making. When I mod the code, I want to make sure those are still enforced and it is easy enough to "break" something without getting a compiler or run-time error.
I have a test-first approach because I want to have a high degree of confidence in my code. Granted, the tests need to be good tests or they don't enforce anything.
I've got some pretty large code bases that I work on and there is a lot of non-trivial stuff going on. It is easy enough to make changes that ripple and suddenly X happens when X should never happen. My tests have saved me on several occasions from making a critical (but subtle) error that might have gone unnoticed by human testers.
When the tests do fail, they are opportunities to look at them and the production code and make sure that it is correct. Sometimes the design changes and the tests will need to be modified. Sometimes I'll write something that passes 99 out of 100 tests. That 1 test that didn't pass is like a co-worker reviewing my code (in a sense) to make sure I'm still building what I'm supposed to be building.
I feel that as a solo developer on a project, especially a larger one, you tend to be spread pretty thin.
You are in the middle of a large refactoring when all of a sudden a couple of critical bugs are detected that for some reason did not show up during pre-release testing. In this case you have to drop everything and fix them and after having spent two weeks tearing your hair out you can finally get back to whatever you were doing before.
A week later one of your largest customers realizes that they absolutely must have this cool new shiny feature or otherwise they won't place the order for those 1M units they should have already ordered a month ago.
Now, three months later you don't even remember why you started refactoring in the first place let alone what the code you are refactoring was supposed to do. Thank god you did a good job writing those unit tests because at least they tell you that your refactored code is still doing what it was supposed to do.
Lather, rinse, repeat.
..story of my life for the past 6 months. :-/
Sole developer should use TDD on his project (track record does not matter), since eventually this project could be passed to some other developer. Or more developers could be brought in.
New people will have extremely have hard time working with the code without the tests. They will break things.
Does your client own the source code when you deliver the product? If you can convince them that delivering the product with unit tests adds value, then you are up-selling your services and delivering a better product. From the client's perspective, test coverage not only ensures quality, it allows future maintainers to understand the code much more readily since the tests isolate functionality from the UI.
I think TDD as a methodology is not just about "having tests when making changes", thus it does not depend on team- nor on project size. It's about noting one's expectations about what a pice of code/an application does BEFORE one starts to really think about HOW the noted behaviour is implemented. The main focus of TDD is not only having test in place for written code but writing less code because you just do what make the test green (and refactor later).
If you're like me and find it quite hard to think about what a part/the whole application does WITHOUT thinking about how to implement it, I think its fine to write your test after your code and thus letting the code "drive" the tests.
If your question isn't so much about test-first (TDD) or test-after (good coding?) I think testing should be standard practise for any developer, wether alone or in a big team, who creates code which stays in production longer than three months. In my expirience that's the time-span after which even the original author has to think hard about what these twenty lines of complex, super-optimized, but sparsely documented code really code do. If you've got tests (which cover all paths throughth the code), there less to think - and less to ERR about, even years later...
Here are a few memes and my responses:
"TDD made me think about how it would fail, which made me a better programmer"
Given enough experience, being higly concerned with failure modes should naturally become part of your process anyway.
"Applications need to work properly"
This assumes you are able to test absolutely everything. You're not going to be any better at covering all possible tests correctly than you were at writing the functional code correctly in the first place. "Applications need to work better" is a much better argument. I agree with that, but it's idealistic and not quite tangible enough to motivate as much as I wish it would. Metrics/anecdotes would be great here.
"Worked great for my <library component X>"
I said in the question I saw value in these cases, but thanks for the anecdote.
"Think of the next developer"
This is probably one of the best arguments to me. However, it is quite likely that the next developer wouldn't practice TDD either, and it would therefore be a waste or possibly even a burden in that case. Back-door evangelism is what it amounts to there. I'm quite sure a TDD developer would really appeciate it, though.
How much are you going to appreciate projects done in deprecated must-do methodologies when you inherit one? RUP, anyone? Think of what TDD means to next developer if TDD isn't as great as everyone thinks it is.
"Refactoring is a lot easier"
Refactoring is a skill like any other, and iterative development certainly requires this skill. I tend to throw away considerable amounts of code if I think the new design will save time in the long run, and it feels like there would be an awful number of tests thrown away too. Which is more efficient? I don't know.
...
I would probably recommend some level of TDD to anyone new... but I'm still having trouble with the benefits for anyone who's been around the block a few times already. I will probably start adding automated tests to libraries. It's possible that after doing that, I'll see more value in doing it generally.
Motivated self interest.
In my case, sole developer translates to small business owner. I've written a reasonable amount of library code to (ostensibly) make my life easier. A lot of these routines and classes aren't rocket science, so I can be pretty sure they work properly (at least in most cases) by reviewing the code, some some spot testing and debugging into the methods to make sure they behave the way I think they do. Brute force, if you will. Life is good.
Over time, this library grows and gets used in more projects for different customers. Testing gets more time consuming. Especially cases where I'm (hopefully) fixing bugs and (even more hopefully) not breaking something else. And this isn't just for bugs in my code. I have to be careful adding functionality (customers keep asking for more "stuff") or making sure code still works when moved to a new version of my compiler (Delphi!), third party code, runtime environment or operating system.
Taken to the extreme, I could spend more time reviewing old code than working on new (read: billable) projects. Think of it as the angle of repose of software (how high can you stack untested software before it falls over :).
Techniques like TDD gives me methods and classes that are more thoughtfully designed, more thoroughly tested (before the customer gets them) and need less maintenance going forward.
Ultimately, it translates to less time doing maintenance and more time to spend doing things that are more profitable, more interesting (almost anything) and more important (like family).
We are all developers with a good track record. After all, we are all reading Stackoverflow. And many of us use TDD and perhaps those people have a great track record. I get hired because people want someone who writes great test automation and can teach that to others. When working alone, I do TDD on my coding projects at home because I found that if I don’t, I spent time doing manual testing or even debugging, and who needs that. (Perhaps those people have only good track records. I don’t know.)
When it comes to being a good automobile driver, everyone believes they are a “good driver.” This is a cognitive bias all drivers have. Programmers have their own biases. The reasons developers such as the OP don’t do TDD are covered in this Agile Thoughts podcast series. The podcast archive also has content on test automation concepts such as the test pyramid, and an intro about what is TDD and why write tests first starting with episode 9 in the podcast archive.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I was watching Rob Connerys webcasts on the MVCStoreFront App, and I noticed he was unit testing even the most mundane things, things like:
public Decimal DiscountPrice
{
get
{
return this.Price - this.Discount;
}
}
Would have a test like:
[TestMethod]
public void Test_DiscountPrice
{
Product p = new Product();
p.Price = 100;
p.Discount = 20;
Assert.IsEqual(p.DiscountPrice,80);
}
While, I am all for unit testing, I sometimes wonder if this form of test first development is really beneficial, for example, in a real process, you have 3-4 layers above your code (Business Request, Requirements Document, Architecture Document), where the actual defined business rule (Discount Price is Price - Discount) could be misdefined.
If that's the situation, your unit test means nothing to you.
Additionally, your unit test is another point of failure:
[TestMethod]
public void Test_DiscountPrice
{
Product p = new Product();
p.Price = 100;
p.Discount = 20;
Assert.IsEqual(p.DiscountPrice,90);
}
Now the test is flawed. Obviously in a simple test, it's no big deal, but say we were testing a complicated business rule. What do we gain here?
Fast forward two years into the application's life, when maintenance developers are maintaining it. Now the business changes its rule, and the test breaks again, some rookie developer then fixes the test incorrectly...we now have another point of failure.
All I see is more possible points of failure, with no real beneficial return, if the discount price is wrong, the test team will still find the issue, how did unit testing save any work?
What am I missing here? Please teach me to love TDD, as I'm having a hard time accepting it as useful so far. I want too, because I want to stay progressive, but it just doesn't make sense to me.
EDIT: A couple people keep mentioned that testing helps enforce the spec. It has been my experience that the spec has been wrong as well, more often than not, but maybe I'm doomed to work in an organization where the specs are written by people who shouldn't be writing specs.
First, testing is like security -- you can never be 100% sure you've got it, but each layer adds more confidence and a framework for more easily fixing the problems that remain.
Second, you can break tests into subroutines which themselves can then be tested. When you have 20 similar tests, making a (tested) subroutine means your main test is 20 simple invocations of the subroutine which is much more likely to be correct.
Third, some would argue that TDD addresses this concern. That is, if you just write 20 tests and they pass, you're not completely confident that they are actually testing anything. But if each test you wrote initially failed, and then you fixed it, then you're much more confident that it's really testing your code. IMHO this back-and-forth takes more time than it's worth, but it is a process that tries to address your concern.
A test being wrong is unlikely to break your production code. At least, not any worse than having no test at all. So it's not a "point of failure": the tests don't have to be correct in order for the product to actually work. They might have to be correct before it's signed off as working, but the process of fixing any broken tests does not endanger your implementation code.
You can think of tests, even trivial tests like these, as being a second opinion what the code is supposed to do. One opinion is the test, the other is the implementation. If they don't agree, then you know you have a problem and you look closer.
It's also useful if someone in future wants to implement the same interface from scratch. They shouldn't have to read the first implementation in order to know what Discount means, and the tests act as an unambiguous back-up to any written description of the interface you may have.
That said, you're trading off time. If there are other tests you could be writing using the time you save skipping these trivial tests, maybe they would be more valuable. It depends on your test setup and the nature of the application, really. If the Discount is important to the app, then you're going to catch any bugs in this method in functional testing anyway. All unit testing does is let you catch them at the point you're testing this unit, when the location of the error will be immediately obvious, instead of waiting until the app is integrated together and the location of the error might be less obvious.
By the way, personally I wouldn't use 100 as the price in the test case (or rather, if I did then I'd add another test with another price). The reason is that someone in future might think that Discount is supposed to be a percentage. One purpose of trivial tests like this is to ensure that mistakes in reading the specification are corrected.
[Concerning the edit: I think it's inevitable that an incorrect specification is a point of failure. If you don't know what the app is supposed to do, then chances are it won't do it. But writing tests to reflect the spec doesn't magnify this problem, it merely fails to solve it. So you aren't adding new points of failure, you're just representing the existing faults in code instead of waffle documentation.]
All I see is more possible points of failure, with no real beneficial return, if the discount price is wrong, the test team will still find the issue, how did unit testing save any work?
Unit testing isn't really supposed to save work, it's supposed to help you find and prevent bugs. It's more work, but it's the right kind of work. It's thinking about your code at the lowest levels of granularity and writing test cases that prove that it works under expected conditions, for a given set of inputs. It's isolating variables so you can save time by looking in the right place when a bug does present itself. It's saving that suite of tests so that you can use them again and again when you have to make a change down the road.
I personally think that most methodologies are not many steps removed from cargo cult software engineering, TDD included, but you don't have to adhere to strict TDD to reap the benefits of unit testing. Keep the good parts and throw out the parts that yield little benefit.
Finally, the answer to your titular question "How do you unit test a unit test?" is that you shouldn't have to. Each unit test should be brain-dead simple. Call a method with a specific input and compare it to its expected output. If the specification for a method changes then you can expect that some of the unit tests for that method will need to change as well. That's one of the reasons that you do unit testing at such a low level of granularity, so only some of the unit tests have to change. If you find that tests for many different methods are changing for one change in a requirement, then you may not be testing at a fine enough level of granularity.
Unit tests are there so that your units (methods) do what you expect. Writing the test first forces you to think about what you expect before you write the code. Thinking before doing is always a good idea.
Unit tests should reflect the business rules. Granted, there can be errors in the code, but writing the test first allows you to write it from the perspective of the business rule before any code has been written. Writing the test afterwards, I think, is more likely to lead to the error you describe because you know how the code implements it and are tempted just to make sure that the implementation is correct -- not that the intent is correct.
Also, unit tests are only one form -- and the lowest, at that -- of tests that you should be writing. Integration tests and acceptance tests should also be written, the latter by the customer, if possible, to make sure that the system operates the way it is expected. If you find errors during this testing, go back and write unit tests (that fail) to test the change in functionality to make it work correctly, then change your code to make the test pass. Now you have regression tests that capture your bug fixes.
[EDIT]
Another thing that I have found with doing TDD. It almost forces good design by default. This is because highly coupled designs are nearly impossible to unit test in isolation. It doesn't take very long using TDD to figure out that using interfaces, inversion of control, and dependency injection -- all patterns that will improve your design and reduce coupling -- are really important for testable code.
How does one test a test? Mutation testing is a valuable technique that I have personally used to surprisingly good effect. Read the linked article for more details, and links to even more academic references, but in general it "tests your tests" by modifying your source code (changing "x += 1" to "x -= 1" for example) and then rerunning your tests, ensuring that at least one test fails. Any mutations that don't cause test failures are flagged for later investigation.
You'd be surprised at how you can have 100% line and branch coverage with a set of tests that look comprehensive, and yet you can fundamentally change or even comment out a line in your source without any of the tests complaining. Often this comes down to not testing with the right inputs to cover all boundary cases, sometimes it's more subtle, but in all cases I was impressed with how much came out of it.
When applying Test-Driven Development (TDD), one begins with a failing test. This step, that might seem unecessary, actually is here to verify the unit test is testing something. Indeed, if the test never fails, it brings no value and worse, leads to wrong confidence as you'll rely on a positive result that is not proving anything.
When following this process strictly, all ''units'' are protected by the safety net the unit tests are making, even the most mundane.
Assert.IsEqual(p.DiscountPrice,90);
There is no reason the test evolves in that direction - or I'm missing something in your reasoning. When the price is 100 and the discount 20, the discount price is 80. This is like an invariant.
Now imagine your software needs to support another kind of discount based on percentage, perhaps depending on the volume bought, your Product::DiscountPrice() method may become more complicated. And it is possible that introducing those changes breaks the simple discount rule we had initially. Then you'll see the value of this test which will detect the regression immediately.
Red - Green - Refactor - this is to remember the essence of the TDD process.
Red refers to JUnit red bar when a tests fails.
Green is the color of JUnit progress bar when all tests pass.
Refactor under green condition: remove any dupliation, improve readability.
Now to address your point about the "3-4 layers above the code", this is true in a traditional (waterfall-like) process, not when the development process is agile. And agile is the world where TDD is coming from ; TDD is the cornerstone of eXtreme Programming.
Agile is about direct communication rather than thrown-over-the-wall requirement documents.
While, I am all for unit testing, I
sometimes wonder if this form of test
first development is really beneficial...
Small, trivial tests like this can be the "canary in the coalmine" for your codebase, alerting of danger before it's too late. The trivial tests are useful to keep around because they help you get the interactions right.
For example, think about a trivial test put in place to probe how to use an API you're unfamiliar with. If that test has any relevance to what you're doing in the code that uses the API "for real" it's useful to keep that test around. When the API releases a new version and you need to upgrade. You now have your assumptions about how you expect the API to behave recorded in an executable format that you can use to catch regressions.
...[I]n a real process, you have 3-4
layers above your code (Business
Request, Requirements Document,
Architecture Document), where the
actual defined business rule (Discount
Price is Price - Discount) could be
misdefined. If that's the situation,
your unit test means nothing to you.
If you've been coding for years without writing tests it may not be immediately obvious to you that there is any value. But if you are of the mindset that the best way to work is "release early, release often" or "agile" in that you want the ability to deploy rapidly/continuously, then your test definitely means something. The only way to do this is by legitimizing every change you make to the code with a test. No matter how small the test, once you have a green test suite you're theoretically OK to deploy. See also "continuous production" and "perpetual beta."
You don't have to be "test first" to be of this mindset, either, but that generally is the most efficient way to get there. When you do TDD, you lock yourself into small two to three minute Red Green Refactor cycle. At no point are you not able to stop and leave and have a complete mess on your hands that will take an hour to debug and put back together.
Additionally, your unit test is another
point of failure...
A successful test is one that demonstrates a failure in the system. A failing test will alert you to an error in the logic of the test or in the logic of your system. The goal of your tests is to break your code or prove one scenario works.
If you're writing tests after the code, you run the risk of writing a test that is "bad" because in order to see that your test truly works, you need to see it both broken and working. When you're writing tests after the code, this means you have to "spring the trap" and introduce a bug into the code to see the test fail. Most developers are not only uneasy about this, but would argue it is a waste of time.
What do we gain here?
There is definitely a benefit to doing things this way. Michael Feathers defines "legacy code" as "untested code." When you take this approach, you legitimize every change you make to your codebase. It's more rigorous than not using tests, but when it comes to maintaining a large codebase, it pays for itself.
Speaking of Feathers, there are two great resources you should check out in regard to this:
Working Effectively with Legacy Code
Brownfield Application Development in .NET
Both of these explain how to work these types of practices and disciplines into projects that aren't "Greenfield." They provide techniques for writing tests around tightly coupled components, hard wired dependencies, and things that you don't necessarily have control over. It's all about finding "seams" and testing around those.
[I]f the discount price is wrong, the
test team will still find the issue,
how did unit testing save any work?
Habits like these are like an investment. Returns aren't immediate; they build up over time. The alternative to not testing is essentially taking on debt of not being able to catch regressions, introduce code without fear of integration errors, or drive design decisions. The beauty is you legitimize every change introduced into your codebase.
What am I missing here? Please teach
me to love TDD, as I'm having a hard
time accepting it as useful so far. I
want too, because I want to stay
progressive, but it just doesn't make
sense to me.
I look at it as a professional responsibility. It's an ideal to strive toward. But it is very hard to follow and tedious. If you care about it, and feel you shouldn't produce code that is not tested, you'll be able to find the will power to learn good testing habits. One thing that I do a lot now (as do others) is timebox myself an hour to write code without any tests at all, then have the discipline to throw it away. This may seem wasteful, but it's not really. It's not like that exercise cost a company physical materials. It helped me to understand the problem and how to write code in such a way that it is both of higher quality and testable.
My advice would ultimately be that if you really don't have a desire to be good at it, then don't do it at all. Poor tests that aren't maintained, don't perform well, etc. can be worse than not having any tests. It's hard to learn on your own, and you probably won't love it, but it is going to be next to impossible to learn if you don't have a desire to do it, or can't see enough value in it to warrant the time investment.
A couple people keep mentioned that
testing helps enforce the spec. It has
been my experience that the spec has
been wrong as well, more often than
not...
A developer's keyboard is where the rubber meets the road. If the spec is wrong and you don't raise the flag on it, then it's highly probable you'll get blamed for it. Or at least your code will. The discipline and rigor involved in testing is difficult to adhere to. It's not at all easy. It takes practice, a lot of learning and a lot of mistakes. But eventually it does pay off. On a fast-paced, quickly changing project, it's the only way you can sleep at night, no matter if it slows you down.
Another thing to think about here is that techniques that are fundamentally the same as testing have been proven to work in the past: "clean room" and "design by contract" both tend to produce the same types of "meta"-code constructs that tests do, and enforce those at different points. None of these techniques are silver bullets, and rigor is going to cost you ultimately in the scope of features you can deliver in terms of time to market. But that's not what it's about. It's about being able to maintain what you do deliver. And that's very important for most projects.
Unit testing works very similar to double entry book keeping. You state the same thing (business rule) in two quite different ways (as programmed rules in your production code, and as simple, representative examples in your tests). It's very unlikely that you make the same mistake in both, so if they both agree with each other, it's rather unlikely that you got it wrong.
How is testing going to be worth the effort? In my experience in at least four ways, at least when doing test driven development:
it helps you come up with a well decoupled design. You can only unit test code that is well decoupled;
it helps you determine when you are done. Having to specify the needed behavior in tests helps to not build functionality that you don't actually need, and determine when the functionality is complete;
it gives you a safety net for refactorings, which makes the code much more amenable to changes; and
it saves you a lot of debugging time, which is horribly costly (I've heard estimates that traditionally, developers spend up to 80% of their time debugging).
Most unit tests, test assumptions. In this case, the discount price should be the price minus the discount. If your assumptions are wrong I bet your code is also wrong. And if you make a silly mistake, the test will fail and you will correct it.
If the rules change, the test will fail and that is a good thing. So you have to change the test too in this case.
As a general rule, if a test fails right away (and you don't use test first design), either the test or the code is wrong (or both if you are having a bad day). You use common sense (and possilby the specs) to correct the offending code and rerun the test.
Like Jason said, testing is security. And yes, sometimes they introduce extra work because of faulty tests. But most of the time they are huge time savers. (And you have the perfect opportunity to punish the guy who breaks the test (we are talking rubber chicken)).
Test everything you can. Even trivial mistakes, like forgetting to convert meters to feet can have very expensive side effects. Write a test, write the code for it to check, get it to pass, move on. Who knows at some point in the future, someone may change the discount code. A test can detect the problem.
I see unit tests and production code as having a symbiotic relationship. Simply put: one tests the other. And both test the developer.
Remember that the cost of fixing defects increases (exponentially) as the defects live through the development cycle. Yes, the testing team might catch the defect, but it will (usually) take more work to isolate and fix the defect from that point than if a unit test had failed, and it will be easier to introduce other defects while fixing it if you don't have unit tests to run.
That's usually easier to see with something more than a trivial example ... and with trivial examples, well, if you somehow mess up the unit test, the person reviewing it will catch the error in the test or the error in the code, or both. (They are being reviewed, right?) As tvanfosson points out, unit testing is just one part of an SQA plan.
In a sense, unit tests are insurance. They're no guarantee that you'll catch every defect, and it may seem at times like you're spending a lot of resources on them, but when they do catch defects that you can fix, you'll be spending a lot less than if you'd had no tests at all and had to fix all defects downstream.
I see your point, but it's clearly overstated.
Your argument is basically: Tests introduce failure. Therefore tests are bad/waste of time.
While that may be true in some cases, it's hardly the majority.
TDD assumes: More Tests = Less Failure.
Tests are more likely to catch points of failure than introduce them.
Even more automation can help here !
Yes, writing unit tests can be a lot of work, so use some tools to help you out.
Have a look at something like Pex, from Microsoft, if you're using .Net
It will automatically create suites of unit tests for you by examining your code. It will come up with tests which give good coverage, trying to cover all paths through your code.
Of course, just by looking at your code it can't know what you were actually trying to do, so it doesn't know if it's correct or not. But, it will generate interesting tests cases for you, and you can then examine them and see if it is behaving as you expect.
If you then go further and write parameterized unit tests (you can think of these as contracts, really) it will generate specific tests cases from these, and this time it can know if something's wrong, because your assertions in your tests will fail.
I've thought a bit about a good way to respond to this question, and would like to draw a parallel to the scientific method. IMO, you could rephrase this question, "How do you experiment an experiment?"
Experiments verify empirical assumptions (hypotheses) about the physical universe. Unit tests will test assumptions about the state or behavior of the code they call. We can talk about the validity of an experiment, but that's because we know, through numerous other experiments, that something doesn't fit. It doesn't have both convergent validity and empirical evidence. We don't design a new experiment to test or verify the validity of an experiment, but we may design a completely new experiment.
So like experiments, we don't describe the validity of a unit test based on whether or not it passes a unit test itself. Along with other unit tests, it describes the assumptions we make about the system it is testing. Also, like experiments, we try to remove as much complexity as we can from what we are testing. "As simple as possible, but no simpler."
Unlike experiments, we have a trick up our sleeve to verify our tests are valid other than just convergent validity. We can cleverly introduce a bug we know should be caught by the test, and see if the test does indeed fail. (If only we could do that in the real world, we'd depend much less on this convergent validity thing!) A more efficient way to do this is watch your test fail before implementing it (the red step in Red, Green, Refactor).
You need to use the correct paradigm when writing tests.
Start by first writing your tests.
Make sure they fail to start off with.
Get them to pass.
Code review before you checkin your code (make sure the tests are reviewed.)
You cant always be sure but they improve overall tests.
Even if you do not test your code, it will surely be tested in production by your users. Users are very creative in trying to crash your soft and finding even non-critical errors.
Fixing bugs in production is much more costly than resolving issues in development phase.
As a side-effect, you will lose income because of an exodus of customers. You can count on 11 lost or not gained customers for 1 angry customer.