Are there any guidelines when quoting estimates for projects/tasks involving TDD?
For example, when compared to normal development of a task taking 1 day to complete, how much more should a TDD driven task take? 50% more time or 70% more time? Are there any statistics available, assuming the developer is well versed with the language and the test framework?
To my experience, writing tests takes as much time and sometimes even more time than development (i.e. add from 100% to 150% more time) but drastically reduce the time spent for bug fixing. Here are some Studies Of Test Driven Development. Also have a look at this paper.
The difference is high at first, decreases with experience, but is probably always a factor
The difference in implementation time between conventional coding and TDD will decline as the developer becomes better at TDD. TDD beginners, and even intermediates, are likely to get caught-up deciding what tests to write and/or will write more tests that end up being thrown out after refactoring. With experience, a TDD'er will become more efficient, as they become better and quicker at choosing what tests to write
I'm not sure what the absolute lower limit of the ratio of conventional to TDD is. I'd guess 1:1.5, but I can't believe that most developers can ever test-drive code as fast as they could just write code, much less write code then write tests.
And as others have said, a significant payoff for that extra time spent doing TDD is that debugging time is vastly reduced for test-driven code.
Coding time should be about the same as testless development. Debugging time should be reduced by approximately 99%.
Another payoff when producing code with TDD is that the tests written become a regression suite. The tests then make these activities much more safe:
Refactoring after the fact.
Further development of the same code.
Bug fixing.
(NB wrt Bug Fixing, I remember a couple of times where either I'd forgotten to implement a couple of cases or there were late additions to the spec, the additional development was a doddle with tests in place.)
If you want a mathematical answer, my finding has been has been time spent in test : production is 2:1 (conservative - majority of time spent in test is THINK time). However you can't apply a 3X factor to your current (non-TDD) estimates because TDD helps you make steady progress.. you don't spend time
coding up things you don't need
running in circles of break and fix. OR breaking existing features.
a distinct bug-find and fix phase at the end
Near-zero attach the debugger and step thru sessions
Off the top of my head, I'd go for a 2X on your existing estimates.
Related
I'm programming from home and I want to know whether I'm more or less productive programming at 10 AM than I am when I'm programming at 8PM.
What metrics should I use to determine an answer to the question?
Ignoring the debate in the question's comments, a bunch of arbitrary productivity-ish metrics you could measure...
lines of code written
user stories/tasks completed
bugs fixed
tests written
tests passing first time
bugs found
code churn vs new code (i.e. "right first time" vs "rewritten repeatedly")
%age of time in IDE vs debugging
%age of time in IDE vs non-work applications
code quality (using another similarly arbitrary measure like FxCop compliance or cyclic complexity)
code performance (against some arbitrary or customer-specified benchmark)
The best metrics tend to be combinations - say, "average of bugs found per line of code written" - rather than a single measure. Still, these are all subjective and innacurate.
I'd suggest the best thing to do is decide what your goal is when you're programming. Is it to produce high-quality code, or super-performant realtime code, or mission-critical-must-be-bug-free code, or do you just need to ship something that works in the shortest time? Until you've defined "productive", it's hard to suggest what would be a meaningful measurement.
I don't know if there is some established method for measuring productivity in programmers but assuming alertness and focus has a direct impact on productivity, I suppose you could set yourself some kind of mental arithmetic test with randomised questions and answers and take it at regular intervals.
It's a tricky one because you can't measure by lines, or problems solved (because they vary in scale and difficulty.) In fact, this article suggests that when attempting to measure programmer productivity, there is almost no correllation between the time it takes to complete a task and the quality of the finished product.
I am currently studying test driven development. I am inquiring about an improvement in design if we refactor after the development of every couple of units as opposed to dedicated refactoring sessions that are more spaced apart in time.
I am aware that the technical debt will be larger. But I am wondering what other impacts. Maybe the refactoring process is not as effective when there is a larger time interval because...?
Thank you..
For the base definition of TDD, you write a failing test, write the minimum amount of code that will make the test pass, refactor, repeat. There isn't a lot of room there for deferring the refactoring. ;)
I think this is so for a lot of reasons, mainly so that people like my boss won't be correct in assuming that the word "refactor" is a geek-speek code word for rewrite. How much easier is it to write one method, that say grabs some info from the web. You write your test, get it to pass, then say, "Ok, now take this hard coded URL and move it to the top of the class, or a property file." How much harder is that to do once you've got your class completed and weighing in a several hundred or more lines of code.
Where the design portion comes in, is less in the big "D"esign, at least in my understanding and use of TDD, rather its in the good practices it encourages and/or demands. Going back to my use-case, ok, I've got my method written to grab my data, now I need to do something with it, do I go back and start adding code to my getData method? No, of course not, its "done.". I go on and write an new method or three to do something with the data. All the methods are kept short, on task, testable, in short, a much better design for the code.
I think that's where the confusion comes in. TDD produces better designed CODE, which will probably produce better software, but if the overall SYSTEM design is no good, then the best coding practices in the universe aren't going to create a good product/system.
YMMV of course.
A beneficial collateral effect of applying TDD is that your design tends to be thinner and sounder. So major refactorings become unnecessary. In my opinion, this is the effect of designing the system by tests, from "top" to "down" (where "top" are the external interfaces, constraints and requirements, and "down" are the internal implementations and contracts). Sometimes I'm even confused by TDD meaning "test driven design" instead of "development".
Thus, redesigns are driven by changes in constraints and requirements, not by technical debt, which is mitigated at the innermost interfaces. The scope of the changes will determine the extent of the refactoring -- if this is true, "major" refactorings are expected only when the topmost interfaces change.
You mentioned planned refactorings as a step in a longer term development cycle. I think this is a fragile strategy, not because we shouldn't clean technical debt -- we should do it as soon as possible, and that's a practice at the core of TDD --, but because it's risky. To illustrate: in order to plan your refactoring step, how would you assess the size of the technical debt? What if the estimated refactoring time is not enough? Will the system be healthy if you cut the refactoring to do other important things? These are important questions, and not the only ones regarding technical debt management. From a risk management perspective, TDD means better control.
I have found that short iterations combined with TDD/continuous integration works best.
Long iterations have the disadvantages of :
1) You lose track of what you are working on
2) Its hard to estimate what you can do in 2months vs say 1 week
3) It gives the business holders less time to re prioritize the development tasks.
Short cycles are easier: "I have to get a and b done this week". You stay focused. You have shorter meetings. Your TDD and continuous integration keep you informed as to the state of your code.
wrt your refactoring question, i think you should probably be refactoring as you go. You don't do things the wrong way for a few weeks until you get around to refactoring. With your tests in place, you will quickly be able to see where your refactoring breaks things.
edit -- another disadvantage to longer development cycles is that people get complacent: "eh, i have 4 more weeks ill do all this stuff later..."
following the text book, I do measure performance whenever I try optimizing my code. Sometimes, however, the performance gain is rather small and I can't decisively decide whether I should implement that optimization.
For example, when a fix shortens an average response time of 100ms to 90ms under some conditions, should I implement that fix? What if it shortens 200ms to 190ms? How many condition should I try before I can conclude that it will be beneficial overall?
I guess it's not possible to give a straight forward answer to this, as it depends on too many things, but is there a good rule of thumb that I should follow? Are there any guideline/best-practices?
EDIT:Thanks for the great answers! I guess the moral of the story is, there is no easy way to tell whether you should, but there ARE guidelines that can aid that process.. Things you should consider, things you shouldn't do etc. This particular time I ended up implementing the fix, even though it made a few line of code into 20-30 lines of code. Because our app. is very performance critical, and it was a consistent 10% gain in various realistic cases.
I think the rule of thumb (at least for me) is two-fold:
"It matters if it matters"--in the business world, this generally means that it matters if the clients care. That is, if the end users will "notice" the difference between 100ms and 90ms (I'm not being facetious here), then it matters.
If "it matters," then you will want to test your code thoroughly against a realistic variety of use cases that are likely to arise or at least may arise. If an optimization speeds up code in 50% of cases, but actually runs slower than what you previously had the other 50% of the time, obviously, it may not be worth implementing.
Regarding point 1 above: by suggesting an end user of your software might "notice" a 10ms difference, I don't mean to suggest that they will actually visibly see a difference. But if your app runs on a server with millions of connections and every little speed increase takes a substantial load off the server, that might matter to the client running the server. Or if your app performs extremely time-critical work, this is another case where the result of a 10ms speedup might be noticeable, even if the speedup itself isn't.
The only sensible approach to your question is something along the lines of "when the benefit is large enough to warrant the time you invest in exploring, implementing and testing the optimization."
The "benefit is large enough" is extremely subjective. Can you or your employer sell more units of software if you make this change? Will your user base notice? Will it give you personal gratification to have the fastest-possible code? Which of those or similar questions apply is something only you can know.
By and large, most of the software I have written (in a 20+ year career) has been "fast enough" out of the box, and the code I cared to optimize presented itself as an obvious bottleneck to the end users: Queries taking a long time, scrolling too slow, that sort of thing.
Donald Knuth made the following two statements on optimization:
"We should forget about small
efficiencies, say about 97% of the
time: premature optimization is the
root of all evil" [2]
and
"In established engineering
disciplines a 12 % improvement, easily
obtained, is never considered marginal
and I believe the same viewpoint
should prevail in software
engineering"[5]
src: http://en.wikipedia.org/wiki/Program_optimization
Is the optimization obfuscating your code too much?
Do you really need an optimization? if your app just runs fine then readability of the code is probably more important
Did you work on the general design and algorithms of your application before trying small hacky optimizations?
You should focus optimisation efforts on the parts of code that account for the most runtime. If a particular piece of code takes up 80% of the total runtime, then optimising it to reduce the time is takes by 5% will have as much impact as reducing the time of the rest of the code by 20%.
In general, optimisations make code less readable (not always, but often). Therefore you should avoid optimising until you are sure that there is a problem.
If it speeds up your program at all, why not implement it? You have already done the work by creating the new implementation, so you are not doing extra work by applying the new implementation.
Unless the code is THAT much harder to understand.
Also, 100 ms to 90 ms is a 10% gain in performance. A 10% gain should not be taken lightly.
The real question is, if it only took 100 ms to run in the first place, what was the point in trying to optimize it?
As long as it's fast enough, then you don't need to optimise any more. But then, you wouldn't even bother profiling if that was the case...
If the performance gain is small, consider the other factors: maintainability, risk of making the change, understandability, etc. If it reduces the ability to maintain or understand the code, it probably isn't worth doing. If it improves those attributes, then it's more reason to implement the change.
In most cases, your time is more valuable than the computer's. If you think it'll take you half an hour longer to work out what the code is doing later (say if there's a bug in it), and it's only saved you a few seconds, ever, you're at a net loss.
It depends very much on the usage scenario. I'll assume here that the code in question has been profiled and thus it is known to be the bottleneck--i.e. not just "this could be faster", but "the program would give results/finish running faster if this were faster". In situations where this is not the case--e.g. if you spend 99% of your time waiting for more data to come over an ethernet connection--then you should care about correctness but not optimize for speed.
If you are writing a piece of user interface code, what you care about is perceived speed. Generally anything under ~100 ms is perceived as "instant"--no point speeding it up.
If you are writing a piece of code for a giant server farm, then if the cost of your salary to make the code fast is less than the cost of the extra electricity for the server farm, it's worthwhile. (But be sure to prioritize your time.)
If you are writing a piece of code that is used rarely or when unattended, as long as it completes in a semi-sane duration, don't worry about it. Install scripts tend to be of this sort (unless you start running into many minutes, at which point users might start abandoning the install because it's taking too long).
If you are writing code to automate a task for someone else, then if (your time spent coding + their time spent using the optimized code) is less than (their time spent using the slow code), it's worthwhile. If you're doing this in a commercial setting, weight this by your respective salaries.
If you are writing library code that will be used by many thousands of people, always make it faster if you have time to.
If you are under time pressure to simply have something working e.g. as a demo, don't optimize (except through sensible choice of algorithms from libraries) unless the result would be so slow that it isn't even "working".
One of the biggest annoyances for me personally is finding software which perhaps initially fell into one category and then later fell into another, but for which nobody went back to do needed optimizations. Until recently Javascript performance was a great example of this. Moral of the story is: don't just decide once; revisit the issue as the situation demands.
Having read Fowler's "Refactoring" for a while, I still often catch myself thinking "I should have done this in smaller steps." -- even when I did not broke my code.
Refactoring in small steps is safe, but cost time. It's a trade off between speed and risk -- I try to be strategic in choosing the way how I am refactoring.
Nevertheless: Most the time I am doing refactorings in larger steps. If I took some of Fowler's "Mechanics" section and compare how I am working, I maybe find that I often leap two or five steps forward at once. This does not mean that I am a refactoring guru. My code maybe stay for 5 - 60 minutes broken or uncompilable.
Do you refactor in smaller steps and try to produce unbroken code in shorter frequencies? And: Are you successful in doing this?
Martin Fowler seems to lean towards the small, gradual refactoring approach. However, after reading his book he does occasionally make some drastic steps but only with unit tests to back up the code.
Refactoring is a controlled technique for improving the design of an existing code base. Its essence is applying a series of small behavior-preserving transformations, each of which "too small to be worth doing". However the cumulative effect of each of these transformations is quite significant. By doing them in small steps you reduce the risk of introducing errors. You also avoid having the system broken while you are carrying out the restructuring - which allows you to gradually refactor a system over an extended period of time. - Martin Fowler
I try :) The one urge I have to resist most while refactoring is actually making other changes along the way. Say I'm refactoring some code and see something unrelated in the code. I have to make a conscious effort not to go "fix" that as well. Make a note of it and move on. For one thing, it's a distraction from the task at hand. It also ends up polluting your change set so your commit message now has to document several seemingly random changes.
Yes, always. I think the real essence of refactoring is picking which steps to start with.
I find the thing with refactoring large changes in a safe manner is always to have a reasonably clear picture of where you want to go. Then consider your existing system and try to find out which pieces you can introduce that have least likelyhood of being a radical change. Then you can introduce these in a controlled and well tested manner.
So what you do is to work in the vincinity of the nastiness. Not always attacking directly from the front, but sometimes just chipping away small pieces. Usually I wait, and only go for the "big prize" after a few rounds of chipping away at minor nastiness. But I know where I want to go.
The nice thing about working this way is that you can maintain progress. You never "stop development to do refactoring". Arguably there are cases where stopping is the correct situation, but most of the time it's not.
The idea here is that if you "start" with cashing in the prize money, you will be spending the next X days doing the drudgery. And there's risk, maybe you chicken out or it doesn't work - or spend 6 months instead of a week. If you do the drudgery first, cashing in the prize will be possible with less risk. And your code will improve as you go. Sometimes you can decide that doing half the job was enough, since your understanding of the problem increases. Sometimes your idea of where you wanted to go was slightly botched, and you can realign your goal as you progress.
But its tempting to go straight for the reward.
I tend to refactor in large steps most of the time so I can see the forest from the trees. It's a "stream of consciousness" kind of programming. As long as you have your last working version safe in your source control of choice...
that's where the "red, green, refactor" approach is useful. At each stage you have the ability to verify that your code's behaviour is unchanged, and the refactoring only has to integrate the new behaviour.
The rule of thumb I use is refactor with tests and only refactor as much code as you are confident too.
At 60 minutes are you certain that your code is doing exactly what it should be. You'd need a lot of tests to pass. I would just try and get one going and then move on to the next.
If I have a clear picture of what I want to do, and if I can easily verify that I haven't broken anything afterwards, I am taking larger steps.
If the refactoring is more complicated, I try to break it down into smaller steps and do heavy testing after each one.
I usually refactor code as I change it. That is, instead of taking a piece of code and rewriting it while maintaining its function, I rewrite it towards a new functionality and in the process of doing so I improve the design of the code.
Often this means that by the time I've implemented the feature I was after I haven't done a complete and satisfactory refactoring of the old code. It is improved though, and I know I'll have the time to improve it further the next time I'm about to change its function.
For testing this means that I get to test both the refactoring and the new feature at the same time, which should save some time.
It also mean that I only spend enough time on refactoring to improve the maintenance situation required for that particular feature. This should help to avoid over engineering and/or wasting time refactoring stuff that already works and wont benefit from a better desing. By focusing only on code I would change anyway, there is also a high probability I will revisit that code in the near time to do further changes while it's in the users attention span.
Small discrete steps is what I'm most comfortable with, though at some points it can be a test of my self-control to reign in what could be a refactoring blood-bath. If I notice any improvements (no-matter how large) that could be made, I make a note of them and consider how it'd be split up into individual refactoring tasks. Plus, having a saga of changes in the commit message doesn't help.
NB. The code-base I work on is quite old, and full of those mystical bugs named after scientists. With large portions still lacking anything near even 50% test coverage it would be careless to get carried away.
Yep. I like to run the tests continually and so a chain of tiny refactors works well. I get really uncomfortable having my code broken for more than a few minutes at a time, and I generally revert if my code is broken when I go home at night, the re-write the next morning ALWAYS works better than trying to pick up where I was.
Reading over the responses to this question Disadvantages of Test Driven Development? I got the impression there is alot of misunderstanding on what TDD is and how it should be conducted. It may prove useful to address these issues here.
I feel the accepted answer was one of the weakest (Disadvantages of Test Driven Development?), and the most up-modded answer smells of someone who might be writing over specified tests.
Big time investment: for the simple
case you lose about 20% of the actual
implementation, but for complicated
cases you lose much more.
TDD is an investment. I've found that once I was fully into TDD, the time I lost is very very little, and what time I did lose was more than made up when it came to maintence time.
For complex cases your test cases are
harder to calculate, I'd suggest in
cases like that to try and use
automatic reference code that will run
in parallel in the debug version /
test run, instead of the unit test of
simplest cases.
If your test are becoming very complex, it might be time to review your design. TDD should lead you down the path smaller, less complex units of code working together
Sometimes you the design is not clear at the start and evolves as you go along - this will force you to redo your test which will generate a big time lose. I would suggest postponing unit tests in this case until you have some grasp of the design in mind.
This is the worst point of them all! TDD should really be "Test Driven Design". TDD is about design, not testing. To fully realise the value of benefits of TDD, you have toy drive your design from your tests. So you should be redoing your production code to make your tests pass, not the other way round as this point suggests
Now the currently most upmodded: Disadvantages of Test Driven Development?
When you get to the point where you have a large number of tests, changing the system might require re-writing some or all of your tests, depending on which ones got invalidated by the changes. This could turn a relatively quick modification into a very time-consuming one.
Like the accepted answers first point, this seems like over specification in the tests and a general lack of understanding of the TDD process. When making changes, start from your test. Change the test for what the new code should do, and make the change. If that change breaks other tests, then your tests are doing what their supposed to do, failing. Unit Tests, for me, are designed to fail, hence why the RED stage is first, and should never be missed.
IMHO The biggest misconception about TDD is that: time spent writing and refactoring tests would be time lost. The thinking goes like "yeah, a test suite is nice, but the feature would be complete much faster if we just coded it".
When done properly, time spend writing and maintaining tests is saved multiple times over the life of the project in time not spent debugging and fixing regressions. Since the testing cost is up-front and the payoff is over time, it is easy to overlook.
Other big misconceptions include ignoring the impact of TDD on the design process, and not realizing that "painful tests" is a serious code smell that needs fixing quickly.
I see a lot of people misunderstanding what tests actually are usefull to TDD. People write big acceptance tests instead of small unit tests and then spend far too much time maintaining their tests and then conclude that TDD doesn't work. I think the BDD people have a point in avoiding the use of the word test entirely.
The other extreme is that people stop doing acceptance testing and think that because they do unit testing their code is tested. This is again a misunderstanding of the function of a unit test. You still need acceptance tests of some sort.
The misconception that I often see is that TDD ensures good results.
Often times tests are written off of flawed requirements, and therefore, the developers produce a product that does not do what the user is expecting. Key to TDD is, in my opinion, working with the users to define requirements while helping manage their expectations.
These are the issues that in my opinion are quite controversial and hence prone to misunderstanding:
In my experience the biggest advantage is producing far better code at the cost of a lot of time spent writing tests. So it's really worthwhile for projects that require high quality, but on some other, less quality centric sites, the extra time will not be worth the effort.
People seem to think that only a major subset of the features must be tested, but that is actually wrong IMHO. You need to test everything in order for your test to be valid after refactoring.
The big drawback of TDD is the false sense of security given by incomplete tests: I've seen sites go down because people assumed that Unit Testing was enough to trigger a deployment.
There is no need of mocking frameworks to do TDD. It's just a tool for testing some cases in an easier way. The best unit tests though are fired high in the stack and should be agnostic on the layers in the code. Testing one layer at a time is meaningless in this context.
Just chucking another answer in the pot.
One of the most common misunderstandings is that your code is fixed, ie. I have this code, now how on earth will I test it? If it's hard to write a test, we should ask the question: how can I change this code to make it easier to test?
Why..?
Well The sort of code that's easy to test is:
Modular - each method does one thing.
Parameterised - each method accepts everything it needs and outputs everything it should.
Well Specified - each method does exactly what it should, no more, no less.
If we write code like this, testing is a doddle. The interesting thing is that code that is easy to test is, coincidentally, better code.
Better as in easier to read, easier to test, easier to understand, easier to debug. This is why TDD is often described as a design exercise.