Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
Our team is having a heated debate as to whether we allow failing unit tests to be checked-in to source control.
On one side the argument is that yes you can as long as it is temporary - to be resolved within the current sprint. Some say even that in the case of bugs that may not be corrected within the current sprint we can check-in a corresponding failing test.
The other side of the argument is that those tests, if they are checked-in must be marked with the Ignore attribute - the reasoning being that the nightly build should not serve as a TODO list for a developer.
The problem with Ignore attribute however is that we tend to forget about the tests.
Does the community have any advice for us ?
We are a team of 8 developers, with a nightly build. Personally I am trying to practice TDD but the team tends to write unit tests after the code is written
I'd say not only should you not check in new failing tests, you should disable your "10 or so long-term failing tests". Better to fix them, of course, but between having a failing build every night and having every included test passing (with some excluded) - you're better off green. As things stand now, when you change something that causes a new failure in your existing suite of tests, you're pretty likely to miss it.
Disable the failing tests and enter a ticket for each of them; that way you'll get to it. You'll feel a lot better about your automated build system, too.
I discussed this with a friend and our first comment was a recent geek&poke :) (attached)
To be more specific - all tests should be written before (sa long as it's supposed to be TDD) but those checking an unimplemented functionality should have their value prepended with negation. If it's not implemented - it shouldn't work. [If it works - the test is bad] After implementing a test you remove the ! and it works [or fails, but then it's there to do so :) ].
You shouldn't think that tests are something written once and always right. Tests can have bugs too. So editing a test should be normal.
I'd say that checking in (known) failing tests should of course be only temporary, if at all. If the build is always failing, it loses its meaning (we've been there and that's not pretty).
I guess it would be ok to check in failing tests if you found a bug and could reproduce it quickly with a test, but the offending code is not "yours" and you don't have the time/responsibility to get into it enough to fix the code. Then give a ticket to someone who knows his way around and point to the test.
Otherwise I'd say use your ticket system as a TODO list, not the build.
It depends how you use tests. In my group, running tests is what you do before a commit in order to check that you (likely) have not broken anything.
When you are about to commit, it is painful to find failed tests that seem vaguely possibly related to your changes but still strange, investigate for a couple of hours, then realize it cannot possibly be because of your changes, do a clean checkout, compile, and find that indeed the test failures come from the trunk.
Obviously you do not use tests in the same way, otherwise you wouldn't even be asking.
If you were using a DVCS (e.g., git) this wouldn't be an issue as you'd commit the failing test to your local repository, make it work, and then push the whole lot (test and fix) back to the team's master repository. Job done, everyone happy.
As it seems you can't do that, the best you can do is to first make sure that the test is written and fails in your sandbox, and then fix that before committing. This might or might not be great TDD, but it's a reasonable compromise with the working practices of everyone else; working with your co-workers is more important than following some ivory tower principle in every aspect, since the author of the principle isn't located in the cubicle next door…
The purpose of your nightly build is to give you confidence that the code written the day before hasn't broken anything. If tests are often failing then you can't have that confidence.
You should first fix any failing tests you can and delete or comment out/ignore the other failing ones. Every nightly build should be green. If its not then there is a problem and that's immediately obvious now since it should have run green.
Secondly, you should never check in failing tests. You should never knowingly break the build. Ever. It causes unnecessary noise, distractions and lowers confidence. It also creates an atmosphere of laziness around quality and discipline. With respect to ignored tests which are checked in, these should be caught and addressed in your code reviews, which should be covering you test code as well.
If people want to write their code first and tests later, this is OK (though I prefer TDD), but only tested code which runs green should be checked in.
Finally, I would recommend changing the nightly build to a continuous integration build (run on each code check in) which might just change people's habits around code check-in.
I can see that you have a number of problems.
1) You are writing failing tests.
This is great! However, someone is intending to check those in "to be fixed within the current sprint". This tells me that it's taking too long to make those unit tests pass. Either the tests are covering more than one or two aspects of behaviour, or your underlying code is far too complex. Refactor the code to break it up and use mocks to separate the responsibilities of the class under test from its collaborators.
2) You tend to forget about tests with [Ignore] attributes.
If you're delivering code that people care about, either it works, or it has bugs which require the behaviour of the system to be changed. If your unit tests describe that behaviour but the behaviour doesn't work yet, then you won't forget because someone will have registered a bug. However, see point 1).
3) Your team is writing tests after the code is written.
This is fairly common for teams learning TDD. They might find it easier if they thought of the tests not as tests to check if their code is broken, but examples of how another developer might want to use their code, with a description of the value that their code provides. Perhaps you could pair with them and help them learn from what they already know about writing tests afterwards?
4) You're trying to practice TDD.
Do or do not. There is no try. Write a test first, or don't. Learning never stops even when you think you're doing TDD well.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
While attempting to adopt more TDD practices lately on a project I've run into to a situation regarding tests that cover future requirements which has me curious about how others are solving this problem.
Say for example I'm developing an application called SuperUberReporting and the current release is 1.4. As I'm developing features which are to be included in SuperUberReporting 1.5 I write a test for a new file export feature that will allow exporting report results to a CSV file. While writing that test it occurs to me that the feature to support exports to some other formats are slated for later versions 1.6, 1.7, and 1.9 which are documented in a issue tracking software. Now the question that I'm faced with is whether I should write up tests for these other formats or should I wait until I actually implement those features? This question hits at something a bit more fundamental about TDD which I would like to ask more broadly.
Can/should tests be written up front as soon as requirements are known or should the degree of stability of the requirements somehow determine whether a test should be written or not?
More generally, how far in advance should tests be written? Is it OK to write a test that will fail for two years until the that feature is slated to be implemented? If so then how would one organize their tests to separate tests that are required to pass versus those that are not yet required to pass? I'm currently using NUnit for a .NET project so I don't mind specifics since they may better demonstrate how to accomplish such organization.
If you're doing TDD properly, you will have a continuous integration server (something like Cruise Control or TeamCity or TFS) that builds your code and runs all your tests every time you check in. If any tests fail, the build fails.
So no, you don't go writing tests in advance. You write tests for what you're working on today, and you check in when they pass.
Failing tests are noise. If you have failing tests that you know fail, it will be much harder for you to notice that another (legitimate) failure has snuck in. If you strive to always have all your tests pass, then even one failing test is a big warning sign -- it tells you it's time to drop everything and fix that bug. But if you always say "oh, it's fine, we always have a few hundred failing tests", then when real bugs slip in, you don't notice. You're negating the primary benefit of having tests.
Besides, it's silly to write tests now for something you won't work on for years. You're delaying the stuff you should be working on now, and you're wasting work if those future features get cut.
I don't have a lot of experience with TDD (just started recently), but I think while practicing TDD, tests and actual code go together. Remember Red-Green-Refactor. So I would write just enough tests to cover my current functionality. Writing tests upfront for future requirements might not be a good idea.
Maybe someone with more experience can provide a better perspective.
Tests for future functionality can exist (I have BDD specs for things I'll implement later), but should either (a) not be run, or (b) run as non-error "pending" tests.
The system isn't expected to make them pass (yet): they're not valid tests, and should not stand as a valid indication of system functionality.
Could anyone explain the word regression test in an understandable way?
Regression test is a test that is performed to make sure that previously working functionality still works, after changes elsewhere in the system. Wikipedia article is pretty good at explaining what it is.
Your unit tests are automatically regression tests, and that's one of their biggest advantages. Once those tests are written, they will be run in future, whenever you add new functionality or change existing functionality. You don't need to explicitly write regression tests.
Notwithstanding the old joke, "Congress" is not the opposite of "progress;" "regress" is. For your code to regress is for it to "move backward," typically meaning that some bad behavior it once had, which you fixed, has come back. A "regression" is the return of a bug (although there can be other interpretations). A regression test, therefore, is a test that validates that you have fixed the bug, and one that you run periodically to ensure that your fix is still in place, still working.
The word regression as coined by Francis Galton means
Regression: The act of going back
I.e. it is the phenomenon/technique in software testing to check any change / bug fixes hasn't impacted the existing functionality of the system. Thus the intent of regression testing is to ensure that a change, such as a bug fix should not result in another fault being uncovered in the application.
Regression Testing is required when
there is a change in requirements and code is modified according to the requirement
a new feature is added to the software
defects are fixed
a performance issue is fixed
Regression testing can be done both manually and automated.
These are some tools for the automation approach:
QTP
AdventNet QEngine
Regression Tester
vTest
Watir
Selenium
actiWate
Rational Functional Tester
SilkTest
During a regression test, testers run through your application testing features that were known to work in the previous build.
They look specifically for parts of the application that may not have been directly modified, but depend on (and could have residual bugs from) code that was modified.
Those bugs (ones caused by bugs in dependent code even though they were working before) are known as regressions (because the feature was working properly and now has a bug...and therefore, regressed).
Regression testing is a part of testing activity, which can be start after modification has been made to check the reliability of each software released.It's nothing but an impact analysis to check wheather it not affecting critical area of the software.
Do unit test
Do integration test
After (1) and (2) are passed, do regression test
In simple term, regression test is to repeat step (1) and (2) again.
Regression testing basically perform after completing of retesting.
The main purpose of regression testing is to check the impact of modification. Whether still our application is acting stable.
Its necessary to perform a regression testing because sometimes it happened after retesting or while fixing bug developer fixed the bug and missed out something on other code or on dependent code
http://en.wikipedia.org/wiki/Regression_testing
Basically, test the code you've updated to make sure you haven't introduced new bugs and that the functionality still works as before.
Regression test:- IF THERE ANY Changes, delate,modification, up dings or adding in my application . In that case I have to know that my application works as it was working before.
Regression test - Is a type of SW testing where we try to cover or check around the bug Original bug Fix.
The functionality around the bug fix should not get changed or altered due to the Fix provided. Issues found in such process are called as Regression Issues.
In a simple way, Regression test is a test to make sure that the functionality of a system still works after a new code change has been introduced. It doesn't really have to be a thorough testing of the whole functionality (such as functional testing), only the areas that are considered to be impacted by the introduced code changes.
Regression test is a test which enables us to find introduced bug by testing some areas in the software that we are testing. Introduced bug means a bug which is caused by the new changes made by the developer.
The key in the regression test is how we can effectively do the test by wisely deciding some areas which might be impacted by the changes since we can't test all the functionalities due to the time constraint (most of the time). 'Effective' in here means we can find bugs in a relatively short period of time.
Regression testing means testing your software/website repeatedly. The main reason for it is to make sure there aren't any new bugs introduced.
Typically, regression tests will be automated, to reduce the cost of rerunning the test. The more high value test cases you can construct, the better. This is one example of a Play and Record regression testing platform
Definition: - Regression testing is defined as a type of software testing to confirm that a recent program or code change has not harmfully affected existing features.
Regression Testing is Re-Testing to make sure that any modification done in a program will not affect the other functionality.
Regression testing is nothing but a full or partial selection of already executed test cases which are re-executed to ensure existing functionalities work fine.
We can do Regression Testing at all the level of testing like Unit Testing, Integration Testing and System Testing Level.
Need of Regression Testing
Common code changed correctly or not.
Correct or incorrect version control.
Bug fixes perfectly.
Bug fixes completely.
Performance issue fix.
6.Change in requirements and code is modified according to the requirement.
The new feature is added to the software perfectly.
For More Visit LINK
I like this definition of regression testing:
[regression testing] tells you if a previously written and tested code broke after you’ve added an update or a fix
[...] it helps you notice if you’ve unknowingly introduced bugs to your software while adding new code. New bugs of this kind are called regressions.
Basically, a regression is returning to a state where your application has bugs.
Regression testing is an activity performed to ensure the different functionalities of the system are still working as expected and the new functionalities added did not break any of the existing ones.
Secondly, you generally write automated tests or do manual testing to the above mentioned testing. It could be a combination of Unit/API/UI tests that are run on a daily basis. Regression testing can be performed in various phases of the SDLC, it all depends on the context.
Hopefully this gives an idea on what is regression testing.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
When I started building a continuous integration server, I ran across the statement "It's bad to break the build [of the code]." After finishing that project I came to the conclusion that
"Breaking the build." was a catchy phrase that was being thrown around a lot because of the alliteration, or
I wasn't understanding a key element of Continuous Integration.
So my question is in the spirit of #2: why is breaking the build a bad thing?
Be very careful in labeling "Breaking the Build" as a bad thing. It is something that needs immediate attention, but it is also a very normal and expected part of the development cycle. This is why Continuous Integration is so useful -- it tells you immediately when the build is broken, and what change set caused it. It helps you get back on track quickly.
If your culture penalizes "Breaking the Build", then you are in danger of cultivating a toxic work environment. Again, consider it to be something that needs immediate attention, but don't label it as "bad".
Because if other people check out your broken changes, they won't be able to work, or if they do they will do so less efficiently.
It also means you're not properly testing your changes before you commit, which is key in CI.
From Martin Fowler http://martinfowler.com/articles/continuousIntegration.html
The whole point of working with CI is
that you're always developing on a
known stable base. It's not a bad
thing for the mainline build to break,
although if it's happening all the
time it suggests people aren't being
careful enough about updating and
building locally before a commit. When
the mainline build does break,
however, it's important that it gets
fixed fast.
Because if other people checkout the changes, they won´t be able to work...
This image is copyrighted to Geek & Poke under a Creative Commons License
It you break the build as has happend to me yesterday. When your team-mates try and use the sourcecode. It will not build. Therfore they will struggle to test the work that they are doing. It get worse the bigger your team.
Surely the whole point of continuous integration is to identify problems early. Daily or more frequent check ins are required to reduce conflicts to a manageable size.
You should get an up to date copy of the repository and build locally. This will tell you if your proposed check in will break the build. You resolve any issues and then check in.
In this way the integration issues are kept local and easy to fix.
Breaking the build has dire implications for the project schedule (and the blood pressure of team-mates)
=> Other developers who then get latest version can no longer build there own changes, delaying them
=> Continuous integration will break, meaning that formal testing can be delayed
Many version control tools (e.g. TFS) can prevent developers from checking in code which does not compile or pass unit or code analysis tests.
Once builds start breaking, people get reluctant to get the latest changes, and you begin the deadly spiral towards Big Bang integration of changes.
I don't think breaking the build is necessarily a bad thing, as long as there is a well-known, working branch or tag in the repository. That said, make your own branch in the repository if you know your code is going the break the build today, but you will fix it next week. Then later you can merge back into trunk.
Because it means someone has done something bad (or at least, some changes have clashed) and you can no longer build and deploy your system.
Breaking the build means that you committed code to a shared repository that either (a) does not compile, or (b) does not work (fails unit tests). Anyone else who's developing from that shared repository is going to have to deal with the broken code you committed until it is fixed. That will cause a loss of productivity for the entire team.
You have some code you want to remove associated with an obsolete piece of functionality from a ruby project. How do ensure that you get rid of all of the code?
Some guidelines that usually help in refactoring ruby apply, but there are added challenges because having code that isn't being called by anything won't break any unit tests.
Update: Has anyone written anything that allows you to guess based on your version control history if there are commits where you have since deleted most, but not all, of the code and can point out the remaining code?
Current thoughts:
Identify the outermost part of the stack associated with the obsolete functionality: the binary script calling it, or the unit tests calling it.
Look for methods that are only called by methods associated with the obsolete functionality. I often use git grep for this.
In theory, running mutation testing and looking for code that used to be mutation resistant when the old test suite applied, but is now mutation prone might help. It only helps if your code was well-tested in the first place! (Or you can use code coverage tools such as rcov rather than mutation testing)
Running test suites will ensure you haven't removed anything you shouldn't have!
Using autotest can save you time if you're constantly running tests.
If your code was well-structured, it should be easier to find related methods that need to be removed.
Especially in a dynamically typed language, there is no easy way to do this. If you have unittests, thank the developer that wrote them because it will help you remove the code correctly. But you're basically SOL. Remove the code, if it breaks, put it back, figure out where it broke, attempt to work around it, and repeat.
Look at your code coverage. Any code which isn't covered may be part of the code you have left to remove (if any). (Just be sure you have removed you tests. =])
I know there's a lot of stuff on TDD and i'm trying to pick up the practice too.
But i wonder is it a good idea to to TDD your bug fix too?
I was thinking along the lines of find the bug and narrow it down.
Write unit test to ensure that it will now pass whatever problem that it previously caused.
Write more unit test for other breakable conditions.
And finally write unit test to test for Integration test, as we don't have any unit test prior to this, so whenever i'm fixing a bug i'm always worried that i might accidentally break something.
So is TDD suitable for debugging
too?
Or is
there other method or resource/book
that would be more useful for this
purpose?
I am more concerned for the
Integration Test part as i've
mentioned above. I'm looking for
anything in LAMP/Axkit/Perl related
method.
Thanks.
Write a test that fails because of
the bug.
Fix the bug in your code.
Re-run test.
Re-run all tests.
So - yes - very much so.
When a tester finds a bug I usually write a unit test for it. When the test succeeds, I know the bug is fixed and it will always be covered again in the future.
When you have a bug in your system it is good practice in TDD to write a test that spots the bug (i.e. a red test that proves the bug). When you have to fix the bug, the test should eventually turn green. It may be a good idea to ferret out any other bugs if they're close enough.
Regarding debugging, TDD should be used to leverage debugging sessions away from the programmer. You can still debug, if you have no idea where some bug is, but it's easier to pinpoint a bug if you have a regression test suite with enough granularity.
You have to remember though that TDD is more about unit testing and not about integration testing. There is nothing wrong with writing integration tests since they're a sanity check to make sure your application or system under test works.
One book that is about testing with xUnit frameworks is the xUnit Patterns book that is general enough to work with any unit testing framework (even for PerlUnit I would guess) a cookbook of interesting tricks you can do with xUnit frameworks.
UPDATE:
There is a chapter about unit testing in perl at Extreme Perl.
The answer in short is yes. The basic structure of doing that is to write a test case which would simulate the bug and fail the test case. Then fix the bug which would pass the test case.
Yes. Of course all the tests you performed during TDD of your release will have been added to a regression test suite. But, in the case of a bug, that regression suite obviously wasn't detailed enough.
The first step in fixing a bug is replicating it and this is TDD. Once you find a test case that replicates the bug, you can expand on it if you wish (to catch other obvious problems of the same class), but I don't tend to do a lot of expansion since we have specific turnaround times for fixing a single bug.
Once you have a fix for that bug, add the test case to the regression suite. The idea is to keep adding test cases to the regression suite for both releases and bug fixes to give it very good coverage.
I always wrote tests before the actual code while fixing a bug.
This way I had the example of what to expect from a working code - and I could just focus on making this test (and all others, for regression) pass.
Yes but beware, If you write as many bugs as I do you will soon have a huge set of tests to cover all the bugs you have written and then fixed.
This will mean tests runs will be slower, and intent of behaviour will become muddied by your bug tests.
You could keep these tests logically separate or revisit your original set of specified behaviour checks (read tests) to see if you really have covered all your expected behaviour.
I think its important to differentiate between the two.
I think you'd fix the bug initially and then the regression tests would be composed of TDD.