Im looking into a BDD solution like SpecFlow and going through various examples and I see references to other TDD frameworks like MsTest and NUnit which Im familiar with. I understand the value of what Specflow and BDD is providing. I read somewhere that Specflow and BDD "wraps" your unit tests. So, with Specflow, does the "step definitions" serve the same purpose as something like MsTest or Nunit would, and these other frameworks are just options to use instead of the Step definitions?
If you go back to the origins of BDD, you'll find out that the first ever tool - JBehave - was originally a replacement for JUnit. Annotations weren't a thing at the time, so JUnit used to look for things starting with the word "test":
testDatabase
Of course, that's pretty meaningless. So JBehave did the same thing, but starting with the word "should":
shouldPersistRecordsUsingHibernate
So now we can see what our class should do, and talk about its behaviour using plain English. "Behaviour" is a more useful word than "test", here.
After a while, and a few conversations between Dan North and Chris Matts, they worked out that what Dan was doing with classes was the same thing that Chris (an analyst at the time) did with whole systems, and so the scenario-running* component was born. Dan ported JBehave to RBehave, which became RSpec's scenario runner, which became SpecFlow and Cucumber and all the rest of them. And of course, JUnit these days uses annotations, so we can start things with the word "should" anyway, and NUnit's pretty much JUnit for .NET.
The point is that you can describe any level of code with BDD. I use BDD for my class-level examples as well as my system-level scenarios.
For the class-level examples, the only audience is technical. It's enough to write comments about how the class behaves in something like NUnit or JUnit. Some frameworks like MSpec or RSpec have other ways of capturing those descriptions, but the audience is still technical. It's still BDD. With class-level examples we mock out dependencies so that we're only looking at one aspect of behaviour at a time. This helps drive good design, same as TDD.
For system-level examples, though, there's another audience; the non-technical stakeholders. Most of them can understand readable code just fine, so you can make a DSL and still use NUnit here too. However, there's benefit to being able to capture natural language a little more directly, without having to worry about making it run. That's where SpecFlow and the like come into play.
The overhead of parsing natural language and matching it to steps isn't small. It wouldn't be worth it at a unit / class level, where behaviours are separated by the Single Responsibility Principle, encapsulation, and other aspects of good design. Natural languages aren't as easy to refactor as code, either.
So we save the natural language scenarios for the outermost layers, where we only want a few examples of how a system behaves. We then end up with a few examples of end-to-end behaviour, more integration / module / library-level scenarios, and still more unit tests (or examples). This is commonly referred to as the "testing pyramid".
For instance, if we want to validate a form, it's enough to have a couple of examples of how our application helps users fill in the form when they make a mistake. We might have a couple of integration tests that check that we validate at both the client and server-side (if that's important to us, for instance). And then we'll have a bunch of unit tests around the actual validation classes, covering all the different possibilities.
To answer your question in shorthand:
SpecFlow, Cucumber, JBehave and other natural-language frameworks are generally used only for system-level scenarios, often automated against the UI.
You don't have to use the frameworks for those; it's possible to do it using a DSL in NUnit (or JUnit, etc.).
Using natural-language framworks at a class level is overkill. Use NUnit here.
BDDers often have one layer of scenarios running at the system level, and a ton of class-level examples (unit tests) sitting beneath them. This is the "wrapping" that you're talking about.
There may be intermediate layers too.
*The words "scenario" and "example" are pretty much synonyms, meaning much the same thing. Traditionally we've used "scenario" to refer only to end-to-end examples, with "example" or "unit test" being used at a class level.
The single most important thing about BDD, though, is to make sure you talk through those aspects of behaviour with someone, even if it's in retrospect or with a rubber duck. BDD is much more about the conversation than it is about the tools.
To be short "step definitions" don't serve the same purpose as something like MsTest or Nunit. Actually, I could write a lot of, but there will be too many text. Let me show some simple example:
Here is a simple feature file
Feature: BingSearchUi
Scenario: Search in Bing
Given I open page 'http://www.bing.com'
And I have entered 'visual studio' into the bing search field
When I press search button on the bing page
Then the result should contain 'www.visualstudio.com' on the bing page
It's valid specflow scenario, but by itself it can't do anything. To make it work you gotta write code and code is not only uses Nunit or MsTest. They are could be part of steps implementation, but not nessecary.
E.g. Here is Given I open page 'http://www.bing.com' implementation, which atually uses Selenium, not any test frameworks:
[Given(#"I open page '(.*)'")]
public void GivenIOpenPage(string ulr)
{
driver.Navigate().GoToUrl(ulr);
}
But to make things more complicated Specflow generates .cs file with steps and if I use NUnit this file is NUnit test class. It's small part of file:
[NUnit.Framework.TestAttribute()]
[NUnit.Framework.DescriptionAttribute("Search in Bing")]
[NUnit.Framework.CategoryAttribute("Selenium")]
public virtual void SearchInBing()
{
TechTalk.SpecFlow.ScenarioInfo scenarioInfo = new TechTalk.SpecFlow.ScenarioInfo("Search in Bing", new string[] {
"Selenium"});
#line 15
this.ScenarioSetup(scenarioInfo);
#line 16
testRunner.Given("I open page \'http://www.bing.com\'", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Given ");
#line 17
testRunner.And("I have entered \'visual studio\' into the bing search field", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "And ");
#line 18
testRunner.When("I press search button on the bing page", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "When ");
#line 19
testRunner.Then("the result should contain \'www.visualstudio.com\' on the bing page", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Then ");
#line hidden
this.ScenarioCleanup();
}
and this generated file actually is called by test runner.
I think The simplest way to se how it works altogether - find any public repository on github with specflow in use and play with it.
Related
I'm using VS 2012, but that's not really important.
What is important is that I'm trying to do some TDD by writing all my tests first and then creating the code.
However, the app will not compile because none of my objects or methods exist.
Now, to my mind, I should be able to create ALL my tests but still run my app so I can debug etc. The tests shouldn't prevent compilation because objects and methods are missing.
I thought the whole point of it was that as you develop your tests you can begin to see duplications etc so that you can refactor before writing a single line of code.
So the question is, is there a way to do this or am I doing this wrong?
EDIT
I am using VS2012 and C#
I see a small problem with
writing all my tests first and then creating the code.
You don't need to write ALL your tests first, you just need one, make it fail, make it pass and repeat. That means ideally at any point you should have ideally one failing test.
A compile failure counts as a failed test in that sense. So the next step is to make it pass - i.e. add stubs or return default values to make it compile. The test would then be red.. then work at getting it to green.
Test Driven Development is about very small iterations. You don't define all your tests up front. You create one test based on one fraction of one requirement. Then you implement the code to pass that test. Once it's passing, you work on another fraction of a requirement.
The idea is that trying to do all the design up front (whether it be creating detailed class diagrams or creating a bunch of tests) means that you will find it too expensive to change a weakness in your design, so you won't improve your code.
Here's an example. Let's say you decide to use inheritance to relate two objects, but when you started implementing the objects, you found that made testing them tough. You discover it would be much easier to test each object individually, and relate them via containment instead. What is happening is the tests are driving your design in a more loosely coupled direction. This is a very good outcome of TDD - you are using tests to improve the design.
If you had written all your tests in advance assuming your design decision of inheritance was a good choice, you would either throw away a lot of work, or you would say "it's too tough to make a change like that now, so I'll just live with this sub-optimal design instead."
You can certainly create business-rule-related acceptance tests in advance. Those are called behavior tests (part of Behavior Driven Development, or BDD) and they are good to test features of the software from the user's point of view. But those are NOT unit tests. Unit tests are for testing code from the developer's point of view. Creating the unit tests in advance defeats the purpose of TDD, because it will make testing harder, it will prevent you from improving your code, and will often lead to rebellion and failure of the practice. That's why it's important to do it right.
What is important is that I'm trying to do some TDD by writing all my tests first and then creating the code.
The problem is that "writing all my tests first" is most emphatically not "do[ing] some TDD". Test driven development consists of lots of small repetitions of the "red-green-refactor" cycle:
Add a unit test to the test suite, run it and watch it fail (red)
Add just enough code to the system under test to make all the tests
pass (green)
Improve the design of the system under test (typically
by removing duplication) while keeping all the tests passing
(refactor)
If you code an entire huge test suite up front, you'll spend forever trying to get to the "green" (all tests passing) state.
However, the app will not compile because none of my objects or methods exist.
That's typical of any compiled language; it's not a TDD issue per se. All it means is that, in order to watch the new test fail, you may have to write a minimal stub for whatever feature you're currently working on to make the compiler happy.
For example, I might write this test (using NUnit):
[Test]
public void DefaultGreetingIsHelloWorld()
{
WorldGreeter target = new WorldGreeter();
string expected = "Hello, world!";
string actual = target.Greet();
Assert.AreEqual(expected, actual);
}
And I'd have to then write this much code to get the app to compile and the test to fail:
public class WorldGreeter
{
public string Greet()
{
return String.Empty;
}
}
Once I've gotten the solution to build and I've seen the one failing test, I can add the code to make the first test pass:
public string Greet()
{
return "Hello, world!";
}
Once all tests pass, I can look through the system under test and see what could be done to improve the design. However, it's essential to the TDD discipline to go through both the "red" and "green" steps before playing around with refactoring.
I thought the whole point of it was that as you develop your tests you can begin to see duplications etc so that you can refactor before writing a single line of code.
Martin Fowler defines refactoring as "a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior" (emphasis added). If you haven't written a single line of code, there's nothing to refactor.
So the question is, is there a way to do this or am I doing this wrong?
If you're looking to do TDD, then, yes, I fear you are doing this wrong. You may well be able to deliver great code doing what you're doing, but it isn't TDD; whether or not that's a problem is for you to decide for yourself.
You should be able to create your empty class with stub functions, no?
class Whatever {
char *foo( const char *name ) {}
int can_wibble( Bongo *myBongo ) {}
}
Then you can compile.
No. It's about coding just enough to verify the implementation of the required use cases
You can define your tests cases early, but to code the test cases them you iteratively write a test, have it fail. Then write some code that ensures that the code passes.
Then rinse and repeat until all your test cases are covered,
Edit to address comment.
As you build out the code, that's where your programming designs and faults are identified. Extreme programming lends it self to you being able to change code with out care as the test base protects your requirements. Your intentions are good but the reality is that you'll refactor/redesign test test cases as you discover design issues and flaws through building out the code and test base.
However IMHO, in a very general case, a test that doesn't compile is effectively a meta test that's failing that needs to be corrected before moving on. It's telling you to write some code!
Use mock, from Wikipiedia: mock objects are simulated objects that mimic the behavior of real objects in controlled ways. A programmer typically creates a mock object to test the behavior of some other object, in much the same way that a car designer uses a crash test dummy to simulate the dynamic behavior of a human in vehicle impacts.
Please refer this.
I found this using dynamic for objects that don't exists yet:
https://coderwall.com/p/0uzmdw
Say a Chef can make Recipes, and Sous-Chefs can create Recipes that must be approved by a Head Chef.
You want to test that, when a Head Chef views her homepage, she sees Recipes that she herself created. You also want to test that she sees there are Recipes awaiting her approval.
I can think of two ways to do this:
Test that the view contains certain words, like "Your recipes" and "Recipes awaiting your approval"
Add unnecessary attributes to the html elements you're using so that you can check for an element with "id=recipe_1" or "data-for-the-sake-of-testing=1"
I very much dislike both of these approaches.
Why approach #1 sucks
Incredibly brittle tests. Every time you want to make minor updates to the copy, tests will break.
i18n? How will that work with this approach?
There are probably more reasons, but those two are pretty huge.
Why approach #2 sucks
How annoying to have superfluous markup just for the sake of testing! The user should not have an increased download size for the sake of tests.
What is a good approach to this? I'm interested to hear any alternatives at all, in whatever language you think in. I mostly think in Ruby, Test::Unit, Minitest, RSpec, and Cucumber (though my Cuke skills are stale), but if other languages/frameworks have this figured out, I'd love to see what they're doing, too.
Use a page paradigm.
Phrase the steps in as human a way as you can, at the level of capabilities (high level) wherever possible, and use specific examples. For instance, if I'm using Cucumber I might say:
Given the sous-chef has created a recipe for Frog Pie
When the chef looks for recipes to approve
Then the recipe for Frog Pie should be in the list.
Inside the code for these steps, instantiate or find the particular page you're looking for, where the page is an object that represents the capabilities of the page. That page can then have all the things that the user can do with the page - look for recipes, approve recipes, move to another page, etc.
This way, if you need to change the underlying code for the step, you only have to change it in one place, and all the changes for a particular page will be together. Because you've phrased the scenario in terms of capabilities you're delivering, it's unlikely that the scenario will need to change much (unless you discover that your business need different capabilities to the ones you're delivering).
This also works pretty well for window-based apps too, with each widget or module being a particular page.
It's also fine to have extra ids just for testing. Sometimes designers like to use them too.
I see at least two options:
Avoid testing business logic through the UI. Write a "service" or "use case controller" object that returns plain data structures. In other words, you build an API to your system. Your unit test accesses the system through the API. Your UI accesses the system through the same API, but then there should be almost no logic in the view. See http://www.confreaks.com/videos/759-rubymidwest2011-keynote-architecture-the-lost-years or http://www.cleancoders.com/codecast/clean-code-episode-7/show.
Use the "page object" pattern. Write an object that reads in the HTML code that your app produces, parses it, and makes interesting data available through getters. This will do wonders to make your test code clear. Your objection could be that you still have problem #2. In fact I don't think it's really a problem. If you use structural HTML markup, it should be fairly easy to extract the information you need. It will be much easier if you attach an ID to a key element of the page; in your example I would have a div with id="my-recipes" and another div with id="to-be-approved". That should be enough; anything else should be easy to find with xpath or css selectors. Why do you find this objectionable? These IDs will probably be useful for other purposes, such as attaching behaviour with unobtrusive JavaScript or attaching styles with a CSS stylesheet.
Live with #2, perhaps using brief comments (no i18n issues and not visble to the end user):
<!-- APPROVAL -->
The documentation of simpletest has a nice take on it:
Next chance you get, look at a circuit board, perhaps the motherboard
of the computer you are looking at right now. On most boards you will
find the odd empty hole, or solder joint with nothing attached or
perhaps a pin or socket that has no obvious function. Chances are that
some of these are for expansion and variations, but most of the
remainder will be for testing.
If a small amount of superfluous markup makes your product more testable and reliable then just live with it!
I personally try to do not test views at all. I mean generated markup, since those tests appearing to be much fragile.
Instead, I focus on "data-provider" side, in case of MVC web framework is Controller. As soon as controller is covered by unit tests that check what kind of data controller prepares, you are pretty much safe. The view you create is easy to test by just running the application and see that it looks OK.
Nethertheless, there some approaches of view testing. First one is based on "end-to-end" testing simulation with Selenium Driver. It runs the browsers and initialize requests to you application. Tests are checking the output HTML. Tests logon to "known" edition, it means tests know that current localization is EN, for instance.
You should basically combine the approaches, where it works use HTML markup values ("Recipies") otherwise use HTML elements id's or classes. I would not add any additional markup for the testing.
Another approach you can try is Approval Testing. I believe there is a Ruby driver for that - http://approvaltests.sourceforge.net/. With approvals you render the view and save the HTML as golden master. The test will fail in case of View had changed. It much more easier to implement than Selenium tests.
Read a lot of materials regarding the behavior-driven testing. Indeed, as it is stated (for example) here, the specs' code should be extremely readable, short and straightforward.
Now I have a question: suppose there are two kinds of team players: one is the developer, the other is a tester (QA). What would be the best test plan if not describe and it stubs, where all the its are marked as pending? So, the QA might want to create such a test plan and pass it to the developer for the proper implementation.
After a few days, the tester (or product manager) wants to have a look on the tests. But... there is a hell of a code in there.
Hence, the question: is there a way to separate between the behavior specification and implementation in RSpec? I can fabricate a plenty of methods, but before inventing the wheel, I would rather prefer to stick to some common, proven and well-known practice.
as according to The RSpec Book RSpec is what developers manage for them internally, but Cucumber is what production manager and business can see. Think about RSpec as like internal tool to code coverage to see the objects internal behaviour: models, controllers, views. Other team members can see overall pictures that Cucumbers covers as well
Since I'm a TDD newbie, I'm currently developing a tiny C# console application in order to practice (because practice makes perfect, right?). I started by making a simple sketchup of how the application could be organized (class-wise) and started developing all domain classes that I could identify, one by one (test first, of course).
In the end, the classes have to be integrated together in order to make the application runnable, i.e. placing necessary code in the Main method which calls the necessary logic. However, I don't see how I can do this last integration step in a "test first" manner.
I suppose I wouldn't be having these issues had I used a "top down" approach. The question is: how would I do that? Should I have started by testing the Main() method?
If anyone could give me some pointers, it will be much appreciated.
"Top-down" is already used in computing to describe an analysis technique. I suggest using the term "outside-in" instead.
Outside-in is a term from BDD, in which we recognise that there are often multiple user interfaces to a system. Users can be other systems as well as people. A BDD approach is similar to a TDD approach; it might help you so I will describe it briefly.
In BDD we start with a scenario - usually a simple example of a user using the system. Conversations around the scenarios help us work out what the system should really do. We write a user interface, and we can automate the scenarios against that user interface if we want.
When we write the user interface, we keep it as thin as possible. The user interface will use another class - a controller, viewmodel, etc. - for which we can define an API.
At this stage, the API will be either an empty class or a (programme) interface. Now we can write examples of how the user interface might use this controller, and show how the controller delivers value.
The examples also show the scope of the controller's responsibility, and how it delegates its responsibilities to other classes like repositories, services, etc. We can express that delegation using mocks. We then write that class to make the examples (unit tests) work. We write just enough examples to make our system-level scenarios pass.
I find it's common to rework mocked examples as the interfaces of the mocks are only guessed at first, then emerge more fully as the class is written. That will help us to define the next layer of interfaces or APIs, for which we describe more examples, and so on until no more mocks are needed and the first scenario passes.
As we describe more scenarios, we create different behaviour in the classes and we can refactor to remove duplication where different scenarios and user interfaces require similar behaviour.
By doing this in an outside-in fashion we get as much information as to what the APIs should be as possible, and rework those APIs as quickly as we can. This fits with the principles of Real Options (never commit early unless you know why). We don't create anything we don't use, and the APIs themselves are designed for usability - rather than for ease of writing. The code tends to be written in a more natural style using domain language more than programming language, making it more readable. Since code is read about 10x more than it's written, this also helps make it maintainable.
For that reason, I'd use an outside-in approach, rather than the intelligent guesswork of bottom-up. My experience is that it results in simpler, more strongly decoupled, more readable and more maintainable code.
If you moved stuff out of main(), could you not test that function?
It makes sense to do that, since you might want to run with different cmd-args and you'd want to test those.
You can test your console application as well. I found it quite hard, look at this example:
[TestMethod]
public void ValidateConsoleOutput()
{
using (StringWriter sw = new StringWriter())
{
Console.SetOut(sw);
ConsoleUser cu = new ConsoleUser();
cu.DoWork();
string expected = string.Format("Ploeh{0}", Environment.NewLine);
Assert.AreEqual<string>(expected, sw.ToString());
}
}
You can look at the full post here.
"bottom up" can save you a lot of work cos you already have low-level classes (tested) and you can use them in higher level tests. in opposite case you'd have to write a lot of (probably complicated) mockups. also "top down" often doesn't work as it's supposed: you think out some nice high-level model, test and implement it, and then moving down you realize that some of your assumptions were wrong (it's quite typical situation even for experienced programmers). of course patterns help here but it's not silver bullet too.
to have complete test coverage, you'll need to test your Main in any case. I'm not sure it's worth the effort in all cases. Just move all logic from Main to separate function (as Marcus Lindblom proposed), test it and let Main to just pass command-line arguments to your function.
console output is a simplest possible testing ability, and if you use TDD it can be used just to output the final result w/o any diagnosis and debug messages. so make your top-level function to return solid result, test it and just output it in Main
I recommend "top down" (I call that high level testing). I wrote about that:
http://www.hardcoded.net/articles/high-level-testing.htm
So yes, you should test your console output directly. Sure, initially it's a pain to setup a test so that it's easy to test console output directly, but if you create appropriate helper code, your next "high level" tests will be much easier to write. By doing that, you empower yourself to unlimited re-factoring potential. With "bottom up", your initial class relationship is very rigid because changing the relationship means changing the tests (lots of work and dangerous).
There was an interesting piece in MSDN magazine of December, which describes "how the BDD cycle wraps the traditional Test-Driven Development (TDD) cycle with feature-level tests that drive unit-level implementation". The details may be too technology specific, but the ideas and process outline sound relevant to your question.
The main should be very simple in the end. I don't know how it looks like in c#, but in c++ it should look something like this :
#include "something"
int main( int argc, char *argv[])
{
return TaskClass::Run( argc, argv );
}
Pass already created objects to the constructors of classes, but in unit tests pass mock objects.
For more info about TDD, take a look at these screencasts. They explain how to do agile development, but it also talks how to do TDD, with examples in c#.
I am just getting into the concept of BDD and have listened to Scott Bellware's talk with the Herding Code guys. I have been playing around with SpecFlow some and like it pretty well.
I understand the distinction between ATDD and TDD as described in the blog post Classifying BDD Tools (Unit-Test-Driven vs. Acceptance Test Driven) and a bit of BDD history, but that leads me to a question.
As described, isn't using a BDD tool (such as MSpec) just another unit testing framework? It seems to me that it is.
Furthermore, this seems to suggest that using SpecFlow to spec out the lower level components (such as your repositories and services) would be wrong. If I can use the same tool for both ATDD and TDD of lower level components, why shouldn't I? There seems to still be some blurry lines here that I feel like I'm not quite understanding.
The Quick Answer
One very important point to bring up is that there are two flavors of Behavior Driven Development. The two flavors are xBehave and xSpec.
xBehave BDD: SpecFlow
SpecFlow (very similar to cucumber from the Ruby stack) is excellent in facilitating xBehave BDD tests as Acceptance Criteria. It does not however provide a good way to write behavioral tests on a unit level. There are a few other xBehave testing frameworks, but SpecFlow has gotten a lot of traction.
xSpec BDD: NSpec
For behavior driven development on a unit level, I would recommend NSpec (inspired directly by RSpec for Ruby). You can accomplish BDD on a unit level by simply using NUnit or MSTest...but they kinda fall short (it's really hard to build up contexts incrementally). MSpec is also an option, there has been a lot of work put into it, but there are just somethings that are just simpilier in NSpec (you can build up context incrementally in MSpec, but it requires inheritance which can become complex).
The Long Answer
The two flavors of BDD primarily exist because of the orthogonal benefits they provide.
Pros and Cons of xBehave (GWT Syntax)
Pros
helps facilitate a conversations with the business through a common dialect called (eg. Given ...., And Given ...., When ......, And When ..... , Then ...., And Then)
the common dialect can then be mapped to executable code which proves to the business that you actually finished what you said you'd finish
the dialect is constricting, so the business has to disambiguate requirements and make it fit into the sentences.
Cons
While the xBehave approach is good for driving high level Acceptance Criteria, the cycles needed to map English to executable code via attributes makes it infeasible for driving out a domain at the unit level.
Mapping the common dialect to tests is PAINFUL (ramp up on your regex). Each sentence the business creates must be mapped to an executable method via attributes.
The common dialect must be tightly controlled so that managing the mapping doesn't get out of hand. Any time you change a sentence, you have to find method that directly relates to that sentence and fix the regex matching.
Pros and Cons of xSpec (Context/Specification)
Pros
Allows the developer to build up context incrementally. A context can be set up for a test and some assertions can be performed against that context. You can then specify more context (building upon the context that already exists) and then specify more tests.
No constricting language. Developers can be more expressive about how a certain part of a system behaves.
No mapping needed between English and a common dialect (because there isn't one).
Cons
Not as approachable by the business. Let's face it, the business don't like to disambiguate what they want. If we gave them a context based approach to BDD then the sentence would just read "Just make it work".
Everything is in the code. The context documentation is intertwined within the code (that's why we don't have to worry about mapping english to code)
Not as readable given a less restrictive verbiage.
Samples
The Bowling Kata is a pretty good example.
SpecFlow Sample
Here is what the specification would look like in SpecFlow (again, this is great as an acceptance test, because it communicates directly with the business):
Feature File
The feature file is the common dialect for the test.
Feature: Score Calculation
In order to know my performance
As a player
I want the system to calculate my total score
Scenario: Gutter game
Given a new bowling game
When all of my balls are landing in the gutter
Then my total score should be 0
Step Definition File
The step definition file is the actual execution of the test, this file includes the mappings for SpecFlow
[Binding]
public class BowlingSteps
{
private Game _game;
[Given(#"a new bowling game")]
public void GivenANewBowlingGame()
{
_game = new Game();
}
[When(#"all of my balls are landing in the gutter")]
public void WhenAllOfMyBallsAreLandingInTheGutter()
{
_game.Frames = "00000000000000000000";
}
[Then(#"my total score should be (\d+)")]
public void ThenMyTotalScoreShouldBe(int score)
{
Assert.AreEqual(0, _game.Score);
}
}
NSpec Sample, xSpec, Context/Specification
Here is a NSpec example of the same bowling kata:
class describe_BowlingGame : nspec
{
Game game;
void before_each()
{
game = new Game();
}
void when_all_my_balls_land_in_the_gutter()
{
before = () =>
{
game.Frames = "00000000000000000000";
};
it["should have a score of 0"] = () => game.Score.should_be(0);
}
}
So Yea...SpecFlow is cool, NSpec is cool
As you do more and more BDD, you'll find that both the xBehave and xSpec flavors of BDD are needed. xBehave is more suited for Acceptance Tests, xSpec is more suited for unit tests and domain driven design.
Relevant Links
RSpec vs Cucumber (RSpec stories)
BDD with Cucumber and rspec - when is this redundant?
NSpec Project Site
Continuous Testing
Introduction to BDD and Mocking
BDD using NUnit and Moq
A true behavior driven tool would be something like Cucumber. We use it at my job against .NET code. This allows us to write features that define behavior of the system as a whole and we can then execute the features and verify that the system does what we expect. The whole process works very well for us.
http://cukes.info/
There is a .net implementation called NStep that connects to cucumber via the wire protocol, it allows you to write step definitions in C# using lambdas...its pretty awesome.
Step definitions look like this:
When("^I go to the \"([^\"]*)\" (?:[Ss]creen|[Pp]age)$", (string pageName) =>
{
var screen = ParseScreen(pageName);
GoToScreen(screen);
World.Browser.Wait(1000);
});
http://github.com/clearwavebuild/nStep
I think your understanding is in line with mine. BDD is more suited for integration testing and generally tests your system as the end user, eg:
Given I am an authorised user
When I go to the front page
Then there should be a link to my profile with my username as the link text.
There is no reason to not to unit test your repositories at a more granular level. I think both are useful and appropriate.
Can't I just use normal unit testing tools?
BDD is a process and mentality and so, yes, you can do it with any tools (or not, you can write your own without a tool if you want). However, the TDD tools had certain assumptions which cause some friction when trying to do things in a BDD way. For instance, TDD assumes you are testing an architectural unit of the software; class, module, service. Whereas BDD assumes you are specifying some functional portion of the system.
Should I use SpecFlow/Cucumber to describe lower-level components?
First of all, I think the question is a bit misguided. You wouldn't tend to describe components unless those components directly represent behavior. I'll still answer what I believe the spirit of the question is.
Story oriented tools like Cucumber are great for talking about behavior from a customer/user perspective. It can allow you to make specifications that are easily approachable by laymen. However, it can be tedious to describe large amounts or complex state with those tools.
Unit testing, or more code oriented specification tools like rSpec and Machine.Specification, can be a lot more convenient when dealing with complex or large state setups. You can use the various tools available to the languages to manage the state. Things like inheritance and fakes/mocks. Machine.Specification has some good approaches to this for the .NET minded.
So, should you use Cucumber to specify lower-level behavior? I'd say only if its important to have high levels of visibility for that particular behavior. On my current project, we've developed an architectural component to represent certain business-rule intensive portions of the system. Those components are specified with Cucumber, but the majority of system is covered with NUnit.
Btw, SpecFlow is really nice and approachable for .NET folks just getting into BDD, but eventually you'll want to graduate to full-blown Cucumber+nStep. The Cucumber ecosystem is HUGE and helpful. SpecFlow's is much smaller.
Also, the lambda syntax offered by nStep is quite a bit nicer than having to decorate methods a la SpecFlow or Cuke4Nuke.
Disclaimer/Background:
I did some of the original development on nStep but I'm using SpecFlow on my current project. I'm working to introduce BDD here and needed something simple and approachable.
It is interesting that this blog on Classifying BDD Tools talks about TDD and ATDD. As Liz Keogh points out: BDD is about conversation and exploration. The easier it is for all involved guys to contribute, communicate intension, share ideas, understand the others, etc. the faster we get to an adequate solution and the better software we build. When we finally follow the tool path, what are the tools that support collaboration between stakeholders of software projects best?
Based on the this blog on the differences between TDD, BDD, and ATDD I would say that there are at least three different flavors of BDD tool:
Unit Testing Frameworks
JUnit changed our view on development and testing. One of its strengths is that tests can be written in the same programming language as the application itself. Thus, we can build on the knowledge we already have in the delivery team. When the tests are even used to drive the development we reach the heaven of TDD.
Programming languages have been optimized to reduce redundancy, which is according to Ron Jeffries one of the worst sins of developers. Therefore, we often rely on these tools when we do technical testing to build the product right as they help us to be most efficient.
Several guys tried to make automated tests more understandable, as unit tests aren't really readable. One of the first attempts was to parse unit tests and provide a concise summary that is also readable to non-developers. For example TestDox / AgileDox creates simple documentation from the method names of JUnit test classes or Pickels generates documentation based on feature files written in Gherkin.
Frameworks such as MSpec help to write tests that are better readable and additionally provide readable output. These flavor of BDD tools focuses on human readable output, which enables the involvement of non-developers after the developers did their job.
Scenario Testing Frameworks
To involve stakeholders earlier in the development cycle, new tools were created that focus more on readable input. Cucumber utilizes plain text files to provide human readable inputs for automated tests. The text files contain scenarios written in a specially structured language based on the given-when-then structure. These frameworks are great tools that support collaborative definition of acceptance criteria.
Acceptance Testing Frameworks
In parallel to the BDD idea, another flavor of tools has been developed, where FIT was an early representative. This Framework for Integrated Test allows to specify examples within tables that are embedded into a documentation related to the examples. To write these documents no development skills are required and they can be easily read and reviewed by non technical guys as they are purely text based. Additionally, the text can be structured as the documents are not plain text files but the output of rich editors.
FitNesse allows to specify the expected behavior collaboratively based on a wiki. As wikis are easy to access and use, it has a low entry and learning curve, which propels the common work of the entire team. Many agile proponents emphasize that the best way for collaboration is face-to-face communication. But, if you write down what you have thought and discussed, it should be as rich and well structured as possible.
Concordion provides a lot of flexibility as you can describe your requirements in normal language using paragraphs, tables and proper punctuation. Any part of your description can be used as input to your automated tests and for the validation of the outputs of your system under test. As it is based on HTML you can structure your documents and integrate images. Simply, you have the expressiveness of the web to describe the expected behavior.
BDD should help to build the right product
You could implement BDD with all three flavors of tools, but each has its strengths and weaknesses. Unit testing frameworks and xSpec like tools perfectly use the strengths of programming. As they are tools from developers for developers, they are a perfect choice if you try to get the technical part right.
When you want to communicate the intention of the application you are probably better off with a tool that is strongly related with tools that editors use for their work. If a specification contains only inputs and expected outputs, anyone who reads it will have to reconstruct your ideas from the relation of inputs to expected outputs. A short description explaining the goal of a specification in the header helps the reader understand the structure of the specification. Documents based on specification-by-example could look like:
Yes, SpecFlow is cool, NSpec is cool ...
FitNesse and Concordion are cool as well