Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
The other day I was showing one of the testers at my company some tests I had written in cucumber (2 features, 5 scenarios).
Then he asked me question that I could not answer:
How is this better than selenium or any other functionality test recording tool?
I understand that cucumber is a different technology and it's placed at a different level of testing, but I can't understand why I should bother to write and maintain Cucumber/Capybara tests.
Can someone give me a reasonable explanation for using Cucumber/Capybara instead of just Selenium?
This question is borderline asking for an opinion. Your question actually reads to me, "What tool is right for me?" I say this because you don't give a reason for why you chose Cucumber and Capybara. I believe to answer that tester's question, you need to answer a couple more questions first:
1.) What stage in the process are you going to be writing these tests?
Cucumber may not be the right choice for unit tests, depending on the language you're using. But it can be used for any level of testing, from unit to integration to end-user.
2.) Who is going to maintaining your tests? You? Other developers? Testers? Business Analysts? Project Managers?
Automated tests must be maintained, and knowing who will be doing that can help you decide on a tool - as some will be too technical for certain users.
3.) Who is going to be defining new tests?
Cucumber is meant to be used collaboratively between development, QA and business owners. It is the perfect tool for leveraging everyone's knowledge into the automated testing process. It requires the development of an ubiquitous language to be effect however. You can read up on that on James Shore's Art of Agile page.
Once you've answered these questions, you're ready to address the tester's question.
However, there are a couple of points to keep in mind when comparing recording tools (such as Selenium IDE, HP Quick Test Pro, IBM Rational Functional Tester) vs. development tools (nUnit, jUnit, RSpec, Selenium webdriver, Capybara) is that they are targeted towards different audiences. They also have different plusses and minuses.
Recording tools are easy for anyone to use, but the scripts they create are fragile. They break easily and require more maintenance. They are great for one-off automated testing, where you need to get it done quickly and have non-technical manpower.
Development tools have a larger learning curve and require programming (or at the least scripting) experience. The scripts are generally more robust, but require more technical knowledge to maintain. They are a good solution when you want repeatability and plan to use tests for a long time.
I strongly suggest you read The Cucumber Book. It will really help you decide if Cucumber is the right choice for you.
Cucumber isn't only a testing tool. Besides testing Cucumber features also take a role of documentation, a mechanism to collaborate with stakeholders and requirements storage (if you write them in declarative style).
You don't have to use Cucumber with Capybara. You can use selenium directly. But Capybara has the same high-level API for all supporting drivers. It's more high-level than Selenium's and allows to write tests a bit faster. You don't have to change code when you switch from one driver to another.
Tests built using test recording tools are generally much worse. Selenium IDE may produce valid programming code but it's not good-looking and thus quiet difficult to maintain.
Cucumber is tool used to make tests readable to business users. It consists of plain English sentences that are matched using regex to your Capybara steps.
Using recording tools won't do you any good in the long run. They were meant for beginners and aren't that powerful so I recommend you go straight to coding.
You can use Selenium alone for your tests, but I would recommend you continue to use Cucumber for documentation purposes, if you find them useful and easy to work with. After all, Cucumber can use Capybara or the Selenium web driver.
Selenium ide is good for testing features that have mostly visual elements (links, text and etc.). But often web apps have features that don't represent itself as visual elements, like sending emails, queueing jobs, communicating with 3rd party services and etc. You could, for example, test if an 'Email has been sent' message is present after submitting a form. But it doesn't really tell you if an email is actually sent and therefore you aren't really testing the whole feature here.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 5 years ago.
Improve this question
From the agile manifesto, agile values:
Individuals and interactions over processes and tools,
Working software over comprehensive documentation,
Customer collaboration over contract negotiation,
Responding to change over following a plan
Yet doesn't TDD create a plan and almost structure out a contract negotiation?
"What are the features you want?"
"1,2,3"
Developer writes tests for 1,2,3 -> Team delivers code
"Here's 1,2,3 give us our money"
It's also a form of comprehensive documentation and also a process. Once the tests are written individuals and interactions no longer matter as much because the "source of truth" is no longer with people but ironed out in the code.
Just wondering how they fit together, if they're opposed or do they work together?
TDD is more like a practice for individual contributors, instead of a process. Test here usually refers to unit test, which is part of development work, instead of comprehensive test suits such as performance, functional, and integration tests.
TDD in certain cases should help individual contributor really think about requirement and implementation (respond to change and come up with working software). I personally do not adopt this practice, but it is an agile practice that can be adopted by a single contributor. Do not confuse it with higher level tests and related documents.
Yet doesn't TDD create a plan
Nope. TDD does not mean "write tests up front" it means "write tests before writing code". The whole "Do as much as you need and no more" comes into play. You are not expected to write the tests for all your features before writing any code, just the feature you are currently on. And then (depending on the level of testing) just a small subset of the feature will need tests now.
It's also a form of comprehensive documentation and also a process
It also aids with working software.
Working software over comprehensive documentation,
Over, not instead of. If you can get both, great.
the tests are written individuals and interactions no longer matter as much because the "source of truth" is no longer with people but ironed out in the code.
The oracle for what it does is always the code. The oracle for what it should do is always people.
TDD done well also aids with the communication.
Any insight as to why some people seem to be getting mad at the question?
The question comes off as very troll-y. You are twisting the manifesto to make it sound like anything that aids the latter is "bad" and you are twisting the definition of TDD to be an all-encompassing, completely up-front process. Neither of which are true.
Individuals and interactions over processes and tools,
BDD is a great tool for aiding interactions at a dev/BA/stake holder level. TDD (xUnit and alikes) are great tools for aiding interactions at a dev level.
Working software over comprehensive documentation
TDD helps create working software.
Customer collaboration over contract negotiation
(BDD) Being able to describe in a common language the specification and have that execute is awesome.
Responding to change over following a plan
A well tested code base can change with ease. An untested or badly tested code base is fixed.
That is, while there is value in the items on
the right, we value the items on the left more.
I also agree with Tom’s answer, ‘if it is possible to do agile well, then I believe that is always a good thing. If it is not possible to do it well, then I believe that it can be harmful.’ Agile simply isn’t the right answer for every company. It is different to do well in a large company and it’s lack of focus on software architecture can really affect the usefulness of the resulting technology. Digital Animal have written an interesting article on Agile and why it doesn’t work for them. http://digitalanimal.com/blog/slaying-the-agile-dragon-the-game-of-thrones-methodology/?AT=D8c953
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I have no experience neither in TDD nor in BDD. Yes I've created unit tests for existing code a lot, but it's not relevant here. Also I cann't use TDD/BDD at my job but want to try in some hobby project.
I am not sure if I currently grasped the difference between TDD and BDD correctly. For now I just see BDD as evolved TDD with the most destinctive feature being the ability to work on higher level of abstraction (user stories) then TDD. In TDD you basically get same user stories but they are not as explicit as in BDD. Is it correct?
In terms of tools, assuming that statements above are correct, for TDD I should use something like TestNG or JUnit, and for BDD I'll benefit from tools like JBehave.
Now the question is should I first start with TestNG and TDD and only after some succesfull experience with it migrate to JBehave and BDD? Or this is just waste of time and there are no reasons at all preventing me from trying to use Jbehave and BDD from the very beginning?
UPDATE:
After receiving two great answers on my question, and spending some time on additional reading on the topic, I couldn't help myself not to add link to a great article I found. It just repeats same ideas as in two answers to this question below, but maybe with more details. My favorite part of the article:
The best way to do that is to leverage BDD and TDD. Here is an approach:
1. Write requirements as user stories using the BDD grammar/structure.
Do this collaboratively with the key stakeholders.
2. Enter the User
Stories (feature + scenarios) in a BDD tool.
3. Write code to map the
User Stories to tests.
4. Write production code using TDD to make the
tests pass.
As you can see, BDD is not just TDD done right. You could use just
the vocabulary of BDD to improve TDD but that would be like using
only some of the benefits that BDD has to offer us. When we use the
the strengths of both these techniques we will have “Software that
matters” along with “Software that works”.
BDD concentrates on testing the functionality of the app, and ensuring that it meets the stated acceptance criteria of the application as described in the user story.
TDD tends to concentrate on the lower-level code, testing on a per-class basis rather than on a per-functional-unit basis.
If you have a good idea of what you want to achieve in the user story - enough to enumerate a list of things the feature should & shouldn't do - you're in a good position to start BDD. You can write a BDD feature file from that list, and doing this before writing a scrap of code will allow you think about the feature and crystallise your thinking on it. This will help you write more concise correct code.
I would suggest looking at Cucumber-JVM as your tool for implementing the BDD tests, but I'm told JBehave is functionally similar.
Finally, to me BDD and TDD are techniques that compliment each other, not compete - doing the BDD beforehand makes it easier to think about what is required, which makes it easier to think of what low-level functionality is required, which in turn makes it easier to write your unit tests up front. Once the code is written, you then have a suite of tests allowing you to prove the app meets the acceptance criteria.
The way I do BDD is this:
1) Examine the user story for User-Acceptance Criteria
2) Write a feature file (English) containing test scenarios to test each of these conditions.
3) Implement the feature
4) Use Cucumber/JBehave to programmatically implement the tests from the feature file
5) Ensure all the tests pass
We then maintain a library of these tests, and run them as part of our Continuous Integration
A feature file contains instructions in a specific wording, such as:
Given a user goes to our app
When the user clicks on the search link
Then the search window opens
And the user enters XXX in the search box
And the user presses enter
The search results are displayed
Although most of the literature focuses on TDD as unit tests and BDD as acceptance tests, this is not what distinguishes them.
On the implementation-level, BDD is basically TDD with extra attention to language. BDD is not significantly harder to learn, but you will be missing most of the value of BDD as it is geared towards team communication.
My advice would be to learn JUnit if you haven't already because it is the most ubiquitous framework. The hardest part of starting TDD/BDD is figuring out how write tests first and design for testability.
If you want to learn pure TDD/BDD then write acceptance tests (executable specifications) first and let them drive your unit tests and code. This is probably overkill for a small project, but a good exercise.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I'm looking to enhance our current test suites, and continuous integration builds with full stack integration/acceptance testing.
I'm looking into tools like Culerity and Selenium that can execute front end javascript while running user stories. I'm looking for something that can provide coverage of front-end javascript and high level features without sucking up tons of development time maintaining a complex test environment. We're currently using Rspec, Cucumber, and CruiseControl.rb, so easy integration with those tools would be ideal.
Are any of the headless browsers and js-capable test environments to a point where they are worth the trouble of setting up and maintaining? What are the best option you've come across, and pitfalls to avoid?
Thanks.
You sound like you are way further down this road than I am, but I'll comment anyway.
I am working on a JavaScript project (with a Java + MySQL back end) and decided to use Selenium for testing, and to try to achieve as thorough coverage as I could. I also poked around with a few other testing tools, but I can't say I really got to know any of them. None of them appeared, from their web sites, to be very polished or popular compared to Selenium. I am planning to integrate to CruiseControl eventually, but haven't done so yet.
This has been an interesting project and at the end of the day, I am quite happy with Selenium. Selenium plusses:
Test 'scripts' can all be written in Java, no obscure scripting language involved. Among other things, you can easily do things like manipulate and verify the data in your database before and after tests.
Se also supports Perl, C#, etc. I think, although that is of no interest to me.
Selenium IDE is a great tool for quickly understanding how Se works, how locators work, etc. You don't want to actually run tests long-term using the IDE, but it's great for getting your feet wet, and for ongoing figuring things out.
Se seems to work flawlessly with jUnit. Probably TestNG as well, but have not tried that yet, it's on my todo list.
Excellent documentation and web site.
Minuses:
I spent a LOT of time figuring out how to locate elements in all cases. This is partially the 'fault' of the framework I am using (ExtJS), not Selenium.
It seems no matter what you do, Se has timing dependencies - eg. places where you have to inject artificial pauses to make it work.
There are also monitor-size dependencies in my tests. I think this is extremely undesirable but in some places it seems to be unavoidable. Basically, this is because there are many element types that JS doesn't support you clicking on programatically.
Related to #3, in places I am forced to drive the mouse. That means you have to have a dedicated test PC. Which is no big deal, but doesn't seem right.
Tests are slow - mainly due to the time it takes Se to invoke Firefox. No doubt this is partially my environment, and I suspect I could do lots of things to improve this. However, it is really noticeable and not obvious why. It takes about 10 minutes to run about 40 tests.
Support forum is very spotty. Well, you get what you pay for. But time and again I found someone had posted about my problem, and the post was ignored or else an invalid solution was offered with no follow-up when the OP pointed out that the suggestion was bogus.
HTH, cheers.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I've been using TDD for server-side development. I'm not really sure if the benefits of having all of my production code surrounded by unit tests outweigh the disadvantage of spending 4x more time than needed on refactoring.
But when I'm developing UI code, I simply cannot apply TDD. To all fundamentalists out there, the first law of TDD states that "you may not write production code until you have written a failing unit test". But how can this be if you are developing UIs?
(One can use acceptance test frameworks like Selenium, but that doesn't count, because you don't interact directly with the source code.)
So, can I tell my manager that because of the new >90% code coverage policy I cannot write user interface code?
If you find that writing TDD causes you to spend 4x more time on refactoring, you need to be writing better, more isolated tests, and really let the tests drive the design, as intended. You are also not counting the time you spend in the debugger when you refactor without tests, not to mention how much time everyone else spends on bugs you introduce when you refactor.
Anyway, here is some good advice about what TDD means for UI development. How much that will translate into code coverage depends heavily on the UI framework.
Definitely don't tell your manager you can't do it, he may just replace you with someone who can.
First off, even Robert Martin has testing challenges with UIs.
When TDDing a UI, you write "behavioral contracts" that get as close to the action as possible. Ideally that means unit tests. But some UI frameworks make that inordinately difficult, requiring that you step back and use integration or "acceptance" tests to capture how you expect the UI to behave.
Does it not count if you can't use unit tests? That depends on which rules you're using to keep score. The "only unit tests count" rule is a good one for beginners to try to live with, in the same vein as "don't split infinitives" or "avoid the passive voice". Eventually, you learn where the boundaries of that rule are. In one podcast, Kent Beck talks about using combinations of unit and integration tests, as appropriate (adding, if I recall correctly, that it doesn't bother him).
And if TDD is your goal, you can most certainly write Selenium tests first, though that can be a slow way to proceed. I've worked on several projects that have used Selenium RC to great effect (and great pain, because the tests run so slowly).
Whatever your framework, you can Google around for TDD tips from people who've fought the same battles.
TDD is about testing methods in isolation. If you want to test your UI you are doing integration tests and not unit tests. So if you carefully separate the concerns in your application you will be able to successfully apply TDD to ANY kind of project.
That policy sounds a little artificial, but I would agree with the answer that UIs require functional test cases, not unit test. I disagree however with the point about which comes first. I've worked in an environment where the UI functional tests had to be written before the UI was developed and found it to work extremely well. Of course, this assumes that you do some design work up front too. As long as the test case author and the developer agree on the design it's possible for someone to write the test cases before you start coding; then your code has to make all the test cases pass. Same basic principle but it doesn't follow the law to the letter.
Unit tests are inappropriate for UI code. Functional tests are used to test the UI, but you cannot feasibly write those first. You should check with your manager to see if the >90% code coverage policy covers UI code as well. If it does, he should probably seriously rethink that move.
Separate the business logic from the UI and ensure that the UI code takes less than 10% of the total? Separation of concerns is the main goal of TDD, so that's actually a good thing.
As far as 90% coverage goes ... well, best course is to review extant literature (I'd focus on Kent Beck and Bob Martin), and I think you'll find support for not following a mindless coverage percentage (in fact, I think Uncle Bob wrote a blog post on this recently).
A having a >90% code coverage is dumb because a smart dev can get 100% coverage in one test. ;)
If you are using WPF, you can test your UI code if you use the MVVM pattern. By test your UI code, i mean you can test the ModleView but there is nothing that I know that can test XAML.
Read Phlip's book
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
We are working on a large project with a measure of new/modified GUI functionality. We've found in the past that we often introduced new problems in related code when adding new functionality.
We have non-technical users perform testing, but they often miss parts and allow bugs to slip through.
Are there any best practices for organizing the UI testing of a WinForms project? Is there any way to automate it?
There are GUI testing tools that will click buttons and stuff for you but they're pretty fragile in my experience.
The best thing to do is to keep your UI layer as thin as possible. Your event handler classes should optimally be only one or two lines that call out to other more testable classes. That way you can test your business logic in unit tests without having to actually do a button click.
You can automate GUI testing using White framework.
Also consider using TDD friendly design, i.e. use MVP/MVC pattern.
I would highly recommend you to read documentation from Microsoft patterns&practies teams.
Especially have a look at the Composite UI application block and CompositeWPF.
These projects specifically designed to give you best practices in GUI apps development including test driven UI.
Keep the GUI layer as thin as possible. Michael Feathers article, The Humble Dialog Box, is a classic. Also check out Martin Fowler's Passive View. I have also heard that the "automatic button clickers" are fragile, and that it's easy to spend more time maintaining the test than you spend maintaining the code.
In the event that someone finds this useful:
List of GUI testing tools found on Wikipedia.
The following book is an introduction to the subject.
There are as many ways as there are developers out there..
http://pragprog.com/titles/idgtr/scripted-gui-testing-with-ruby
There are many tools and libraries available that can automate WinForms testing, ranging from open source solutions like White to the expensive commercial solutions such as HP QuickTest Pro. There is also the UIAutomation namespace in .NET if you want to roll your own automation framework. But the real cost of automation is in the time and specialised skills it requires to implement. Maintainability is also one of the most important aspects of automated test design; you dont want to expend excessive resource keeping the automation assets current with your application. There are also lots of factors influencing the decision to automate which will be specific to your specific application and organisation.
Your best bet will be to do some more research on the subject and check out some of the specialised testing sites such as http://www.sqaforums.com.
I found this quick and dirty way to test web page layouts in various browsers. It's called browsershots.org. Our client requires support in 5 browsers right now and that takes about a week for full regression testing. This service will deliver screenshots of some 70+ browsers and versions. I print them out and hold up the pages to the light. If they don't line up, there must be a layout problem.
I can't really help with organization or best practices, but an NUnit extension appears to be available for forms testing, NUnitForms.
I used a trial download similar to this product (http://www.tethyssolutions.com/product.htm) and this product (http://www.mjtnet.com/macro_scheduler.htm) years ago and I was happy with the results. These are pretty cheap solutions and some of these macro recorder products can actually be used for automated testing.
A new method available now is using Ruby, via the Ruby gem called win32-autogui. This provides a framework for testing Windows GUI apps. Combine it with Ruby tools RSpec and Cucumber, and it makes for quite a powerful testing framework.
How to organize UI testing depends on how you design the test cases.
Automating Windows Forms application in unit test level can use TDD framework, such as NUnit; or use BDD framework, such as NSpec.
Automating Windows Forms applications in functional testing level can use White, CodedUI, or even directly using Windows Automation API 3.0 (UI Automation and MSAA).
Since all these stuffs mentioned above are technologies rather than solutions, it's better to build some fundamental test automation/scripting frameworks based on these technologies before you start to write automated tests.