We are considering using Cucumber for testing web applications (not in rails, most of them are asp.net actually).
The applications are in production, our main goal is to test if everything is fine with the services, from time to time, infra people would run it.
We have two questions:
1) Is this a good use for cucumber? Do community people encourage this use of cucumber feature definitions?
2) We have some captchas in our applications. Is there an widely adopted solution for this common problem?
Cucumber looks relatively new. I am a Java person and have used Selenium, HtmlUnit, JWebUnit, etc. Selenium runs in .net, ruby, java, and some other scripting languages.
Selenium has been around for a while (2004 whereas Cucumber is relatively new, 2007).
Selenium has an IDE so you can easily record tests in firefox, save them, and then run them in your integration tests.
I am biased towards Selenium, but it does a good job and allows you to test your applications in several browsers (firefox, safari, IE). It also has support for distributing tests across several servers (if your environment is that large, it supports it).
Ideally, you would have developers or the infrastructure people writing the tests. Then if you have a CI server, you could automatically run the tests you have recorded/written, and then continuously check your application still works as a whole. This works really great for catching errors as soon as they happen so if the developer makes a change and breaks something out of his scope, it will be fresh in his or her head.
As far as CAPTCHA goes, there are various libraries out there. I am unfortunately more knowledgeable with the Java equivalent and not so much with the .Net. Don't write your own, there should be a library you can use.
Walter
ad 1. in my opinion cucumber is great, also we were able to convince our customers to actually understand and verify the tests we wrote in cucumber. We used cucumber + watir for webtesting
ad 2. as far as captcha goes, do you mean how to ignore the captcha for testing? we do not show captcha for our own ip addresses, you could also always accept a specific value for the captcha if the request comes from your development or test environments ip
I can recommend Cucumber. I trained a team of developers and managers to use it on my last project (a PHP application). It worked very well in most cases.
I think your two questions are mutually exclusive. Captcha is designed to prevent something automated so you're going to have to solve that problem for which ever automated test runner you use. You can probably mock something up or work out how to disable it in your test environment. I would opt for the latter. I don't imagine it would be critical to cover your captcha in your test suite.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
The other day I was showing one of the testers at my company some tests I had written in cucumber (2 features, 5 scenarios).
Then he asked me question that I could not answer:
How is this better than selenium or any other functionality test recording tool?
I understand that cucumber is a different technology and it's placed at a different level of testing, but I can't understand why I should bother to write and maintain Cucumber/Capybara tests.
Can someone give me a reasonable explanation for using Cucumber/Capybara instead of just Selenium?
This question is borderline asking for an opinion. Your question actually reads to me, "What tool is right for me?" I say this because you don't give a reason for why you chose Cucumber and Capybara. I believe to answer that tester's question, you need to answer a couple more questions first:
1.) What stage in the process are you going to be writing these tests?
Cucumber may not be the right choice for unit tests, depending on the language you're using. But it can be used for any level of testing, from unit to integration to end-user.
2.) Who is going to maintaining your tests? You? Other developers? Testers? Business Analysts? Project Managers?
Automated tests must be maintained, and knowing who will be doing that can help you decide on a tool - as some will be too technical for certain users.
3.) Who is going to be defining new tests?
Cucumber is meant to be used collaboratively between development, QA and business owners. It is the perfect tool for leveraging everyone's knowledge into the automated testing process. It requires the development of an ubiquitous language to be effect however. You can read up on that on James Shore's Art of Agile page.
Once you've answered these questions, you're ready to address the tester's question.
However, there are a couple of points to keep in mind when comparing recording tools (such as Selenium IDE, HP Quick Test Pro, IBM Rational Functional Tester) vs. development tools (nUnit, jUnit, RSpec, Selenium webdriver, Capybara) is that they are targeted towards different audiences. They also have different plusses and minuses.
Recording tools are easy for anyone to use, but the scripts they create are fragile. They break easily and require more maintenance. They are great for one-off automated testing, where you need to get it done quickly and have non-technical manpower.
Development tools have a larger learning curve and require programming (or at the least scripting) experience. The scripts are generally more robust, but require more technical knowledge to maintain. They are a good solution when you want repeatability and plan to use tests for a long time.
I strongly suggest you read The Cucumber Book. It will really help you decide if Cucumber is the right choice for you.
Cucumber isn't only a testing tool. Besides testing Cucumber features also take a role of documentation, a mechanism to collaborate with stakeholders and requirements storage (if you write them in declarative style).
You don't have to use Cucumber with Capybara. You can use selenium directly. But Capybara has the same high-level API for all supporting drivers. It's more high-level than Selenium's and allows to write tests a bit faster. You don't have to change code when you switch from one driver to another.
Tests built using test recording tools are generally much worse. Selenium IDE may produce valid programming code but it's not good-looking and thus quiet difficult to maintain.
Cucumber is tool used to make tests readable to business users. It consists of plain English sentences that are matched using regex to your Capybara steps.
Using recording tools won't do you any good in the long run. They were meant for beginners and aren't that powerful so I recommend you go straight to coding.
You can use Selenium alone for your tests, but I would recommend you continue to use Cucumber for documentation purposes, if you find them useful and easy to work with. After all, Cucumber can use Capybara or the Selenium web driver.
Selenium ide is good for testing features that have mostly visual elements (links, text and etc.). But often web apps have features that don't represent itself as visual elements, like sending emails, queueing jobs, communicating with 3rd party services and etc. You could, for example, test if an 'Email has been sent' message is present after submitting a form. But it doesn't really tell you if an email is actually sent and therefore you aren't really testing the whole feature here.
Can BDD (Behavior Driven Design) tests be implemented using a UI driver?
For example, given a web application, instead of:
Writing tests for the back-end, and then more tests in Javascript for the front-end
Should I:
Write the tests as Selenium macros, which simulate mouse-clicks, etc in the actual browser?
The advantages I see in doing it this way are:
The tests are written in one language, rather than several
They're focussed on the UI, which gets developers thinking outside-in
They run in the real execution environment (the browser), which allows us to
Test different browsers
Test different servers
Get insight into real-world performance
Thoughts?
We've done this for a C# application using a WPF testing tool (WipFlash) and writing NUnit tests in a BDD-like fashion.
e.g.
Given.TheApplicationWindowIsOpen();
When.I.Press.OKButton();
The.Price.ShouldBeCalculated();
We had to code a lot of the DSL ourselves, needless to say. But it becomes a business/customer readable solution.
Try using SpecFlow with WatiN: (I'm not sure if you're using .NET here)
http://msdn.microsoft.com/en-us/magazine/gg490346.aspx
For web testing, you could try WebDriver. The Selenium team are busy integrating WebDriver at the moment. Simon Stewart from Google, who created WebDriver, blogged here about how it works differently to Selenium.
WebDriver uses different technologies for each browser. For Internet Explorer, WebDriver uses Microsoft's UI automation - the same technology on which WipFlash which #Brian Agnew mentioned is based. This is as close as you'll get to pretending to click buttons. Simon's blog shows why this approach can be more powerful than Selenium's Javascript solution.
WebDriver is available from the Selenium site but hasn't been fully implemented as part of Selenium yet.
For BDD, and any use-case driven tests, it is important to be able to communicate what a test is doing. The problem with many test suites is that post-writting nobody is quite certain exactly what the test is doing. This will come up very often if you write in a non-specialized language. Specialization doesn't necessarily mean a special language, but just enough of an abstraction in the one language so it is clear what is happening.
For example, a lot of tests have code that looks like this (pseudo-code, I won't pick on any particular framework):
object = createBrowser()
response = object.gotoURL( "http://someurl.com" );
element = response.getLink( "Click Here" );
response = element.doClick();
This is hard for somebody to quickly translate to a business driver (product manager perhaps, or user). Instead you want to create specialized functions, or a language if you're adventurous, so you can have this:
GotoURL http://someurl.com/
Click link:Click Here
Selenium, and its macros or interface, are still fairly low-level in this regards. If you do use them then at least build some wrappers around them.
You can of course also use a product called TestPlan. It has Selenium in the back-end and exposes a high-level API, and a custom langauge for testing. It also goes beyond just the web to included Email, FTP, etc. The sample language above is a TestPlan snippet
You can certainly do some of your acceptance tests this way, but I think most BDD advocates would not advise using this for all tests. And of course, true BDD advocates wouldn't call them tests...
The RSpec Book advocates a two-level cycle with acceptance tests (or Scenarios) written first (primarily in Cucumber), and unit tests written (in RSpec) in an inner cycle more resembling the traditional TDD.
The outer cycle of acceptance testing can also use tools like Selenium to drive the entire application through the UI (and the authors of The RSpec Book spend a chapter on this). But it is not appropriate for unit tests.
Tests exercising the entire application through the UI are harder to make repeatable, and have a tendency to be slower and more fragile than unit tests.
Actually you could do both - make a user-centric Driver interface (agnostic of GUI / tech / impl). You could then write a UIDriver and a APIDriver and choose a driver to run a specific test. Running through the UI is usually slower (out of proc, control repaints but somehow creates a higher level of confidence initially). Running through the API is much faster (in proc, easy setup-teardown).
The trick here is to separate the What from the How. Otherwise you will end up with ObscureTests and high test maintenance. Ensure the primary focus on testing and not automation.
I can find some articles on how to use them but I can't seem to find anywhere peoples impression of them using them in real projects. I have been trying to figure out how to use them and I've had alot of problems.
Can someone out there who uses these tools on the job give me thier impression? Are there better alternate tools available? Using these really just a waste of time?
With Coded UI Tests I see how they are good for basic javascript checking but its so basic of a example I don't think it is worth it. With web tests I like how they work but when I activate code coverage/ASP.NET profiling it doesnt work half the time.
I tried both Selenium and Coded UI for integration testing the project I am working on. I wrote a blog article with my impressions.
The bottom line for me is that both Selenium and Coded UI are still far from perfect, but better than manual testing. I am using Selenium right now, though I might revisit CodedUI if a few things would be fixed in a later version.
experience with Selenium and VS Coded UI test for web apps.
Better than manual testing in my eyes, though the tests are at times hard to maintain and often need to be changed manually to make them more robust. Both tools have browser plugins which can generate tests automatically, but this code is generally not very resilient towards changes in the HTML (in case of web apps).
You will be disappointed if you expect an automated test run orders of magnitude faster than a manual tester. I think the aim of the vendors is to get the automated test execute aboit twice as fast as a manual tester (that can feel painfully slow if you have a large set of tests).
I have used both selenium 1.0 (currently deprecated 2.0 for my dev environment) and visual studio 2010 web testing. There is no doubt, vs 2010 web testing tool is by far (and i mean by far) better than selenium 1.0 and 2.0 (in beta now) in terms of tooling, robustness and the different kinds of metric you can measure (load balance analysis etc). The one good thing also is that it can run your tests in firefox and other browsers as well(provided you have those browsers installed - i believe. Not sure on that one). Selenium is good and have been around for a while. Has a lot of community support but... it can be frustrating. They are currently in transition to 2.0 so I think things will get better and improve after 2.0. In short, if your development environment is .NET with vs 2010, you will get the best ROI by using of the web testing tools.
Disclaimer: I don't work for Microsoft and I am in no way affiliated with MS. My comment above is based on my experience and my only.
As with (most) all Microsoft tools they will do the job--provided you read plenty of materials to understand how to use them and you understand their limitations. If you are looking for full-featured, high-quality, relatively intuitive tools then look to RedGate.
Microsoft tools will do everything, but nothing terribly well.
I would like to know how to go about doing regression testing effectively for a web application.
I m spending a lot of time in regression testing for every build.
I also hear that automated GUI testing is bad and not worth it.
Is there any better way of automating regression testing. if so any good tools to automate GUI tests.
My web app is designed for stupid IE. cant help it, but that how it is.Any good tool that can help me?
I know that we must keep GUI layer thin and all, but I m not a GUI expert or anything , but i need to test it properly and effectively.
Thanks all
Try looking into Cucumber or Selenium
There is also a screencast on it over at railscasts.
That should really help.
You can use standard unit tests and integration tests for the non-GUI components, of course. For the GUI, it's worth looking at UI automation packages.
Many of them are pretty terrible, but a lot of the horror stories come from people using them incorrectly. If you can stay away from specifying click locations by (x,y) coordinates, and you can detect when state transitions happen, rather than using delay(), GUI testing can be effectively automated.
As long as your webapp works with Firefox, too, I recommend using Selenium for that scenario. It allows you to record your test cases inside the browser (by means of a Firefox plugin) but has Internet Explorer support for playback.
If Firefox for recording tests is not an option, you could have a look at the WebDriver project. It uses a code based approach to test specification an works well with TDD, too.
You might also want to have a look at CubicTest, which is a Eclipse plugin for graphically defining Selenium and Watir test cases.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
What are the good automated web UI testing tools?
I want to be able to use it in the .Net world - but it doesn't have to written in .net.
Features such as a record mode, integration into build process\ continuous integration would be nice.
Im going to look at:
Watir
Selenium
Are there any others I should look at?
I definitively recommend Selenium, you can use it from .NET, supports different browsers, works in automatic builds and CI processes (we use it from CCNet). The code is stable. It has a few quirks, but after all they all do.
Whichever tool you choose, I recommend making your own test facade class(es) around it. The facade should be designed to suite your concrete testing needs, without exposing too much the details of the testing tool's API. This will make the test code easier to write and maintain.
UPDATE: if you use ASP.NET view state in your app, you could have problems using a pure HTTP test tool. This is where browser-controllers (like Selenium) are much better.
WatiN
Automates FF and IE
[Test]
public void SearchForWatiNOnGoogle()
{
using (IE ie = new IE("http://www.google.com"))
{
ie.TextField(Find.ByName("q")).TypeText("WatiN");
ie.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(ie.ContainsText("WatiN"));
}
}
http://watin.sourceforge.net/
Watin is pretty unstable to use it in serious projects. It often fails with unexpected reasons like "EI is busy" or something like "Error with COM object".
Selenium is much more stable and it already has a lot of supporting tools. For example Selenium GRID is a solution which allows significantly decrease run time of tests. (Our smoke tests on Watin takes 6 hours to run).
Currently in my job i use QTP and it so far atleast can handle pretty much anything we throw at it both on the UI and it has a special mode for testing non gui services allowing us to check both and help us narrow down where some problems occur when we change the system. It is in my opinion very configurable and the inclusion of vbscript as its language allows integration with lots and lots of things on windows to allow you to do pretty much anything you want! For instance we use it to control the excel com object to make custom excel reports of success and failure so the format of the results is the same wether a test was run manually and also on another project used the adodb object to check that when a page submits information to the database that the database contains the correct data for that record!
As for integration into the build process i have not tried this myself but it is possible to launch qtp and a test from a vbs file so i would assume this should be fairly trvial as ms tools tend to allow you to run vbs files pretty easily from most tools.
I would reccomend it to anyone assuming you can get someone to buy the license!
You can also try VSTT - http://blogs.msdn.com/b/slumley/archive/2009/05/28/vsts-2010-feature-enhancements-for-web-test-playback-ui.aspx
Telerik Test Tools - http://www.telerik.com/automated-testing-tools.aspx
Visual Studio UI Test Extensibility–Scenarios & Guiding Principles - http://blogs.msdn.com/b/mathew_aniyan/archive/2011/03/28/visual-studio-ui-test-extensibility-scenarios-amp-guiding-principles.aspx
VSTS Web Test Step-by-Step Primer - http://blogs.msdn.com/b/jimmymay/archive/2009/02/23/vsts-web-test-step-by-step-primer-7-minute-video-by-microsoft-a-c-e-performance-engineer-chris-lundquist-with-copious-notes-screen-shots-from-your-humble-correspondent.aspx
you might also be interested in taking a look at what the ASP.NET team cooked up itself: Lightweight Test Automation Framework.
There's also a dedicated forum for it.
Having used several different automated testing solutions (TestComplete, QTP, etc), I have to put a vote in for Telerik + Visual Studio. Telerik has great support forums, and is very compatible with whatever testing framework you come up with. Our Devs put unique IDs into their HTML code so that our scripts don't need to be rewritten even with pretty drastic UI refactors. It's definitely more challenging than record and playback, but once you have your unique IDs in place, the automation code requires little or no maintenance.
Try httpunit
Depend on what you would like to achieve.
You can use web test built in the Visual Studio Tester Edition. It's quite good and easy to automate. You can use external data as a test data source and it integrates nicely with VS.
There is also test tool by Automated QA (forgot the name) which looks good but expensive.
And there is Selenium. That's the one we are using in Symantec. The biggest advantage is that it actually uses a browser you want to test. VS mimic a browser by changing http request parameters only so you may not be able to test your site for cross-browser compatibility. Selenium on the other hand uses browser and automates it so you can actually test your site in IE, Firefox etc. It can be also integrated with VS unit tests so you can see test results in VS.
So I would recommend Selenium or VS.
I've used Selenium. The features were good, and it was usable but it was buggy.
The IDE would often record events incorrectly (so tests would need to be manually changed), and test files occasionally became completely unusable for no apparent reason, which meant they would have to be recreated all over again. Also development on Selenium IDE seems to have stopped; there hasn't been any bug fixes and patches for a while, and bug reports seem to go unnoticed.
Molybdenum is an alternative, built on Selenium that's worth looking into.
http://www.molyb.org/
Just to throw out another option (of which I haven't tried but I do like Telerik) is Telerik's new WebUI Testing Studio. I will also echo Selenium up votes.
I forget one nice tools and can find link on it but find this ... http://weblogs.asp.net/bsimser/archive/2008/02/21/automated-ui-testing-with-project-white.aspx maybe can help.
If you are looking for simple, cross-browser tool with record and playback, multithreaded playback, intergration with build processes, powerful scripting, good reporting and excellent support, go for Sahi. It will be much easier for your testers/devs to learn and maintain.
you might want to take in consideration near Selenium also Rational Functional Tester ! whether you are familiar with coding in .Net or Java and want to just play around with record & replay or want to create more sophisticated programmatic testing I would recommend it.
WebDriver is another possibility: http://code.google.com/p/webdriver
They are working on a .NET wrapper that may be interesting for you.
Try QEngine. It has all the features of QTP.
You may want to look at RIATest for cross-platform cross-browser testing of web applications.
It works on Windows and Mac, supported browsers are Firefox, IE and Chrome. Automated testing scripts written on one platform/browser can be run against all other supported platforms/browsers.
It has the features that you want: user interaction recording mode and integration with CI servers (outputs results in JUnit format which can be consumed by CI servers such as Hudson).
(Disclaimer: I am a RIATest team member).