I would like to know how to go about doing regression testing effectively for a web application.
I m spending a lot of time in regression testing for every build.
I also hear that automated GUI testing is bad and not worth it.
Is there any better way of automating regression testing. if so any good tools to automate GUI tests.
My web app is designed for stupid IE. cant help it, but that how it is.Any good tool that can help me?
I know that we must keep GUI layer thin and all, but I m not a GUI expert or anything , but i need to test it properly and effectively.
Thanks all
Try looking into Cucumber or Selenium
There is also a screencast on it over at railscasts.
That should really help.
You can use standard unit tests and integration tests for the non-GUI components, of course. For the GUI, it's worth looking at UI automation packages.
Many of them are pretty terrible, but a lot of the horror stories come from people using them incorrectly. If you can stay away from specifying click locations by (x,y) coordinates, and you can detect when state transitions happen, rather than using delay(), GUI testing can be effectively automated.
As long as your webapp works with Firefox, too, I recommend using Selenium for that scenario. It allows you to record your test cases inside the browser (by means of a Firefox plugin) but has Internet Explorer support for playback.
If Firefox for recording tests is not an option, you could have a look at the WebDriver project. It uses a code based approach to test specification an works well with TDD, too.
You might also want to have a look at CubicTest, which is a Eclipse plugin for graphically defining Selenium and Watir test cases.
Related
Can BDD (Behavior Driven Design) tests be implemented using a UI driver?
For example, given a web application, instead of:
Writing tests for the back-end, and then more tests in Javascript for the front-end
Should I:
Write the tests as Selenium macros, which simulate mouse-clicks, etc in the actual browser?
The advantages I see in doing it this way are:
The tests are written in one language, rather than several
They're focussed on the UI, which gets developers thinking outside-in
They run in the real execution environment (the browser), which allows us to
Test different browsers
Test different servers
Get insight into real-world performance
Thoughts?
We've done this for a C# application using a WPF testing tool (WipFlash) and writing NUnit tests in a BDD-like fashion.
e.g.
Given.TheApplicationWindowIsOpen();
When.I.Press.OKButton();
The.Price.ShouldBeCalculated();
We had to code a lot of the DSL ourselves, needless to say. But it becomes a business/customer readable solution.
Try using SpecFlow with WatiN: (I'm not sure if you're using .NET here)
http://msdn.microsoft.com/en-us/magazine/gg490346.aspx
For web testing, you could try WebDriver. The Selenium team are busy integrating WebDriver at the moment. Simon Stewart from Google, who created WebDriver, blogged here about how it works differently to Selenium.
WebDriver uses different technologies for each browser. For Internet Explorer, WebDriver uses Microsoft's UI automation - the same technology on which WipFlash which #Brian Agnew mentioned is based. This is as close as you'll get to pretending to click buttons. Simon's blog shows why this approach can be more powerful than Selenium's Javascript solution.
WebDriver is available from the Selenium site but hasn't been fully implemented as part of Selenium yet.
For BDD, and any use-case driven tests, it is important to be able to communicate what a test is doing. The problem with many test suites is that post-writting nobody is quite certain exactly what the test is doing. This will come up very often if you write in a non-specialized language. Specialization doesn't necessarily mean a special language, but just enough of an abstraction in the one language so it is clear what is happening.
For example, a lot of tests have code that looks like this (pseudo-code, I won't pick on any particular framework):
object = createBrowser()
response = object.gotoURL( "http://someurl.com" );
element = response.getLink( "Click Here" );
response = element.doClick();
This is hard for somebody to quickly translate to a business driver (product manager perhaps, or user). Instead you want to create specialized functions, or a language if you're adventurous, so you can have this:
GotoURL http://someurl.com/
Click link:Click Here
Selenium, and its macros or interface, are still fairly low-level in this regards. If you do use them then at least build some wrappers around them.
You can of course also use a product called TestPlan. It has Selenium in the back-end and exposes a high-level API, and a custom langauge for testing. It also goes beyond just the web to included Email, FTP, etc. The sample language above is a TestPlan snippet
You can certainly do some of your acceptance tests this way, but I think most BDD advocates would not advise using this for all tests. And of course, true BDD advocates wouldn't call them tests...
The RSpec Book advocates a two-level cycle with acceptance tests (or Scenarios) written first (primarily in Cucumber), and unit tests written (in RSpec) in an inner cycle more resembling the traditional TDD.
The outer cycle of acceptance testing can also use tools like Selenium to drive the entire application through the UI (and the authors of The RSpec Book spend a chapter on this). But it is not appropriate for unit tests.
Tests exercising the entire application through the UI are harder to make repeatable, and have a tendency to be slower and more fragile than unit tests.
Actually you could do both - make a user-centric Driver interface (agnostic of GUI / tech / impl). You could then write a UIDriver and a APIDriver and choose a driver to run a specific test. Running through the UI is usually slower (out of proc, control repaints but somehow creates a higher level of confidence initially). Running through the API is much faster (in proc, easy setup-teardown).
The trick here is to separate the What from the How. Otherwise you will end up with ObscureTests and high test maintenance. Ensure the primary focus on testing and not automation.
I can find some articles on how to use them but I can't seem to find anywhere peoples impression of them using them in real projects. I have been trying to figure out how to use them and I've had alot of problems.
Can someone out there who uses these tools on the job give me thier impression? Are there better alternate tools available? Using these really just a waste of time?
With Coded UI Tests I see how they are good for basic javascript checking but its so basic of a example I don't think it is worth it. With web tests I like how they work but when I activate code coverage/ASP.NET profiling it doesnt work half the time.
I tried both Selenium and Coded UI for integration testing the project I am working on. I wrote a blog article with my impressions.
The bottom line for me is that both Selenium and Coded UI are still far from perfect, but better than manual testing. I am using Selenium right now, though I might revisit CodedUI if a few things would be fixed in a later version.
experience with Selenium and VS Coded UI test for web apps.
Better than manual testing in my eyes, though the tests are at times hard to maintain and often need to be changed manually to make them more robust. Both tools have browser plugins which can generate tests automatically, but this code is generally not very resilient towards changes in the HTML (in case of web apps).
You will be disappointed if you expect an automated test run orders of magnitude faster than a manual tester. I think the aim of the vendors is to get the automated test execute aboit twice as fast as a manual tester (that can feel painfully slow if you have a large set of tests).
I have used both selenium 1.0 (currently deprecated 2.0 for my dev environment) and visual studio 2010 web testing. There is no doubt, vs 2010 web testing tool is by far (and i mean by far) better than selenium 1.0 and 2.0 (in beta now) in terms of tooling, robustness and the different kinds of metric you can measure (load balance analysis etc). The one good thing also is that it can run your tests in firefox and other browsers as well(provided you have those browsers installed - i believe. Not sure on that one). Selenium is good and have been around for a while. Has a lot of community support but... it can be frustrating. They are currently in transition to 2.0 so I think things will get better and improve after 2.0. In short, if your development environment is .NET with vs 2010, you will get the best ROI by using of the web testing tools.
Disclaimer: I don't work for Microsoft and I am in no way affiliated with MS. My comment above is based on my experience and my only.
As with (most) all Microsoft tools they will do the job--provided you read plenty of materials to understand how to use them and you understand their limitations. If you are looking for full-featured, high-quality, relatively intuitive tools then look to RedGate.
Microsoft tools will do everything, but nothing terribly well.
I would like to know how test automation tools like winrunner, QTP etc work. Whether these tool use any test API provided by windows or they depened on IPC and events. I could not figure out how they work. For me QTP record and play feature seems to be a magic.Any guidance will be highly appreciated?
Actually WatiN is not bad place to start. It is not exactly unit testing framework. It may look like one at firs sight but it is used to write functional test. It is totally up to you if you run them like unit tests. I was writing the same test using WatiN in two ways (functional test that was simulating user actions on the web page):
1) Writing script in Powershell at running from command line like any other PS script. It was fun, although you need to write lots of code for reporting, exception handling, and other.
2) Writing unit test in MSVS in C# using C# Unit Test from MSVS project type. This actually was fun because you would just run it in MSVS like unit test but you have environment support for code writing, reporting, running etc.
So if you want to start with something take a look at WatiN, especially as WatiN has dedicated recorder that records actions and outputs code in C#. Looking at internals of library and tool would give you some start.
Just one thing to mention - it is web only. Desktop is totally different. With web you have hook into IE you can query html document for objects, checking browser state and so. With desktop it may be more difficult. You need to hook into application, maybe via mentioned Win32hooks. Maybe try with Microsoft Scripting Host.
I dont know much about QTP. But if you wanted to know the internals then you can download the open source projects like Watin- for dotnet and Watir- for ruby and see whats happening inside. Both are used for web test automation. And the code is freely available..
If you are looking for Unit testing frameworks like Nunit. they are attribute driven.. Nunit identifies the classes by "TestFixtures" and methods by "Test" attributes. It scans the entire application for these test methods and trigger the tests.
I guess they're using Win32 Hooks.
edit: here is an example of defining hooks in .NET.
It should be Win32/64 Programming in C, C++ using the technologies COM, OLE, and they might be using Active accesibility API. Definitely they might have created a big API on these technologies. Even they might have used a bit of Assembly programming as well.
We are considering using Cucumber for testing web applications (not in rails, most of them are asp.net actually).
The applications are in production, our main goal is to test if everything is fine with the services, from time to time, infra people would run it.
We have two questions:
1) Is this a good use for cucumber? Do community people encourage this use of cucumber feature definitions?
2) We have some captchas in our applications. Is there an widely adopted solution for this common problem?
Cucumber looks relatively new. I am a Java person and have used Selenium, HtmlUnit, JWebUnit, etc. Selenium runs in .net, ruby, java, and some other scripting languages.
Selenium has been around for a while (2004 whereas Cucumber is relatively new, 2007).
Selenium has an IDE so you can easily record tests in firefox, save them, and then run them in your integration tests.
I am biased towards Selenium, but it does a good job and allows you to test your applications in several browsers (firefox, safari, IE). It also has support for distributing tests across several servers (if your environment is that large, it supports it).
Ideally, you would have developers or the infrastructure people writing the tests. Then if you have a CI server, you could automatically run the tests you have recorded/written, and then continuously check your application still works as a whole. This works really great for catching errors as soon as they happen so if the developer makes a change and breaks something out of his scope, it will be fresh in his or her head.
As far as CAPTCHA goes, there are various libraries out there. I am unfortunately more knowledgeable with the Java equivalent and not so much with the .Net. Don't write your own, there should be a library you can use.
Walter
ad 1. in my opinion cucumber is great, also we were able to convince our customers to actually understand and verify the tests we wrote in cucumber. We used cucumber + watir for webtesting
ad 2. as far as captcha goes, do you mean how to ignore the captcha for testing? we do not show captcha for our own ip addresses, you could also always accept a specific value for the captcha if the request comes from your development or test environments ip
I can recommend Cucumber. I trained a team of developers and managers to use it on my last project (a PHP application). It worked very well in most cases.
I think your two questions are mutually exclusive. Captcha is designed to prevent something automated so you're going to have to solve that problem for which ever automated test runner you use. You can probably mock something up or work out how to disable it in your test environment. I would opt for the latter. I don't imagine it would be critical to cover your captcha in your test suite.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
What are the good automated web UI testing tools?
I want to be able to use it in the .Net world - but it doesn't have to written in .net.
Features such as a record mode, integration into build process\ continuous integration would be nice.
Im going to look at:
Watir
Selenium
Are there any others I should look at?
I definitively recommend Selenium, you can use it from .NET, supports different browsers, works in automatic builds and CI processes (we use it from CCNet). The code is stable. It has a few quirks, but after all they all do.
Whichever tool you choose, I recommend making your own test facade class(es) around it. The facade should be designed to suite your concrete testing needs, without exposing too much the details of the testing tool's API. This will make the test code easier to write and maintain.
UPDATE: if you use ASP.NET view state in your app, you could have problems using a pure HTTP test tool. This is where browser-controllers (like Selenium) are much better.
WatiN
Automates FF and IE
[Test]
public void SearchForWatiNOnGoogle()
{
using (IE ie = new IE("http://www.google.com"))
{
ie.TextField(Find.ByName("q")).TypeText("WatiN");
ie.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(ie.ContainsText("WatiN"));
}
}
http://watin.sourceforge.net/
Watin is pretty unstable to use it in serious projects. It often fails with unexpected reasons like "EI is busy" or something like "Error with COM object".
Selenium is much more stable and it already has a lot of supporting tools. For example Selenium GRID is a solution which allows significantly decrease run time of tests. (Our smoke tests on Watin takes 6 hours to run).
Currently in my job i use QTP and it so far atleast can handle pretty much anything we throw at it both on the UI and it has a special mode for testing non gui services allowing us to check both and help us narrow down where some problems occur when we change the system. It is in my opinion very configurable and the inclusion of vbscript as its language allows integration with lots and lots of things on windows to allow you to do pretty much anything you want! For instance we use it to control the excel com object to make custom excel reports of success and failure so the format of the results is the same wether a test was run manually and also on another project used the adodb object to check that when a page submits information to the database that the database contains the correct data for that record!
As for integration into the build process i have not tried this myself but it is possible to launch qtp and a test from a vbs file so i would assume this should be fairly trvial as ms tools tend to allow you to run vbs files pretty easily from most tools.
I would reccomend it to anyone assuming you can get someone to buy the license!
You can also try VSTT - http://blogs.msdn.com/b/slumley/archive/2009/05/28/vsts-2010-feature-enhancements-for-web-test-playback-ui.aspx
Telerik Test Tools - http://www.telerik.com/automated-testing-tools.aspx
Visual Studio UI Test Extensibility–Scenarios & Guiding Principles - http://blogs.msdn.com/b/mathew_aniyan/archive/2011/03/28/visual-studio-ui-test-extensibility-scenarios-amp-guiding-principles.aspx
VSTS Web Test Step-by-Step Primer - http://blogs.msdn.com/b/jimmymay/archive/2009/02/23/vsts-web-test-step-by-step-primer-7-minute-video-by-microsoft-a-c-e-performance-engineer-chris-lundquist-with-copious-notes-screen-shots-from-your-humble-correspondent.aspx
you might also be interested in taking a look at what the ASP.NET team cooked up itself: Lightweight Test Automation Framework.
There's also a dedicated forum for it.
Having used several different automated testing solutions (TestComplete, QTP, etc), I have to put a vote in for Telerik + Visual Studio. Telerik has great support forums, and is very compatible with whatever testing framework you come up with. Our Devs put unique IDs into their HTML code so that our scripts don't need to be rewritten even with pretty drastic UI refactors. It's definitely more challenging than record and playback, but once you have your unique IDs in place, the automation code requires little or no maintenance.
Try httpunit
Depend on what you would like to achieve.
You can use web test built in the Visual Studio Tester Edition. It's quite good and easy to automate. You can use external data as a test data source and it integrates nicely with VS.
There is also test tool by Automated QA (forgot the name) which looks good but expensive.
And there is Selenium. That's the one we are using in Symantec. The biggest advantage is that it actually uses a browser you want to test. VS mimic a browser by changing http request parameters only so you may not be able to test your site for cross-browser compatibility. Selenium on the other hand uses browser and automates it so you can actually test your site in IE, Firefox etc. It can be also integrated with VS unit tests so you can see test results in VS.
So I would recommend Selenium or VS.
I've used Selenium. The features were good, and it was usable but it was buggy.
The IDE would often record events incorrectly (so tests would need to be manually changed), and test files occasionally became completely unusable for no apparent reason, which meant they would have to be recreated all over again. Also development on Selenium IDE seems to have stopped; there hasn't been any bug fixes and patches for a while, and bug reports seem to go unnoticed.
Molybdenum is an alternative, built on Selenium that's worth looking into.
http://www.molyb.org/
Just to throw out another option (of which I haven't tried but I do like Telerik) is Telerik's new WebUI Testing Studio. I will also echo Selenium up votes.
I forget one nice tools and can find link on it but find this ... http://weblogs.asp.net/bsimser/archive/2008/02/21/automated-ui-testing-with-project-white.aspx maybe can help.
If you are looking for simple, cross-browser tool with record and playback, multithreaded playback, intergration with build processes, powerful scripting, good reporting and excellent support, go for Sahi. It will be much easier for your testers/devs to learn and maintain.
you might want to take in consideration near Selenium also Rational Functional Tester ! whether you are familiar with coding in .Net or Java and want to just play around with record & replay or want to create more sophisticated programmatic testing I would recommend it.
WebDriver is another possibility: http://code.google.com/p/webdriver
They are working on a .NET wrapper that may be interesting for you.
Try QEngine. It has all the features of QTP.
You may want to look at RIATest for cross-platform cross-browser testing of web applications.
It works on Windows and Mac, supported browsers are Firefox, IE and Chrome. Automated testing scripts written on one platform/browser can be run against all other supported platforms/browsers.
It has the features that you want: user interaction recording mode and integration with CI servers (outputs results in JUnit format which can be consumed by CI servers such as Hudson).
(Disclaimer: I am a RIATest team member).