As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
What are the good automated web UI testing tools?
I want to be able to use it in the .Net world - but it doesn't have to written in .net.
Features such as a record mode, integration into build process\ continuous integration would be nice.
Im going to look at:
Watir
Selenium
Are there any others I should look at?
I definitively recommend Selenium, you can use it from .NET, supports different browsers, works in automatic builds and CI processes (we use it from CCNet). The code is stable. It has a few quirks, but after all they all do.
Whichever tool you choose, I recommend making your own test facade class(es) around it. The facade should be designed to suite your concrete testing needs, without exposing too much the details of the testing tool's API. This will make the test code easier to write and maintain.
UPDATE: if you use ASP.NET view state in your app, you could have problems using a pure HTTP test tool. This is where browser-controllers (like Selenium) are much better.
WatiN
Automates FF and IE
[Test]
public void SearchForWatiNOnGoogle()
{
using (IE ie = new IE("http://www.google.com"))
{
ie.TextField(Find.ByName("q")).TypeText("WatiN");
ie.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(ie.ContainsText("WatiN"));
}
}
http://watin.sourceforge.net/
Watin is pretty unstable to use it in serious projects. It often fails with unexpected reasons like "EI is busy" or something like "Error with COM object".
Selenium is much more stable and it already has a lot of supporting tools. For example Selenium GRID is a solution which allows significantly decrease run time of tests. (Our smoke tests on Watin takes 6 hours to run).
Currently in my job i use QTP and it so far atleast can handle pretty much anything we throw at it both on the UI and it has a special mode for testing non gui services allowing us to check both and help us narrow down where some problems occur when we change the system. It is in my opinion very configurable and the inclusion of vbscript as its language allows integration with lots and lots of things on windows to allow you to do pretty much anything you want! For instance we use it to control the excel com object to make custom excel reports of success and failure so the format of the results is the same wether a test was run manually and also on another project used the adodb object to check that when a page submits information to the database that the database contains the correct data for that record!
As for integration into the build process i have not tried this myself but it is possible to launch qtp and a test from a vbs file so i would assume this should be fairly trvial as ms tools tend to allow you to run vbs files pretty easily from most tools.
I would reccomend it to anyone assuming you can get someone to buy the license!
You can also try VSTT - http://blogs.msdn.com/b/slumley/archive/2009/05/28/vsts-2010-feature-enhancements-for-web-test-playback-ui.aspx
Telerik Test Tools - http://www.telerik.com/automated-testing-tools.aspx
Visual Studio UI Test Extensibility–Scenarios & Guiding Principles - http://blogs.msdn.com/b/mathew_aniyan/archive/2011/03/28/visual-studio-ui-test-extensibility-scenarios-amp-guiding-principles.aspx
VSTS Web Test Step-by-Step Primer - http://blogs.msdn.com/b/jimmymay/archive/2009/02/23/vsts-web-test-step-by-step-primer-7-minute-video-by-microsoft-a-c-e-performance-engineer-chris-lundquist-with-copious-notes-screen-shots-from-your-humble-correspondent.aspx
you might also be interested in taking a look at what the ASP.NET team cooked up itself: Lightweight Test Automation Framework.
There's also a dedicated forum for it.
Having used several different automated testing solutions (TestComplete, QTP, etc), I have to put a vote in for Telerik + Visual Studio. Telerik has great support forums, and is very compatible with whatever testing framework you come up with. Our Devs put unique IDs into their HTML code so that our scripts don't need to be rewritten even with pretty drastic UI refactors. It's definitely more challenging than record and playback, but once you have your unique IDs in place, the automation code requires little or no maintenance.
Try httpunit
Depend on what you would like to achieve.
You can use web test built in the Visual Studio Tester Edition. It's quite good and easy to automate. You can use external data as a test data source and it integrates nicely with VS.
There is also test tool by Automated QA (forgot the name) which looks good but expensive.
And there is Selenium. That's the one we are using in Symantec. The biggest advantage is that it actually uses a browser you want to test. VS mimic a browser by changing http request parameters only so you may not be able to test your site for cross-browser compatibility. Selenium on the other hand uses browser and automates it so you can actually test your site in IE, Firefox etc. It can be also integrated with VS unit tests so you can see test results in VS.
So I would recommend Selenium or VS.
I've used Selenium. The features were good, and it was usable but it was buggy.
The IDE would often record events incorrectly (so tests would need to be manually changed), and test files occasionally became completely unusable for no apparent reason, which meant they would have to be recreated all over again. Also development on Selenium IDE seems to have stopped; there hasn't been any bug fixes and patches for a while, and bug reports seem to go unnoticed.
Molybdenum is an alternative, built on Selenium that's worth looking into.
http://www.molyb.org/
Just to throw out another option (of which I haven't tried but I do like Telerik) is Telerik's new WebUI Testing Studio. I will also echo Selenium up votes.
I forget one nice tools and can find link on it but find this ... http://weblogs.asp.net/bsimser/archive/2008/02/21/automated-ui-testing-with-project-white.aspx maybe can help.
If you are looking for simple, cross-browser tool with record and playback, multithreaded playback, intergration with build processes, powerful scripting, good reporting and excellent support, go for Sahi. It will be much easier for your testers/devs to learn and maintain.
you might want to take in consideration near Selenium also Rational Functional Tester ! whether you are familiar with coding in .Net or Java and want to just play around with record & replay or want to create more sophisticated programmatic testing I would recommend it.
WebDriver is another possibility: http://code.google.com/p/webdriver
They are working on a .NET wrapper that may be interesting for you.
Try QEngine. It has all the features of QTP.
You may want to look at RIATest for cross-platform cross-browser testing of web applications.
It works on Windows and Mac, supported browsers are Firefox, IE and Chrome. Automated testing scripts written on one platform/browser can be run against all other supported platforms/browsers.
It has the features that you want: user interaction recording mode and integration with CI servers (outputs results in JUnit format which can be consumed by CI servers such as Hudson).
(Disclaimer: I am a RIATest team member).
Related
I can find some articles on how to use them but I can't seem to find anywhere peoples impression of them using them in real projects. I have been trying to figure out how to use them and I've had alot of problems.
Can someone out there who uses these tools on the job give me thier impression? Are there better alternate tools available? Using these really just a waste of time?
With Coded UI Tests I see how they are good for basic javascript checking but its so basic of a example I don't think it is worth it. With web tests I like how they work but when I activate code coverage/ASP.NET profiling it doesnt work half the time.
I tried both Selenium and Coded UI for integration testing the project I am working on. I wrote a blog article with my impressions.
The bottom line for me is that both Selenium and Coded UI are still far from perfect, but better than manual testing. I am using Selenium right now, though I might revisit CodedUI if a few things would be fixed in a later version.
experience with Selenium and VS Coded UI test for web apps.
Better than manual testing in my eyes, though the tests are at times hard to maintain and often need to be changed manually to make them more robust. Both tools have browser plugins which can generate tests automatically, but this code is generally not very resilient towards changes in the HTML (in case of web apps).
You will be disappointed if you expect an automated test run orders of magnitude faster than a manual tester. I think the aim of the vendors is to get the automated test execute aboit twice as fast as a manual tester (that can feel painfully slow if you have a large set of tests).
I have used both selenium 1.0 (currently deprecated 2.0 for my dev environment) and visual studio 2010 web testing. There is no doubt, vs 2010 web testing tool is by far (and i mean by far) better than selenium 1.0 and 2.0 (in beta now) in terms of tooling, robustness and the different kinds of metric you can measure (load balance analysis etc). The one good thing also is that it can run your tests in firefox and other browsers as well(provided you have those browsers installed - i believe. Not sure on that one). Selenium is good and have been around for a while. Has a lot of community support but... it can be frustrating. They are currently in transition to 2.0 so I think things will get better and improve after 2.0. In short, if your development environment is .NET with vs 2010, you will get the best ROI by using of the web testing tools.
Disclaimer: I don't work for Microsoft and I am in no way affiliated with MS. My comment above is based on my experience and my only.
As with (most) all Microsoft tools they will do the job--provided you read plenty of materials to understand how to use them and you understand their limitations. If you are looking for full-featured, high-quality, relatively intuitive tools then look to RedGate.
Microsoft tools will do everything, but nothing terribly well.
Does anyone know of any tools out there that will let me run and debug a VXML application visually? There are a ton of VXML development tools, but they all require you to build your application within them.
I have an existing application that uses JSPs to generate VXML, and I'm looking for a way to navigate through and debug the rendered VXML in much the same way that Firebug allows one to do this with HTML. I have some proxy-like tools that let me inspect the rendered code as it is sent to the VXML browser, but there's a ton of JS, which makes traversing the code by hand rather difficult.
Has anyone worked with a product that allows for this?
Thanks!
IVR Avenger
There is JigSaw Test suite - has free trial license and reasonably priced.
There is IBM's debugger - part of WebSphere Voice Toolkit.
Many other products have debuggers - a very good summary is here
Disclaimer: I am the development manager for Voiyager (www.voiyager.com), a VoiceXML testing tool. It doesn't meet your criteria nor do I believe it is the type of tool you want, but I thought it was worth mentioning it.
As far as I know, there isn't such a test tool for VoiceXML. In fact there are very few VoiceXML tools on the market and hardly any of them test or analysis. The vendors that created development tools, have all been acquired by other companies. Some of them offered did offer various forms of debugging that were specific to their tool set or stayed at the Dialog (caller input) level. From your question, I'm assuming you need much lower level debugging capabilities.
I think the alternative paths are minimal and somewhat difficult. I believe your primary goal is to debug or rewrite an existing application, but you haven't provided any specific challenges beyond the JavaScript. Some thoughts or approaches that may help:
Isolate the JavaScript and place the code into a unit test harness. That will go a long way to understanding the logic of the application. Any encapsulation of the JavaScript you perform will probably go a long way towards better code maintainability.
Attempt to run the VoiceXML through a translation layer to HTML so you could use FireBug. The largest challenge would involve caller input (ie processing the SRGS grammars). You could probably cheat this by just having the form accept a JSON string the populates the field values. There are tools on the market to test grammars. Depending on the nature of your problems, you could take a simple and light approach and attempt this over just the trouble areas.
Plumb the application with a lot of logging. This can be done through the VoiceXML LOG element, or push the variable space back to the server. By adding intermediate forms, you may be able to provide a dump from each via the VoiceXML Data element.
See if your application will run in one of the open source VoiceXML browsers (not sure of the state of the open source browsers as we've built and bought for our various product lines). If you can get it mostly working, you can use the development debugger to provide some ability to step through the logic. However, it is probably one of the more difficult paths as you'll really need to understand the browser to know when and where to stick your breakpoints and to figure out how to expose the data you want.
Good luck on the challenge. If you find another approach, I would be interested in seeing it posted.
An alternative debug env is to use something like Asterisk with a voicexml browser plugin like the one from http://www.voiceglue.org/ or for a limited licence, i6net.
You can keep all the pieces separate(dynamic html and vxml application in php/jsp/j2ee/, tts processing, and optional asr processing as separate virtual machines with something like virtualbox. If the logic can be kept the same, then it is just a matter of changing the UI based on the channel.
A softphone is all you need to call a minimal asterisk machine, which has the voicexml browser with the url of the vxml in the call plan.
I just used Zend Framework as php is used in this environment, and changed view suffixes(phtml vs vxml) based on the user-agent string.
Flite for tts is fine for debugging, and when your app is ready you can either record phrases, and there was a page on the ubuntu forums with directions for how to increase flite quality with some additional sound files.
Do you have tried Eclipse VTP or InVision Studio?
Eclipse VTP
This is Eclipse plugin. But I feel that it is user-unfriendly a little (of Japanese viewpoint).
InVision Studio *Required create user account*
This is Convergys's IVR tool. It has to edit standard VXML mode. (Unfortunately, It's not exact matching.)
For just debugging vxml, I use Nuance Cafe's VoiceXML checker. It doesn't give you a visual tree or anything, but it's pretty good at spotting syntax errors and is free. I think they might also have more advanced debugging tools if you look into it, but I haven't had the need. (Note: I have no association with them)
http://cafe.bevocal.com/tools/vxmlchecker/vxmlchecker.jsp
I'm looking for the same problem that most of the links are down. I found a document where they propose an open source solution, which works as a plugin for Asterisk (https://www.researchgate.net/publication/228873959_Open_Source_VoiceXML_Interpreter_over_Asterisk_for_Use_in_IVR_Applications) and is available at https://sourceforge.net/projects/voxy/
I would like to know if there are current options to create a VXML structure graphically, like the next image.
I would like to know how test automation tools like winrunner, QTP etc work. Whether these tool use any test API provided by windows or they depened on IPC and events. I could not figure out how they work. For me QTP record and play feature seems to be a magic.Any guidance will be highly appreciated?
Actually WatiN is not bad place to start. It is not exactly unit testing framework. It may look like one at firs sight but it is used to write functional test. It is totally up to you if you run them like unit tests. I was writing the same test using WatiN in two ways (functional test that was simulating user actions on the web page):
1) Writing script in Powershell at running from command line like any other PS script. It was fun, although you need to write lots of code for reporting, exception handling, and other.
2) Writing unit test in MSVS in C# using C# Unit Test from MSVS project type. This actually was fun because you would just run it in MSVS like unit test but you have environment support for code writing, reporting, running etc.
So if you want to start with something take a look at WatiN, especially as WatiN has dedicated recorder that records actions and outputs code in C#. Looking at internals of library and tool would give you some start.
Just one thing to mention - it is web only. Desktop is totally different. With web you have hook into IE you can query html document for objects, checking browser state and so. With desktop it may be more difficult. You need to hook into application, maybe via mentioned Win32hooks. Maybe try with Microsoft Scripting Host.
I dont know much about QTP. But if you wanted to know the internals then you can download the open source projects like Watin- for dotnet and Watir- for ruby and see whats happening inside. Both are used for web test automation. And the code is freely available..
If you are looking for Unit testing frameworks like Nunit. they are attribute driven.. Nunit identifies the classes by "TestFixtures" and methods by "Test" attributes. It scans the entire application for these test methods and trigger the tests.
I guess they're using Win32 Hooks.
edit: here is an example of defining hooks in .NET.
It should be Win32/64 Programming in C, C++ using the technologies COM, OLE, and they might be using Active accesibility API. Definitely they might have created a big API on these technologies. Even they might have used a bit of Assembly programming as well.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
This question isn't about unit-testing. And it is for a desktop product.
This is about testing of the gui and testing that the right stuff is input in the right text box at the right time.
A company I used to work at used WinRunner (different department so I don't know much more that that), but that has now been shutdown by HP but they don't seem that bothered whether you stay with HP or go elsewhere. You can't read about the product until you've signed up which is annoying.
The tool has to work with MFC (non-negotiable) and the ideal tool will also...
be automated.
be scriptable.
work with different screen resolutions automatically.
be able to 'spy' on individual static text boxes, etc.
intuitive enough so non-programmers can create the scripts.
have reporting tools, including email of individual users.
What do other SO users do for automated GUI testing?
We use the SAFS framework for Rational Robot (RRAFS). There are also SAFS implementations for WinRunner (WRAFS) and it looks like they have a new "Image-Based Testing" implementation, which I'm not familiar with.
This framework does a nice job of seperating the UI implementation from the test scripts. I've tested four releases of a web application developed by two different teams (one team using classic ASP, one using ASP.NET) and I only had to change the application map of my UI objects, the tests themselves didn't need to change.
That said, the language of the framework is cumbersome and takes getting used to. It's not very robust, in terms of language constructs, but with some effort you can do anything you need to. It's sort of like "programming" in Windows Batch language, but for tests ;)
To address your individual requirements above:
1) The tool has to work with MFC (non-negotiable).
The SAFS framework uses a 3rd party "record-playback" tool to drive the tests, like Rational Robot or Mercury WinRunner. If that tool can interact with MFC apps, then the framework can. I don't know how the "Image-Based Testing" implementation drives the tests, but I'd guess it can also work with MFC.
2) be automated.
The SAFS framework integrates with the STAF framework, which will allow you to automate the execution of your tests. I have a proof-of-concept test that uses STAF to start a VM image from a pool of images, install the application under test, run the RRAFS test, and put the results on a web server for others to get at.
3) be scriptable.
Yes, but as mentioned, it's not the most robust programming language. I wrote an Excel add-in that our testers use to write their tests that simplifies things a little bit.
4) work with different screen resolutions automatically.
Yes, since it's looking "under the covers" at the UI objects and not the screen. Except for maybe the "Image-based Testing" option...
5) be able to 'spy' on individual static text boxes, etc.
Yes, you can wait for a UI object to appear, disapper, to have a value, for a value to be changed, etc.
6) intuitive enough so non-programmers can create the scripts.
With some training. We've had limited success. Some QA folks can write the tests, some struggle.
7) have reporting tools, including email of individual users.
Yes, using the STAF framework you can post results to a web server, send out emails, etc.
Lots of good answers here, but I wanted to address this goal statement, specifically:
intuitive enough so non-programmers
can create the scripts
I can understand why you'd want this, but it's a lot harder than you might think. While you can find any number of tools out there that'll claim to make writing scripts easy, in practice, you're going to need at least some people on your automation team who understand programming. Writing scripts that are reasonably robust is going to involve one or more of looping, if/then/else, and subroutine calls. Not the kind of thing that non-programmers are going to find intuitive.
Be especially wary of the idea that you can "record" a person using a tool, then play it back for testing. That sort of "automation" is often so brittle that you'll end up modifying or re-recording the script for nearly every change in the software.
Coming from a strong Mercury/HP background, I would highly recommend using QuickTest Professional for your GUI testing. It has a lot of the same functionality as WinRunner, but without a lot of code. Simple GUI checks can be done through the QTP interface with minimal, if any, custom VB code. Checks for text next to boxes can be done with simple compares using the datasheet in QTP.
If you are used to WinRunner, and know VBScript (not so much TSL), then I would definantly look at QTP.
As far as your other requirements, QTP also has the Spy feature, like WinRunner, that will list all properties and actions you can perform on objects. And as for simplicity of use, at my old job, we would have business or system testers create simple smoke scripts, then I would take them and code them for more in-depth testing (multiple data values, error checking, etc). And as for reporting, QTP will do simple reporting of pass/fail/warning on tags that you put in, along with custom data you can input. So you could use a case statement to populate your output values based on your results. It won't do e-mail naitevly, but if you integrate with TestDirector/QualityCenter, you can setup through there, along with automating the kick-off of your scripts, and parameterizing data right from there (which is nice to send back to testers to have populate data without being involved in the script itseld).
Pat
See User Interface Testing.
Desktop or Web apps alike have the same testing pattern here (I've worked in both).
Place as little logic as possible in the UI and test everything below it. So you say, but what if I want to test that such and such happens when a button is clicked? The method that is called when that button is clicked should call out to another class that actually does the thinking.
You might say, but I'm using some static classes/methods that can only exist in my UI, wrap those with an adapter and utilize that interface in order to make your code testable.
The parts of your GUI tests that you want to automate should be automated below the UI. There are parts of your GUI that you can't automate. Checking to make sure things "look right", or testing that you can see certain elements, etc. All of that is what your humans should be doing. Ensuring that events are properly firing, and that values are being appropriately returned from your business objects, that's all unit tests.
I can see from your question that you've already pretty much decided you need an automated GUI tester, but that's not the right tool for this job. If you decide to use that you're trying to find the best way of doing the wrong thing.
If you think that this isn't about unit testing because you're testing GUI interactions, then I can guarantee you're not unit testing close enough to your UI. If you were, you'd feel like most of what you'd be testing was redundant.
If you disagree with me, please post some reasons and we'll hash this out.
There are a lot more (open source) alternatives if you are testing a web product. For a desktop product, some popular general purpose desktop GUI automation tools below (in no particular order). I've worked with all of these personally, and they all get the job done. If you choose to go with a vendor tool, get POCs for the ones you are considering, and make the decision based on what tool is the best fit for the company in general. One tool may be a better fit for a particular application, but there may be other projects/applications to consider.
QuickTest Pro (HP's successor to WinRunner)
Rational Functional Tester (IBM's successor to Robot)
TestPartner
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
We are working on a large project with a measure of new/modified GUI functionality. We've found in the past that we often introduced new problems in related code when adding new functionality.
We have non-technical users perform testing, but they often miss parts and allow bugs to slip through.
Are there any best practices for organizing the UI testing of a WinForms project? Is there any way to automate it?
There are GUI testing tools that will click buttons and stuff for you but they're pretty fragile in my experience.
The best thing to do is to keep your UI layer as thin as possible. Your event handler classes should optimally be only one or two lines that call out to other more testable classes. That way you can test your business logic in unit tests without having to actually do a button click.
You can automate GUI testing using White framework.
Also consider using TDD friendly design, i.e. use MVP/MVC pattern.
I would highly recommend you to read documentation from Microsoft patterns&practies teams.
Especially have a look at the Composite UI application block and CompositeWPF.
These projects specifically designed to give you best practices in GUI apps development including test driven UI.
Keep the GUI layer as thin as possible. Michael Feathers article, The Humble Dialog Box, is a classic. Also check out Martin Fowler's Passive View. I have also heard that the "automatic button clickers" are fragile, and that it's easy to spend more time maintaining the test than you spend maintaining the code.
In the event that someone finds this useful:
List of GUI testing tools found on Wikipedia.
The following book is an introduction to the subject.
There are as many ways as there are developers out there..
http://pragprog.com/titles/idgtr/scripted-gui-testing-with-ruby
There are many tools and libraries available that can automate WinForms testing, ranging from open source solutions like White to the expensive commercial solutions such as HP QuickTest Pro. There is also the UIAutomation namespace in .NET if you want to roll your own automation framework. But the real cost of automation is in the time and specialised skills it requires to implement. Maintainability is also one of the most important aspects of automated test design; you dont want to expend excessive resource keeping the automation assets current with your application. There are also lots of factors influencing the decision to automate which will be specific to your specific application and organisation.
Your best bet will be to do some more research on the subject and check out some of the specialised testing sites such as http://www.sqaforums.com.
I found this quick and dirty way to test web page layouts in various browsers. It's called browsershots.org. Our client requires support in 5 browsers right now and that takes about a week for full regression testing. This service will deliver screenshots of some 70+ browsers and versions. I print them out and hold up the pages to the light. If they don't line up, there must be a layout problem.
I can't really help with organization or best practices, but an NUnit extension appears to be available for forms testing, NUnitForms.
I used a trial download similar to this product (http://www.tethyssolutions.com/product.htm) and this product (http://www.mjtnet.com/macro_scheduler.htm) years ago and I was happy with the results. These are pretty cheap solutions and some of these macro recorder products can actually be used for automated testing.
A new method available now is using Ruby, via the Ruby gem called win32-autogui. This provides a framework for testing Windows GUI apps. Combine it with Ruby tools RSpec and Cucumber, and it makes for quite a powerful testing framework.
How to organize UI testing depends on how you design the test cases.
Automating Windows Forms application in unit test level can use TDD framework, such as NUnit; or use BDD framework, such as NSpec.
Automating Windows Forms applications in functional testing level can use White, CodedUI, or even directly using Windows Automation API 3.0 (UI Automation and MSAA).
Since all these stuffs mentioned above are technologies rather than solutions, it's better to build some fundamental test automation/scripting frameworks based on these technologies before you start to write automated tests.