Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
This question isn't about unit-testing. And it is for a desktop product.
This is about testing of the gui and testing that the right stuff is input in the right text box at the right time.
A company I used to work at used WinRunner (different department so I don't know much more that that), but that has now been shutdown by HP but they don't seem that bothered whether you stay with HP or go elsewhere. You can't read about the product until you've signed up which is annoying.
The tool has to work with MFC (non-negotiable) and the ideal tool will also...
be automated.
be scriptable.
work with different screen resolutions automatically.
be able to 'spy' on individual static text boxes, etc.
intuitive enough so non-programmers can create the scripts.
have reporting tools, including email of individual users.
What do other SO users do for automated GUI testing?
We use the SAFS framework for Rational Robot (RRAFS). There are also SAFS implementations for WinRunner (WRAFS) and it looks like they have a new "Image-Based Testing" implementation, which I'm not familiar with.
This framework does a nice job of seperating the UI implementation from the test scripts. I've tested four releases of a web application developed by two different teams (one team using classic ASP, one using ASP.NET) and I only had to change the application map of my UI objects, the tests themselves didn't need to change.
That said, the language of the framework is cumbersome and takes getting used to. It's not very robust, in terms of language constructs, but with some effort you can do anything you need to. It's sort of like "programming" in Windows Batch language, but for tests ;)
To address your individual requirements above:
1) The tool has to work with MFC (non-negotiable).
The SAFS framework uses a 3rd party "record-playback" tool to drive the tests, like Rational Robot or Mercury WinRunner. If that tool can interact with MFC apps, then the framework can. I don't know how the "Image-Based Testing" implementation drives the tests, but I'd guess it can also work with MFC.
2) be automated.
The SAFS framework integrates with the STAF framework, which will allow you to automate the execution of your tests. I have a proof-of-concept test that uses STAF to start a VM image from a pool of images, install the application under test, run the RRAFS test, and put the results on a web server for others to get at.
3) be scriptable.
Yes, but as mentioned, it's not the most robust programming language. I wrote an Excel add-in that our testers use to write their tests that simplifies things a little bit.
4) work with different screen resolutions automatically.
Yes, since it's looking "under the covers" at the UI objects and not the screen. Except for maybe the "Image-based Testing" option...
5) be able to 'spy' on individual static text boxes, etc.
Yes, you can wait for a UI object to appear, disapper, to have a value, for a value to be changed, etc.
6) intuitive enough so non-programmers can create the scripts.
With some training. We've had limited success. Some QA folks can write the tests, some struggle.
7) have reporting tools, including email of individual users.
Yes, using the STAF framework you can post results to a web server, send out emails, etc.
Lots of good answers here, but I wanted to address this goal statement, specifically:
intuitive enough so non-programmers
can create the scripts
I can understand why you'd want this, but it's a lot harder than you might think. While you can find any number of tools out there that'll claim to make writing scripts easy, in practice, you're going to need at least some people on your automation team who understand programming. Writing scripts that are reasonably robust is going to involve one or more of looping, if/then/else, and subroutine calls. Not the kind of thing that non-programmers are going to find intuitive.
Be especially wary of the idea that you can "record" a person using a tool, then play it back for testing. That sort of "automation" is often so brittle that you'll end up modifying or re-recording the script for nearly every change in the software.
Coming from a strong Mercury/HP background, I would highly recommend using QuickTest Professional for your GUI testing. It has a lot of the same functionality as WinRunner, but without a lot of code. Simple GUI checks can be done through the QTP interface with minimal, if any, custom VB code. Checks for text next to boxes can be done with simple compares using the datasheet in QTP.
If you are used to WinRunner, and know VBScript (not so much TSL), then I would definantly look at QTP.
As far as your other requirements, QTP also has the Spy feature, like WinRunner, that will list all properties and actions you can perform on objects. And as for simplicity of use, at my old job, we would have business or system testers create simple smoke scripts, then I would take them and code them for more in-depth testing (multiple data values, error checking, etc). And as for reporting, QTP will do simple reporting of pass/fail/warning on tags that you put in, along with custom data you can input. So you could use a case statement to populate your output values based on your results. It won't do e-mail naitevly, but if you integrate with TestDirector/QualityCenter, you can setup through there, along with automating the kick-off of your scripts, and parameterizing data right from there (which is nice to send back to testers to have populate data without being involved in the script itseld).
Pat
See User Interface Testing.
Desktop or Web apps alike have the same testing pattern here (I've worked in both).
Place as little logic as possible in the UI and test everything below it. So you say, but what if I want to test that such and such happens when a button is clicked? The method that is called when that button is clicked should call out to another class that actually does the thinking.
You might say, but I'm using some static classes/methods that can only exist in my UI, wrap those with an adapter and utilize that interface in order to make your code testable.
The parts of your GUI tests that you want to automate should be automated below the UI. There are parts of your GUI that you can't automate. Checking to make sure things "look right", or testing that you can see certain elements, etc. All of that is what your humans should be doing. Ensuring that events are properly firing, and that values are being appropriately returned from your business objects, that's all unit tests.
I can see from your question that you've already pretty much decided you need an automated GUI tester, but that's not the right tool for this job. If you decide to use that you're trying to find the best way of doing the wrong thing.
If you think that this isn't about unit testing because you're testing GUI interactions, then I can guarantee you're not unit testing close enough to your UI. If you were, you'd feel like most of what you'd be testing was redundant.
If you disagree with me, please post some reasons and we'll hash this out.
There are a lot more (open source) alternatives if you are testing a web product. For a desktop product, some popular general purpose desktop GUI automation tools below (in no particular order). I've worked with all of these personally, and they all get the job done. If you choose to go with a vendor tool, get POCs for the ones you are considering, and make the decision based on what tool is the best fit for the company in general. One tool may be a better fit for a particular application, but there may be other projects/applications to consider.
QuickTest Pro (HP's successor to WinRunner)
Rational Functional Tester (IBM's successor to Robot)
TestPartner
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I'm looking for a tool for automated testing. I will use scripting for testing, rather than just automating, which means that it can't just send some key-presses. It needs to examine the app's window to see if it was updated correctly. The cheaper the better, with no downside limit on cost. ;-)
I need a real language, with functions, variables, loops, branching, etc., and the apps are developed in Delphi.
I need to be able to focus controls*, wait for windows to appear, search for controls/windows, send keyboard input and check that various windows/controls have changed state (e.g a button is now disabled, a text box contains the correct text, a dialog box opens, etc.).
I used to use AutoIt a few years ago and am now coming back to scripting. I wondered if there has been any new development while I was gone.
Any suggestions?
* I do not want to move the cursor to absolute coordinates in case the app's layout changes. This quote from Wikipedia explains why:
Regression testing becomes a problem with GUIs as well. This is because the GUI may change significantly across versions of the application, even though the underlying application may not. A test designed to follow a certain path through the GUI may not be able to follow that path since a button, menu item, or dialog may have changed location or appearance.
I believe you just described TestComplete by SmartBear Software.
Autoit does everything you listed. I use it for the same porpoises all the time. Also, Delphi is good in the sense that uses standard windows controls which makes it very easy to hook into.
I've had some success with sikuli, which has the advantage of being python (jython) code. It is built on top of OpenCV, and uses a full image processing engine to pattern match parts of the screen. It also contains editors and functions for screenshots, waiting on changes, as well as keystroke and mouse injection.
Something different from my other answer: PyWinAuto
It has full scripting (because it's Python), it uses control names instead of X, Y coordinates, it waits for windows and you can send keypresses. And it's free.
The downsides should be obvious :)
To automate a delphi application main problem i found was that the any testing tools (QTP with delphi addon , Test Complete , Testanywhere, autoit, sikuli) they all dont detect any Tlabels and Tlist and and other components, which for an application build in delphi forms a prominent part of the UI. Even to do func based testing we need to ultimately examine a component in the front end to see whether a func is giving right out. Even to right a script to automate we need to first be able to detect and add the object to the repository.
We have an application in which we have heavy dependency on a lot of third party components. I have tried to check feasibility with almost all the tools available in market.
The top two tools in market were of no use.
QTP with Delphi addon is of no use even in there website they have openly mentioned it wont support many classes build in delphi.
TestComplete its better than QTP where in it detects many components menu components and if i remember correctly grid etc.
Then after few googling i found
TestAnywhere which was certainly better in detecting the thirdparty build components. Only problem i found with this was that it was not detecting Tlabel classes and some components dont remember which.
When i speak of detecting components it may be partial or fully ie some tools were able to detect that there is a component at XY co-ordinates but failed to get the component classes or details (like components property name etc) without which it was difficult in automating and writing a script.
Rest all was more or less combination or subset of these above there.
Still Searching !!!!!!!!!!!
With TestComplete, if you compile your app as an open app - i.e. include their libs, it makes your app super inspectable. Great for testing, probably not best for security. Test complete was the best, it is just TOOOO expensive for my budget.
The scripting in it is great, the understanding of Delphi is great. The pricetag is just not great, especially if you are trying to get traction for GUI testing within an org, like it sounds like you are.
Currently I am preparing exercises about networks and mobile communications for students. I was thinking about creating an interactive user-interface which enables the user to drag&drop predefined elements and then implement a logic based upon element distances etc.
An example would be to place two base stations (a predefined element with several properties), set the scale in the interface and then check the signal interferrence in the environment (user-interface).
The first part might be too abstract whereas the example might be too specific, but I was wondering whether there already exists any friendly framework or language which enables developers to create interactive interfaces (for teaching/learning purpouses) in short ammount of time. Usually I write applications for PC environment in .NET but in this case it would take too much time to create a specific interface for every exercise.
I would appreciate if anyone could suggest any way to create interactive user-interface in short ammount of time. Are there any special programming languages or development tools for this kind of applications or are there any useful frameworks for .NET, Java or any other language to speed up the development of user-interfaces?
Thank you!
Perhaps sketchflow could be of interest
although more intended as a tool for mocking up GUI's. It does also have the ability to hook up behaviours and other short snippets of code to your testgui.
Take a look at Mockingbird. I'm not sure if it's exactly what you are looking for, but it is a pretty sweet web application and works well for the development of UIs (at least the wireframes).
There are lots of tools for this purpose -- but most of them only work well if your user interface follows one of a few fairly well-defined (and widely used) patterns. Just for example, if you're creating a database front-end, there are quite a few tools to make the job quite easy.
One often-overlooked tool for this kind of job is Visio. Visio is normally thought of primarily for drawing and diagramming, but you can also attach behaviors to objects so double-clicking an object can run a macro of your choice, opening forms, doing calculations, etc. The macros are (at least normally) written in VBA, which probably isn't the greatest choice for big projects, but as you said, for a situation like this you're mostly interested in short snippets.
There are, of course, a number of downsides to this, such as the inability (at least AFAIK) to deploy such a diagram as a web-based application. Like Henry Ford's "they can have any color they want as long as it's black", you can respond to any user action you want as long as it's a double-click.
Microsoft expression blend might bee something cool too
http://www.microsoft.com/france/expression/default.aspx
Does anyone know of any tools out there that will let me run and debug a VXML application visually? There are a ton of VXML development tools, but they all require you to build your application within them.
I have an existing application that uses JSPs to generate VXML, and I'm looking for a way to navigate through and debug the rendered VXML in much the same way that Firebug allows one to do this with HTML. I have some proxy-like tools that let me inspect the rendered code as it is sent to the VXML browser, but there's a ton of JS, which makes traversing the code by hand rather difficult.
Has anyone worked with a product that allows for this?
Thanks!
IVR Avenger
There is JigSaw Test suite - has free trial license and reasonably priced.
There is IBM's debugger - part of WebSphere Voice Toolkit.
Many other products have debuggers - a very good summary is here
Disclaimer: I am the development manager for Voiyager (www.voiyager.com), a VoiceXML testing tool. It doesn't meet your criteria nor do I believe it is the type of tool you want, but I thought it was worth mentioning it.
As far as I know, there isn't such a test tool for VoiceXML. In fact there are very few VoiceXML tools on the market and hardly any of them test or analysis. The vendors that created development tools, have all been acquired by other companies. Some of them offered did offer various forms of debugging that were specific to their tool set or stayed at the Dialog (caller input) level. From your question, I'm assuming you need much lower level debugging capabilities.
I think the alternative paths are minimal and somewhat difficult. I believe your primary goal is to debug or rewrite an existing application, but you haven't provided any specific challenges beyond the JavaScript. Some thoughts or approaches that may help:
Isolate the JavaScript and place the code into a unit test harness. That will go a long way to understanding the logic of the application. Any encapsulation of the JavaScript you perform will probably go a long way towards better code maintainability.
Attempt to run the VoiceXML through a translation layer to HTML so you could use FireBug. The largest challenge would involve caller input (ie processing the SRGS grammars). You could probably cheat this by just having the form accept a JSON string the populates the field values. There are tools on the market to test grammars. Depending on the nature of your problems, you could take a simple and light approach and attempt this over just the trouble areas.
Plumb the application with a lot of logging. This can be done through the VoiceXML LOG element, or push the variable space back to the server. By adding intermediate forms, you may be able to provide a dump from each via the VoiceXML Data element.
See if your application will run in one of the open source VoiceXML browsers (not sure of the state of the open source browsers as we've built and bought for our various product lines). If you can get it mostly working, you can use the development debugger to provide some ability to step through the logic. However, it is probably one of the more difficult paths as you'll really need to understand the browser to know when and where to stick your breakpoints and to figure out how to expose the data you want.
Good luck on the challenge. If you find another approach, I would be interested in seeing it posted.
An alternative debug env is to use something like Asterisk with a voicexml browser plugin like the one from http://www.voiceglue.org/ or for a limited licence, i6net.
You can keep all the pieces separate(dynamic html and vxml application in php/jsp/j2ee/, tts processing, and optional asr processing as separate virtual machines with something like virtualbox. If the logic can be kept the same, then it is just a matter of changing the UI based on the channel.
A softphone is all you need to call a minimal asterisk machine, which has the voicexml browser with the url of the vxml in the call plan.
I just used Zend Framework as php is used in this environment, and changed view suffixes(phtml vs vxml) based on the user-agent string.
Flite for tts is fine for debugging, and when your app is ready you can either record phrases, and there was a page on the ubuntu forums with directions for how to increase flite quality with some additional sound files.
Do you have tried Eclipse VTP or InVision Studio?
Eclipse VTP
This is Eclipse plugin. But I feel that it is user-unfriendly a little (of Japanese viewpoint).
InVision Studio *Required create user account*
This is Convergys's IVR tool. It has to edit standard VXML mode. (Unfortunately, It's not exact matching.)
For just debugging vxml, I use Nuance Cafe's VoiceXML checker. It doesn't give you a visual tree or anything, but it's pretty good at spotting syntax errors and is free. I think they might also have more advanced debugging tools if you look into it, but I haven't had the need. (Note: I have no association with them)
http://cafe.bevocal.com/tools/vxmlchecker/vxmlchecker.jsp
I'm looking for the same problem that most of the links are down. I found a document where they propose an open source solution, which works as a plugin for Asterisk (https://www.researchgate.net/publication/228873959_Open_Source_VoiceXML_Interpreter_over_Asterisk_for_Use_in_IVR_Applications) and is available at https://sourceforge.net/projects/voxy/
I would like to know if there are current options to create a VXML structure graphically, like the next image.
Background
I'm at the planning stages of a DIY project that'll help me automate some hardware at my house. It's probably also worthwhile to mention that I've got almost no experience with web-related development.
The Basics
http://img7.imageshack.us/img7/4706/drawingo.png -- I can't seem to embed the diagram.
In order to simplify management, I want to implement my UI in the browser.
The meat of my application will reside inside a Windows service or Linux daemon; this does not mean, however, that I'm after a cross-platform solution -- I'm not tied to any particular platform, so I'll pick one (probably based on the responses that I get) and stick with it.
I would prefer to use "free" tools (e.g., LAMP/WAMP), but it's not a deal breaker.
It would be nice to be able to communicate back to the user that some action is in progress (I think AJAX would be one way to go?)
Questions
The only thing that's not entirely clear to me is the implementation of step № 3. I'd like to hear possible implementation ideas (on Windows or Linux) as to how this should be done. Hopefully some of you can share how this sort of thing is done in the real world.
Miscellaneous
As always, if there's a problem with my thinking, please point it out!
There are many people better qualified to help with step 3 so I'll leave that to them.
My question is whether is you are looking forward to learning the mess of web technologies required for the front end or consider it a necessary evil on the way to what you really want to accomplish? If the latter (and assuming you are working in C/C++) consider taking a look at WT. It's a toolkit that makes the developing the web interface seem more like a desktop gui while handling much of the ugliness for you. It could potentially cut a lot of time off your development.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
We are working on a large project with a measure of new/modified GUI functionality. We've found in the past that we often introduced new problems in related code when adding new functionality.
We have non-technical users perform testing, but they often miss parts and allow bugs to slip through.
Are there any best practices for organizing the UI testing of a WinForms project? Is there any way to automate it?
There are GUI testing tools that will click buttons and stuff for you but they're pretty fragile in my experience.
The best thing to do is to keep your UI layer as thin as possible. Your event handler classes should optimally be only one or two lines that call out to other more testable classes. That way you can test your business logic in unit tests without having to actually do a button click.
You can automate GUI testing using White framework.
Also consider using TDD friendly design, i.e. use MVP/MVC pattern.
I would highly recommend you to read documentation from Microsoft patterns&practies teams.
Especially have a look at the Composite UI application block and CompositeWPF.
These projects specifically designed to give you best practices in GUI apps development including test driven UI.
Keep the GUI layer as thin as possible. Michael Feathers article, The Humble Dialog Box, is a classic. Also check out Martin Fowler's Passive View. I have also heard that the "automatic button clickers" are fragile, and that it's easy to spend more time maintaining the test than you spend maintaining the code.
In the event that someone finds this useful:
List of GUI testing tools found on Wikipedia.
The following book is an introduction to the subject.
There are as many ways as there are developers out there..
http://pragprog.com/titles/idgtr/scripted-gui-testing-with-ruby
There are many tools and libraries available that can automate WinForms testing, ranging from open source solutions like White to the expensive commercial solutions such as HP QuickTest Pro. There is also the UIAutomation namespace in .NET if you want to roll your own automation framework. But the real cost of automation is in the time and specialised skills it requires to implement. Maintainability is also one of the most important aspects of automated test design; you dont want to expend excessive resource keeping the automation assets current with your application. There are also lots of factors influencing the decision to automate which will be specific to your specific application and organisation.
Your best bet will be to do some more research on the subject and check out some of the specialised testing sites such as http://www.sqaforums.com.
I found this quick and dirty way to test web page layouts in various browsers. It's called browsershots.org. Our client requires support in 5 browsers right now and that takes about a week for full regression testing. This service will deliver screenshots of some 70+ browsers and versions. I print them out and hold up the pages to the light. If they don't line up, there must be a layout problem.
I can't really help with organization or best practices, but an NUnit extension appears to be available for forms testing, NUnitForms.
I used a trial download similar to this product (http://www.tethyssolutions.com/product.htm) and this product (http://www.mjtnet.com/macro_scheduler.htm) years ago and I was happy with the results. These are pretty cheap solutions and some of these macro recorder products can actually be used for automated testing.
A new method available now is using Ruby, via the Ruby gem called win32-autogui. This provides a framework for testing Windows GUI apps. Combine it with Ruby tools RSpec and Cucumber, and it makes for quite a powerful testing framework.
How to organize UI testing depends on how you design the test cases.
Automating Windows Forms application in unit test level can use TDD framework, such as NUnit; or use BDD framework, such as NSpec.
Automating Windows Forms applications in functional testing level can use White, CodedUI, or even directly using Windows Automation API 3.0 (UI Automation and MSAA).
Since all these stuffs mentioned above are technologies rather than solutions, it's better to build some fundamental test automation/scripting frameworks based on these technologies before you start to write automated tests.