I would like to know how test automation tools like winrunner, QTP etc work. Whether these tool use any test API provided by windows or they depened on IPC and events. I could not figure out how they work. For me QTP record and play feature seems to be a magic.Any guidance will be highly appreciated?
Actually WatiN is not bad place to start. It is not exactly unit testing framework. It may look like one at firs sight but it is used to write functional test. It is totally up to you if you run them like unit tests. I was writing the same test using WatiN in two ways (functional test that was simulating user actions on the web page):
1) Writing script in Powershell at running from command line like any other PS script. It was fun, although you need to write lots of code for reporting, exception handling, and other.
2) Writing unit test in MSVS in C# using C# Unit Test from MSVS project type. This actually was fun because you would just run it in MSVS like unit test but you have environment support for code writing, reporting, running etc.
So if you want to start with something take a look at WatiN, especially as WatiN has dedicated recorder that records actions and outputs code in C#. Looking at internals of library and tool would give you some start.
Just one thing to mention - it is web only. Desktop is totally different. With web you have hook into IE you can query html document for objects, checking browser state and so. With desktop it may be more difficult. You need to hook into application, maybe via mentioned Win32hooks. Maybe try with Microsoft Scripting Host.
I dont know much about QTP. But if you wanted to know the internals then you can download the open source projects like Watin- for dotnet and Watir- for ruby and see whats happening inside. Both are used for web test automation. And the code is freely available..
If you are looking for Unit testing frameworks like Nunit. they are attribute driven.. Nunit identifies the classes by "TestFixtures" and methods by "Test" attributes. It scans the entire application for these test methods and trigger the tests.
I guess they're using Win32 Hooks.
edit: here is an example of defining hooks in .NET.
It should be Win32/64 Programming in C, C++ using the technologies COM, OLE, and they might be using Active accesibility API. Definitely they might have created a big API on these technologies. Even they might have used a bit of Assembly programming as well.
Related
Can BDD (Behavior Driven Design) tests be implemented using a UI driver?
For example, given a web application, instead of:
Writing tests for the back-end, and then more tests in Javascript for the front-end
Should I:
Write the tests as Selenium macros, which simulate mouse-clicks, etc in the actual browser?
The advantages I see in doing it this way are:
The tests are written in one language, rather than several
They're focussed on the UI, which gets developers thinking outside-in
They run in the real execution environment (the browser), which allows us to
Test different browsers
Test different servers
Get insight into real-world performance
Thoughts?
We've done this for a C# application using a WPF testing tool (WipFlash) and writing NUnit tests in a BDD-like fashion.
e.g.
Given.TheApplicationWindowIsOpen();
When.I.Press.OKButton();
The.Price.ShouldBeCalculated();
We had to code a lot of the DSL ourselves, needless to say. But it becomes a business/customer readable solution.
Try using SpecFlow with WatiN: (I'm not sure if you're using .NET here)
http://msdn.microsoft.com/en-us/magazine/gg490346.aspx
For web testing, you could try WebDriver. The Selenium team are busy integrating WebDriver at the moment. Simon Stewart from Google, who created WebDriver, blogged here about how it works differently to Selenium.
WebDriver uses different technologies for each browser. For Internet Explorer, WebDriver uses Microsoft's UI automation - the same technology on which WipFlash which #Brian Agnew mentioned is based. This is as close as you'll get to pretending to click buttons. Simon's blog shows why this approach can be more powerful than Selenium's Javascript solution.
WebDriver is available from the Selenium site but hasn't been fully implemented as part of Selenium yet.
For BDD, and any use-case driven tests, it is important to be able to communicate what a test is doing. The problem with many test suites is that post-writting nobody is quite certain exactly what the test is doing. This will come up very often if you write in a non-specialized language. Specialization doesn't necessarily mean a special language, but just enough of an abstraction in the one language so it is clear what is happening.
For example, a lot of tests have code that looks like this (pseudo-code, I won't pick on any particular framework):
object = createBrowser()
response = object.gotoURL( "http://someurl.com" );
element = response.getLink( "Click Here" );
response = element.doClick();
This is hard for somebody to quickly translate to a business driver (product manager perhaps, or user). Instead you want to create specialized functions, or a language if you're adventurous, so you can have this:
GotoURL http://someurl.com/
Click link:Click Here
Selenium, and its macros or interface, are still fairly low-level in this regards. If you do use them then at least build some wrappers around them.
You can of course also use a product called TestPlan. It has Selenium in the back-end and exposes a high-level API, and a custom langauge for testing. It also goes beyond just the web to included Email, FTP, etc. The sample language above is a TestPlan snippet
You can certainly do some of your acceptance tests this way, but I think most BDD advocates would not advise using this for all tests. And of course, true BDD advocates wouldn't call them tests...
The RSpec Book advocates a two-level cycle with acceptance tests (or Scenarios) written first (primarily in Cucumber), and unit tests written (in RSpec) in an inner cycle more resembling the traditional TDD.
The outer cycle of acceptance testing can also use tools like Selenium to drive the entire application through the UI (and the authors of The RSpec Book spend a chapter on this). But it is not appropriate for unit tests.
Tests exercising the entire application through the UI are harder to make repeatable, and have a tendency to be slower and more fragile than unit tests.
Actually you could do both - make a user-centric Driver interface (agnostic of GUI / tech / impl). You could then write a UIDriver and a APIDriver and choose a driver to run a specific test. Running through the UI is usually slower (out of proc, control repaints but somehow creates a higher level of confidence initially). Running through the API is much faster (in proc, easy setup-teardown).
The trick here is to separate the What from the How. Otherwise you will end up with ObscureTests and high test maintenance. Ensure the primary focus on testing and not automation.
I would like to know how to go about doing regression testing effectively for a web application.
I m spending a lot of time in regression testing for every build.
I also hear that automated GUI testing is bad and not worth it.
Is there any better way of automating regression testing. if so any good tools to automate GUI tests.
My web app is designed for stupid IE. cant help it, but that how it is.Any good tool that can help me?
I know that we must keep GUI layer thin and all, but I m not a GUI expert or anything , but i need to test it properly and effectively.
Thanks all
Try looking into Cucumber or Selenium
There is also a screencast on it over at railscasts.
That should really help.
You can use standard unit tests and integration tests for the non-GUI components, of course. For the GUI, it's worth looking at UI automation packages.
Many of them are pretty terrible, but a lot of the horror stories come from people using them incorrectly. If you can stay away from specifying click locations by (x,y) coordinates, and you can detect when state transitions happen, rather than using delay(), GUI testing can be effectively automated.
As long as your webapp works with Firefox, too, I recommend using Selenium for that scenario. It allows you to record your test cases inside the browser (by means of a Firefox plugin) but has Internet Explorer support for playback.
If Firefox for recording tests is not an option, you could have a look at the WebDriver project. It uses a code based approach to test specification an works well with TDD, too.
You might also want to have a look at CubicTest, which is a Eclipse plugin for graphically defining Selenium and Watir test cases.
Does anyone know of any tools out there that will let me run and debug a VXML application visually? There are a ton of VXML development tools, but they all require you to build your application within them.
I have an existing application that uses JSPs to generate VXML, and I'm looking for a way to navigate through and debug the rendered VXML in much the same way that Firebug allows one to do this with HTML. I have some proxy-like tools that let me inspect the rendered code as it is sent to the VXML browser, but there's a ton of JS, which makes traversing the code by hand rather difficult.
Has anyone worked with a product that allows for this?
Thanks!
IVR Avenger
There is JigSaw Test suite - has free trial license and reasonably priced.
There is IBM's debugger - part of WebSphere Voice Toolkit.
Many other products have debuggers - a very good summary is here
Disclaimer: I am the development manager for Voiyager (www.voiyager.com), a VoiceXML testing tool. It doesn't meet your criteria nor do I believe it is the type of tool you want, but I thought it was worth mentioning it.
As far as I know, there isn't such a test tool for VoiceXML. In fact there are very few VoiceXML tools on the market and hardly any of them test or analysis. The vendors that created development tools, have all been acquired by other companies. Some of them offered did offer various forms of debugging that were specific to their tool set or stayed at the Dialog (caller input) level. From your question, I'm assuming you need much lower level debugging capabilities.
I think the alternative paths are minimal and somewhat difficult. I believe your primary goal is to debug or rewrite an existing application, but you haven't provided any specific challenges beyond the JavaScript. Some thoughts or approaches that may help:
Isolate the JavaScript and place the code into a unit test harness. That will go a long way to understanding the logic of the application. Any encapsulation of the JavaScript you perform will probably go a long way towards better code maintainability.
Attempt to run the VoiceXML through a translation layer to HTML so you could use FireBug. The largest challenge would involve caller input (ie processing the SRGS grammars). You could probably cheat this by just having the form accept a JSON string the populates the field values. There are tools on the market to test grammars. Depending on the nature of your problems, you could take a simple and light approach and attempt this over just the trouble areas.
Plumb the application with a lot of logging. This can be done through the VoiceXML LOG element, or push the variable space back to the server. By adding intermediate forms, you may be able to provide a dump from each via the VoiceXML Data element.
See if your application will run in one of the open source VoiceXML browsers (not sure of the state of the open source browsers as we've built and bought for our various product lines). If you can get it mostly working, you can use the development debugger to provide some ability to step through the logic. However, it is probably one of the more difficult paths as you'll really need to understand the browser to know when and where to stick your breakpoints and to figure out how to expose the data you want.
Good luck on the challenge. If you find another approach, I would be interested in seeing it posted.
An alternative debug env is to use something like Asterisk with a voicexml browser plugin like the one from http://www.voiceglue.org/ or for a limited licence, i6net.
You can keep all the pieces separate(dynamic html and vxml application in php/jsp/j2ee/, tts processing, and optional asr processing as separate virtual machines with something like virtualbox. If the logic can be kept the same, then it is just a matter of changing the UI based on the channel.
A softphone is all you need to call a minimal asterisk machine, which has the voicexml browser with the url of the vxml in the call plan.
I just used Zend Framework as php is used in this environment, and changed view suffixes(phtml vs vxml) based on the user-agent string.
Flite for tts is fine for debugging, and when your app is ready you can either record phrases, and there was a page on the ubuntu forums with directions for how to increase flite quality with some additional sound files.
Do you have tried Eclipse VTP or InVision Studio?
Eclipse VTP
This is Eclipse plugin. But I feel that it is user-unfriendly a little (of Japanese viewpoint).
InVision Studio *Required create user account*
This is Convergys's IVR tool. It has to edit standard VXML mode. (Unfortunately, It's not exact matching.)
For just debugging vxml, I use Nuance Cafe's VoiceXML checker. It doesn't give you a visual tree or anything, but it's pretty good at spotting syntax errors and is free. I think they might also have more advanced debugging tools if you look into it, but I haven't had the need. (Note: I have no association with them)
http://cafe.bevocal.com/tools/vxmlchecker/vxmlchecker.jsp
I'm looking for the same problem that most of the links are down. I found a document where they propose an open source solution, which works as a plugin for Asterisk (https://www.researchgate.net/publication/228873959_Open_Source_VoiceXML_Interpreter_over_Asterisk_for_Use_in_IVR_Applications) and is available at https://sourceforge.net/projects/voxy/
I would like to know if there are current options to create a VXML structure graphically, like the next image.
I had just started working on some automated testing tools, namely test complete, which I would be using along with test partner. So I had also started learning VB scripts for this.
I just want to know if anyone could provide me a link so that I could learn this tool, as well as VB scripting more effectively.
Microsoft
w3schools
devguru
Windows Scripting Host
The Test Complete Help can sometimes be helpful, but not usually.
Test complete itself has keywords for VB. You can use them or you can define your own keywords by giving it keywords.
It has a tool like object spy with which you can spy on objects properties and make your own repository, through which your automation can easily be done.
When I was using Test Complete a few years ago, we mainly programmed in 'C# script' really was very much JavaScript like, but they had lots of examples on the AutomatedQA website.
That and we recorded scripts of us clicking on things then looked at what that generated.
The third thing we did a lot use their UI inspector widget, to find what was the name of UI components, and there properties.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
What are the good automated web UI testing tools?
I want to be able to use it in the .Net world - but it doesn't have to written in .net.
Features such as a record mode, integration into build process\ continuous integration would be nice.
Im going to look at:
Watir
Selenium
Are there any others I should look at?
I definitively recommend Selenium, you can use it from .NET, supports different browsers, works in automatic builds and CI processes (we use it from CCNet). The code is stable. It has a few quirks, but after all they all do.
Whichever tool you choose, I recommend making your own test facade class(es) around it. The facade should be designed to suite your concrete testing needs, without exposing too much the details of the testing tool's API. This will make the test code easier to write and maintain.
UPDATE: if you use ASP.NET view state in your app, you could have problems using a pure HTTP test tool. This is where browser-controllers (like Selenium) are much better.
WatiN
Automates FF and IE
[Test]
public void SearchForWatiNOnGoogle()
{
using (IE ie = new IE("http://www.google.com"))
{
ie.TextField(Find.ByName("q")).TypeText("WatiN");
ie.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(ie.ContainsText("WatiN"));
}
}
http://watin.sourceforge.net/
Watin is pretty unstable to use it in serious projects. It often fails with unexpected reasons like "EI is busy" or something like "Error with COM object".
Selenium is much more stable and it already has a lot of supporting tools. For example Selenium GRID is a solution which allows significantly decrease run time of tests. (Our smoke tests on Watin takes 6 hours to run).
Currently in my job i use QTP and it so far atleast can handle pretty much anything we throw at it both on the UI and it has a special mode for testing non gui services allowing us to check both and help us narrow down where some problems occur when we change the system. It is in my opinion very configurable and the inclusion of vbscript as its language allows integration with lots and lots of things on windows to allow you to do pretty much anything you want! For instance we use it to control the excel com object to make custom excel reports of success and failure so the format of the results is the same wether a test was run manually and also on another project used the adodb object to check that when a page submits information to the database that the database contains the correct data for that record!
As for integration into the build process i have not tried this myself but it is possible to launch qtp and a test from a vbs file so i would assume this should be fairly trvial as ms tools tend to allow you to run vbs files pretty easily from most tools.
I would reccomend it to anyone assuming you can get someone to buy the license!
You can also try VSTT - http://blogs.msdn.com/b/slumley/archive/2009/05/28/vsts-2010-feature-enhancements-for-web-test-playback-ui.aspx
Telerik Test Tools - http://www.telerik.com/automated-testing-tools.aspx
Visual Studio UI Test Extensibility–Scenarios & Guiding Principles - http://blogs.msdn.com/b/mathew_aniyan/archive/2011/03/28/visual-studio-ui-test-extensibility-scenarios-amp-guiding-principles.aspx
VSTS Web Test Step-by-Step Primer - http://blogs.msdn.com/b/jimmymay/archive/2009/02/23/vsts-web-test-step-by-step-primer-7-minute-video-by-microsoft-a-c-e-performance-engineer-chris-lundquist-with-copious-notes-screen-shots-from-your-humble-correspondent.aspx
you might also be interested in taking a look at what the ASP.NET team cooked up itself: Lightweight Test Automation Framework.
There's also a dedicated forum for it.
Having used several different automated testing solutions (TestComplete, QTP, etc), I have to put a vote in for Telerik + Visual Studio. Telerik has great support forums, and is very compatible with whatever testing framework you come up with. Our Devs put unique IDs into their HTML code so that our scripts don't need to be rewritten even with pretty drastic UI refactors. It's definitely more challenging than record and playback, but once you have your unique IDs in place, the automation code requires little or no maintenance.
Try httpunit
Depend on what you would like to achieve.
You can use web test built in the Visual Studio Tester Edition. It's quite good and easy to automate. You can use external data as a test data source and it integrates nicely with VS.
There is also test tool by Automated QA (forgot the name) which looks good but expensive.
And there is Selenium. That's the one we are using in Symantec. The biggest advantage is that it actually uses a browser you want to test. VS mimic a browser by changing http request parameters only so you may not be able to test your site for cross-browser compatibility. Selenium on the other hand uses browser and automates it so you can actually test your site in IE, Firefox etc. It can be also integrated with VS unit tests so you can see test results in VS.
So I would recommend Selenium or VS.
I've used Selenium. The features were good, and it was usable but it was buggy.
The IDE would often record events incorrectly (so tests would need to be manually changed), and test files occasionally became completely unusable for no apparent reason, which meant they would have to be recreated all over again. Also development on Selenium IDE seems to have stopped; there hasn't been any bug fixes and patches for a while, and bug reports seem to go unnoticed.
Molybdenum is an alternative, built on Selenium that's worth looking into.
http://www.molyb.org/
Just to throw out another option (of which I haven't tried but I do like Telerik) is Telerik's new WebUI Testing Studio. I will also echo Selenium up votes.
I forget one nice tools and can find link on it but find this ... http://weblogs.asp.net/bsimser/archive/2008/02/21/automated-ui-testing-with-project-white.aspx maybe can help.
If you are looking for simple, cross-browser tool with record and playback, multithreaded playback, intergration with build processes, powerful scripting, good reporting and excellent support, go for Sahi. It will be much easier for your testers/devs to learn and maintain.
you might want to take in consideration near Selenium also Rational Functional Tester ! whether you are familiar with coding in .Net or Java and want to just play around with record & replay or want to create more sophisticated programmatic testing I would recommend it.
WebDriver is another possibility: http://code.google.com/p/webdriver
They are working on a .NET wrapper that may be interesting for you.
Try QEngine. It has all the features of QTP.
You may want to look at RIATest for cross-platform cross-browser testing of web applications.
It works on Windows and Mac, supported browsers are Firefox, IE and Chrome. Automated testing scripts written on one platform/browser can be run against all other supported platforms/browsers.
It has the features that you want: user interaction recording mode and integration with CI servers (outputs results in JUnit format which can be consumed by CI servers such as Hudson).
(Disclaimer: I am a RIATest team member).