I can find some articles on how to use them but I can't seem to find anywhere peoples impression of them using them in real projects. I have been trying to figure out how to use them and I've had alot of problems.
Can someone out there who uses these tools on the job give me thier impression? Are there better alternate tools available? Using these really just a waste of time?
With Coded UI Tests I see how they are good for basic javascript checking but its so basic of a example I don't think it is worth it. With web tests I like how they work but when I activate code coverage/ASP.NET profiling it doesnt work half the time.
I tried both Selenium and Coded UI for integration testing the project I am working on. I wrote a blog article with my impressions.
The bottom line for me is that both Selenium and Coded UI are still far from perfect, but better than manual testing. I am using Selenium right now, though I might revisit CodedUI if a few things would be fixed in a later version.
experience with Selenium and VS Coded UI test for web apps.
Better than manual testing in my eyes, though the tests are at times hard to maintain and often need to be changed manually to make them more robust. Both tools have browser plugins which can generate tests automatically, but this code is generally not very resilient towards changes in the HTML (in case of web apps).
You will be disappointed if you expect an automated test run orders of magnitude faster than a manual tester. I think the aim of the vendors is to get the automated test execute aboit twice as fast as a manual tester (that can feel painfully slow if you have a large set of tests).
I have used both selenium 1.0 (currently deprecated 2.0 for my dev environment) and visual studio 2010 web testing. There is no doubt, vs 2010 web testing tool is by far (and i mean by far) better than selenium 1.0 and 2.0 (in beta now) in terms of tooling, robustness and the different kinds of metric you can measure (load balance analysis etc). The one good thing also is that it can run your tests in firefox and other browsers as well(provided you have those browsers installed - i believe. Not sure on that one). Selenium is good and have been around for a while. Has a lot of community support but... it can be frustrating. They are currently in transition to 2.0 so I think things will get better and improve after 2.0. In short, if your development environment is .NET with vs 2010, you will get the best ROI by using of the web testing tools.
Disclaimer: I don't work for Microsoft and I am in no way affiliated with MS. My comment above is based on my experience and my only.
As with (most) all Microsoft tools they will do the job--provided you read plenty of materials to understand how to use them and you understand their limitations. If you are looking for full-featured, high-quality, relatively intuitive tools then look to RedGate.
Microsoft tools will do everything, but nothing terribly well.
Related
I am starting a maintenance project where I need to work with legacy code and also create new ones. New ones I can create proper interface based development and I can use Moq to do proper unit testing. I can't use Moq against the legacy code since they do not have proper coding to mock out the objects. Based on the reading, Typemock looks like an excellent fit since I can isolate the object and return what we want when a method is called on legacy objects. I really like the simplicity and ease of development. I would like to know is there any other tools out there like Typemock I should look at before committing to it since I have to pay for it.
Thanks
PS: ours is Microsoft shop we work with C#/ASP/ASp.Net/Silverlight and VB.Net
** I just found that Infragistics have one mocking tool as well.
What you're looking for is a mocking framework that uses the Profiler API. These are the ones that can "mock anything" because they sort of run "underneath" the code being tested.
When considering one of these, you'll want to think about things like:
Integration support: Does it work with the tools you're using? For example, your build server, coverage tool, unit testing tool, etc. Are there any additional tools it comes with (e.g., Visual Studio plugins) that might help you work with it?
Technical support: If something goes wrong or you find a bug in the product, can you ask someone about it?
Syntax/patterns: Does it support AAA testing or is it only record/replay? How easy is it to start working with?
Community and learning resources: How's the documentation? Are there people out there who can answer your more general usage questions or are you "on your own?"
There are three primary mocking frameworks to consider that I'm aware of:
Typemock Isolator - The most mature with a free license for open source projects, but otherwise not free.
Telerik JustMock - New to the scene, but good features. There is a free/limited version.
Moles - Free from Microsoft, though somewhat less easy to integrate with tools outside the VSTS realm.
Full disclosure: I'm a Typemock MVP and I've been using Isolator for a while. It has thus far been worth its weight in gold, particularly in Brownfield development. I will admit in some cases, where it's more Greenfield, I use Rhino or Moq. It really depends on what I'm doing.
In any case, I recommend evaluating each of the tools yourself to see which works best in your environment. Be sure to see how it integrates, how easy it is to get going, etc. for each one, and then make your decision based on your experience in your setting.
I would like to know how to go about doing regression testing effectively for a web application.
I m spending a lot of time in regression testing for every build.
I also hear that automated GUI testing is bad and not worth it.
Is there any better way of automating regression testing. if so any good tools to automate GUI tests.
My web app is designed for stupid IE. cant help it, but that how it is.Any good tool that can help me?
I know that we must keep GUI layer thin and all, but I m not a GUI expert or anything , but i need to test it properly and effectively.
Thanks all
Try looking into Cucumber or Selenium
There is also a screencast on it over at railscasts.
That should really help.
You can use standard unit tests and integration tests for the non-GUI components, of course. For the GUI, it's worth looking at UI automation packages.
Many of them are pretty terrible, but a lot of the horror stories come from people using them incorrectly. If you can stay away from specifying click locations by (x,y) coordinates, and you can detect when state transitions happen, rather than using delay(), GUI testing can be effectively automated.
As long as your webapp works with Firefox, too, I recommend using Selenium for that scenario. It allows you to record your test cases inside the browser (by means of a Firefox plugin) but has Internet Explorer support for playback.
If Firefox for recording tests is not an option, you could have a look at the WebDriver project. It uses a code based approach to test specification an works well with TDD, too.
You might also want to have a look at CubicTest, which is a Eclipse plugin for graphically defining Selenium and Watir test cases.
Does anyone know of any tools out there that will let me run and debug a VXML application visually? There are a ton of VXML development tools, but they all require you to build your application within them.
I have an existing application that uses JSPs to generate VXML, and I'm looking for a way to navigate through and debug the rendered VXML in much the same way that Firebug allows one to do this with HTML. I have some proxy-like tools that let me inspect the rendered code as it is sent to the VXML browser, but there's a ton of JS, which makes traversing the code by hand rather difficult.
Has anyone worked with a product that allows for this?
Thanks!
IVR Avenger
There is JigSaw Test suite - has free trial license and reasonably priced.
There is IBM's debugger - part of WebSphere Voice Toolkit.
Many other products have debuggers - a very good summary is here
Disclaimer: I am the development manager for Voiyager (www.voiyager.com), a VoiceXML testing tool. It doesn't meet your criteria nor do I believe it is the type of tool you want, but I thought it was worth mentioning it.
As far as I know, there isn't such a test tool for VoiceXML. In fact there are very few VoiceXML tools on the market and hardly any of them test or analysis. The vendors that created development tools, have all been acquired by other companies. Some of them offered did offer various forms of debugging that were specific to their tool set or stayed at the Dialog (caller input) level. From your question, I'm assuming you need much lower level debugging capabilities.
I think the alternative paths are minimal and somewhat difficult. I believe your primary goal is to debug or rewrite an existing application, but you haven't provided any specific challenges beyond the JavaScript. Some thoughts or approaches that may help:
Isolate the JavaScript and place the code into a unit test harness. That will go a long way to understanding the logic of the application. Any encapsulation of the JavaScript you perform will probably go a long way towards better code maintainability.
Attempt to run the VoiceXML through a translation layer to HTML so you could use FireBug. The largest challenge would involve caller input (ie processing the SRGS grammars). You could probably cheat this by just having the form accept a JSON string the populates the field values. There are tools on the market to test grammars. Depending on the nature of your problems, you could take a simple and light approach and attempt this over just the trouble areas.
Plumb the application with a lot of logging. This can be done through the VoiceXML LOG element, or push the variable space back to the server. By adding intermediate forms, you may be able to provide a dump from each via the VoiceXML Data element.
See if your application will run in one of the open source VoiceXML browsers (not sure of the state of the open source browsers as we've built and bought for our various product lines). If you can get it mostly working, you can use the development debugger to provide some ability to step through the logic. However, it is probably one of the more difficult paths as you'll really need to understand the browser to know when and where to stick your breakpoints and to figure out how to expose the data you want.
Good luck on the challenge. If you find another approach, I would be interested in seeing it posted.
An alternative debug env is to use something like Asterisk with a voicexml browser plugin like the one from http://www.voiceglue.org/ or for a limited licence, i6net.
You can keep all the pieces separate(dynamic html and vxml application in php/jsp/j2ee/, tts processing, and optional asr processing as separate virtual machines with something like virtualbox. If the logic can be kept the same, then it is just a matter of changing the UI based on the channel.
A softphone is all you need to call a minimal asterisk machine, which has the voicexml browser with the url of the vxml in the call plan.
I just used Zend Framework as php is used in this environment, and changed view suffixes(phtml vs vxml) based on the user-agent string.
Flite for tts is fine for debugging, and when your app is ready you can either record phrases, and there was a page on the ubuntu forums with directions for how to increase flite quality with some additional sound files.
Do you have tried Eclipse VTP or InVision Studio?
Eclipse VTP
This is Eclipse plugin. But I feel that it is user-unfriendly a little (of Japanese viewpoint).
InVision Studio *Required create user account*
This is Convergys's IVR tool. It has to edit standard VXML mode. (Unfortunately, It's not exact matching.)
For just debugging vxml, I use Nuance Cafe's VoiceXML checker. It doesn't give you a visual tree or anything, but it's pretty good at spotting syntax errors and is free. I think they might also have more advanced debugging tools if you look into it, but I haven't had the need. (Note: I have no association with them)
http://cafe.bevocal.com/tools/vxmlchecker/vxmlchecker.jsp
I'm looking for the same problem that most of the links are down. I found a document where they propose an open source solution, which works as a plugin for Asterisk (https://www.researchgate.net/publication/228873959_Open_Source_VoiceXML_Interpreter_over_Asterisk_for_Use_in_IVR_Applications) and is available at https://sourceforge.net/projects/voxy/
I would like to know if there are current options to create a VXML structure graphically, like the next image.
Or in another way, is there any reason to install NUnit or any other library and use it instead of the one that comes with Visual Studio 2008?
We have been using MS test tools since VS2005, and I did look at NUnit as a replacement at one point because of some problems we had with the MS tools. Here are some of my findings:
MS test tools pros
Easy to convince management to use MS supplied tools (of course this is not a technical issue, but it is probably relevant in a lot of companies)
Test libraries, test runner and code coverage are all nicely integrated with VS
Central reporting and other features in a Team Foundation setup
More or less the same feature set as with NUnit (I get the feeling that NUnit may be a tad better than MS test tools, but the differences are not huge)
MS test tools cons
VS2005 - really slow, VS2008 a lot better, but still sluggish for large solutions
Test host is STA which makes it harder to test multi-threaded code
VS support is a bit clunky because they support test-first and test-after in the same interface (so there are two "new unit test" commands -> more support work)
In the end switching to NUnit was not attractive to us for a couple of reasons:
Code changes, not a lot but with a large number of tests it still amounts to a bit of work
Increased cost and complexity per developer seat: if you swap MS Test with NUnit, you also loose the test runner and code coverage. Both can be obtained through other tools such as R#, TestDriven.NET, NCover etc, but it all adds to both the cost and the complexity.
Do you want developers who don't have Visual Studio 2008, or who only have the Express edition, to run the unit tests? This is particularly relevant for Open Source projects. If so, I'd go with one of the open source unit test frameworks.
I think most test projects created before VS2008 will nowadays still use other libraries.
For newer projects it comes down to personal taste I guess, or company policies.
Also, not all visual studio versions support the testing framework I think...
I have just started to use Visual Studio test projects. I have no experience of using thrid party test products, but I have found that the Visual Studio one appears to be perfectly adequate for most points.
I think that it would boild down to what you have experience in.
If you already know how to use NUnit and/or Rhino mocks then go with them.
If you have no experience of either or aren't too bothered about gaining experience in them then go with the Visual Studio supplied one.
A useful pointer is that the Visual Studio one will create test method stubs and accessors automatically for you if you right click and select "Create Unit Test" from anywhere in your code. Particularly useful if you're not following a strict TDD of writing tests before coding.
Some advantages:
(Jon's) You can run your tests in VS 2008 Express
Lot's of cutting edge features get built on top of the free libraries. Eg. BDD
In general there is rich extensability when it comes to the OSS testing frameworks.
You can debug into the source of the OSS frameworks, and make changes if required.
Arguably, some test frameworks like xUnit are easier to learn and enforce a cleaner side-effect free testing pattern.
Given the awesome testdriven.net supports pretty much all of the OSS testing frameworks, you get the same (if not better) IDE integration that you get with VS 2008.
A couple of notes RE MSTest...
1) There is an entire team dedicated to brining features to market (it's literally their full time job). These guys end up having resources for writing documents, integration points into other Microsoft products, discussing advances from the general test community and how these get into Microsoft test products and in general an overall, fully functioning product unit dedicated to the effort. When you go with MSTest, you get the full benefit of these peoples efforts and the dollars Ms invests.
2) Web Testing and Stress Testing. There is yet a WHOLE other dedicated group (separate from the MSTest group) that works on Web testing and stress testing. These are huge additions to the Test stack from Microsoft. These offer ways to simulate online users, use complex bandwidth manipulation, check server performance and much much much more. The Web and Stress testing stuff is a BIG reason to really give the MS Test stack a hard look.
3) Functional testing ... Ms is adding full UI automated testing support and this will really round out their offering. It was missing in terms of earlier versions of MS Test tools, but it must be considered that a fully integrated set of UI testing tools will really amp up the overall stock of an investment in MS test technology.
4) As noted above - Impact Testing. From a developers point of view - this is WAY cool. As mentioned - the tools select the right unit tests passed on code coverage of the unit test. So a dev edits and existing class, etc... and the tool specifically points to the tests that impact the checkin. BTW - this is a GREAT example of the research you gain from that I mentioned in #1.
.... BTW - Full disclosure - I worked on VSTS directly - and totally consider it a part of my family - so yes - I am utterly compromised in terms of favoritism.
As for Non-MS test tools ...
A) You can't disregard the efforts of other companies nor the entire open source community. Really MS still loses if you consider on one hand the entire open source community and all of other company efforts VS several Microsoft product teams. HOWEVER, the MS efforts are mostly coordinated (though not always) and do tend to have at least soft points of integration (though again, not always).
B) Faster turn-around from other vendors / the open source community. Ms is improving - BUT - no - other communities and vendors DO out pace MS in getting things to market. This is not different from test tool innovations.
C) ***** Other testing tools will do a GREAT job with non-ms technology stack stuff. Ms tools will be 'ok' in some cases - and in other cases utterly miss the boat. Many times the big ticket wins from MS tools aren't there for other technology (read NON-MS) technology stacks - ... Other vendors and the open source community (obviously) do provide much more compelling support.
In the end... I'd suggest this: If you're on the MS stack AND have an investment in Visual Studio - particuarly with MSDN - and thus have a major amount of access and investment in buying MS tools .... REALLY, using SOME FORM of MS development and test tools is a no brainer. I didn't even discuss the test tools integration and usage of the rest of VSTS - holy cow.
However - if you're using other stacks, don't have the end-to-end investment in the MS stack / MS provided developer tools ... If you're operating in a shop where other OS's and such are huge deals for desktop or server installs, etc... Then perhaps minor elements of the MS tool set might be worth checking out - but obviously starting in the MS test tools direction will be somewhat more difficult / costly.
Hope this helps.
NUnit allows you to run tests in a defined order whereas the Visual Studio test framework runs the tests concurrently.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
What are the good automated web UI testing tools?
I want to be able to use it in the .Net world - but it doesn't have to written in .net.
Features such as a record mode, integration into build process\ continuous integration would be nice.
Im going to look at:
Watir
Selenium
Are there any others I should look at?
I definitively recommend Selenium, you can use it from .NET, supports different browsers, works in automatic builds and CI processes (we use it from CCNet). The code is stable. It has a few quirks, but after all they all do.
Whichever tool you choose, I recommend making your own test facade class(es) around it. The facade should be designed to suite your concrete testing needs, without exposing too much the details of the testing tool's API. This will make the test code easier to write and maintain.
UPDATE: if you use ASP.NET view state in your app, you could have problems using a pure HTTP test tool. This is where browser-controllers (like Selenium) are much better.
WatiN
Automates FF and IE
[Test]
public void SearchForWatiNOnGoogle()
{
using (IE ie = new IE("http://www.google.com"))
{
ie.TextField(Find.ByName("q")).TypeText("WatiN");
ie.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(ie.ContainsText("WatiN"));
}
}
http://watin.sourceforge.net/
Watin is pretty unstable to use it in serious projects. It often fails with unexpected reasons like "EI is busy" or something like "Error with COM object".
Selenium is much more stable and it already has a lot of supporting tools. For example Selenium GRID is a solution which allows significantly decrease run time of tests. (Our smoke tests on Watin takes 6 hours to run).
Currently in my job i use QTP and it so far atleast can handle pretty much anything we throw at it both on the UI and it has a special mode for testing non gui services allowing us to check both and help us narrow down where some problems occur when we change the system. It is in my opinion very configurable and the inclusion of vbscript as its language allows integration with lots and lots of things on windows to allow you to do pretty much anything you want! For instance we use it to control the excel com object to make custom excel reports of success and failure so the format of the results is the same wether a test was run manually and also on another project used the adodb object to check that when a page submits information to the database that the database contains the correct data for that record!
As for integration into the build process i have not tried this myself but it is possible to launch qtp and a test from a vbs file so i would assume this should be fairly trvial as ms tools tend to allow you to run vbs files pretty easily from most tools.
I would reccomend it to anyone assuming you can get someone to buy the license!
You can also try VSTT - http://blogs.msdn.com/b/slumley/archive/2009/05/28/vsts-2010-feature-enhancements-for-web-test-playback-ui.aspx
Telerik Test Tools - http://www.telerik.com/automated-testing-tools.aspx
Visual Studio UI Test Extensibility–Scenarios & Guiding Principles - http://blogs.msdn.com/b/mathew_aniyan/archive/2011/03/28/visual-studio-ui-test-extensibility-scenarios-amp-guiding-principles.aspx
VSTS Web Test Step-by-Step Primer - http://blogs.msdn.com/b/jimmymay/archive/2009/02/23/vsts-web-test-step-by-step-primer-7-minute-video-by-microsoft-a-c-e-performance-engineer-chris-lundquist-with-copious-notes-screen-shots-from-your-humble-correspondent.aspx
you might also be interested in taking a look at what the ASP.NET team cooked up itself: Lightweight Test Automation Framework.
There's also a dedicated forum for it.
Having used several different automated testing solutions (TestComplete, QTP, etc), I have to put a vote in for Telerik + Visual Studio. Telerik has great support forums, and is very compatible with whatever testing framework you come up with. Our Devs put unique IDs into their HTML code so that our scripts don't need to be rewritten even with pretty drastic UI refactors. It's definitely more challenging than record and playback, but once you have your unique IDs in place, the automation code requires little or no maintenance.
Try httpunit
Depend on what you would like to achieve.
You can use web test built in the Visual Studio Tester Edition. It's quite good and easy to automate. You can use external data as a test data source and it integrates nicely with VS.
There is also test tool by Automated QA (forgot the name) which looks good but expensive.
And there is Selenium. That's the one we are using in Symantec. The biggest advantage is that it actually uses a browser you want to test. VS mimic a browser by changing http request parameters only so you may not be able to test your site for cross-browser compatibility. Selenium on the other hand uses browser and automates it so you can actually test your site in IE, Firefox etc. It can be also integrated with VS unit tests so you can see test results in VS.
So I would recommend Selenium or VS.
I've used Selenium. The features were good, and it was usable but it was buggy.
The IDE would often record events incorrectly (so tests would need to be manually changed), and test files occasionally became completely unusable for no apparent reason, which meant they would have to be recreated all over again. Also development on Selenium IDE seems to have stopped; there hasn't been any bug fixes and patches for a while, and bug reports seem to go unnoticed.
Molybdenum is an alternative, built on Selenium that's worth looking into.
http://www.molyb.org/
Just to throw out another option (of which I haven't tried but I do like Telerik) is Telerik's new WebUI Testing Studio. I will also echo Selenium up votes.
I forget one nice tools and can find link on it but find this ... http://weblogs.asp.net/bsimser/archive/2008/02/21/automated-ui-testing-with-project-white.aspx maybe can help.
If you are looking for simple, cross-browser tool with record and playback, multithreaded playback, intergration with build processes, powerful scripting, good reporting and excellent support, go for Sahi. It will be much easier for your testers/devs to learn and maintain.
you might want to take in consideration near Selenium also Rational Functional Tester ! whether you are familiar with coding in .Net or Java and want to just play around with record & replay or want to create more sophisticated programmatic testing I would recommend it.
WebDriver is another possibility: http://code.google.com/p/webdriver
They are working on a .NET wrapper that may be interesting for you.
Try QEngine. It has all the features of QTP.
You may want to look at RIATest for cross-platform cross-browser testing of web applications.
It works on Windows and Mac, supported browsers are Firefox, IE and Chrome. Automated testing scripts written on one platform/browser can be run against all other supported platforms/browsers.
It has the features that you want: user interaction recording mode and integration with CI servers (outputs results in JUnit format which can be consumed by CI servers such as Hudson).
(Disclaimer: I am a RIATest team member).