We are about to get a canned package in that has been modified to our needs. I'm part of the team setup to prepare tests for it. It has an Oracle back end and I believe it's written in C++ .NET.
My question is what free or open source testing tools would you recommend.
Thanks
Ken
For regression testing of our applications I use a free tool called AutoHotKey http://www.autohotkey.com/. It is simple, batch configurable, and can work for virtually any application you have. Not exactly designed for black box testing, but a good free tool to add to your toolbox.
While there are a few good commercial applications for black box testing of applications (HoloDeck http://www.sisecure.com/holodeck/index.shtml, Cenzic Hailstorm http://www.cenzic.com/), the only open source applications that I know about only test network security (Spike http://www.immunitysec.com/resources-freesoftware.shtml, OWASP WebScarab http://www.owasp.org/index.php/Category:OWASP_WebScarab_Project, and Nikto http://www.cirt.net/nikto2)
Value checking. See if only valid dates are exempted, number fields except the full range, ect.
What do you expect from such a tool? I don't know of any tool that can arbitrarily test any piece of software.
For what is sounds like you already know what it is that you want to check. Being a custom application your best bet would be to devise a test plan and manually test the values that you would like to validate.
Agree with the others - since the application has been modified to your needs, you should make sure that it actually is modified to your needs.
If the assembly isn't obfuscated, you can use FxCop to analyze the binaries and see if there are any critical bugs (note - if you're not familiar with fxcop and static analysis, find someone who is before reporting a ton of bugs that won't be fixed).
Beyond that, you're looking at more techniques vs. tools to get the job done.
Testing, either functional or non-functional, without reference to the
internal structure of the component or system.
Related
I am starting a maintenance project where I need to work with legacy code and also create new ones. New ones I can create proper interface based development and I can use Moq to do proper unit testing. I can't use Moq against the legacy code since they do not have proper coding to mock out the objects. Based on the reading, Typemock looks like an excellent fit since I can isolate the object and return what we want when a method is called on legacy objects. I really like the simplicity and ease of development. I would like to know is there any other tools out there like Typemock I should look at before committing to it since I have to pay for it.
Thanks
PS: ours is Microsoft shop we work with C#/ASP/ASp.Net/Silverlight and VB.Net
** I just found that Infragistics have one mocking tool as well.
What you're looking for is a mocking framework that uses the Profiler API. These are the ones that can "mock anything" because they sort of run "underneath" the code being tested.
When considering one of these, you'll want to think about things like:
Integration support: Does it work with the tools you're using? For example, your build server, coverage tool, unit testing tool, etc. Are there any additional tools it comes with (e.g., Visual Studio plugins) that might help you work with it?
Technical support: If something goes wrong or you find a bug in the product, can you ask someone about it?
Syntax/patterns: Does it support AAA testing or is it only record/replay? How easy is it to start working with?
Community and learning resources: How's the documentation? Are there people out there who can answer your more general usage questions or are you "on your own?"
There are three primary mocking frameworks to consider that I'm aware of:
Typemock Isolator - The most mature with a free license for open source projects, but otherwise not free.
Telerik JustMock - New to the scene, but good features. There is a free/limited version.
Moles - Free from Microsoft, though somewhat less easy to integrate with tools outside the VSTS realm.
Full disclosure: I'm a Typemock MVP and I've been using Isolator for a while. It has thus far been worth its weight in gold, particularly in Brownfield development. I will admit in some cases, where it's more Greenfield, I use Rhino or Moq. It really depends on what I'm doing.
In any case, I recommend evaluating each of the tools yourself to see which works best in your environment. Be sure to see how it integrates, how easy it is to get going, etc. for each one, and then make your decision based on your experience in your setting.
Right now we are maintaining some old project written in VB6 we are planning to implement continuous integration server for it. We would like to implement some code analysis as well to track that maintanability at least not getting worse. Basically there is only one requirement - the tool should be command line so we can call it from continuous integration server and it should work with VB6 projects. I will really appreciate any recommendations regards tools to try.
Thank you, Maksym
You could look at SourceMonitor as this says that it can produce metrics for VB6, and can be made to work from the command line.
I've only used it for C++ myself.
Have you looked at the recommendations in this question about tools for analysing VB6 source code?
I don't know whether any of these tools can be run from the command-line.
The Code Advisor for Visual Basic 6 is an add-in used to review your code to ensure that it meets predetermined coding standards.
The coding standards are based on best practices developed by Microsoft to produce robust and easy-to-maintain code.
http://www.microsoft.com/en-us/download/details.aspx?id=1222
You can use VBDepend for VB6 and VBA code, static analysis tool based on CQL, here is some of its functionality:
Compare Builds, 60 code metrics, manage Complexity and Dependencies
However VBDepend does not have a rule to check for duplicate/repeated code in the project. This was the most important rule that I was looking for.
Our SD Source Code Search Engine is a tool for lightning-fast search across large code bases.
It words by breaking the text into language atoms, indexing all those atoms, and then executing queries using the indexes to guide the search. The SCSE has a VB6-specific front end (as well as front ends for a wide variety of other languages).
As a side effect of the indexing process, it produces metrics: SLOC, number of comments, Cyclomatic Complexity, Halstead measures, deepest-nested conditionals, etc. These metrics are emitted to an XML file that can be formatted/interrogated any way you like.
The indexing process is a command-line step, so you can integrate it into your scripts.
We also have a clone detector for VB6 that finds exact and near-miss duplicate blocks of code across large systems of source code.
Does anyone know of any tools out there that will let me run and debug a VXML application visually? There are a ton of VXML development tools, but they all require you to build your application within them.
I have an existing application that uses JSPs to generate VXML, and I'm looking for a way to navigate through and debug the rendered VXML in much the same way that Firebug allows one to do this with HTML. I have some proxy-like tools that let me inspect the rendered code as it is sent to the VXML browser, but there's a ton of JS, which makes traversing the code by hand rather difficult.
Has anyone worked with a product that allows for this?
Thanks!
IVR Avenger
There is JigSaw Test suite - has free trial license and reasonably priced.
There is IBM's debugger - part of WebSphere Voice Toolkit.
Many other products have debuggers - a very good summary is here
Disclaimer: I am the development manager for Voiyager (www.voiyager.com), a VoiceXML testing tool. It doesn't meet your criteria nor do I believe it is the type of tool you want, but I thought it was worth mentioning it.
As far as I know, there isn't such a test tool for VoiceXML. In fact there are very few VoiceXML tools on the market and hardly any of them test or analysis. The vendors that created development tools, have all been acquired by other companies. Some of them offered did offer various forms of debugging that were specific to their tool set or stayed at the Dialog (caller input) level. From your question, I'm assuming you need much lower level debugging capabilities.
I think the alternative paths are minimal and somewhat difficult. I believe your primary goal is to debug or rewrite an existing application, but you haven't provided any specific challenges beyond the JavaScript. Some thoughts or approaches that may help:
Isolate the JavaScript and place the code into a unit test harness. That will go a long way to understanding the logic of the application. Any encapsulation of the JavaScript you perform will probably go a long way towards better code maintainability.
Attempt to run the VoiceXML through a translation layer to HTML so you could use FireBug. The largest challenge would involve caller input (ie processing the SRGS grammars). You could probably cheat this by just having the form accept a JSON string the populates the field values. There are tools on the market to test grammars. Depending on the nature of your problems, you could take a simple and light approach and attempt this over just the trouble areas.
Plumb the application with a lot of logging. This can be done through the VoiceXML LOG element, or push the variable space back to the server. By adding intermediate forms, you may be able to provide a dump from each via the VoiceXML Data element.
See if your application will run in one of the open source VoiceXML browsers (not sure of the state of the open source browsers as we've built and bought for our various product lines). If you can get it mostly working, you can use the development debugger to provide some ability to step through the logic. However, it is probably one of the more difficult paths as you'll really need to understand the browser to know when and where to stick your breakpoints and to figure out how to expose the data you want.
Good luck on the challenge. If you find another approach, I would be interested in seeing it posted.
An alternative debug env is to use something like Asterisk with a voicexml browser plugin like the one from http://www.voiceglue.org/ or for a limited licence, i6net.
You can keep all the pieces separate(dynamic html and vxml application in php/jsp/j2ee/, tts processing, and optional asr processing as separate virtual machines with something like virtualbox. If the logic can be kept the same, then it is just a matter of changing the UI based on the channel.
A softphone is all you need to call a minimal asterisk machine, which has the voicexml browser with the url of the vxml in the call plan.
I just used Zend Framework as php is used in this environment, and changed view suffixes(phtml vs vxml) based on the user-agent string.
Flite for tts is fine for debugging, and when your app is ready you can either record phrases, and there was a page on the ubuntu forums with directions for how to increase flite quality with some additional sound files.
Do you have tried Eclipse VTP or InVision Studio?
Eclipse VTP
This is Eclipse plugin. But I feel that it is user-unfriendly a little (of Japanese viewpoint).
InVision Studio *Required create user account*
This is Convergys's IVR tool. It has to edit standard VXML mode. (Unfortunately, It's not exact matching.)
For just debugging vxml, I use Nuance Cafe's VoiceXML checker. It doesn't give you a visual tree or anything, but it's pretty good at spotting syntax errors and is free. I think they might also have more advanced debugging tools if you look into it, but I haven't had the need. (Note: I have no association with them)
http://cafe.bevocal.com/tools/vxmlchecker/vxmlchecker.jsp
I'm looking for the same problem that most of the links are down. I found a document where they propose an open source solution, which works as a plugin for Asterisk (https://www.researchgate.net/publication/228873959_Open_Source_VoiceXML_Interpreter_over_Asterisk_for_Use_in_IVR_Applications) and is available at https://sourceforge.net/projects/voxy/
I would like to know if there are current options to create a VXML structure graphically, like the next image.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
This question isn't about unit-testing. And it is for a desktop product.
This is about testing of the gui and testing that the right stuff is input in the right text box at the right time.
A company I used to work at used WinRunner (different department so I don't know much more that that), but that has now been shutdown by HP but they don't seem that bothered whether you stay with HP or go elsewhere. You can't read about the product until you've signed up which is annoying.
The tool has to work with MFC (non-negotiable) and the ideal tool will also...
be automated.
be scriptable.
work with different screen resolutions automatically.
be able to 'spy' on individual static text boxes, etc.
intuitive enough so non-programmers can create the scripts.
have reporting tools, including email of individual users.
What do other SO users do for automated GUI testing?
We use the SAFS framework for Rational Robot (RRAFS). There are also SAFS implementations for WinRunner (WRAFS) and it looks like they have a new "Image-Based Testing" implementation, which I'm not familiar with.
This framework does a nice job of seperating the UI implementation from the test scripts. I've tested four releases of a web application developed by two different teams (one team using classic ASP, one using ASP.NET) and I only had to change the application map of my UI objects, the tests themselves didn't need to change.
That said, the language of the framework is cumbersome and takes getting used to. It's not very robust, in terms of language constructs, but with some effort you can do anything you need to. It's sort of like "programming" in Windows Batch language, but for tests ;)
To address your individual requirements above:
1) The tool has to work with MFC (non-negotiable).
The SAFS framework uses a 3rd party "record-playback" tool to drive the tests, like Rational Robot or Mercury WinRunner. If that tool can interact with MFC apps, then the framework can. I don't know how the "Image-Based Testing" implementation drives the tests, but I'd guess it can also work with MFC.
2) be automated.
The SAFS framework integrates with the STAF framework, which will allow you to automate the execution of your tests. I have a proof-of-concept test that uses STAF to start a VM image from a pool of images, install the application under test, run the RRAFS test, and put the results on a web server for others to get at.
3) be scriptable.
Yes, but as mentioned, it's not the most robust programming language. I wrote an Excel add-in that our testers use to write their tests that simplifies things a little bit.
4) work with different screen resolutions automatically.
Yes, since it's looking "under the covers" at the UI objects and not the screen. Except for maybe the "Image-based Testing" option...
5) be able to 'spy' on individual static text boxes, etc.
Yes, you can wait for a UI object to appear, disapper, to have a value, for a value to be changed, etc.
6) intuitive enough so non-programmers can create the scripts.
With some training. We've had limited success. Some QA folks can write the tests, some struggle.
7) have reporting tools, including email of individual users.
Yes, using the STAF framework you can post results to a web server, send out emails, etc.
Lots of good answers here, but I wanted to address this goal statement, specifically:
intuitive enough so non-programmers
can create the scripts
I can understand why you'd want this, but it's a lot harder than you might think. While you can find any number of tools out there that'll claim to make writing scripts easy, in practice, you're going to need at least some people on your automation team who understand programming. Writing scripts that are reasonably robust is going to involve one or more of looping, if/then/else, and subroutine calls. Not the kind of thing that non-programmers are going to find intuitive.
Be especially wary of the idea that you can "record" a person using a tool, then play it back for testing. That sort of "automation" is often so brittle that you'll end up modifying or re-recording the script for nearly every change in the software.
Coming from a strong Mercury/HP background, I would highly recommend using QuickTest Professional for your GUI testing. It has a lot of the same functionality as WinRunner, but without a lot of code. Simple GUI checks can be done through the QTP interface with minimal, if any, custom VB code. Checks for text next to boxes can be done with simple compares using the datasheet in QTP.
If you are used to WinRunner, and know VBScript (not so much TSL), then I would definantly look at QTP.
As far as your other requirements, QTP also has the Spy feature, like WinRunner, that will list all properties and actions you can perform on objects. And as for simplicity of use, at my old job, we would have business or system testers create simple smoke scripts, then I would take them and code them for more in-depth testing (multiple data values, error checking, etc). And as for reporting, QTP will do simple reporting of pass/fail/warning on tags that you put in, along with custom data you can input. So you could use a case statement to populate your output values based on your results. It won't do e-mail naitevly, but if you integrate with TestDirector/QualityCenter, you can setup through there, along with automating the kick-off of your scripts, and parameterizing data right from there (which is nice to send back to testers to have populate data without being involved in the script itseld).
Pat
See User Interface Testing.
Desktop or Web apps alike have the same testing pattern here (I've worked in both).
Place as little logic as possible in the UI and test everything below it. So you say, but what if I want to test that such and such happens when a button is clicked? The method that is called when that button is clicked should call out to another class that actually does the thinking.
You might say, but I'm using some static classes/methods that can only exist in my UI, wrap those with an adapter and utilize that interface in order to make your code testable.
The parts of your GUI tests that you want to automate should be automated below the UI. There are parts of your GUI that you can't automate. Checking to make sure things "look right", or testing that you can see certain elements, etc. All of that is what your humans should be doing. Ensuring that events are properly firing, and that values are being appropriately returned from your business objects, that's all unit tests.
I can see from your question that you've already pretty much decided you need an automated GUI tester, but that's not the right tool for this job. If you decide to use that you're trying to find the best way of doing the wrong thing.
If you think that this isn't about unit testing because you're testing GUI interactions, then I can guarantee you're not unit testing close enough to your UI. If you were, you'd feel like most of what you'd be testing was redundant.
If you disagree with me, please post some reasons and we'll hash this out.
There are a lot more (open source) alternatives if you are testing a web product. For a desktop product, some popular general purpose desktop GUI automation tools below (in no particular order). I've worked with all of these personally, and they all get the job done. If you choose to go with a vendor tool, get POCs for the ones you are considering, and make the decision based on what tool is the best fit for the company in general. One tool may be a better fit for a particular application, but there may be other projects/applications to consider.
QuickTest Pro (HP's successor to WinRunner)
Rational Functional Tester (IBM's successor to Robot)
TestPartner
I've been tasked with setting up a continuous integration server for our project and am weighing up the options.
I would like to use cc.rb rather than cc for the simple fact that I'd rather be writing Ruby than XML. From what I can see cc is a more mature product and comes with more features out of the box.
So I would like to know if you were setting up an integration server with the following requirements, which system would you choose and how would you go about it?
Mercurial support
Run selenium test suite on multiple platforms
x10 support for build lights (not essential)
Any guidance would be appreciated.
Regards,
Chris
yup I agree, hudson is much more easier to manage and has a variety of nifty plugins that make it flexible
It does have mercurial support
http://wiki.hudson-ci.org/display/HUDSON/Mercurial+Plugin
Selenium support is also available
http://thinkvitamin.com/dev/easy-automated-web-application-testing-with-hudson-and-selenium/
Have you considered using hudson. It has support for multiple languages and pluins and it generally very flexible.
I believe it has mercurial support and you could prob hook in selenium test somehow.
Karl
Well personally I would use the original CruiseControl, but that's probably because I helped write it. ;)
There are lots of tools available to do what you want (CC, cc.rb, Hudson, etc.). If you just care about running those tests then most of them will do. If you want a more polished experience and a few more features (like IDE integration) then tools like Zutubi Pulse (or Bamboo or Team City) start to look good. If you want to get into the full lifecycle (CI->test->deployment) and reporting from multiple tools then something higher end like AnthillPro might be the way to go.
So really a question of what else you think you might want to do.