Related
I am starting a maintenance project where I need to work with legacy code and also create new ones. New ones I can create proper interface based development and I can use Moq to do proper unit testing. I can't use Moq against the legacy code since they do not have proper coding to mock out the objects. Based on the reading, Typemock looks like an excellent fit since I can isolate the object and return what we want when a method is called on legacy objects. I really like the simplicity and ease of development. I would like to know is there any other tools out there like Typemock I should look at before committing to it since I have to pay for it.
Thanks
PS: ours is Microsoft shop we work with C#/ASP/ASp.Net/Silverlight and VB.Net
** I just found that Infragistics have one mocking tool as well.
What you're looking for is a mocking framework that uses the Profiler API. These are the ones that can "mock anything" because they sort of run "underneath" the code being tested.
When considering one of these, you'll want to think about things like:
Integration support: Does it work with the tools you're using? For example, your build server, coverage tool, unit testing tool, etc. Are there any additional tools it comes with (e.g., Visual Studio plugins) that might help you work with it?
Technical support: If something goes wrong or you find a bug in the product, can you ask someone about it?
Syntax/patterns: Does it support AAA testing or is it only record/replay? How easy is it to start working with?
Community and learning resources: How's the documentation? Are there people out there who can answer your more general usage questions or are you "on your own?"
There are three primary mocking frameworks to consider that I'm aware of:
Typemock Isolator - The most mature with a free license for open source projects, but otherwise not free.
Telerik JustMock - New to the scene, but good features. There is a free/limited version.
Moles - Free from Microsoft, though somewhat less easy to integrate with tools outside the VSTS realm.
Full disclosure: I'm a Typemock MVP and I've been using Isolator for a while. It has thus far been worth its weight in gold, particularly in Brownfield development. I will admit in some cases, where it's more Greenfield, I use Rhino or Moq. It really depends on what I'm doing.
In any case, I recommend evaluating each of the tools yourself to see which works best in your environment. Be sure to see how it integrates, how easy it is to get going, etc. for each one, and then make your decision based on your experience in your setting.
I can find some articles on how to use them but I can't seem to find anywhere peoples impression of them using them in real projects. I have been trying to figure out how to use them and I've had alot of problems.
Can someone out there who uses these tools on the job give me thier impression? Are there better alternate tools available? Using these really just a waste of time?
With Coded UI Tests I see how they are good for basic javascript checking but its so basic of a example I don't think it is worth it. With web tests I like how they work but when I activate code coverage/ASP.NET profiling it doesnt work half the time.
I tried both Selenium and Coded UI for integration testing the project I am working on. I wrote a blog article with my impressions.
The bottom line for me is that both Selenium and Coded UI are still far from perfect, but better than manual testing. I am using Selenium right now, though I might revisit CodedUI if a few things would be fixed in a later version.
experience with Selenium and VS Coded UI test for web apps.
Better than manual testing in my eyes, though the tests are at times hard to maintain and often need to be changed manually to make them more robust. Both tools have browser plugins which can generate tests automatically, but this code is generally not very resilient towards changes in the HTML (in case of web apps).
You will be disappointed if you expect an automated test run orders of magnitude faster than a manual tester. I think the aim of the vendors is to get the automated test execute aboit twice as fast as a manual tester (that can feel painfully slow if you have a large set of tests).
I have used both selenium 1.0 (currently deprecated 2.0 for my dev environment) and visual studio 2010 web testing. There is no doubt, vs 2010 web testing tool is by far (and i mean by far) better than selenium 1.0 and 2.0 (in beta now) in terms of tooling, robustness and the different kinds of metric you can measure (load balance analysis etc). The one good thing also is that it can run your tests in firefox and other browsers as well(provided you have those browsers installed - i believe. Not sure on that one). Selenium is good and have been around for a while. Has a lot of community support but... it can be frustrating. They are currently in transition to 2.0 so I think things will get better and improve after 2.0. In short, if your development environment is .NET with vs 2010, you will get the best ROI by using of the web testing tools.
Disclaimer: I don't work for Microsoft and I am in no way affiliated with MS. My comment above is based on my experience and my only.
As with (most) all Microsoft tools they will do the job--provided you read plenty of materials to understand how to use them and you understand their limitations. If you are looking for full-featured, high-quality, relatively intuitive tools then look to RedGate.
Microsoft tools will do everything, but nothing terribly well.
I know that unit testing is desirable, and I am interested in doing unit testing. The only problem is I have no idea how, or even where to start really. So my question is: How do I learn about and start unit testing? Specifically I frequently write Java code in NetBeans and C# code in Visual Studio and am interested in what tools to use and how to get my feet wet. Can anyone offer any advice for an absolute unit testing n00b?
I know that there are a lot of somewhat similar questions out there but I am less interested in why and more interested in how.
Try to read on StackOverflow, tag unit-testing :)
Is Unit Testing worth the effort?
How to make junior programmers write tests?
What is unit testing?
How do you know what to test when writing unit tests?
Another entry point would be the tags junit and nunit
There are lots of question dealing this.
If you're searching books about Unit Testing, try this thread:
Good C# Unit testing book. There the famous Kent Beck book is mentioned, "Test Driven Development By Example".
It's worth reading!
Good luck!
This Tutorial for writing JUnit tests in NetBeans should give you an idea how unit testing is done technically. NUnit for C# works pretty much the same.
For an advanced view of how to integrate unit testing into you daily development, the standard reference is Kent Beck's "Test Driven Development By Example". Here's a broad overview.
If you really want to understand unit testing (and get hooked), try it out, it should only take a few hours!
First, I recommend downloading a unit testing framework, such as NUnit (if you want to start with .NET / C#).
Most of these frameworks have online documentation that provides a brief introduction, like the NUnit Quick Start. Read that documentation, then choose a fairly simple, self-contained class for which you are responsible. If you can, try to choose a class that:
Has few or no dependencies on other classes - at least not on complex classes.
Has some behavior: a simple container with a bunch of properties won't really show you much about unit testing.
Try writing some tests to get good coverage on that class, then compile and run the tests.
Unit testing is simple to learn and hard to master (apologies for the cliché, but it is appropriate here), so once you've done this, start reading around: for example, guerda has provided several excellent links in another answer to this question.
Start small.
Unit testing (and automated testing in general) isn't a silver bullet, doesn't always apply to every situation and can be a bit of a culture shock. That said, if you're writing software that you're selling or that your company relies on, I highly recommend adopting it. You'd be surprised how many professional development shops don't.
First, get comfortable the mechanics of creating and running unit tests with your development tools.
Then, start with a new (preferably small, but not necessarily trivial) class or method that you want to test. Writing tests for existing code has its own challenges, which is why you should start with either something brand new or something that you are going to rewrite.
You should notice that making this class or method testable (and therefore reusable) has an impact on how you write the code. You should also find that thinking about how to test the code up front forces you to think about and refine the design now instead of some time down the road "when there's more time". Things as simple as "What should be returned if a bad parameter is passed in?". You should also feel a certain amount of confidence that the code behaves exactly the way you expect it to.
If you see a benefit from this exercise, then build on it and start applying it to other parts of your code. Over time, you'll have confidence in more and more of your software as it becomes more provably correct.
The hands on approach helped get my head around the subject better than a lot of the reading material and helped fill in the gaps of things I just didn't understand. Especially where TDD was concerned. It was counter-intuitive until I actually tried it.
Find-a-bug-write-a-test
The next time you find a bug in your code base, before fixing it, write a test. The test should fail. Then, fix the bug. The test should pass.
If the test doesn't pass, there's either a bug in your test, or a bug in your fix.
A person will never find that bug in your code again. The unit tests will find it (and faster than a person can).
This is definitely a small start, but it gets you into testing. Once you've got the hang of it, you'll probably start writing more tests, and eventually get a knack for how code will fail and which tests you need (for example: a test for every business rule).
Later in your progression you setup a continuous integration server, which makes sure your codebase is always solid.
A good start is to buy a good book where you can read about unit-testing.
I have a tips on a book called "Software testing with visual studio team system 2008" and it takes you trough the basics stuff and background to more higher levels of unit-testing and practises.
Check out The Art of Unit Testing by Roy Osherove, it's a good book for beginners since it starts at the very beginning.
I would recommend reading Michael Feathers' "Working Effectively with Legacy Code". Old code often ends up being code that's hard to Unit Test. Feathers' book is a great guide in how to refactor your code to the point that unit tests are a snap to write.
Not exactly an answer to the question you asked, but might be a missing step between where you are, and where you need to be to implement some of the answers others have given.
Or in another way, is there any reason to install NUnit or any other library and use it instead of the one that comes with Visual Studio 2008?
We have been using MS test tools since VS2005, and I did look at NUnit as a replacement at one point because of some problems we had with the MS tools. Here are some of my findings:
MS test tools pros
Easy to convince management to use MS supplied tools (of course this is not a technical issue, but it is probably relevant in a lot of companies)
Test libraries, test runner and code coverage are all nicely integrated with VS
Central reporting and other features in a Team Foundation setup
More or less the same feature set as with NUnit (I get the feeling that NUnit may be a tad better than MS test tools, but the differences are not huge)
MS test tools cons
VS2005 - really slow, VS2008 a lot better, but still sluggish for large solutions
Test host is STA which makes it harder to test multi-threaded code
VS support is a bit clunky because they support test-first and test-after in the same interface (so there are two "new unit test" commands -> more support work)
In the end switching to NUnit was not attractive to us for a couple of reasons:
Code changes, not a lot but with a large number of tests it still amounts to a bit of work
Increased cost and complexity per developer seat: if you swap MS Test with NUnit, you also loose the test runner and code coverage. Both can be obtained through other tools such as R#, TestDriven.NET, NCover etc, but it all adds to both the cost and the complexity.
Do you want developers who don't have Visual Studio 2008, or who only have the Express edition, to run the unit tests? This is particularly relevant for Open Source projects. If so, I'd go with one of the open source unit test frameworks.
I think most test projects created before VS2008 will nowadays still use other libraries.
For newer projects it comes down to personal taste I guess, or company policies.
Also, not all visual studio versions support the testing framework I think...
I have just started to use Visual Studio test projects. I have no experience of using thrid party test products, but I have found that the Visual Studio one appears to be perfectly adequate for most points.
I think that it would boild down to what you have experience in.
If you already know how to use NUnit and/or Rhino mocks then go with them.
If you have no experience of either or aren't too bothered about gaining experience in them then go with the Visual Studio supplied one.
A useful pointer is that the Visual Studio one will create test method stubs and accessors automatically for you if you right click and select "Create Unit Test" from anywhere in your code. Particularly useful if you're not following a strict TDD of writing tests before coding.
Some advantages:
(Jon's) You can run your tests in VS 2008 Express
Lot's of cutting edge features get built on top of the free libraries. Eg. BDD
In general there is rich extensability when it comes to the OSS testing frameworks.
You can debug into the source of the OSS frameworks, and make changes if required.
Arguably, some test frameworks like xUnit are easier to learn and enforce a cleaner side-effect free testing pattern.
Given the awesome testdriven.net supports pretty much all of the OSS testing frameworks, you get the same (if not better) IDE integration that you get with VS 2008.
A couple of notes RE MSTest...
1) There is an entire team dedicated to brining features to market (it's literally their full time job). These guys end up having resources for writing documents, integration points into other Microsoft products, discussing advances from the general test community and how these get into Microsoft test products and in general an overall, fully functioning product unit dedicated to the effort. When you go with MSTest, you get the full benefit of these peoples efforts and the dollars Ms invests.
2) Web Testing and Stress Testing. There is yet a WHOLE other dedicated group (separate from the MSTest group) that works on Web testing and stress testing. These are huge additions to the Test stack from Microsoft. These offer ways to simulate online users, use complex bandwidth manipulation, check server performance and much much much more. The Web and Stress testing stuff is a BIG reason to really give the MS Test stack a hard look.
3) Functional testing ... Ms is adding full UI automated testing support and this will really round out their offering. It was missing in terms of earlier versions of MS Test tools, but it must be considered that a fully integrated set of UI testing tools will really amp up the overall stock of an investment in MS test technology.
4) As noted above - Impact Testing. From a developers point of view - this is WAY cool. As mentioned - the tools select the right unit tests passed on code coverage of the unit test. So a dev edits and existing class, etc... and the tool specifically points to the tests that impact the checkin. BTW - this is a GREAT example of the research you gain from that I mentioned in #1.
.... BTW - Full disclosure - I worked on VSTS directly - and totally consider it a part of my family - so yes - I am utterly compromised in terms of favoritism.
As for Non-MS test tools ...
A) You can't disregard the efforts of other companies nor the entire open source community. Really MS still loses if you consider on one hand the entire open source community and all of other company efforts VS several Microsoft product teams. HOWEVER, the MS efforts are mostly coordinated (though not always) and do tend to have at least soft points of integration (though again, not always).
B) Faster turn-around from other vendors / the open source community. Ms is improving - BUT - no - other communities and vendors DO out pace MS in getting things to market. This is not different from test tool innovations.
C) ***** Other testing tools will do a GREAT job with non-ms technology stack stuff. Ms tools will be 'ok' in some cases - and in other cases utterly miss the boat. Many times the big ticket wins from MS tools aren't there for other technology (read NON-MS) technology stacks - ... Other vendors and the open source community (obviously) do provide much more compelling support.
In the end... I'd suggest this: If you're on the MS stack AND have an investment in Visual Studio - particuarly with MSDN - and thus have a major amount of access and investment in buying MS tools .... REALLY, using SOME FORM of MS development and test tools is a no brainer. I didn't even discuss the test tools integration and usage of the rest of VSTS - holy cow.
However - if you're using other stacks, don't have the end-to-end investment in the MS stack / MS provided developer tools ... If you're operating in a shop where other OS's and such are huge deals for desktop or server installs, etc... Then perhaps minor elements of the MS tool set might be worth checking out - but obviously starting in the MS test tools direction will be somewhat more difficult / costly.
Hope this helps.
NUnit allows you to run tests in a defined order whereas the Visual Studio test framework runs the tests concurrently.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
What are the good automated web UI testing tools?
I want to be able to use it in the .Net world - but it doesn't have to written in .net.
Features such as a record mode, integration into build process\ continuous integration would be nice.
Im going to look at:
Watir
Selenium
Are there any others I should look at?
I definitively recommend Selenium, you can use it from .NET, supports different browsers, works in automatic builds and CI processes (we use it from CCNet). The code is stable. It has a few quirks, but after all they all do.
Whichever tool you choose, I recommend making your own test facade class(es) around it. The facade should be designed to suite your concrete testing needs, without exposing too much the details of the testing tool's API. This will make the test code easier to write and maintain.
UPDATE: if you use ASP.NET view state in your app, you could have problems using a pure HTTP test tool. This is where browser-controllers (like Selenium) are much better.
WatiN
Automates FF and IE
[Test]
public void SearchForWatiNOnGoogle()
{
using (IE ie = new IE("http://www.google.com"))
{
ie.TextField(Find.ByName("q")).TypeText("WatiN");
ie.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(ie.ContainsText("WatiN"));
}
}
http://watin.sourceforge.net/
Watin is pretty unstable to use it in serious projects. It often fails with unexpected reasons like "EI is busy" or something like "Error with COM object".
Selenium is much more stable and it already has a lot of supporting tools. For example Selenium GRID is a solution which allows significantly decrease run time of tests. (Our smoke tests on Watin takes 6 hours to run).
Currently in my job i use QTP and it so far atleast can handle pretty much anything we throw at it both on the UI and it has a special mode for testing non gui services allowing us to check both and help us narrow down where some problems occur when we change the system. It is in my opinion very configurable and the inclusion of vbscript as its language allows integration with lots and lots of things on windows to allow you to do pretty much anything you want! For instance we use it to control the excel com object to make custom excel reports of success and failure so the format of the results is the same wether a test was run manually and also on another project used the adodb object to check that when a page submits information to the database that the database contains the correct data for that record!
As for integration into the build process i have not tried this myself but it is possible to launch qtp and a test from a vbs file so i would assume this should be fairly trvial as ms tools tend to allow you to run vbs files pretty easily from most tools.
I would reccomend it to anyone assuming you can get someone to buy the license!
You can also try VSTT - http://blogs.msdn.com/b/slumley/archive/2009/05/28/vsts-2010-feature-enhancements-for-web-test-playback-ui.aspx
Telerik Test Tools - http://www.telerik.com/automated-testing-tools.aspx
Visual Studio UI Test Extensibility–Scenarios & Guiding Principles - http://blogs.msdn.com/b/mathew_aniyan/archive/2011/03/28/visual-studio-ui-test-extensibility-scenarios-amp-guiding-principles.aspx
VSTS Web Test Step-by-Step Primer - http://blogs.msdn.com/b/jimmymay/archive/2009/02/23/vsts-web-test-step-by-step-primer-7-minute-video-by-microsoft-a-c-e-performance-engineer-chris-lundquist-with-copious-notes-screen-shots-from-your-humble-correspondent.aspx
you might also be interested in taking a look at what the ASP.NET team cooked up itself: Lightweight Test Automation Framework.
There's also a dedicated forum for it.
Having used several different automated testing solutions (TestComplete, QTP, etc), I have to put a vote in for Telerik + Visual Studio. Telerik has great support forums, and is very compatible with whatever testing framework you come up with. Our Devs put unique IDs into their HTML code so that our scripts don't need to be rewritten even with pretty drastic UI refactors. It's definitely more challenging than record and playback, but once you have your unique IDs in place, the automation code requires little or no maintenance.
Try httpunit
Depend on what you would like to achieve.
You can use web test built in the Visual Studio Tester Edition. It's quite good and easy to automate. You can use external data as a test data source and it integrates nicely with VS.
There is also test tool by Automated QA (forgot the name) which looks good but expensive.
And there is Selenium. That's the one we are using in Symantec. The biggest advantage is that it actually uses a browser you want to test. VS mimic a browser by changing http request parameters only so you may not be able to test your site for cross-browser compatibility. Selenium on the other hand uses browser and automates it so you can actually test your site in IE, Firefox etc. It can be also integrated with VS unit tests so you can see test results in VS.
So I would recommend Selenium or VS.
I've used Selenium. The features were good, and it was usable but it was buggy.
The IDE would often record events incorrectly (so tests would need to be manually changed), and test files occasionally became completely unusable for no apparent reason, which meant they would have to be recreated all over again. Also development on Selenium IDE seems to have stopped; there hasn't been any bug fixes and patches for a while, and bug reports seem to go unnoticed.
Molybdenum is an alternative, built on Selenium that's worth looking into.
http://www.molyb.org/
Just to throw out another option (of which I haven't tried but I do like Telerik) is Telerik's new WebUI Testing Studio. I will also echo Selenium up votes.
I forget one nice tools and can find link on it but find this ... http://weblogs.asp.net/bsimser/archive/2008/02/21/automated-ui-testing-with-project-white.aspx maybe can help.
If you are looking for simple, cross-browser tool with record and playback, multithreaded playback, intergration with build processes, powerful scripting, good reporting and excellent support, go for Sahi. It will be much easier for your testers/devs to learn and maintain.
you might want to take in consideration near Selenium also Rational Functional Tester ! whether you are familiar with coding in .Net or Java and want to just play around with record & replay or want to create more sophisticated programmatic testing I would recommend it.
WebDriver is another possibility: http://code.google.com/p/webdriver
They are working on a .NET wrapper that may be interesting for you.
Try QEngine. It has all the features of QTP.
You may want to look at RIATest for cross-platform cross-browser testing of web applications.
It works on Windows and Mac, supported browsers are Firefox, IE and Chrome. Automated testing scripts written on one platform/browser can be run against all other supported platforms/browsers.
It has the features that you want: user interaction recording mode and integration with CI servers (outputs results in JUnit format which can be consumed by CI servers such as Hudson).
(Disclaimer: I am a RIATest team member).