In our QA team, we run a suite of automated tests on every commit the developers do. Since there are quite a few such commits daily and no developer wants to wait more than a few minutes to get feedback we're limited to 5 minutes of testing. In these 5 minutes we want to run as many tests as possible.
We've found that selenium tests are the best for our needs: mostly because they're reliable. If a selenium test reports a JS error you're 95% sure it's a real error. (This is an extremely important property as we've learned from our experience using HTMLUnit). However, running selenium tests is slow and heavy. (We maintain a small cpu farm so we can run many selenium servers and many scripts asynchronously).
Recently we've proposed a new approach - use selenium only for the places where you REALLY need it: popups, ajax, JS in general,.. In other places use a "Textual Browser". For example, if you want to check that the following link "works":
<a href='/somewhere'> link </a>
You don't really need selenium. You can perform a GET request to the page and then use regex/parse the page and use xpath/.. Bottom line - you don't need a JS engine. Clearly this is a much lighter and faster test.
We've had much success with this approach and we ran into the following links:
<a href='/somewhere-1' onclick="foo()" > link 1 </a>
<a href='/somewhere-2' onclick="foo()" > link 2 </a>
... many more such links ...
So in this case, you don't really have to run a selenium script that pressed each and every link. Just click on one of them using selenium (so you test the functionality of the JS function foo()) and then use the textual browser to verify the hrefs of the other links.
My question is where do you think we should draw the line? and I'd be happy to hear your opinions - are there "Textual Browser" tools out there (we haven't worked with WebDriver)?
It sounds to me like the line between what you are doing and what I would expect your developers to be doing is a little blurred.
In my mind the developers should be doing unit tests of thier foo() function. TDD in Javascript is a bit low on the tool support, but something that should happen.
If the functions are being unit tests, then selenium becomes the place to test more against the user requirements rather than at the unit of code level.
That said, interactions between QA and Dev teams are pretty complicated socially and it may be hard to get agreement to follow this approach.
Related
I have created a website that allows users to search a database. It is a Perl script that searches oracle using Perl DBI then writes in HTML and JavaScript.
I have found many websites that will quantitatively test the initial loading of the website. I can't help but think that the figures I have are false because the test is not actually performing a search and loading any data.
Are there any tools for testing the speed and performance of the interactive operations of a site beyond its initial load?
You can look to wiring up load testing with something like WebDriver and JMeter. Lots of folks use these or similar tools for just these sorts of scenarios. They're great tools, but require a pretty significant investment in time to get up and running.
You can also use Telerik's Test Studio which makes it easier to quickly get good performance and load tests up and running. Please note I said "easier" and not "easy." Load and performance testing of websites takes anywhere from a moderate amount of work to "OMG! This is nuts!"
Disclaimer: I'm the director of engineering for Test Studio, so I'm a bit biased about it. :)
For load testing you have to user load testing tool like Jmeter or Loadrunner.
Jmeter is an open source tool and Load runner is paid tool but both are user to find load of the website there is other tool also available in the marker which is used to find the load of a website and that tool is free for 1 month.
But you have to use tool to find load of a website
I know this question looks a lot like this one, but I don't have enough rep points to comment seeking further clarification VS2010 Coded UI Tests vs. Web Performance test (Whats the difference??)
Tom E. gives a great explanation, but I'm still a bit vague on one point. I see why Coded UI tests cannot be replaced by Web Performance tests (the extra resources needed for a browser interface) but why can Web Performance tests not replace Coded UI tests?
If you record a webperf test and generate the code from that, couldn't you add validation and extraction rules (inspecting the DOM) to achieve the same result as a Coded UI test without the overhead of the browser?
I realize that this wouldn't be exactly the same as testing in different browsers, but is there a reason this wouldn't at least test whether you're receiving the appropriate response from the server?
Thanks!
Dave, good point. I think you would see the difference fairly quickly if you were trying to build an automated functional test suite (think 500 tests or more) with VS web performance tests and having to parse the DOM for querying and interacting with the application. You would essentially be writing your own Coded UI test playback mechanism. You could do it without the Coded UI test functionality, but it would be quite painful. The level of pain would be dependent on how many test cases you need to automate and how many screens there are in your app and how complex the interactions are.
Can BDD (Behavior Driven Design) tests be implemented using a UI driver?
For example, given a web application, instead of:
Writing tests for the back-end, and then more tests in Javascript for the front-end
Should I:
Write the tests as Selenium macros, which simulate mouse-clicks, etc in the actual browser?
The advantages I see in doing it this way are:
The tests are written in one language, rather than several
They're focussed on the UI, which gets developers thinking outside-in
They run in the real execution environment (the browser), which allows us to
Test different browsers
Test different servers
Get insight into real-world performance
Thoughts?
We've done this for a C# application using a WPF testing tool (WipFlash) and writing NUnit tests in a BDD-like fashion.
e.g.
Given.TheApplicationWindowIsOpen();
When.I.Press.OKButton();
The.Price.ShouldBeCalculated();
We had to code a lot of the DSL ourselves, needless to say. But it becomes a business/customer readable solution.
Try using SpecFlow with WatiN: (I'm not sure if you're using .NET here)
http://msdn.microsoft.com/en-us/magazine/gg490346.aspx
For web testing, you could try WebDriver. The Selenium team are busy integrating WebDriver at the moment. Simon Stewart from Google, who created WebDriver, blogged here about how it works differently to Selenium.
WebDriver uses different technologies for each browser. For Internet Explorer, WebDriver uses Microsoft's UI automation - the same technology on which WipFlash which #Brian Agnew mentioned is based. This is as close as you'll get to pretending to click buttons. Simon's blog shows why this approach can be more powerful than Selenium's Javascript solution.
WebDriver is available from the Selenium site but hasn't been fully implemented as part of Selenium yet.
For BDD, and any use-case driven tests, it is important to be able to communicate what a test is doing. The problem with many test suites is that post-writting nobody is quite certain exactly what the test is doing. This will come up very often if you write in a non-specialized language. Specialization doesn't necessarily mean a special language, but just enough of an abstraction in the one language so it is clear what is happening.
For example, a lot of tests have code that looks like this (pseudo-code, I won't pick on any particular framework):
object = createBrowser()
response = object.gotoURL( "http://someurl.com" );
element = response.getLink( "Click Here" );
response = element.doClick();
This is hard for somebody to quickly translate to a business driver (product manager perhaps, or user). Instead you want to create specialized functions, or a language if you're adventurous, so you can have this:
GotoURL http://someurl.com/
Click link:Click Here
Selenium, and its macros or interface, are still fairly low-level in this regards. If you do use them then at least build some wrappers around them.
You can of course also use a product called TestPlan. It has Selenium in the back-end and exposes a high-level API, and a custom langauge for testing. It also goes beyond just the web to included Email, FTP, etc. The sample language above is a TestPlan snippet
You can certainly do some of your acceptance tests this way, but I think most BDD advocates would not advise using this for all tests. And of course, true BDD advocates wouldn't call them tests...
The RSpec Book advocates a two-level cycle with acceptance tests (or Scenarios) written first (primarily in Cucumber), and unit tests written (in RSpec) in an inner cycle more resembling the traditional TDD.
The outer cycle of acceptance testing can also use tools like Selenium to drive the entire application through the UI (and the authors of The RSpec Book spend a chapter on this). But it is not appropriate for unit tests.
Tests exercising the entire application through the UI are harder to make repeatable, and have a tendency to be slower and more fragile than unit tests.
Actually you could do both - make a user-centric Driver interface (agnostic of GUI / tech / impl). You could then write a UIDriver and a APIDriver and choose a driver to run a specific test. Running through the UI is usually slower (out of proc, control repaints but somehow creates a higher level of confidence initially). Running through the API is much faster (in proc, easy setup-teardown).
The trick here is to separate the What from the How. Otherwise you will end up with ObscureTests and high test maintenance. Ensure the primary focus on testing and not automation.
I would like to know how to go about doing regression testing effectively for a web application.
I m spending a lot of time in regression testing for every build.
I also hear that automated GUI testing is bad and not worth it.
Is there any better way of automating regression testing. if so any good tools to automate GUI tests.
My web app is designed for stupid IE. cant help it, but that how it is.Any good tool that can help me?
I know that we must keep GUI layer thin and all, but I m not a GUI expert or anything , but i need to test it properly and effectively.
Thanks all
Try looking into Cucumber or Selenium
There is also a screencast on it over at railscasts.
That should really help.
You can use standard unit tests and integration tests for the non-GUI components, of course. For the GUI, it's worth looking at UI automation packages.
Many of them are pretty terrible, but a lot of the horror stories come from people using them incorrectly. If you can stay away from specifying click locations by (x,y) coordinates, and you can detect when state transitions happen, rather than using delay(), GUI testing can be effectively automated.
As long as your webapp works with Firefox, too, I recommend using Selenium for that scenario. It allows you to record your test cases inside the browser (by means of a Firefox plugin) but has Internet Explorer support for playback.
If Firefox for recording tests is not an option, you could have a look at the WebDriver project. It uses a code based approach to test specification an works well with TDD, too.
You might also want to have a look at CubicTest, which is a Eclipse plugin for graphically defining Selenium and Watir test cases.
We are considering using Cucumber for testing web applications (not in rails, most of them are asp.net actually).
The applications are in production, our main goal is to test if everything is fine with the services, from time to time, infra people would run it.
We have two questions:
1) Is this a good use for cucumber? Do community people encourage this use of cucumber feature definitions?
2) We have some captchas in our applications. Is there an widely adopted solution for this common problem?
Cucumber looks relatively new. I am a Java person and have used Selenium, HtmlUnit, JWebUnit, etc. Selenium runs in .net, ruby, java, and some other scripting languages.
Selenium has been around for a while (2004 whereas Cucumber is relatively new, 2007).
Selenium has an IDE so you can easily record tests in firefox, save them, and then run them in your integration tests.
I am biased towards Selenium, but it does a good job and allows you to test your applications in several browsers (firefox, safari, IE). It also has support for distributing tests across several servers (if your environment is that large, it supports it).
Ideally, you would have developers or the infrastructure people writing the tests. Then if you have a CI server, you could automatically run the tests you have recorded/written, and then continuously check your application still works as a whole. This works really great for catching errors as soon as they happen so if the developer makes a change and breaks something out of his scope, it will be fresh in his or her head.
As far as CAPTCHA goes, there are various libraries out there. I am unfortunately more knowledgeable with the Java equivalent and not so much with the .Net. Don't write your own, there should be a library you can use.
Walter
ad 1. in my opinion cucumber is great, also we were able to convince our customers to actually understand and verify the tests we wrote in cucumber. We used cucumber + watir for webtesting
ad 2. as far as captcha goes, do you mean how to ignore the captcha for testing? we do not show captcha for our own ip addresses, you could also always accept a specific value for the captcha if the request comes from your development or test environments ip
I can recommend Cucumber. I trained a team of developers and managers to use it on my last project (a PHP application). It worked very well in most cases.
I think your two questions are mutually exclusive. Captcha is designed to prevent something automated so you're going to have to solve that problem for which ever automated test runner you use. You can probably mock something up or work out how to disable it in your test environment. I would opt for the latter. I don't imagine it would be critical to cover your captcha in your test suite.