I am new to load testing.
So please help in learning gatling and Apache Jmeter for stress testing.
Please help in installing both on Windows and Linux.
How to implement them in my application?
Which one is better for stress testing?
You are asking very generic questions in terms of Stress/Load testing. I think it would be best if you take a look at their documentation then formulate a more specific question.
Installation documentation is best served from the creators of the software.
Implementing these load/stress testing tools into your application isn't really a thing. If you are looking for unit testing (test to utilize in validating your functions/classes/etc work then look at your languages specific go-to libraries - ie. Java is junit/jboss, Nodejs is Karma/Protractor, Python is TestCase/Nose, etc). These tools (jmeter/gatling) are used for stressing your application outside of your build process so they should be treated as end-users (meaning you run the stress testing from remote machines if it is a web service).
Either are best for the right scenario. I think jmeter clusters easier (built-in, where gatling is more manual) but gatling is more programatic and can be manipulated more.
These are opinions and shouldn't be taken as fact or the best so your milage may vary
I strongly doubt that you need them both, if you want a piece of advice in regards which one to choose take a look at Open Source Load Testing Tools: Which One Should You Use? guide.
Once you have clear vision on what tool better suits the needs - you could start ramping up on the selected tool and ask questions in its community communication channels.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
The other day I was showing one of the testers at my company some tests I had written in cucumber (2 features, 5 scenarios).
Then he asked me question that I could not answer:
How is this better than selenium or any other functionality test recording tool?
I understand that cucumber is a different technology and it's placed at a different level of testing, but I can't understand why I should bother to write and maintain Cucumber/Capybara tests.
Can someone give me a reasonable explanation for using Cucumber/Capybara instead of just Selenium?
This question is borderline asking for an opinion. Your question actually reads to me, "What tool is right for me?" I say this because you don't give a reason for why you chose Cucumber and Capybara. I believe to answer that tester's question, you need to answer a couple more questions first:
1.) What stage in the process are you going to be writing these tests?
Cucumber may not be the right choice for unit tests, depending on the language you're using. But it can be used for any level of testing, from unit to integration to end-user.
2.) Who is going to maintaining your tests? You? Other developers? Testers? Business Analysts? Project Managers?
Automated tests must be maintained, and knowing who will be doing that can help you decide on a tool - as some will be too technical for certain users.
3.) Who is going to be defining new tests?
Cucumber is meant to be used collaboratively between development, QA and business owners. It is the perfect tool for leveraging everyone's knowledge into the automated testing process. It requires the development of an ubiquitous language to be effect however. You can read up on that on James Shore's Art of Agile page.
Once you've answered these questions, you're ready to address the tester's question.
However, there are a couple of points to keep in mind when comparing recording tools (such as Selenium IDE, HP Quick Test Pro, IBM Rational Functional Tester) vs. development tools (nUnit, jUnit, RSpec, Selenium webdriver, Capybara) is that they are targeted towards different audiences. They also have different plusses and minuses.
Recording tools are easy for anyone to use, but the scripts they create are fragile. They break easily and require more maintenance. They are great for one-off automated testing, where you need to get it done quickly and have non-technical manpower.
Development tools have a larger learning curve and require programming (or at the least scripting) experience. The scripts are generally more robust, but require more technical knowledge to maintain. They are a good solution when you want repeatability and plan to use tests for a long time.
I strongly suggest you read The Cucumber Book. It will really help you decide if Cucumber is the right choice for you.
Cucumber isn't only a testing tool. Besides testing Cucumber features also take a role of documentation, a mechanism to collaborate with stakeholders and requirements storage (if you write them in declarative style).
You don't have to use Cucumber with Capybara. You can use selenium directly. But Capybara has the same high-level API for all supporting drivers. It's more high-level than Selenium's and allows to write tests a bit faster. You don't have to change code when you switch from one driver to another.
Tests built using test recording tools are generally much worse. Selenium IDE may produce valid programming code but it's not good-looking and thus quiet difficult to maintain.
Cucumber is tool used to make tests readable to business users. It consists of plain English sentences that are matched using regex to your Capybara steps.
Using recording tools won't do you any good in the long run. They were meant for beginners and aren't that powerful so I recommend you go straight to coding.
You can use Selenium alone for your tests, but I would recommend you continue to use Cucumber for documentation purposes, if you find them useful and easy to work with. After all, Cucumber can use Capybara or the Selenium web driver.
Selenium ide is good for testing features that have mostly visual elements (links, text and etc.). But often web apps have features that don't represent itself as visual elements, like sending emails, queueing jobs, communicating with 3rd party services and etc. You could, for example, test if an 'Email has been sent' message is present after submitting a form. But it doesn't really tell you if an email is actually sent and therefore you aren't really testing the whole feature here.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
What are the good automated web UI testing tools?
I want to be able to use it in the .Net world - but it doesn't have to written in .net.
Features such as a record mode, integration into build process\ continuous integration would be nice.
Im going to look at:
Watir
Selenium
Are there any others I should look at?
I definitively recommend Selenium, you can use it from .NET, supports different browsers, works in automatic builds and CI processes (we use it from CCNet). The code is stable. It has a few quirks, but after all they all do.
Whichever tool you choose, I recommend making your own test facade class(es) around it. The facade should be designed to suite your concrete testing needs, without exposing too much the details of the testing tool's API. This will make the test code easier to write and maintain.
UPDATE: if you use ASP.NET view state in your app, you could have problems using a pure HTTP test tool. This is where browser-controllers (like Selenium) are much better.
WatiN
Automates FF and IE
[Test]
public void SearchForWatiNOnGoogle()
{
using (IE ie = new IE("http://www.google.com"))
{
ie.TextField(Find.ByName("q")).TypeText("WatiN");
ie.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(ie.ContainsText("WatiN"));
}
}
http://watin.sourceforge.net/
Watin is pretty unstable to use it in serious projects. It often fails with unexpected reasons like "EI is busy" or something like "Error with COM object".
Selenium is much more stable and it already has a lot of supporting tools. For example Selenium GRID is a solution which allows significantly decrease run time of tests. (Our smoke tests on Watin takes 6 hours to run).
Currently in my job i use QTP and it so far atleast can handle pretty much anything we throw at it both on the UI and it has a special mode for testing non gui services allowing us to check both and help us narrow down where some problems occur when we change the system. It is in my opinion very configurable and the inclusion of vbscript as its language allows integration with lots and lots of things on windows to allow you to do pretty much anything you want! For instance we use it to control the excel com object to make custom excel reports of success and failure so the format of the results is the same wether a test was run manually and also on another project used the adodb object to check that when a page submits information to the database that the database contains the correct data for that record!
As for integration into the build process i have not tried this myself but it is possible to launch qtp and a test from a vbs file so i would assume this should be fairly trvial as ms tools tend to allow you to run vbs files pretty easily from most tools.
I would reccomend it to anyone assuming you can get someone to buy the license!
You can also try VSTT - http://blogs.msdn.com/b/slumley/archive/2009/05/28/vsts-2010-feature-enhancements-for-web-test-playback-ui.aspx
Telerik Test Tools - http://www.telerik.com/automated-testing-tools.aspx
Visual Studio UI Test Extensibility–Scenarios & Guiding Principles - http://blogs.msdn.com/b/mathew_aniyan/archive/2011/03/28/visual-studio-ui-test-extensibility-scenarios-amp-guiding-principles.aspx
VSTS Web Test Step-by-Step Primer - http://blogs.msdn.com/b/jimmymay/archive/2009/02/23/vsts-web-test-step-by-step-primer-7-minute-video-by-microsoft-a-c-e-performance-engineer-chris-lundquist-with-copious-notes-screen-shots-from-your-humble-correspondent.aspx
you might also be interested in taking a look at what the ASP.NET team cooked up itself: Lightweight Test Automation Framework.
There's also a dedicated forum for it.
Having used several different automated testing solutions (TestComplete, QTP, etc), I have to put a vote in for Telerik + Visual Studio. Telerik has great support forums, and is very compatible with whatever testing framework you come up with. Our Devs put unique IDs into their HTML code so that our scripts don't need to be rewritten even with pretty drastic UI refactors. It's definitely more challenging than record and playback, but once you have your unique IDs in place, the automation code requires little or no maintenance.
Try httpunit
Depend on what you would like to achieve.
You can use web test built in the Visual Studio Tester Edition. It's quite good and easy to automate. You can use external data as a test data source and it integrates nicely with VS.
There is also test tool by Automated QA (forgot the name) which looks good but expensive.
And there is Selenium. That's the one we are using in Symantec. The biggest advantage is that it actually uses a browser you want to test. VS mimic a browser by changing http request parameters only so you may not be able to test your site for cross-browser compatibility. Selenium on the other hand uses browser and automates it so you can actually test your site in IE, Firefox etc. It can be also integrated with VS unit tests so you can see test results in VS.
So I would recommend Selenium or VS.
I've used Selenium. The features were good, and it was usable but it was buggy.
The IDE would often record events incorrectly (so tests would need to be manually changed), and test files occasionally became completely unusable for no apparent reason, which meant they would have to be recreated all over again. Also development on Selenium IDE seems to have stopped; there hasn't been any bug fixes and patches for a while, and bug reports seem to go unnoticed.
Molybdenum is an alternative, built on Selenium that's worth looking into.
http://www.molyb.org/
Just to throw out another option (of which I haven't tried but I do like Telerik) is Telerik's new WebUI Testing Studio. I will also echo Selenium up votes.
I forget one nice tools and can find link on it but find this ... http://weblogs.asp.net/bsimser/archive/2008/02/21/automated-ui-testing-with-project-white.aspx maybe can help.
If you are looking for simple, cross-browser tool with record and playback, multithreaded playback, intergration with build processes, powerful scripting, good reporting and excellent support, go for Sahi. It will be much easier for your testers/devs to learn and maintain.
you might want to take in consideration near Selenium also Rational Functional Tester ! whether you are familiar with coding in .Net or Java and want to just play around with record & replay or want to create more sophisticated programmatic testing I would recommend it.
WebDriver is another possibility: http://code.google.com/p/webdriver
They are working on a .NET wrapper that may be interesting for you.
Try QEngine. It has all the features of QTP.
You may want to look at RIATest for cross-platform cross-browser testing of web applications.
It works on Windows and Mac, supported browsers are Firefox, IE and Chrome. Automated testing scripts written on one platform/browser can be run against all other supported platforms/browsers.
It has the features that you want: user interaction recording mode and integration with CI servers (outputs results in JUnit format which can be consumed by CI servers such as Hudson).
(Disclaimer: I am a RIATest team member).
I've been tasked with setting up a continuous integration server for our project and am weighing up the options.
I would like to use cc.rb rather than cc for the simple fact that I'd rather be writing Ruby than XML. From what I can see cc is a more mature product and comes with more features out of the box.
So I would like to know if you were setting up an integration server with the following requirements, which system would you choose and how would you go about it?
Mercurial support
Run selenium test suite on multiple platforms
x10 support for build lights (not essential)
Any guidance would be appreciated.
Regards,
Chris
yup I agree, hudson is much more easier to manage and has a variety of nifty plugins that make it flexible
It does have mercurial support
http://wiki.hudson-ci.org/display/HUDSON/Mercurial+Plugin
Selenium support is also available
http://thinkvitamin.com/dev/easy-automated-web-application-testing-with-hudson-and-selenium/
Have you considered using hudson. It has support for multiple languages and pluins and it generally very flexible.
I believe it has mercurial support and you could prob hook in selenium test somehow.
Karl
Well personally I would use the original CruiseControl, but that's probably because I helped write it. ;)
There are lots of tools available to do what you want (CC, cc.rb, Hudson, etc.). If you just care about running those tests then most of them will do. If you want a more polished experience and a few more features (like IDE integration) then tools like Zutubi Pulse (or Bamboo or Team City) start to look good. If you want to get into the full lifecycle (CI->test->deployment) and reporting from multiple tools then something higher end like AnthillPro might be the way to go.
So really a question of what else you think you might want to do.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
We are working on a large project with a measure of new/modified GUI functionality. We've found in the past that we often introduced new problems in related code when adding new functionality.
We have non-technical users perform testing, but they often miss parts and allow bugs to slip through.
Are there any best practices for organizing the UI testing of a WinForms project? Is there any way to automate it?
There are GUI testing tools that will click buttons and stuff for you but they're pretty fragile in my experience.
The best thing to do is to keep your UI layer as thin as possible. Your event handler classes should optimally be only one or two lines that call out to other more testable classes. That way you can test your business logic in unit tests without having to actually do a button click.
You can automate GUI testing using White framework.
Also consider using TDD friendly design, i.e. use MVP/MVC pattern.
I would highly recommend you to read documentation from Microsoft patterns&practies teams.
Especially have a look at the Composite UI application block and CompositeWPF.
These projects specifically designed to give you best practices in GUI apps development including test driven UI.
Keep the GUI layer as thin as possible. Michael Feathers article, The Humble Dialog Box, is a classic. Also check out Martin Fowler's Passive View. I have also heard that the "automatic button clickers" are fragile, and that it's easy to spend more time maintaining the test than you spend maintaining the code.
In the event that someone finds this useful:
List of GUI testing tools found on Wikipedia.
The following book is an introduction to the subject.
There are as many ways as there are developers out there..
http://pragprog.com/titles/idgtr/scripted-gui-testing-with-ruby
There are many tools and libraries available that can automate WinForms testing, ranging from open source solutions like White to the expensive commercial solutions such as HP QuickTest Pro. There is also the UIAutomation namespace in .NET if you want to roll your own automation framework. But the real cost of automation is in the time and specialised skills it requires to implement. Maintainability is also one of the most important aspects of automated test design; you dont want to expend excessive resource keeping the automation assets current with your application. There are also lots of factors influencing the decision to automate which will be specific to your specific application and organisation.
Your best bet will be to do some more research on the subject and check out some of the specialised testing sites such as http://www.sqaforums.com.
I found this quick and dirty way to test web page layouts in various browsers. It's called browsershots.org. Our client requires support in 5 browsers right now and that takes about a week for full regression testing. This service will deliver screenshots of some 70+ browsers and versions. I print them out and hold up the pages to the light. If they don't line up, there must be a layout problem.
I can't really help with organization or best practices, but an NUnit extension appears to be available for forms testing, NUnitForms.
I used a trial download similar to this product (http://www.tethyssolutions.com/product.htm) and this product (http://www.mjtnet.com/macro_scheduler.htm) years ago and I was happy with the results. These are pretty cheap solutions and some of these macro recorder products can actually be used for automated testing.
A new method available now is using Ruby, via the Ruby gem called win32-autogui. This provides a framework for testing Windows GUI apps. Combine it with Ruby tools RSpec and Cucumber, and it makes for quite a powerful testing framework.
How to organize UI testing depends on how you design the test cases.
Automating Windows Forms application in unit test level can use TDD framework, such as NUnit; or use BDD framework, such as NSpec.
Automating Windows Forms applications in functional testing level can use White, CodedUI, or even directly using Windows Automation API 3.0 (UI Automation and MSAA).
Since all these stuffs mentioned above are technologies rather than solutions, it's better to build some fundamental test automation/scripting frameworks based on these technologies before you start to write automated tests.
I've been looking at ways people test their apps in order decide where to do caching or apply some extra engineering effort, and so far httperf and a simple sesslog have been quite helpful.
What tools and tricks did you apply on your projects?
I use httperf for a high level view of performance.
Rails has a performance script built in, that uses the ruby-prof gem to analyse calls deep within the Rails stack. There is an awesome Railscast on Request Profiling using this technique.
NewRelic have some seriously cool analysis tools that give near real-time data.
They just made it a "Lite" version available for free.
I use jmeter for session-based testing - it allows very fine-grained control over pages you want to hit, parameters to inject, loops to go through, etc. It's great for simulating how many real users your site can handle, rather than just performance testing a set of static urls. You can distribute tests over multiple machines quite easily by loading up the jmeter-server on computers with publicly accessible IP's. I have found some limitations in the number of users/threads any one machine can throw at a server at once (it depends on the test), but jmeter has helped my team improve our apps capacity for users to 6x.
It doesn't have any fancy graphing -- I actually use my own in-house graphing with gruff that can do performance analysis on request time for certain pages and actions.
I'm evaluating a new opensource web page instrumentation and measurement suite called Jiffy. It's not particularly for ruby, it works for all kind of webapps
There's also a Jiffy Firebug Extension for rendering the metrics inside the browser.
I also suggest you look at Browser Mob for load testing.
A colleague of mine has also posted some interesting thoughts on this.