There is no NFR data in my project and no performance testing has ever been performed.
data given is there are 136 concurrent users during peak hours and how many time times each activity has been done during the peak hours.
How to design load profile in performance testing
Go to the HTTP logs for an objective view of session duration, users online blocked by session duration length, and hourly hits on unique pages tied to business processes.
Make the business sign off on your analysis and own it. As QA, you cannot both own the creation of the requirement and the assessment of the requirement, particularly since the requirement was absent from the guidance provided to development. Otherwise, no matter what you discover development, ops, architecture will claim your test is wrong because your assumptions don't match theirs.
Related
Could any team assist me for sharing standard all J-Meter performance mark criteria while performing J-Meter Performance testing for min 1000 user .it means that how we can decide what parameter(or threshold) caused for deciding load /performance testing with route cause and proposed solution after generating html report with non-GUI with completed scenario.
Thanks
Amit G
There is no "standard", the acceptance criteria normally are being dictated by the business requirements which might differ depending on the nature of the application
for real-time trading a couple of milliseconds delay is critical, companies invest into locating their highly performing servers physically close to exchange servers because even light speed matters for their scenarios
for normal applications used by people (i.e. news portals, e-commerce websites, etc.) the accepted load time would be 2-3 seconds, if people have to wait more - most probably they will switch somewhere else and never return. Moreover, search engines tend to rank slow websites lower
for internal applications which are being used inside the company response time doesn't really matter cause people will have to use particular this application, but you still might report large response times like: "if a person who earns $18 per hour has to wait for 10 seconds for each operation and the number of operations per day is 100 and number of personnel is 3000 the organization looses $15 000 every single day or $5 475 000 a year"
So I would recommend taking the following steps:
Check for existing SLAs or NFRs, it might be the case the document, you got these 1000 users contains some information regarding maximum response time or minimum throughput (requests per unit of time)
If there are no formal acceptance criteria defined you could go for other performance testing types:
Soak testing - putting your application under the prolonged load to see whether it performs consistently and there are no obvious memory leaks
Stress testing - starting with low number of users and gradually increasing the load until errors start occurring. This way you will be able to report correlation between increasing number of users and increasing response time/number of errors, identify saturation point (the maximum number of users which application can efficiently support), bottleneck (breaking point), etc.
I have User Registration, Flight Search, Book Tickets modules in my application. I have created my JMeter test & I have different thread groups for each module in my test. I verified & it works well.
Thread Group 1: XX number of users - access the site - click on regression , enter the details & register. (bold -> loop- happening again and again)
Thread Group 2: XX number of users - access the site - login, - search for flights - (bold -> loop - happening again and again)
Thread Group 3: XX number of users - access the site - login, book ticket - (bold -> loop - happening again and again)
Issue:
My manager says we need to run all modules (all thread groups) together with appropriate users as that is how It is going to be in Production. Even though i can run them all together, - in case of issues - i would not know which feature of the application caused the issue.
My aim is to run each module separately & find its performance. I think that doing the module wise would be the correct approach to get the response time, resource utilization etc.
Clarify:
I do not have much experience in performance testing. What is the correct approach / How do you do your test for your application?
If i have to find server's optimal load (at which it performs better) - what should my approach be?
Intentionally tagging loadrunner as this question is not specific to JMeter & it is generic.
If your goal is to represent human behavior to assess the risk of deployment then testing each business process atomically will not accomplish your goal.
You appear to be engaging in a process that is more appropriately termed performance unit testing. This is very common with developers (as differentiated from performance testers) who seek to qualify the performance of an individual business process across some number of users. These are also typically classified with non-normal think times (often eliminated altogether), small data sets, smaller than useful test environments and extremely short test durations, such as 5-15 minutes.
You can mark this business scenario as transactions it means the HTTP requests for each module will be grouped for ex- Login requests in one group or transaction , Search flight as one group or transaction and similarly for Book tickets etc. By following this you will be testing it in a integrated manner and it would be a production like scenario too. After your run due to grouping you can easily find out which group of request taking more time either search, book tickets etc... In this way you will get the accurate performance statistics and you will achieve the production like scenario too
The approach really depends on what your goal for the testing exercise is. If you're looking to optimize or profile a particular module, it makes sense to test it in isolation.
However, if you're trying to check if your server scales, or if you have enough capacity, you should test all your modules at once, at or above your expected load levels.
A counter example to your isolated approach:
Say you have to modules A and B. They are both CPU intensive and take up 80% CPU when you run them. You first tested A, it used 80% CPU, you had 20% to spare and it performed fine. Now you test B alone, same result.
Now you go to production and users try to use both A and B modules, both are trying to use 80% CPU and suddenly you don't have sufficient CPU and your performance suffers.
I know this is kind of late but still.
I do not have much experience in performance testing. What is the
correct approach / How do you do your test for your application?
As James mentioned, the approach to conducting a performance test in a normal scenario would be to run all the critical business flows at the same time and not in an isolated fashion.
In order to identify issues, we would group the requests under transactions and name the business flows appropriately. This will help in identifying which requests have failed and which feature/portion of the application is at fault.
Running them individually will not provide you more insights simply because, a load testing tool will only be able to confirm the presence of a bottleneck but not the root cause irrespective of the number of business flows involved.
If i have to find server's optimal load (at which it performs better)
- what should my approach be?
In order to identify the optimal load for the server, it is mandatory to run all the scripts together as the end users are going to access the application (all critical scenarios) concurrently and not in a modularized manner.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I know this may be a repeat of the questions but I started using the WebPerformanceTest and LoadTest in my web projects.I could run the WebPerformanceTest and Loadttest.
Now what are the parameters/statistics that I need to share with the Dev team or Busniess team?I think of these..But it would be great if somoeone share what are the other parameters I might have to consider sharing..
1.No.of users the application can support
2.Reposne time what the application can give under the sustainable load
following things you can consider for sharing,
if SLA's are mentioned by Dev team or stakeholders and if your performance test shows that the web application is not matching those SLA's then you can share that
Next question comes in your and their mind is why? (try finding out which part/tier is taking most time or a bottleneck). This can be done by analyzing logs or use profiler which will give you costly things,slow compnonents
Next question is job of performance engineer (how to resolve them and improve the performance of my application). If you know application very well then try tuning it and get the improvement results after tuning which should be shared with Dev team or stakeholders.
Maximum number of users may be confusing if you do not limit response time. For 100ms requests 10 simultaneos users mean 100 rps (requests per second) and for 10s requests 100 simultaneous users mean only 10 rps.
If you use simple hit-based testing (e.g. testing single page or specific request performance) it could be better to use rps metric instead.
For response time - mean time could be confusing as well, especially in case of high variance of response time, it's better to provide response time for some percentiles.
I.e. 50% in 50ms, 75% in 55 ms, 90% in 60 ms, 95% in 70 ms, 99% in 90 ms and 100% in 10 sec. With average time of 150 ms. For some services 150 ms is very good, but about 1% of really slow answers is unacceptable and you hardly can find that problem using just mean and medium response time.
Also, in my experience, collecting resource usage stats (cpu, memory, I/O intensity and network usage) is very helpful for determining bottlenecks (i.e. service slow-down due to high I/O because of insufficient amount of memory for caches).
Are you asking the right question?
For me a big part of load and performance testing is deciding what my customer wants to learn about the system being tested. There is an element of "what data can I show the customer?" but that is based on interpreting what they ask for. The customer may not know what to ask, your job as a tester is to understand what the customer wants and provide them with the answers they want.
The two topics you list show how the system appears to its users: when it will break and how fast it responds. There are several variations on those factors based on rate-of-change of user load and on duration of the test.
Other factors include the performance of the various parts of the server computers that are being tested. Visual Studio load tests can collect performance data from other computers while the test runs. So they can monitor the web server(s), database server(s), application server(s) and so on. On each of these servers data about CPU and memory usage, SQL and IIS performance, and many more can be collected. All this data can be compared (most easily via graphs) against user load, error rates and transaction times to determine which parts of the system have plenty of headroom, which are busy and where the bottlenecks occur. Monitoring all this data may also reveal threshold warnings from the various servers, they should be checked against the Microsoft documentation and, perhaps, other sources to determine whether they are adversely affecting system performance and whether they should be investigated in more detail.
These and many other ideas are possible but it all goes back to working out what your customer wants to learn.
The same question was asked on another forum and the above words are almost identical to the answer I posted there.
You can furnish following details to your clients:
Response Time
Hits per Second
Throughput
Connections Per Second
First Time to buffer
Number of Errors
Transactions Graph
CPU, Memory, and Disk Utilization
Network Utilization (if applicable)
Number of database inserts/updates/deletes records
It sounds like you simply have no (or exceedingly poor) requirements and you don't have a great depth in the field of performance testing and engineering. As far as what to collect
Before the test:
Full load profile of business functions that make up the load.
Documentation of each business function. Items to time within each business function.
Expected response times for each of the timed business functions
Pay special attention to think times and iteration pacing
Web logs from the current system so you can objectively measure how many people are on the system at any given time, not how many sessions are alive and have not yet timed out.
Test Environment with some defined match level to the production environment to scale your load appropriately.
In the test
Response times matched to the timing of the business functions on the requirements / user stories
Other enumerated datapoints for requirements (hits, volume returned, etc...)
A measurement of any finite resource in the system under test for bottleneck identification for slow response times. You can start at the top level (CPU, DISK, MEMORY, NETWORK) and work your way down through those stats as you find a resource constriction at the top level.
Post Test:
Executive overview: Did you hit the requrements (YES|NO)
Detailed data: response times, monitor peaks
Analysis: Where is the likely bottleneck holding your back
If you are attempting to represent human behavior then under no circumstance should you eliminate think time. Think time, or time between requests on an individual session, is baked into the definition of the client-server model and as you reduce it to zero your test becomes less and less a predictor of what will happen in production
Typically, it is based on the benchmark that you want to achieve with the given hardware and environment.
Following are the key parameters
No.of concurrent users (manual and system threads)
Load of transactional and existing data
Response time (typically page)
Throughput Utilization of CPU, Memory and Disk IOs and Network
Bandwidth(applicable where there is an integration with peripheral
systems )
Success percentage
Suppose you have a web application, no specific stack (Java/.NET/LAMP/Django/Rails, all good).
How would you decide on which hardware to deploy it? What rules of thumb exist when determining how many machines you need?
How would you formulate parameters such as concurrent users, simultaneous connections, daily hits and DB read/write ratio to a decision on how much, and which, hardware you need?
Any resources on this issue would be very helpful...
Specifically - any hard numbers from real world experience and case studies would be great.
Capacity Planning is quite a detailed and extensive area. You'll need to accept an iterative model with a "Theoretical Baseline > Load Testing > Tuning & Optimizing" approach.
Theory
The first step is to decide on the Business requirements: how many users are expected for peak usage ? Remember - these numbers are usually inaccurate by some margin.
As an example, let's assume that all the peak traffic (at worst case) will be over 4 hours of the day. So if the website expects 100K hits per day, we dont divide that over 24 hours, but over 4 hours instead. So my site now needs to support a peak traffic of 25K hits per hour.
This breaks down to 417 hits per minute, or 7 hits per second. This is on the front end alone.
Add to this the number of internal transactions such as database operations, any file i/o per user, any batch jobs which might run within the system, reports etc.
Tally all these up to get the number of transactions per second, per minute etc that your system needs to support.
This gets further complicated when you have requirements such as "Avg response time must be 3 seconds etc" which means you have to figure in network latency / firewall / proxy etc
Finally - when it comes to choosing hardware, check out the published datasheets from each manufacturer such as Sun, HP, IBM, Windows etc. These detail the maximum transactions per second under test conditions. We usually accept 50% of those peaks under real conditions :)
But ultimately the choice of the hardware is usually a commercial decision.
Also you need to keep a minimum of 2 servers at each tier : web / app / even db for failover clustering.
Load testing
It's recommended to have a separate reference testing environment throughout the project lifecycle and post-launch so you can come back to run dedicated performance tests on the app. Scale this to be a smaller version of production, so if Prod has 4 servers and Ref has 1, then you test for 25% of the peak transactions etc.
Tuning & Optimizing
Too often, people throw some expensive hardware together and expect it all to work beautifully. You'll need to tune the hardware and OS for various parameters such as TCP timeouts etc - these are published by the software vendors, and these have to be done once the software are finalized. Set these tuning params on the Ref env, test and then decide which ones you need to carry over to Production.
Determine your expected load.
Setup a machine and run some tests against it with a Load testing tool.
How close are you if you only accomplished 10% of the peak load with some margin for error then you know you are going to need some load balancing. Design and implement a solution and test again. Make sure you solution is flexible enough to scale.
Trial and error is pretty much the way to go. It really depends on the individual app and usage patterns.
Test your app with a sample load and measure performance and load metrics. DB queries, disk hits, latency, whatever.
Then get an estimate of the expected load when deployed (go ask the domain expert) (you have to consider average load AND spikes).
Multiply the two and add some just to be sure. That's a really rough idea of what you need.
Then implement it, keeping in mind you usually won't scale linearly and you probably won't get the expected load ;)
I'm about to start testing an intranet web application. Specifically, I've to determine the application's performance.
Please could someone suggest formal/informal standards for how I can judge the application's performance.
Use some tool for stress and load testing. If you're using Java take a look at JMeter. It provides different methods to test you application performance. You should focus on:
Response time: How fast your application is running for normal requests. Test some read/write use case
Load test: How your application behaves in high traffic times. The tool will submit several requests (you can configure that properly) during a period of time.
Stress test: Do your application can operate during a long period of time? This test will push your application to the limits
Start with this, if you're interested, there are other kinds of tests.
"Specifically, I have to determine the application's performance...."
This comes full circle to the issue of requirements, the captured expectations of your user community for what is considered reasonable and effective. Requirements have a number of components
General Response time, " Under a load of .... The Site shall have a general response time of less than x, y% of the time..."
Specific Response times, " Under a load of .... Credit Card processing shall take less than z seconds, a% of the time..."
System Capacity items, " Under a load of .... CPU|Network|RAM|DISK shall not exceed n% of capacity.... "
The load profile, which is the mix of the number of users and transactions which will take place under which the specific, objective, measures are collected to determine system performance.
You will notice the the response times and other measures are no absolutes. Taking a page from six sigma manufacturing principals, the cost to move from 1 exception in a million to 1 exception in a billion is extraordinary and the cost to move to zero exceptions is usually a cost not bearable by the average organization. What is considered acceptable response time for a unique application for your organization will likely be entirely different from a highly commoditized offering which is a public internet facing application. For highly competitive solutions response time expectations on the internet are trending towards the 2-3 second range where user abandonment picks up severely. This has dropped over the past decade from 8 seconds, to 4 seconds and now into the 2-3 second range. Some applications, like Facebook, shoot for almost imperceptible response times in the sub one second range for competitive reasons. If you are looking for a hard standard, they just don't exist.
Something that will help your understanding is to read through a couple of industry benchmarks for style, form, function.
TPC-C Database Benchmark Document
SpecWeb2009 Benchmark Design Document
Setting up a solid set of performance tests which represents your needs is a non-trivial matter. You may want to bring in a specialist to handle this phase of your QA efforts.
On your tool selection, make sure you get one that can
Exercise your interface
Report against your requirements
You or your team has the skills to use
You can get training on and will attend with management's blessing
Misfire on any of the four elements above and you as well have purchased the most expensive tool on the market and hired the most expensive firm to deploy it.
Good luck!
To test the front-end then YSlow is great for getting statistics for how long your pages take to load from a user perspective. It breaks down into stats for each specfic HTTP request, the time it took, etc. Get it at http://developer.yahoo.com/yslow/
Firebug, of course, also is essential. You can profile your JS explicitly or in real time by hitting the profile button. Making optimisations where necessary and seeing how long all your functions take to run. This changed the way I measure the performance of my JS code. http://getfirebug.com/js.html
Really the big thing I would think is response time, but other indicators I would look at are processor and memory usage vs. the number of concurrent users/processes. I would also check to see that everything is performing as expected under normal and then peak load. You might encounter scenarios where higher load causes application errors due to various requests stepping on each other.
If you really want to get detailed information you'll want to run different types of load/stress tests. You'll probably want to look at a step load test (a gradual increase of users on system over time) and a spike test (a significant number of users all accessing at the same time where almost no one was accessing it before). I would also run tests against the server right after it's been rebooted to see how that affects the system.
You'll also probably want to look at a concept called HEAT (Hostile Environment Application Testing). Really this shows what happens when some part of the system goes offline. Does the system degrade successfully? This should be a key standard.
My one really big piece of suggestion is to establish what the system is supposed to do before doing the testing. The main reason is accountability. Get people to admit that the system is supposed to do something and then test to see if it holds true. This is key because because people will immediately see the results and that will be the base benchmark for what is acceptable.