i have to loadtest apis with 10k rpm,what should be the ramp up time and execution time.
10k rpm = 10k/60=166 req/sec i.e 166 concurrent users
but don't know to calculate exact ramptime time and execution time.
currently standard set is 60min execution and 15 min ramp up in the org
10k rpm = 10k/60=166 req/sec i.e 166 concurrent users
166 concurrent users don't necessarily mean 166 requests per second, moreover it will happen only if response time for each Sampler will be 1 second sharp.
If response time will be 2 seconds - you will have 83 requests per second
If response time will be 0.5 seconds - you will have 332 requests per second
So you need to supply sufficient amount of threads and add a relevant Timer like Constant Throughput Timer or Throughput Shaping Timer to slow down JMeter throughput to the desired value. The latter one can be connected with the Concurrency Thread Group so JMeter would be able to kick off extra threads if the current amount is not sufficient to conduct the required load.
And the system under test need to be able to respond fast enough because JMeter waits for the response from the previous request before starting a new one.
With regards to ramp-up, there are no exact numbers recommendations, the only approach you can follow is increasing the load gradually so you would be able to correlate the increasing load with increasing throughput, response time, number of errors and so on.
If you don't have better ideas you can go for the quote from the JMeter user manual:
The ramp-up period tells JMeter how long to take to "ramp-up" to the full number of threads chosen. If 10 threads are used, and the ramp-up period is 100 seconds, then JMeter will take 100 seconds to get all 10 threads up and running. Each thread will start 10 (100/10) seconds after the previous thread was begun. If there are 30 threads and a ramp-up period of 120 seconds, then each successive thread will be delayed by 4 seconds.
Ramp-up needs to be long enough to avoid too large a work-load at the start of a test, and short enough that the last threads start running before the first ones finish (unless one wants that to happen).
Start with Ramp-up = number of threads and adjust up or down as needed.
It completely depends on what exactly needs to be achieved
Things to consider:
Test execution duration and ramp up/ramp down of users, needs to evaluated based on the AUT(Application under Test) Real user utilization in Production.(Capture the number of users active at particular point of time in system)
Watch out for pattern of users entering/exiting the system.
Calculate User load/ RPM using Little's Law.
Some of the common test Types includes.
Baseline Test (Initial testing conducted with very minimal load to test the stability of the application)
Load Test (To evaluate applications performance under gradually increasing User load with expected concurrent number of users
Stress Test (Testing beyond normal conditions, increasing the user load until the Response time exceeds acceptable criteria. Usually done to find the breakpoint of the application, responsiveness of the application for higher volume traffic
Endurance Test ( To test for a longer durations, by applying varying loads.)
Related
How to calculate inputs (virtual users, ram up period) for performance and load testing using Jmeter.
If you're doing Load testing - you are supposed to be checking your system behaviour against the anticipated load, it means that you should know the expected number of users from NFR or SLA
If you don't have any requirements - go for the stress testing, i.e. start with 1 user and gradually increase the load until response time starts increasing or errors start occurring (whatever comes the first)
With regards to the ramp-up period duration - there are no strict requirements, as per JMeter Documentation
The ramp-up period tells JMeter how long to take to "ramp-up" to the full number of threads chosen. If 10 threads are used, and the ramp-up period is 100 seconds, then JMeter will take 100 seconds to get all 10 threads up and running. Each thread will start 10 (100/10) seconds after the previous thread was begun. If there are 30 threads and a ramp-up period of 120 seconds, then each successive thread will be delayed by 4 seconds.
Ramp-up needs to be long enough to avoid too large a work-load at the start of a test, and short enough that the last threads start running before the first ones finish (unless one wants that to happen).
Start with Ramp-up = number of threads and adjust up or down as needed.
so if you don't have any better ideas - just go for the above recommendation.
I have a requirement like API should be capable of handle 5k Concurrent user in 5 Sec.
How can i test this scenario in jmeter.
For 1k Concurrent user what should the RampUp time to be set in Jmeter.
5000 for 5 seconds is quite easy to achieve:
With normal Thread Group
With Concurrency Thread Group
With regards to the ramp-up, I can only give you:
a quote from JMeter documentation:
The ramp-up period tells JMeter how long to take to "ramp-up" to the full number of threads chosen. If 10 threads are used, and the ramp-up period is 100 seconds, then JMeter will take 100 seconds to get all 10 threads up and running. Each thread will start 10 (100/10) seconds after the previous thread was begun. If there are 30 threads and a ramp-up period of 120 seconds, then each successive thread will be delayed by 4 seconds.
Ramp-up needs to be long enough to avoid too large a work-load at the start of a test, and short enough that the last threads start running before the first ones finish (unless one wants that to happen).
Start with Ramp-up = number of threads and adjust up or down as needed.
and a piece of advice on ramp-up in general: if you're doing some form of a stress test it's a good idea to increase the load gradually, this way you will be able to correlate increasing load with the throughput and response time because running 5000 threads for 5 seconds will give you only a basic information on the response time for this workload and you won't be able to tell what is the maximum number of users/requests per second the application can support, when performance starts degrading, etc. So you could consider a load pattern like this:
and look at Response Times Over Time chart to see the relationship between the number of concurrent users and the application response time
I need to design a very basic Performance test for REST API.
no of concurrent users=40,
response time= less than 500ms
Usually what is ideal ratio between no of threads and ramp up period?
I read somewhere - Virtual users arrival rate should be something like 25-50 users per second. But I don't know its authenticity.
For my case shall I give ramp up period as 2 seconds?
Thanks in Advance!!!
The main purpose of the ramp-up period is that the load should be increased gradually (unless you're doing Spike Testing)
This way you will be able to correlate increasing load with increasing (hopefully) number of requests per second, response time, number of errors, etc. and if you will be kicking off threads without any ramp-up period - you will be able to state only that your application can (or cannot) support 40 concurrent users with (reasonable?) response time of 500 milliseconds.
As per JMeter documentation
Ramp-up needs to be long enough to avoid too large a work-load at the start of a test, and short enough that the last threads start running before the first ones finish (unless one wants that to happen).
Start with Ramp-up = number of threads and adjust up or down as needed.
So if you don't have better ideas you can go for the following setup:
40 seconds of ramp-up
couple of minutes to hold the load
optionally 40 seconds of ramp-down (you might be interested in decreasing the load gradually as well to see whether application goes back to normal when the load decreases)
More information: JMeter Ramp-Up - The Ultimate Guide
I'd not be overly concern about determining the ideal ramp-up time. Your goal, in general, should be to evaluate the application performance during the steady-state (after the ramp-up period and before the ramp-down period). Once you find a reasonable ramp-up period, use the following plugin (script) to discard such period(s), and then focus on analyzing the mentioned steady-state.
I am given a task in which I have to test my app for an hour with 300 TPS.
I have added my HTTP req in Jmeter and I need to configure below parameters to achieve my target
target concurrency
Ramp up time
Ramp-up steps
Hold target
thread iteration limit
I could guess by name few of these but still I am not quiet sure how to work around it to achieve my target. I am new to Jmeter can any one please help.
I assume you're talking about Concurrency Thread Group so answers are:
target concurrency - we don't know, it depends on your application response time, i.e. if response time is 1 second - go for 300, if response time is 2 seconds - go for 600, etc. If you don't know the response time you can put "sufficient" number of concurrent threads, for example 1000 and use Throughput Shaping Timer for limiting the load to 300 TPS
Ramp up time - again, we don't know your requirements. I can only recommend increasing the load gradually so you could correlate increasing load with increasing response time or number of errors so for example start with 1 TPS and increase the load to 300 TPS in i.e. 15 minutes and leave it running for 1 hour
Ramp-up steps - again, we don't know your requirements, start with above recommendation and adjust ramp-up period up or down as needed
Hold target - at least 60 minutes or 3600 seconds plus any ramp-up time
thread iteration limit - leave blank otherwise your test may end earlier than you expect
Example Concurrency Thread Group combination:
the values given in Schedule Feedback Function are for reference only, the example means that the Concurrency Thread Group will start 1000 threads initially with maximum of 1500 threads and have 300 spare threads in pool, hopefully it will be sufficient to produce 300 TPS load which is set by the Throughput Shaping Timer
I'm getting the following results, where the throughput does not have a change, even when I increase the number of threads.
Scenario#1:
Number of threads: 10
Ramp-up period: 60
Throughput: 5.8/s
Avg: 4025
Scenario#2:
Number of threads: 20
Ramp-up period: 60
Throughput: 7.8/s
Avg: 5098
Scenario#3:
Number of threads: 40
Ramp-up period: 60
Throughput: 6.8/s
Avg: 4098
The my JMeter file consists of a single ThreadGroup that contains a single GET.
When I perform the request for an endpoit where the response time faster (less than 300 ms) I can achieve throughput greater than 50 requests per seconds.
Can you see the bottleneck of this?
Is there a relationship between response time and throughput?
It's simple as JMeter user manual states:
Throughput = (number of requests) / (total time)
Now assuming your test contains only a single GET then Throughput will be correlate average response time of your requests.
Notice Ramp-up period: 60 will start to create threads over 1 minute, so it will add to total time of execution, you can try to reduce it to 10 or equal to Number of threads.
But you may have other sampler/controllers/component that may effect total time.
Also in your case especially in Scenario 3, maybe some requests failed then you are not calculating Throughput of successful transactions.
In ideal world if you increase number of threads by factor of 2x - throughput should increase by the same factor.
In reality the "ideal" scenario is hardly achievable so it looks like a bottleneck in your application. The process of identifying the bottleneck normally looks as follows:
Amend your test configuration to increase the load gradually so i.e. start with 1 virtual user and increase the load to i.e. 100 virtual users in 5 minutes
Run your test and look into Active Threads Over Time, Response Times Over Time and Server Hits Per Second listeners. This way you will be able to correlate increasing load with increasing response time and identify the point where performance starts degrading. See What is the Relationship Between Users and Hits Per Second? for more information
Once you figure out what is the saturation point you need to know what prevents your application from from serving more requests, the reasons could be in:
Application simply lacks resources (CPU, RAM, Network, Disk, etc.), make sure to monitor the aforementioned resources, this could be done using i.e JMeter PerfMon Plugin
The infrastructure configuration is not suitable for high loads (i.e. application or database thread pool settings incorrect)
The problem is in your application code (inefficient algorithm, large objects, slow DB queries). These items can be fetched using a profiler tool
Also make sure you're following JMeter Best Practices as it might be the case JMeter is not capable of sending requests fast enough due to either lack of resources on JMeter load generator side or incorrect JMeter configuration (too low heap, running test in GUI mode, using listeners, etc)