I am using Jmeter for load testing and I'm new to this. I have an API where I want to send around 36000 requests in a given time, which is- 5 minutes. What should be the configuration of threads, ramp-up time, loop-count, and constant throughput timer for this scenario?
I am using the following configurations, but I am unable to reach the decided RPS-
Thread- 1000
Ramp-up- 5 Minute
loop-count 36
constant throughput timer- 7200
Where is my configuration wrong?
You can try to reduce the ramp-up period to be close to zero and increase the number of loops to "infinite", the total number of requests can be limited using Throughput Controller
In general there could be 2 main reasons of not being able to conduct the required load:
JMeter cannot produce the desired number of hits per second. Things to try:
Make sure to follow JMeter Best Practices
Increase number of threads in Thread Group
Consider switching to Distributed Testing mode
Application cannot handle that many requests per second. Things to try:
Inspect configuration and make sure it's suitable for high loads
Inspect CPU, RAM, Disk, etc. usage during the load test, it might be simply lack of resources, it can be done using JMeter PerfMon Plugin
Re-run your test with profiler tool telemetry enabled
Raise a ticket as it is a performance bottleneck
Related
I have used the normal thread group of 40 threads,180000 of throughput, and expected TPS is 3000.., But , Resulted only 2750 tps., How can I improve the TPS?
I have tried the concurrency thread group along with the shaping timer combination..
Can u pls guide me..,Thanks in Advance.
TPS or the throughput is controlled by sending the required number of requests to the target server in a unit time (seconds, minutes, hours, etc)
It is not easy or good practice to control the throughput by changing the number of threads. When the server gets slow the throughput gets low and vice-versa.
You need to introduce delays when the server responds faster and introduce more threads when the server responds slower. It is not easy to achieve the desired throughput by setting the thread count.
Please note that
throughput will be lower if the server is not capable of handling it, or if other timers or time-consuming test elements prevent it.
JMeter has a number of ways to achieve the desired throughput by introducing delays to the requests.
Using Timers
Constant Throughput Timer
Precise Throughput Timer
Througput Shaping Timer
In the above cases, you need to ensure a sufficient number of threads are available to generate the required throughputs. This solution is not efficient if a large number of threads are created than the required number of threads.
The best approach would be the test should create additional threads on-demand and introduce delays between requests as required.
In JMeter, you can achieve this with Concurrency Thread Group along with the Throughput Shaping Timer.
This video demonstrate the use of the controllers.
First of all your application must be able to handle such a load. I don't know what your 180000 of throughput, stanza means,if you put it to the Constant Throughput Timer be aware that:
it is precise enough on "minute" level
it can only "pause" JMeter to "limit" its throughput to the given value
In general you can reach out 3000 TPS with 40 users only if 1 user will make 75 requests per second which means that your application response time should be below 75 milliseconds.
If response time will be higher - you won't be able to reach 3000 requests per second with 40 users.
Also JMeter should be able to send requests fast enough, so make sure to follow JMeter Best Practices and if this is not sufficient consider using Distributed Testing
More information: What is the Relationship Between Users and Hits Per Second?
I'm using jmeter to generate a performance test, to keep things short and straight i read the initial data from a json file, i have a single thread group in which after reading the data i randomize certain values to prevent data duplication when i need it, then i'm passing the final data to the endpoint using variables, this will end up in a json body that is recieved by the endpoint and it will basically generate a new transaction in the database. Also i added a constant timer to add a 7 seconds delay between requests, with a test duration of 10 minutes and no ramp up, i calculated the requests per second like this:
1 minute has 60 seconds and i have a delay of 7 seconds per request then it's logical to say that every minute i'm sending approximately 8.5 requests per minute, this is my calculation (60/7) = 8.5 now if the test lasts for 10 minutes then i multiply (8.5*10) = 85 giving me a total of 85 transactions in 10 minutes, so i should be able to see that exact same amount of transactions created in the database after the test completes.
This is true when i'm running 10-20-40 users, after the load test run i query the db and i get the exact same number of transaction however, as i increase the users in the thread group this doesn't happen anymore, for example if i set 1000 users i should be able to generate 8500 transactions in 10 minutes, but this is not the case, the db only creates around 5.1k transactions.
What is happening, what is wrong? Why it initially works as expected and as i increase the users it doesn't? I can provide more information if needed. Please help.
There could be 2 possible reasons for this:
You discovered your application bottleneck. When you add more users the application response time increases therefore throughput decreases. There is a term called saturation point which stands for the maximum performance of the system, if you go beyond this point - the system will respond slower and you will get less TPS than initially. From the application under test side you should take a look into the following areas:
It might be the case your application simply lacks resources (CPU, RAM, Network, etc.), make sure that it has enough headroom to operate using i.e. JMeter PerfMon Plugin
Your application middleware (application server, database, load balancer, etc.) are not properly set up for the high loads. Identify your application infrastructure stack and make sure to follow performance tuning guidelines for each component
It is also possible that your application code needs optimization, you can detect the most time/resource consuming functions, largest objects, slowest DB queries, idle times, etc. using profiling tools
JMeter is not sending requests fast enough
Just like for the application under test check that JMeter machine(s) have enough resources (CPU, RAM, etc.)
Make sure to follow JMeter Best Practices
Consider going for Distributed Testing
Can you please check once CPU and Memory utilization(RAM and java heap utilization) of jmeter load generator while running jemter for 1000 users? If it is higher or reaching to max then it may affect requests/sec. Also just to confirm requests/sec from Jmeter side, can you please add listener in Jmeter script to track Hit/sec or TPS?
This will also be true(8.5K requests in 10 mins test duration) if your API response time is 1 second and also you have provided enough ramp-up time for those 1000 users.
So possible reason is:
You did not provide enough ramp-up time for 1000 users.
Your API average response time is more than 1 second while you performing tests for 1000 users.
Possible workarounds:
First, try to measure the API response time for 1 user.
Then calculate accordingly that how many users you need to reach 8500 requests in 10 mins. Use this formula:
TPS* max response time in second
Give proper ramp-up time for 1000 users. Check this thread to understand how you should calculate ramp-up time.
Check that your load generator is able to generate 1000 users without any memory or health (i.e CPU usage) issues. If requires, try to use distributed architecture.
Unable to achieve expected throughput by running Jmeter scripts as expected throughput is more however getting very less.
Running Jmeter script by targeting 1000 requests per second (Business SLA) hence used 'Constant Throughput Timer' or 'Throughput shaping timer' as suggested in below queries.
Constant Throughput Timer:
Target - 60,000/min (60 secs) - all active threads, Threads (Users) - 200 with Ramp up - 1 sec, Duration: 1 hour.
or Users - 2000 or tried with 10,000 users as well.
Result: Ended the execution with throughput showing 50/sec with average response time 50 seconds.
Jmeter: Scenario to test 5 users with ramp up 1 hour to trigger 10 thousand requests
unable to increase average throughput in jmeter
Throughput shaping timer: Reflecting setup as above.
In both the cases getting 'Throughput' around 50/sec with average response around 30 secs.
When looked at server metrics - CPU and memory consumption is very less around 3% only.
Hence expected 'Throughput' is high as the server resources are not used much, if 'Throughput' is low and continuously sending requests with Forever to see if can get 500 response code errors, it just increases the average response time and reduces Throughput but not getting 500 response code errors.
After some time getting socket exceptions, connection reset issues where Jmeter is running but no failures seen at server side.
Gone through similar queries here however unable to understand when server resources not used much and as per Server platform SLAs, it supports 1000 RPS but unable to achieve through Jmeter.
As per CTT calculation: RPS * / 1000
1000 * 50000/1000 = 50000 (Should give threads upto 50K? however our SLA is only for 200 users).
It might be the case your server isn't capable of responding fast enough. Low CPU and Memory consumption means that the server has enough headroom, however the application server configuration can be incorrect so the server doesn't fully utilise it's hardware resources. Another reason could be inefficient algorithms used in your application code, you can use a profiler tool to see what's going on when you're conducting the load
It might be the case JMeter is not capable of sending requests fast enough. Make sure that the machine where JMeter is running is not overloaded, that you run your JMeter test in non-GUI mode and in general follow JMeter Best Practices. You can also try running JMeter in distributed mode in case if one machine is not capable of creating the required load.
Our requirement is to execute
1 user/thread, 10 requests/sec, Total requests = 10,000
With current Thread Properties not able to achieve.
Only one user/thread send 10 request/sec and total should be 10000 requests. Any other way to achieve this in Jmeter ?
Is following approach is correct ? We used Loop controller. So each request will repeat 10 times
You will be able to achieve 10 requests/second with 1 virtual user only if your response time will be 100 milliseconds. If response time will be more than 100 milliseconds - you will not be able to reach the desired load.
If your application fails to respond in 100 milliseconds time frame - most probably you've found a performance bottleneck and you can report this to your application developers.
If you have some time to investigate the issue you can try to provide more information, i.e. what is the average response time, what are minimum, maximum, quantiles, the actual number of requests per second, etc, all this information can be retrieved using Aggregate Report listener.
Normally when people are looking for the answer if the application under test can support X requests per second they use > 1 virtual user as load test should represent real life usage of the system under test and 1 thread (virtual user) isn't normally associated with performance testing and derivatives.
So probably you should reconsider your approach to testing and try increasing the number of threads (virtual users). The throughput can be controlled using Precise Throughput Timer or Constant Throughput Timer. However be aware that the above timers can only pause JMeter to slow it down to the desired throughput. Another approach is using Concurrency Thread Group and Throughput Shaping Timer combination, they can be connected using Feedback Function so JMeter will kick off extra threads in order to reach/maintain the desired number of requests per second.
You can install Concurrency Thread Group and Throughput Shaping Timer using JMeter Plugins Manager
I'm getting the following results, where the throughput does not have a change, even when I increase the number of threads.
Scenario#1:
Number of threads: 10
Ramp-up period: 60
Throughput: 5.8/s
Avg: 4025
Scenario#2:
Number of threads: 20
Ramp-up period: 60
Throughput: 7.8/s
Avg: 5098
Scenario#3:
Number of threads: 40
Ramp-up period: 60
Throughput: 6.8/s
Avg: 4098
The my JMeter file consists of a single ThreadGroup that contains a single GET.
When I perform the request for an endpoit where the response time faster (less than 300 ms) I can achieve throughput greater than 50 requests per seconds.
Can you see the bottleneck of this?
Is there a relationship between response time and throughput?
It's simple as JMeter user manual states:
Throughput = (number of requests) / (total time)
Now assuming your test contains only a single GET then Throughput will be correlate average response time of your requests.
Notice Ramp-up period: 60 will start to create threads over 1 minute, so it will add to total time of execution, you can try to reduce it to 10 or equal to Number of threads.
But you may have other sampler/controllers/component that may effect total time.
Also in your case especially in Scenario 3, maybe some requests failed then you are not calculating Throughput of successful transactions.
In ideal world if you increase number of threads by factor of 2x - throughput should increase by the same factor.
In reality the "ideal" scenario is hardly achievable so it looks like a bottleneck in your application. The process of identifying the bottleneck normally looks as follows:
Amend your test configuration to increase the load gradually so i.e. start with 1 virtual user and increase the load to i.e. 100 virtual users in 5 minutes
Run your test and look into Active Threads Over Time, Response Times Over Time and Server Hits Per Second listeners. This way you will be able to correlate increasing load with increasing response time and identify the point where performance starts degrading. See What is the Relationship Between Users and Hits Per Second? for more information
Once you figure out what is the saturation point you need to know what prevents your application from from serving more requests, the reasons could be in:
Application simply lacks resources (CPU, RAM, Network, Disk, etc.), make sure to monitor the aforementioned resources, this could be done using i.e JMeter PerfMon Plugin
The infrastructure configuration is not suitable for high loads (i.e. application or database thread pool settings incorrect)
The problem is in your application code (inefficient algorithm, large objects, slow DB queries). These items can be fetched using a profiler tool
Also make sure you're following JMeter Best Practices as it might be the case JMeter is not capable of sending requests fast enough due to either lack of resources on JMeter load generator side or incorrect JMeter configuration (too low heap, running test in GUI mode, using listeners, etc)