We are attempting to stress test our system using Jmeter and we hit a wall with inconsistent throughput, so we installed the Jmeter plugin Concurrency Thread Group with the Throughput Shaping Timer. We're currently running it on a single endpoint, to make sure we can achieve a consistent throughput. we set:
Throughput Shaping Timer to 30 RPS for 60 seconds, which is scoped directly under the Concurrency Thread Group
Target Concurrency using the tstFeedback function as tstFeedback(tst,13,50,10), since the max response time was around 1000
we initially achieved the expected RPS, it would reach close to 30 RPS every time we attempted the test:
However, after a couple of hours with no change to the configuration or the system, the RPS no longer reached the expected, we would consistently get lower RPS (ranging between 20 and 27 RPS):
We tried increasing the Target Concurrency, but it made no difference to the RPS and it was constantly being less than the expected. We attempted this using the GUI and CLI from a local machine and CLI on a remote instance with Jmeter installed, we are running the latest Jmeter version (5.5 at this time) locally and on the remote instance but it made no difference:
Increasing the HEAP didn’t make a difference either
When we increased the logging level on the local machine to diagnose the problem, this error would appear frequently:
o.a.j.p.h.s.HTTPHC4Impl: IOException
java.net.SocketException: Socket closed
Test instance specs
The remote instance is set up on an AWS m6i.8xlarge instance
Local machine has 16GB Memory with intel i7 8th gen processor
All tests were run on Ubuntu machines
Problem/Question
We need to simulate our customer behavior as accurately as possible, so it's critical that we're able to achieve consistent throughput in our requests. How can we do that?
Looking at your Throughput Shaping Timer configuration I can see that it's set up to kick off 50 maximum threads.
You can reach 30 requests per second with 50 threads only if your response time will be less than 1.6 seconds or less.
Looking into response times from your reports I can see that average response time is above 2 seconds and this perfectly explains the fact you're not able to reach the throughput.
Things to check:
Make sure to follow JMeter Best Practices
Make sure that JMeter has enough headroom to operate in terms of CPU, RAM, Network, Disk, etc. It can be done using JMeter PerfMon Plugin
If you cannot conduct required load from the single JMeter instance - consider going for Distributed Testing
And finally it might be your application which cannot handle that many requests per second so it worth checking its logs, telemetry, and so on.
Related
I am searching how to increase the number of sent requests per second .
I used 80 threads and set a Constant Throughput Timer and Variable throughput timer and i set 10 transactions per second and i when i get Jmeter logs i see max 4 transactions per second .
I also changed the heap variable to this and i have a virtual machine with 32 GO
but when i supervise the conssumed RAM for the jmeter process , i find only 1.4 % .
JMeter waits for getting the response before sending the next request therefore the throughput (number of transactions per unit of time) mainly depends on the system under test response time.
One possible situation I can think of is that you have too few loops (iterations) defined on Thread Group level
if you don't have sufficient number of iterations to cover the ramp-up period and "plateau" you may run into the situation when some threads had already have finished their work and had been shut down and some were not started yet. So double check that you have anticipated concurrency using i.e. Active Threads Over Time listener as it is possible that you have much less than 80 online virtual users. See JMeter Test Results: Why the Actual Users Number is Lower than Expected article for more comprehensive explanation if needed.
If you follow JMeter Best Practices and ensure that JMeter has enough headroom to operate in terms of CPU, RAM, network, etc. - it might be the case that the application under test cannot process more than 4 requests per second so whatever you do on JMeter side won't have any impact. So I'd rather check your application logs, CPU, RAM usage, look at profiler tool output, etc.
I am using Jmeter for load testing and I'm new to this. I have an API where I want to send around 36000 requests in a given time, which is- 5 minutes. What should be the configuration of threads, ramp-up time, loop-count, and constant throughput timer for this scenario?
I am using the following configurations, but I am unable to reach the decided RPS-
Thread- 1000
Ramp-up- 5 Minute
loop-count 36
constant throughput timer- 7200
Where is my configuration wrong?
You can try to reduce the ramp-up period to be close to zero and increase the number of loops to "infinite", the total number of requests can be limited using Throughput Controller
In general there could be 2 main reasons of not being able to conduct the required load:
JMeter cannot produce the desired number of hits per second. Things to try:
Make sure to follow JMeter Best Practices
Increase number of threads in Thread Group
Consider switching to Distributed Testing mode
Application cannot handle that many requests per second. Things to try:
Inspect configuration and make sure it's suitable for high loads
Inspect CPU, RAM, Disk, etc. usage during the load test, it might be simply lack of resources, it can be done using JMeter PerfMon Plugin
Re-run your test with profiler tool telemetry enabled
Raise a ticket as it is a performance bottleneck
I'm using jmeter to generate a performance test, to keep things short and straight i read the initial data from a json file, i have a single thread group in which after reading the data i randomize certain values to prevent data duplication when i need it, then i'm passing the final data to the endpoint using variables, this will end up in a json body that is recieved by the endpoint and it will basically generate a new transaction in the database. Also i added a constant timer to add a 7 seconds delay between requests, with a test duration of 10 minutes and no ramp up, i calculated the requests per second like this:
1 minute has 60 seconds and i have a delay of 7 seconds per request then it's logical to say that every minute i'm sending approximately 8.5 requests per minute, this is my calculation (60/7) = 8.5 now if the test lasts for 10 minutes then i multiply (8.5*10) = 85 giving me a total of 85 transactions in 10 minutes, so i should be able to see that exact same amount of transactions created in the database after the test completes.
This is true when i'm running 10-20-40 users, after the load test run i query the db and i get the exact same number of transaction however, as i increase the users in the thread group this doesn't happen anymore, for example if i set 1000 users i should be able to generate 8500 transactions in 10 minutes, but this is not the case, the db only creates around 5.1k transactions.
What is happening, what is wrong? Why it initially works as expected and as i increase the users it doesn't? I can provide more information if needed. Please help.
There could be 2 possible reasons for this:
You discovered your application bottleneck. When you add more users the application response time increases therefore throughput decreases. There is a term called saturation point which stands for the maximum performance of the system, if you go beyond this point - the system will respond slower and you will get less TPS than initially. From the application under test side you should take a look into the following areas:
It might be the case your application simply lacks resources (CPU, RAM, Network, etc.), make sure that it has enough headroom to operate using i.e. JMeter PerfMon Plugin
Your application middleware (application server, database, load balancer, etc.) are not properly set up for the high loads. Identify your application infrastructure stack and make sure to follow performance tuning guidelines for each component
It is also possible that your application code needs optimization, you can detect the most time/resource consuming functions, largest objects, slowest DB queries, idle times, etc. using profiling tools
JMeter is not sending requests fast enough
Just like for the application under test check that JMeter machine(s) have enough resources (CPU, RAM, etc.)
Make sure to follow JMeter Best Practices
Consider going for Distributed Testing
Can you please check once CPU and Memory utilization(RAM and java heap utilization) of jmeter load generator while running jemter for 1000 users? If it is higher or reaching to max then it may affect requests/sec. Also just to confirm requests/sec from Jmeter side, can you please add listener in Jmeter script to track Hit/sec or TPS?
This will also be true(8.5K requests in 10 mins test duration) if your API response time is 1 second and also you have provided enough ramp-up time for those 1000 users.
So possible reason is:
You did not provide enough ramp-up time for 1000 users.
Your API average response time is more than 1 second while you performing tests for 1000 users.
Possible workarounds:
First, try to measure the API response time for 1 user.
Then calculate accordingly that how many users you need to reach 8500 requests in 10 mins. Use this formula:
TPS* max response time in second
Give proper ramp-up time for 1000 users. Check this thread to understand how you should calculate ramp-up time.
Check that your load generator is able to generate 1000 users without any memory or health (i.e CPU usage) issues. If requires, try to use distributed architecture.
Unable to achieve expected throughput by running Jmeter scripts as expected throughput is more however getting very less.
Running Jmeter script by targeting 1000 requests per second (Business SLA) hence used 'Constant Throughput Timer' or 'Throughput shaping timer' as suggested in below queries.
Constant Throughput Timer:
Target - 60,000/min (60 secs) - all active threads, Threads (Users) - 200 with Ramp up - 1 sec, Duration: 1 hour.
or Users - 2000 or tried with 10,000 users as well.
Result: Ended the execution with throughput showing 50/sec with average response time 50 seconds.
Jmeter: Scenario to test 5 users with ramp up 1 hour to trigger 10 thousand requests
unable to increase average throughput in jmeter
Throughput shaping timer: Reflecting setup as above.
In both the cases getting 'Throughput' around 50/sec with average response around 30 secs.
When looked at server metrics - CPU and memory consumption is very less around 3% only.
Hence expected 'Throughput' is high as the server resources are not used much, if 'Throughput' is low and continuously sending requests with Forever to see if can get 500 response code errors, it just increases the average response time and reduces Throughput but not getting 500 response code errors.
After some time getting socket exceptions, connection reset issues where Jmeter is running but no failures seen at server side.
Gone through similar queries here however unable to understand when server resources not used much and as per Server platform SLAs, it supports 1000 RPS but unable to achieve through Jmeter.
As per CTT calculation: RPS * / 1000
1000 * 50000/1000 = 50000 (Should give threads upto 50K? however our SLA is only for 200 users).
It might be the case your server isn't capable of responding fast enough. Low CPU and Memory consumption means that the server has enough headroom, however the application server configuration can be incorrect so the server doesn't fully utilise it's hardware resources. Another reason could be inefficient algorithms used in your application code, you can use a profiler tool to see what's going on when you're conducting the load
It might be the case JMeter is not capable of sending requests fast enough. Make sure that the machine where JMeter is running is not overloaded, that you run your JMeter test in non-GUI mode and in general follow JMeter Best Practices. You can also try running JMeter in distributed mode in case if one machine is not capable of creating the required load.
I'm trying to stress-test my Spring RESTful Web Service.
I run my Tomcat server on a Intel Core 2 Duo notebook, 4 GB of RAM. I know it's not a real server machine, but i've only this and it's only for study purpose.
For the test, I run JMeter on a remote machine and communication is through a private WLAN with a central wireless router. I prefer to test this from wireless connection because it would be accessed from mobile clients. With JMeter i run a group of 50 threads, starting one thread per second, then after 50 seconds all threads are running. Each thread sends repeatedly an HTTP request to the server, containing a small JSON object to be processed, and sleeping on each iteration for an amount of time equals to the sum of a 100 milliseconds constant delay and a random value of gaussian distribution with standard deviation of 100 milliseconds. I use some JMeter plugins for graphs.
Here are the results:
I can't figure out why mi hits per seconds doesn't pass the 100 threshold (in the graph they are multiplied per 10), beacuse with this configuration it should have been higher than this value (50 thread sending at least three times would generate 150 hit/sec). I don't get any error message from server, and all seems to work well. I've tried even more and more configurations, but i can't get more than 100 hit/sec.
Why?
[EDIT] Many time I notice a substantial performance degradation from some point on without any visible cause: no error response messages on client, only ok http response messages, and all seems to work well on the server too, but looking at the reports:
As you can notice, something happens between 01:54 and 02:14: hits per sec decreases, and response time increase, okay it could be a server overload, but what about the cpu decreasing? This is not compatible with the congestion hypothesis.
I want to notice that you've chosen very well which rows to display on Composite Graph. It's enough to make some conclusions:
Make note that Hits Per Second perfectly correlates with CPU usage. This means you have "CPU-bound" system, and the maximum performance is mostly limited by CPU. This is very important to remember: server resources spent by Hits, not active users. You may disable your sleep timers at all and still will receive the same 80-90 Hits/s.
The maximum level of CPU is somewhere at 80%, so I assume you run Windows OS (Win7?) on your machine. I used to see that it's impossible to achieve 100% CPU utilization on Windows machine, it just does not allow to spend the last 20%. And if you achieved the maximum, then you see your installation's capacity limit. It just has not enough CPU resources to serve more requests. To fight this bottleneck you should either give more CPU (use another server with higher level CPU hardware), or configure OS to let you use up to 100% (I don't know if it is applicable), or optimize your system (code, OS settings) to spend less CPU to serve single request.
For the second graph I'd suppose something is downloaded via the router, or something happens on JMeter machine. "Something happens" means some task is running. This may be your friend who just wanted to do some "grep error.log", or some scheduled task is running. To pin this down you should look at the router resources and jmeter machine resources at the degradation situation. There must be a process that swallows CPU/DISK/Network.