JMeter: More HTTP Requests Result in Increased Performance? - performance

I'm trying to understand a significant performance increase in my Jmeter test.
In a multi-tenancy database environment, I have a single RESTful service test containing a Thread Group with a single HTTP Request sampler posting an XML payload. The XML payload is then evaluated via stored procedures, and a response is received stating if the claim was qualified. I run this test from a .bat file (non-gui mode) in an Apache 7 environment with a single JVM running.
Test Thread Group Properties
# of Threads: ${__P(test.threads,200)}
Ramp-Up Period: ${__P(test.rampup,1)}
Loop Count: Forever
Delay Thread: Enabled
Scheduler: Enabled
Duration: ${__P(test.duration,1800)}
HTTP Request
Method: POST
https://serverName:port/database/.../${__P(tenant,1111)}/Claim/${__property(contractId)}
When I duplicate the HTTP Request sampler within the TG and change the tenant ID within the URL, for some reason the performance seems to increase by > 55%. (i.e., the # of claims/second is increased by 55%) It appears the test did not fail, so I cannot attribute the performance increase to an increased error rate.
I would have expected an increase if I had enabled another JVM to let the Load Balancer perform optimization, but this is not the case. (still using only 1 JVM)
HTTP Request 1
https://serverName:port/database/.../${__P(tenant,1111)}/Claim/${__property(contractId)}
HTTP Request 2
https://serverName:port/database/.../${__P(tenant,2222)}/Claim/${__property(contractId)}
The theory going around here is that Jmeter generates a workload at a higher rate for multiple requests than for a single request. I'm skeptical, but haven't found anything "solid" to support my skepticism.
Is this theory true? If so, why would two HTTP Requests increase the performance?

In short: it's OK.
Longer version:
Here is how JMeter works:
JMeter starts all the threads during ramp-up period
Each thread starts executing samplers upside down (or according to the Logic Controllers)
When request doesn't have more samplers to execute and no more loops to iterate it's being shut down.
So how does number of virtual users correlate with the "performance". When you increase virtual users number of requests number for a load test it affects Throughput
Throughput is calculated as requests/unit of time. The time is calculated from the start of the first sample to the end of the last sample. This includes any intervals between samples, as it is supposed to represent the load on the server.
The formula is: Throughput = (number of requests) / (total time).
So if you increase load on well-behaved system throughput should increase by the same factor or linearly.
When you increase load but throughput does not increase, such situation is known as "saturation point" when you get the maximum performance from the system. Further load increasing will lead to throughput going down.
References:
Apache JMeter Glossary
An extended Glossary version

And how do you messure your performance? According to your "theory" your messurements includes jmeter overhead and this would be wrong. More over, is the response the same for both cases? What I mean, is the backend doing the same work work both cases?
Maybe first request returns different output then the other one. Maybe it is more expensive to generate output in one of the request. That is why you will notice "incresed" performance as normally you would do N x Heavy task in X seconds, and in second case G x heavy tasks + H X light tasks in the same time where G < N/2 - more requests in the same time? Sure! Incresed perfomance? Nope.
So to completly investigate what is happening, you need to review your measurement method. I would start with comparing the the actual time for both requests.

Related

Request per second in jemeter do not exceed 4 per second

I am searching how to increase the number of sent requests per second .
I used 80 threads and set a Constant Throughput Timer and Variable throughput timer and i set 10 transactions per second and i when i get Jmeter logs i see max 4 transactions per second .
I also changed the heap variable to this and i have a virtual machine with 32 GO
but when i supervise the conssumed RAM for the jmeter process , i find only 1.4 % .
JMeter waits for getting the response before sending the next request therefore the throughput (number of transactions per unit of time) mainly depends on the system under test response time.
One possible situation I can think of is that you have too few loops (iterations) defined on Thread Group level
if you don't have sufficient number of iterations to cover the ramp-up period and "plateau" you may run into the situation when some threads had already have finished their work and had been shut down and some were not started yet. So double check that you have anticipated concurrency using i.e. Active Threads Over Time listener as it is possible that you have much less than 80 online virtual users. See JMeter Test Results: Why the Actual Users Number is Lower than Expected article for more comprehensive explanation if needed.
If you follow JMeter Best Practices and ensure that JMeter has enough headroom to operate in terms of CPU, RAM, network, etc. - it might be the case that the application under test cannot process more than 4 requests per second so whatever you do on JMeter side won't have any impact. So I'd rather check your application logs, CPU, RAM usage, look at profiler tool output, etc.

Getting so high average response time in Jmeter

I am testing a scenario with 400 threads. Although I am almost getting no errors, I have very high average response. What can bring about this problem? Seems like server gives no time-out but gives response so late. I've addded the summary report. It is as follows:
This table doesn't tell the full story, if response time seems "so high" to you - this is definitely the bottleneck and you can report it already.
What you can do to localize the problem is:
Consider using a longer ramp-up period, i.e. start with 1 user and add 1 more user every 5 seconds (adjust these numbers according to your scenario) so you would have arrival phase, the "plateau" and the load decrease phase. This approach will allow you to correlate increasing load and increasing response time by looking at Active Threads Over Time and Response Times Over Time charts. This way you will be able to state that:
response time remains the same up to X concurrent users
after X concurrent users it starts growing so throughput is going down
after Z concurrent users response time exceeds acceptable threshold
It would also be good to see CPU, RAM, etc. usage on the server side as increased response time might be due to lack of resources, you can use JMeter PerfMon Plugin for this
Inspect your server configuration as you might need to tune it for high loads (same applies to JMeter, make sure to follow JMeter Best Practices)
Use a profiler tool on server side during the next test execution, it will show you the slowest places in your application code

JMeter load is not increasing when we increase the threads count

Load is not getting increased in AppD when we increase the threads count in JMeter. For example, if we are able to achieve 100k calls/min for 500 users with 20 ms avg response time. Load remains at 135 k with 25ms avg resp time when we give 1200 as threads in JMeter. Even we increase the load thrice, load is not going to the application. Didn't observe errors in JMeter as well. We are giving host entries in all the load generators. Is that could be a reason for controlling the load from JMeter? Request help to resolve this issue please
If you increase the number of threads and the throughput is not increasing there could be 2 possible reasons for this:
The throughput is not increasing as response time is increasing which indicates your application performance bottleneck, look at Response Times vs Threads and the point where the response time starts increasing will match the maximum number of users your application can support
JMeter is not capable of sending requests fast enough due to lack of resources or improper configuration, make sure to follow JMeter Best Practices and if it doesn't help - consider allocating more load generator machines and switch to Distributed Mode of JMeter tests execution
As depicted in the following graph, the relationship between the load (Uvsers) and KPIs such as Throughput (TP) is non-linear. At certain point, increasing the load will not result on an increased Throughput that is proportional/linear (the Heavy Load zone).

Limitations in achieving a target JMeter throughput value

I want to understand the actual JMeter throughput behavior achieved at runtime.
Scenario - I'm increasing the JMeter throughput at runtime using Constant Throughput Timer and beanshell script as described here - https://www.blazemeter.com/blog/how-to-change-jmeters-load-during-runtime.
Test plan - Along with CTT as described above, simple threadgroup with fixed #threads and infinite loop iterations is configured. HTTP Sampler for GET call is used. No other timer or plugin is added in the test plan.
As I keep on increasing the target throughput of JMeter at runtime, I noted that the actual achieved throughput value is limited by mainly 2 factors -
The threads in my thread group.
The performance bottleneck of target app.
I have questions regarding both the limitations -
Once the highest throughput is achieved using all threads in current threadgroup (assuming there're no errors from the target app yet), is there a way to increase the #threads at this point dynamically at runtime to achieve a higher JMeter throughput?
Now as I keep on increasing JMeter throughput, it can't be increased further due to errors from the target app. How does the JMeter identify performance bottleneck of my target app and react to it? Does it add any delay or kill threads or apply any such mechanism to reduce it's throughput to the max that the target app can sustain?
In continuation with point #2, if JMeter identifies and reacts to the performance bottleneck by any method, what are the factors (like error rate, response latency etc.) that control it's throughput to keep it within the max limit of the target app? Are these factors configurable or extensible?
You can play the same trick with the number of threads in the thread group, just define it using __P() function and you will be able to manipulate it using Beanshell server. Another option is using a JSR223 Test Element and Groovy language to add the new thread(s) where/when required like:
ctx.getThreadGroup().addNewThread(0, ctx.getEngine())
JMeter doesn't "identify" anything, it just tries to execute Samplers as fast as it can and the number of requests per second depends on 2 factors:
your application needs to be able to respond fast enough
JMeter itself needs to be able to send requests fast enough, i.e. you need to follow JMeter Best Practices
there are no "mechanisms" which detect the application under test behaviour, the closest solution is Auto Stop Listener
See point 2

JMeter sending less requests than expected

I'm using jmeter to generate a performance test, to keep things short and straight i read the initial data from a json file, i have a single thread group in which after reading the data i randomize certain values to prevent data duplication when i need it, then i'm passing the final data to the endpoint using variables, this will end up in a json body that is recieved by the endpoint and it will basically generate a new transaction in the database. Also i added a constant timer to add a 7 seconds delay between requests, with a test duration of 10 minutes and no ramp up, i calculated the requests per second like this:
1 minute has 60 seconds and i have a delay of 7 seconds per request then it's logical to say that every minute i'm sending approximately 8.5 requests per minute, this is my calculation (60/7) = 8.5 now if the test lasts for 10 minutes then i multiply (8.5*10) = 85 giving me a total of 85 transactions in 10 minutes, so i should be able to see that exact same amount of transactions created in the database after the test completes.
This is true when i'm running 10-20-40 users, after the load test run i query the db and i get the exact same number of transaction however, as i increase the users in the thread group this doesn't happen anymore, for example if i set 1000 users i should be able to generate 8500 transactions in 10 minutes, but this is not the case, the db only creates around 5.1k transactions.
What is happening, what is wrong? Why it initially works as expected and as i increase the users it doesn't? I can provide more information if needed. Please help.
There could be 2 possible reasons for this:
You discovered your application bottleneck. When you add more users the application response time increases therefore throughput decreases. There is a term called saturation point which stands for the maximum performance of the system, if you go beyond this point - the system will respond slower and you will get less TPS than initially. From the application under test side you should take a look into the following areas:
It might be the case your application simply lacks resources (CPU, RAM, Network, etc.), make sure that it has enough headroom to operate using i.e. JMeter PerfMon Plugin
Your application middleware (application server, database, load balancer, etc.) are not properly set up for the high loads. Identify your application infrastructure stack and make sure to follow performance tuning guidelines for each component
It is also possible that your application code needs optimization, you can detect the most time/resource consuming functions, largest objects, slowest DB queries, idle times, etc. using profiling tools
JMeter is not sending requests fast enough
Just like for the application under test check that JMeter machine(s) have enough resources (CPU, RAM, etc.)
Make sure to follow JMeter Best Practices
Consider going for Distributed Testing
Can you please check once CPU and Memory utilization(RAM and java heap utilization) of jmeter load generator while running jemter for 1000 users? If it is higher or reaching to max then it may affect requests/sec. Also just to confirm requests/sec from Jmeter side, can you please add listener in Jmeter script to track Hit/sec or TPS?
This will also be true(8.5K requests in 10 mins test duration) if your API response time is 1 second and also you have provided enough ramp-up time for those 1000 users.
So possible reason is:
You did not provide enough ramp-up time for 1000 users.
Your API average response time is more than 1 second while you performing tests for 1000 users.
Possible workarounds:
First, try to measure the API response time for 1 user.
Then calculate accordingly that how many users you need to reach 8500 requests in 10 mins. Use this formula:
TPS* max response time in second
Give proper ramp-up time for 1000 users. Check this thread to understand how you should calculate ramp-up time.
Check that your load generator is able to generate 1000 users without any memory or health (i.e CPU usage) issues. If requires, try to use distributed architecture.

Resources