This is with respect to Jmeter's loop count behavior.
Number of threads 4000
ramp up period 800
Thread count 2
Action to be taken after a sample error Continue
Same user on each iteration Yes
Delay thread creation until needed Yes
This is resulting in 8000 requests being made in 800 seconds. However, my use case is, 4000 requests in 800 seconds (count=1), then another 4000 in next 800 seconds (count=2).
What changes can I make for this?
Ramp-up period doesn't mean 8000 requests in 800 seconds, it results in:
JMeter starts with 1 virtual user and adds 5 virtual users each second for 800 seconds
Each virtual user executes Samplers upside down for the specified number of iterations
When there are no more Samplers to execute and loops to iterate the thread is being shut down
My expectation is that you only have 1 Sampler and its response time is relatively low (less than 1 second), you can check the actual amount of virtual users and produced load using Active Threads Over Time and Transactions Per Second listeners
If you need to implement 4000 requests for 800 seconds twice the easiest option would be going for the Throughput Shaping Timer and configuring it to reach/maintain 5 requests per second throughput for 800 seconds twice with 10 seconds time frame of doing nothing between loops.
Related
i've started distributed performance testing using jmeter. If i give scenario 1:
no.of threads: 10
ramp up period: 1
loop count: 300
Everything runs smooth, as scenario 1 translates to 3000 requests in 300 seconds. i.e. 10 requests per second.
If i give scenario 2:
no.of threads: 100
ramp up period: 10
loop count: 30
Afaik, scenario2 is also executing 3000 requests in 300 seconds, i.e. 10 requests per second.
But things started failing i.e. sever is facing heavy load and requests fail. In theory both scenario1 and scenario2 should be same, right? Am i missing something?
All of these are heavy calls, each one will take 1-2 seconds under normal load.
In ideal world for scenario 2 you would have 100 requests per second and the test would finish in 30 seconds.
The fact that in 2nd case you have the same execution time indicates that your application cannot process incoming requests faster than 10 per second.
Try increasing ramp-up time for 2nd scenario and look into the following charts:
Active Threads Over Time
Response Times Over Time
Transactions Per Second
Normally when you increase the load the number of "Transactions Per Second" should increase by the same factor and "Response Time" should remain the same. Once response time starts growing and number of transactions per second starts decreasing it means that you passed the saturation point and discovered the bottleneck. You should report the point of maximum performance and investigate the reasons of the first bottleneck
More information: What is the Relationship Between Users and Hits Per Second?
In scenario 2 after 10 seconds you have 100 concurrent users which execute requests in parallel, your server may not handle well or prevent such load
Concurrent user load testing sends simultaneous artificial traffic to a web application in order to stress the infrastructure and record system response times during periods of sustained heavy load.
In scenario 1 after 10 seconds you have 10 concurrent users looping through the flow, without causing a load on server
Notice your server may have restriction on number of users requesting only on specific request(s)
We shall be very clear about the Rampup time
Following is extract from the official documentation
Scenario 1 : no.of threads: 10
ramp up period: 1
loop count: 300
In the above scenario 10 threads(virtual users) are to be created in 1 seconds. Each user will loop 300 times. Hence there will be 3000 requests to the server. Throughput cannot be calculated in advance with above configuration. It fluctuates based on the server capability, network etc. You could control the throughput with some components and plugins.
Scenario 2 : no.of threads: 100
ramp up period: 10
loop count: 30
In scenario 2 100 threads (virtual users) are created in 10 seconds. 100 virtual users will send requests concurrently to the server. Each user will send 30 requests. In the second scenario you will have higher throughput (number of requests per seconds) as compared to the scenario 1. Looks like server cannot handle the 100 users sending requests concurrently.
Ramp up time is applicable for the first cycle of each thread. It will simulate delays between first request of each user in their first iteration.
I am trying to send 1100 requests per minute to my API endpoint for a period of 5 minutes, so in total, I will make 5500 requests to the endpoint.
Based on the above requirement, Here is how I have set up my Jmeter:
It seems like I have misconfigured Jmeter, because in the end I can see Jmeter has made 8401 requests to the API instead of 5500.
What have I missed in the configuration?
Does the "Infinite" check-box need to be checked or unchecked?
The number of requests you're expecting (which is 5500) for this test plan is wrong.
As per your expectation, I think, you are making confusion between the Threads and Requests.
Ramp up is the time in which all the users arrive on your tested application server.
Requests are simulated by samplers but threads are the simulation of users.
According to your test plan:
- Total Number of Threads: 1100 vitual users
- Ramp-up time: 60 (1 min)
- Loop count: Infinite
- Test duration: 300 sec (5 min)
- No of requests/sampler in Test Plan: Unknown, assuming it 1
JMeter Execution:
JMeter will kick off those 1100 virtual users in 60 seconds according to the ramp-up time. So, (1100/60)~ 18 users will be active in every second for the first minute of your test. Each thread/user will execute the requests or samplers you have defined in your test plan hierarchy. As you have defined the loop count to infinite, each thread will execute the requests repeatedly until the test finishes. After 60 seconds, all of your 1100 users will be actively hitting those samplers/requests for the remaining 4 mins.
So the total number of requests you will be able to make through JMeter depends on your application response time.
If the avg response time of your requests is 1 second (assuming you have only 1 requests in your test plan), then you will be able to hit total 264000 (264000/240 = 1100 requests per second) for the last 4 mins (when all of your 1100 users are active) of your tests. You can also verify this accordingly by using Active Threads Over Time and Hits Per Second listeners.
So, Please double check the avg response time of your application or the requests/samplers you are using in your test plan.
If you want to control JMeter's throughput to 1100 per minute, you can use a Constant Throughput Timer at your test plan level and use target throughput value as 1100.
Do not forget to add the count of ramp-down time in your test duration. Yes, when you ramp up i.e. in the first minute generally you get more requests as threads are starting. In your case, your test duration should be 7 mins (60 seconds for the ramp-up 1100 users + 300 seconds for 5500 requests + 60 seconds for the ramp down for 1100 users).
You can also check this thread for more : How should I calculate Ramp-up time in Jmeter.
Please note- The total number of requests is related to throughput, Whereas the number of active threads performing the same activity is related to concurrency.
I am trying to send 1100 requests per minutes to my API endpoint for a period of 5 minutes, so in total I will make 5500 requests to the endpoint.
I don't know how you came with this 5500 number but this is not very correct.
Your setup means that:
JMeter will add 18 virtual users each second for 60 seconds
After 60 seconds pass JMeter will run 1100 users for another 4 minutes
The total number of requests JMeter will be able to make will mostly depend on application response time.
If you want to limit JMeter's throughput to 1100 requests per minute consider using Constant Throughput Timer or Precise Throughput Timer
If you don't want to limit JMeter's throughput but want 5500 executions either:
Set "Loop Count" to 5 (but in that case you might fail to get 1100 concurrent users)
Or use Throughput Controller in Total Executions mode so JMeter would stop after executing 5500 requests
I have a problem with JMeter that I can't figure out.
I have 3 requests in one thread group. Each request needs to be executed 5000 times in 5 minutes which means I need to execute 15 000 requests in 5 minutes altogether.
The thread group is set up like this:
Number of threads (users): 50
Ramp-up period: 1
Loop count: forever
Duration (seconds): 300
Each request has a Constant throughput Timer:
Target throughput: 1000.0
Calculate Throughput based on: all active threads in current thread group
And the results after the test run is complete are around 2000-3000 requests and that is too low. I can't figure out which settings are wrong.
Tests are ran in non-GUI mode on an Azure virtual machine with a Intel Xeon E5-2673 #2.3 GHz with 16 cores and 32gb of RAM and the server is more than capable of executing more requests since the response time is about 1 second so hardware should not pose a problem. JMeter heap memory is set to 12 GB.
Any help is appreciated!
I have 3 requests in one thread group. Each request needs to be executed 5000 times in 5 minutes which means I need to execute 15 000 requests in 5 minutes altogether.
That means you have to execute thread group for 5000 times in 5 mins => 300 seconds. If you execute thread group that much of time, you will automatically generate 15k requests.
From Documentation:-
Loop Count Number of times to perform the test case. Alternatively, "forever" can be selected causing the test to run until manually stopped.
When you set loop count: forever or any value in loop count for that matter, it means you want next set of users(threads) [which you have defined as 50 now], only when you are done with the first set 50 users request (150 requests as each thread group as 3 requests). Now, this 50 users request may take unexpected time and your next set will be returned only after this 50 users are done.
So to solve the problem at hand, you should have the following configuration for the thread group :
Number of threads (users): 5000
Ramp-up period: 300
Loop count: 1
So this means after every second around 17 Threads would be on-boarded.
To achieve your desired value configure the following
Number of threads (users): 50 / 60
Ramp-up period: 30
Loop count: forever
Duration (seconds): 300
It is good practice to give some ramp up time.
Also Constant Throughput Timer works on minute level and your desired throughput should be
Target throughput (in minutes): 3000.0
Calculate Throughput based on: all active threads in current thread group
So now, in 1 minute, JMeter will try to send 3000 requests and in 5 minutes, request count will be more or less around 15000.
Here are a few recommendations to tackle tests like this:
Use Ultimate Thread Group to create the threads required to run your tests.
Increase the total number of threads to 150-200 - Depending on the response time of your requests, 50 threads may be busy and you are not able to scale beyond certain throughput.
Add throughput shaping timer under Ultimate thread group, to maintain the throughput of 50RPS for 5 minutes(300 Seconds).
The combination of Ultimate Thread Group and Throughput Shaping Timer will allow you to have granular control over your requests and RPS you want to achieve.
Set Ulitmate Thread Group and Throughput Shaping Timer as below.
Put all your requests below Throughput Shaping Timer and you should be able to achieve the expected load.
Try to use Stepping thread group. It will allow you to send 50 user per second upto 300 second.
Download jar from below link.
https://jmeter-plugins.org/wiki/SteppingThreadGroup/
I am using Jmeter 4.0 with Throughput Shaping Timer and I have mentioned my configuration as follows:
bzm - Concurrency Thread Group:
Target Concurrency 1000
ramp up time : 1
ramp up steps: 1
Hold Target Rate: 100 min
jp#gc - Throughput Shaping Timer
Start RPS: 333 || End Rps: 333 || Duration(sec): 1200
Since the test duration is mentioned as 1200 seconds and Rps is 333/sec so number of request hits through the test should be (333*1200) = 399600. But actual number of hits coming in range of 400000 - 410000 Requests per second.
How can Throughput Shaping Timer be restricted to not send extra requests?
Your Total test duration isn't 1200 seconds. Looking into your Concurrency Thread Group Configurations, your test duration is exactly 6001 seconds (Ramp up for 1000 user is 1 sec and the Hold Target Rate time is 6000 seconds).
To get your desired RPS, You have to follow the below formula to define the number of threads in Concurrency Thread Groups:
Threads pool size can be calculated like RPS * <max response time> / 1000
If your response time is 1 second, then 333 Threads are enough to achieve this RPS. You have used more threads in this case I guess.
According to your given test plan, it is working like 1000 users are active in 1 second and then they will try to achieve 333 RPS for 1200 seconds and then they will maintain 1000 users requests for remaining time (6001-1220=4801 seconds) as you mentioned 1000 users will hold the load for 100 mins. For this reason, you are getting extra requests than desired.
So, Define number of threads and ramp up time accordingly in your Thread groups and also sync your test duration properly (in this case hold load time could be 20 mins not 100 mins).
JMeter is not capable of immediately stopping 1000 threads when the Throughput Shaping Timer reaches its duration limit, JMeter "tells" threads to stop one 1200 seconds pass and it might take a while to gracefully shut the threads down.
Given your setup the only way of having exactly 399600 samplers is using Throughput Controller in Total Executions mode like:
This way you will get confidence that not more than 339600 samplers will be executed (the number can be less by the way if your application response time will be higher than 300 ms)
There are many questions/answers available here to understand Ramp up time but I want to get something in detail for my test case.
Test Case : Expecting 1200 users per 5 minutes on the home page. So it will be like 5 users/second.
I have set following thread properties :
No. of Threads : 1200
Ramp-up Time - ? [I am not sure what to set]
Loop count - Forever
Scheduler - 300 Seconds[5 Minutes]
Can anyone please help me to set ramp up time for my test case? I am running the test on my local machine.
I want to check that how many users server can handle in 5 minutes. Our expectation is 1200 users.
As per you test case and duration settings, I think, you are making confusion with the Threads and Requests.
In short: You must use Ramp-up as 240 seconds for this scenario.
Let me describe you in details:
Ramp up is the time in which all the users arrive on your tested application server.
Requests are simulated by samplers but threads are the simulation of users.
Please note- The total number of requests are related to throughput, Whereas the number of active threads performing the same activity is related to concurrency.
From your requirements, I assume you want to measure the throughput which is related to the requests/second not the users per second. To achieve this, you can use a Constant Throughput Timer at your test plan level.
Constant Throughput timer allows you maintain throughput of your server (requests/sec). Here requests are samplers. Threads are users/clients which are requesting server using samplers.
You can achieve this by using both "Thread Group" or "Ultimate Thread Group".
As far as I know, it doesn't matter which type of Thread group you are using with Constant Throughput timer, at the end of the test in the results, you will get your desired Throughput which you mentioned in Constant Throughput Timer i.e. if you have mentioned 1200/min in Constant Throughput Timer with "Calculate Throughput based on" value as "All active threads" and there are 3 requests (samplers) in your thread group then JMeter will manage the requests in a way that it will generate only 6.6 requests/sec for each sampler i.e. 1200/min is divided among 3 requests, it doesn't matter whether you are using Ultimate Thread Group or the Thread Group; Throughput timer works in same way for all types of thread groups.
Now as per your requirement: (Using Thread Group):
No. of Threads : 1200
Ramp-up Time - 240 (Since you want 5 users per second, 1200/5= 240)
Loop count - Forever
Scheduler (Duration) - 780 Seconds [13 Minutes]
IMPORTANT:
Thread Group configurations will be the same which you have already mentioned. Number of threads will be the client load which you want to put(in you case it's 1200), duration : (240+300+240), loop : forever, ramp-up : 240 yes when you ramp up i.e. in the first minute generally you get more requests as threads are starting so it is normal behavior. To test your scenario run for exactly 5 minutes, you should test for 13 minutes. Exclude first 4 min and last 4 min of the test as that is warm up time for your test as well as server and last 4 min are warm down period.
Now as per your requirement: (Using Ultimate Thread Group):
Start Thread Count: 1200
Initial delay, sec: 0
Startup, sec: 240 (I assume you want 5 users come to your application for requesting expected samples per second)
Hold Load for: 300 sec (Since you required 5 minutes to test for 1200 users)
Shutdown Time, sec: 240
You can set just like this:
Don't get confuse the Thread number with the number of requests, each thread will create multiple requests for seconds in field Hold Load for.
Rampup is the speed at which users arrive on your application.
The number you put means that once it is reached all threads have been started.
The tiniest it is , the faster users arrive.
So nobody can answer that for you, you need to analyse your traffic.
Regarding the number of threads, 1200 threads might not be the good number, as 1200 users can navigate on application slowly or rapidly.
What you need is add a Constant Throughput Timer to control how much transactions per second you'll be reaching.
You should use the Ultimate Thread Group plugins available in Standard set.
To maintain concurrency the configuration should be like this:
Start Thread Count: 1200
Initial delay, sec: 0
Startup, sec: 60
Hold Load: 180
Shutdown Time, sec: 60
Ultimate Thread Group
Here all the 1200 threads will be generated and up within 60 sec of Startup time(Ramp up time) .Those 1200 threads will be active for next 180 sec, and after that, those threads will shut down within next 60 sec.
So overall duration of the test is 300 sec as you desire. Throughout the Hold Load time(180sec) the requests will be looped.
This ultimate thread group will give you more control over Threads.
Now if you want to control the rate of transactions per second or the number of samples to execute you can use Constant Throughput Timer. Constant Throughput Timer will let you control the flow of requests to be sent. The default "Thread Group" only control the way the users or threads will be generated.It doesn't control the rate of requests to be hit your application.So for this purpose, you could use "Constant Throughput Timer" to control your requests per second. For further detail, you could go through JMeter documentation.