I am trying to understand why I am seeing such different results when running the basic Thread Group vs Concurrency Thread group load test.
To narrow down the issue, I tried running Thread Group with just one thread/user for some time and got average, median, and p90 - all around 1,500 milliseconds.
But when I run with the Concurrency Thread group, the aggregate report shows below 70ms no matter if it's 1 or 300 RPS. Postman shows a response time of 1.3 seconds on average. So, I am thinking something is wrong with metrics captured by concurrency thread group.
My setup is very simple. Just one HTTP request that runs as a part of Thread Group in one case, and Blazemeter's Concurrency Thread group in 2nd case.
Thread Group report, with just 1 thread, and 1 RPS:
Concurrency Thread Group report, with 30 threads and 300 RPS:
Active threads for concurrent thread group:
Active threads for the basic thread group with 1 user:
Response time graph for concurrent group:
Sampler detailed view for basic thread group:
Sampler detailed view for concurrent thread group:
Take a look at i.e. Active Threads Over Time chart (can be installed using JMeter Plugins Manager) or equivalent from the HTML Reporting Dashboard and you will find a lot of new information regarding your setup because the tests are not really the same
With regards to response time of 70 ms - I can think of 2 possible cases:
Establishing TCP and especially SSL connection can take a lot of time for first iteration and subsequent iterations will be much faster. If you let "normal" Thread Group or Postman iterate more than once you will see similar results
It might be the case your server responds very fast with some form of an error message with HTTP Status Code 200 so it worth adding i.e. Response Assertion to check that the system under test returns expected response.
Related
I need to run a load test of 25 threads, what will be the most efficient configuration to use? (ramp-up period....). I ran the load test with the below configuration and some of the threads failed but pass if I just ran the script individually.
Your configuration will mean that:
JMeter start with 1 thread and add another thread each 4 seconds
Once started each thread will begin to execute Samplers upside down (or according to the logic controllers)
When the thread executes the last sampler it will be shut down
When the last thread executes the last sampler the test ends
Depending on the number of samplers and application response time you may or may not achieve 25 users concurrency, you might want to check the actual number of concurrent users using Active Threads Over Time listener
If you want to make sure to have 25 online users set "Loop Count" to Infinite and "Specify Thread Lifetime" duration to be more than your ramp-up period. See JMeter Test Results: Why the Actual Users Number is Lower than Expected article for more details.
With regards to the failures - we cannot state anything meaningful without seeing request and response details, make sure to save them using i.e. View Results Tree listener and inspect response body for the failed requests
i am running a simple test with 10 users for 5 mins. my test has a single thread and different transactions within it. By the end of the test, there is a different number of samples for each transaction. Is there a way or a setting so that we don't start any new thread when the test is finishing, i tried ultimate thread group plugin with ramp down , hoping to see same samples for all requests but it didn't happen. Not sure if that is even possible.
Each JMeter thread (virtual user) executes Samplers upside down (or according to the Logic Controllers)
When you're defining test duration it might be the case that some virtual users are somewhere in the "middle" of the Samplers and they stop when they receive shutdown signal.
It means that the very first sampler will be executed with 100% of users and for further samplers the chance of getting executed becomes lower and lower.
The only way to have the same number of Sample Results for all Samplers is using normal JMeter Thread Group and providing fixed amount of loops.
Given above setup you will have strictly 80 executions of each and every Sampler in the Test Plan
More information: Getting Started with JMeter - A Basic Tutorial
I want to understand the relation between Stepping up Thread Groups and Number of Samples.
Stepping Thread Group
My Test Plan looks like this:
Test Plan
But in the Summary report I am getting #Samples more than 10 for certain Requests. Not able to understand how is it working.
The actual number of Sample Results will not necessarily be 10 as:
JMeter starts 1 thread each 2 second until the number of threads will be 10
Once started thread starts executing Samplers upside down
When thread finishes executing Samplers and 60 seconds of duration has not yet passed the threads starts over
So the actual number of Samplers which will be executed in 60 seconds by 10 threads will depend mostly on your application response time.
Check out What is the Relationship Between Users and Hits Per Second? article to learn more about correlation of number of virtual users and delivered load
While understanding concept of Concurrent thread and ultimate thread group, i am confused to understand result of summary/aggregate report while running concurrent thread or ultimate thread group .For example if i have 200 user and ramp up time 60 sec then i didn't see all sampler request as 200 sample after completing execution successfully but only few sampler request have 200 sample. when i use normal thread group then i always got thread count same for each sampler request after completing execution .
for realistic load testing with more user , could you please suggest me which thread group should prefer.
Could you please provide valuable guidance with some valuable link/book and also share me standard performance bench mark criteria or key parameter detail while doing load testing.(if any bench load parameter value does not meet standard then we can say that there is a performance issue)
Thanks for giving valuable time in advance.
Thanks
amit
This is due to the fact that:
Your application response time is too high
Your test duration is too low
For example I can see response times > 80 seconds:
it means that if a single virtual user has cumulative response time for 2 samplers > 160 seconds and the test duration is 120 seconds it will not be able to execute all the requests. Just increase your test duration to be i.e. 10 minutes and you should see more virtual users capable of executing all the Samplers you defined in the test plan.
Also given first user is capable of executing all the requests successfully and in time it looks like that your application gets overloaded hence cannot respond fast enough when the number of concurrent users reaches some "critical threshold", you can add listeners like Active Threads Over Time and Response Times vs Threads, this way you will be able to correlate increasing load with the increasing response time.
If also makes sense to collect:
Baseline health metrics of your application (CPU, RAM, Network, Disk usage, etc.), it can be done using JMeter PerfMon Plugin.
Lower level details like slowest methods, largest objects, heaviest database queries, etc. This form of information can be obtained using profiling tools specific to your application programming language(s).
I have concerns about the duration of executing JMeter Test Scenario.
In fact, if we execute a manual test with 1 user in 1 hour, we will find the same duration or little more with 3 user.
But, with JMeter test the duration with 1 user will be multiplied by the number of users.
During my scripting with this tool, I've noticed that JMeter always wait for the response of the request to pass to another request. It's like we have 1 user doing the work of multiple users.
Does any body have explanations about this issue?
Can we configure JMeter to perform like we have x users working in // ?
Depends how you are simulating users..
Each thread will wait for a response, but users are represented by different threads, and threads do not wait for each other to complete samples, they are independent.
If you are using threads in thread group to represent users (which is the intended usage), the threads will start concurrently and run independently, so you should see the 3 users complete in an hour, as with manual tests. If this is not happening, then you should check the resources used on the client running jmeter during single thread and multi thread runs. Or perhaps the bottleneck may be in the transport (ie bandwidth). All you have determined so far is that the bottleneck is prior to the server, you need to determine where it is.
It is also possible you have created a thread group for each simulated user. If this is the case you can set the groups to run consecutive or concurrent. The setting for this is on the Test Plan element at the very top of the tree. You want concurrent, so untick 'Run Thread Groups Consecutively'.
So it comes down to how you are simulating users.
In ideal world if you test your site with 1 user all set of actions completes in 1 hour. If you add another 100 users - still one hour, 1000 users - 1 hour.
In real world server response time increases with the number of concurrent threads. But it isn't something like
1 user - average response time 10ms
2 users - 20 ms
3 users - 30 ms
But
1 user - 10 ms
2 users - 10 ms
3 users - 10 ms
10 users - 11 ms
The whole idea of the performance testing is to determine the upper limit of server load and identify bottlenecks and issues which happen under the load so you could state something like: using this hardware and this configuration the application is capable of serving 500 concurrent users without serious delays. In case of 1000 users response time will be more than 15 seconds.
In regards to threads, JMeter thread starts, executes samples one by one for defined amount of loops and exits. It is applicable for each thread which are independent. Basic throughput control can be done via ramp-up and loops count (see Thread Group documentation on details)
More advanced load scenario definition can be done using following Test Elements:
Ultimate Thread Group - which provides easy visual way of defining ramp-up, ramp-down and time to hold the load
Constant Throughput Timer - to set desired load rate in requests per minute
Synchronizing Timer - to pause all the test threads and release them at the same moment to produce a "spike"