I configured a load test in visual studio with the following settings:
Load Pattern: Step
Initial User Count: 500
Maximum User Count: 1000
Step Duration (seconds): 300
Step Ramp Time (seconds): 0
Step User Count: 250
I need to start the test with 500 users, then increment to 750 users and finish with 1000 users.
So, in my inform, in the Key Indicators section, I expect to see:
Load users:
min: 500
max: 1000
avg: 750
But really I see:
Load users:
min: 500
max: 500
avg: 500
I don't know what I'm doing wrong.
Related
I have an api that returns student data when i send a student id to it .Now I want to test my api with 1000 users hitting every seconds.
My laptop configuration - Core i5 , ram 8 gB
My jmeter completes test but the threads show an error
summary + 2642 in 00:00:30 = 88.0/s Avg: 938 Min: 59 Max: 130375 Err: 3 (0.11%) Active: 993 Started: 1000 Finished: 7
Generate Summary Results = 14824 in 00:02:54 = 85.3/s Avg: 1699 Min: 59 Max: 130375 Err: 54 (0.36%)
summary = 14824 in 00:02:54 = 85.3/s Avg: 1699 Min: 59 Max: 130375 Err: 54 (0.36%)
Generate Summary Results + 2636 in 00:00:30 = 87.9/s Avg: 613 Min: 59 Max: 15489 Err: 2 (0.08%) Active: 977 Started: 1000 Finished: 23
Generate Summary Results = 17460 in 00:03:24 = 85.7/s Avg: 1535 Min: 59 Max: 130375 Err: 56 (0.32%)
summary + 2636 in 00:00:30 = 87.9/s Avg: 614 Min: 59 Max: 15489 Err: 2 (0.08%) Active: 977 Started: 1000 Finished: 23
summary = 17460 in 00:03:24 = 85.7/s Avg: 1535 Min: 59 Max: 130375 Err: 56 (0.32%)
:
:
:
:
summary = 17460 in 00:010:24 = 123.7/s Avg: 5535 Min: 59 Max: 130375 Err: 723 (70.3%)
By the end around 723 threads failed
My api is returning response if i run the test with 100 users the test is successful but when i run with 1000 users most of the threads get failed or remain active after completion
JMeter default configuration is not suitable for high loads, you need to tune it in order to be able to kick off 1000 threads.
Make sure to use the latest JMeter version and 64-bit version of Server JRE or JDK
Increase JVM Heap size allocated to JMeter to ~6 gigabytes
Disable (or delete) listeners in Test Plan (if any)
Make sure to monitor CPU and RAM usage on the machine where JMeter is running during the test, you can use JMeter PerfMon plugin for that. JMeter must have enough headroom to operate, if it will lack RAM or CPU it will not be able to send requests fast enough. If you see that JMeter machine is overloaded - you will have to consider distributed testing
Harwdare requirements will greatly vary depending on your test nature, i.e. number of pre/post processors, assertions, request and response sizes, application response time, etc. so there is no mapping like X hardware = Y virtual users, you will need to assess it for each and every different test plan. Remember to increase the load gradually and keep an eye on health metrics
Lets calculate it using simple math. At max thread size of HTTP request will be approx. 1MB(Apart from payload).
1MB*1000= 1000MB = 1 GB at-least heap size is required for 1000 concurrent users.
500 MB - for additional tasks like listeners, aggregate result
Total is at-least 1500 MB is required to run this test. configure jmeter to use memory like this in /apache-jmeter/bin/jmeter file.
Multithreading is also depends on CPU capability, so try to use multicore CPU.
What is a thread on a CPU?
In computer architecture, multithreading is the ability of a central processing unit (CPU) (or a single core in a multi-core processor) to execute multiple processes or threads concurrently, supported by the operating system.
I am still confused with some JMeter logs displayed here. Can someone please give me some light into this?
Below is a log generated by JMeter for my tests.
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary + 1 in 00:00:02 = 0.5/s Avg: 1631 Min: 1631 Max: 1631 Err: 0 (0.00%) Active: 2 Started: 2 Finished: 0
summary + 218 in 00:00:25 = 8.6/s Avg: 816 Min: 141 Max: 1882 Err: 1 (0.46%) Active: 10 Started: 27 Finished: 17
summary = 219 in 00:00:27 = 8.1/s Avg: 820 Min: 141 Max: 1882 Err: 1 (0.46%)
summary + 81 in 00:00:15 = 5.4/s Avg: 998 Min: 201 Max: 2096 Err: 1 (1.23%) Active: 0 Started: 30 Finished: 30
summary = 300 in 00:00:42 = 7.1/s Avg: 868 Min: 141 Max: 2096 Err: 2 (0.67%)
Tidying up ... # Fri Jun 09 04:19:15 IDT 2017 (1496971155116)
Does this log means [ last step ] 300 requests were fired, 00.00:42 secs took for the whole tests, 7.1 threads/sec or 7.1 requests/sec fired?
How can i make sure to increase the TPS? Same tests were done in a different site and they are getting 132TPS for the same tests and on the same server. Can someone put some light into this?
In here, total number of requests is 300. Throughput is 7 requests per second. These 300 requests generated by your given number of threads in Thread group configuration. You can also see the number of active threads in the log results. These threads become active depend on your ramp-up time.
Ramp-up time is the speed at which users or threads arrive on your application.
Check this for an example: How should I calculate Ramp-up time in Jmeter
You can give enough duration in your script and also check the loop count forever, so that all of the threads will be hitting those requests in your application server until the test finishes.
When all the threads become active on the server, then they will hit those requests in server.
To increase the TPS, you must have to increase the number of threads because those threads will hit your desired requests in server.
It also depends on the response time of your requests.
Suppose,
If you have 500 virtual users and application response time is 1 second - you will have 500 RPS
If you have 500 virtual users and application response time is 2 seconds - you will have 250 RPS
If you have 500 virtual users and application response time is 500 ms - you will have 1000 RPS.
First of all, a little of theory:
You have Sampler(s) which should mimic real user actions
You have Threads (virtual users) defined under Thread Group which mimic real users
JMeter starts threads which execute samplers as fast as they can and generate certain amount of requests per second. This "request per second" value depends on 2 factors:
number of virtual users
your application response time
JMeter Summarizer doesn't tell the full story, I would recommend generating the HTML Reporting Dashboard from the .jtl results file, it will provide more comprehensive load test result data which is much easier to analyze looking into tables and charts, it can be done as simple as:
jmeter -g /path/to/testresult.jtl -o /path/to/dashboard/output/folder
Looking into current results, you achieved maximum throughput of 7.1 requests second with average response time of 868 milliseconds.
So in order to have more "requests per second" you need to increase the number of "virtual users". If you increase the number of virtual users and "requests per second" is not increasing - it means that you identified so called saturation point and your application is not capable of handling more.
can somebody please advise how to create a jmeter thread properties correctly with the following requirements
55 user a min ramp-up over an hour with a test running for 4 hours.
If you need 55 users added each minute during 1 hour, you set up should look like:
Number of Threads: 3300 (55 users x 60 minutes)
Ramp-up: 3600 (1 hour == 3600 seconds)
Loop Count: Forever
Scheduler -> Duration: 14400 (3600 seconds in hour x 4)
Be aware that 3300 concurrent threads is quite a high load, make sure that you're following recommendations from JMeter Performance and Tuning Tips guide.
If you won't be able to create such a load from a single machine consider Distributed Testing when one JMeter master machine orchestrates several slaves, for instance 3 slaves having 1100 virtual users each.
So you think about something like this setup I show on the screenshot below, isn't ot? It is set on jp#gc - Stepping Thread Group that comes with jmeter plugins in standard set. You have there 55 users tat will ramp up to that value through 3600 seconds and will hold that load for next 3 hours (10800 sec).
If I run distributed testing in GUI mode with 4 client servers(1 Master + 3 Slave) and set the following values in my Plan -
Number of Threads = 12000
Ramp Up time = 1000
Loop count = 1
After completion of Test I get 36000 Samples (which is ok as 12000 * 3 = 36000) but my question is for ramp up time - will it be 3000 for 36000 users??
or will it remain 1000 for 36000
Thanks in Advance
It will be 1000, the same for all client.
Note such a load test profile seems strange as running only 1 request is not what happens with usual usage, what are you trying to simulate ?
I need to test if our system can perform N requests per second.
Technically, it's 2 requests to one API, 2 requests to another, and 6 requests to third one.
But the important thing that they should happen simultaneously - so 10 requests per second.
So, in JMeter I've created three Thread Groups, first defines number of threads 1, and ramp-up time 0.
Second thread group is the same, and third thread group defines number of threads 6 and ramp-up time 0.
But that doesn't really guarantee it's going to run them per second
How do I emulate that? And how do I see the results -- if it was able to perform or wasn't?
Thanks!
You could use ConstantThroughputTimer.
Quote from JMeter help files below:
18.6.4 Constant Throughput Timer
This timer introduces variable pauses, calculated to keep the total throughput (in terms of samples per minute) as close as possible to a give figure. Of course the throughput will be lower if the server is not capable of handling it, or if other timers or time-consuming test elements prevent it.
N.B. although the Timer is called the Constant Throughput timer, the throughput value does not need to be constant. It can be defined in terms of a variable or function call, and the value can be changed during a test.
For example I've used it to generate 40 requests per second:
<ConstantThroughputTimer guiclass="TestBeanGUI" testclass="ConstantThroughputTimer" testname="Constant Throughput Timer" enabled="true">
<stringProp name="calcMode">all active threads in current thread group</stringProp>
<doubleProp>
<name>throughput</name>
<value>2400.0</value>
<savedValue>0.0</savedValue>
</doubleProp>
</ConstantThroughputTimer>
And thats a summary:
Created the tree successfully using performance/search-performance.jmx
Starting the test # Tue Mar 15 16:28:39 CET 2011 (1300202919244)
Waiting for possible shutdown message on port 4445
Generate Summary Results + 3247 in 80,3s = 40,4/s Avg: 18 Min: 0 Max: 1328 Err: 108 (3,33%)
Generate Summary Results + 7199 in 180,0s = 40,0/s Avg: 15 Min: 1 Max: 2071 Err: 378 (5,25%)
Generate Summary Results = 10446 in 260,3s = 40,1/s Avg: 16 Min: 0 Max: 2071 Err: 486 (4,65%)
Generate Summary Results + 7200 in 180,0s = 40,0/s Avg: 14 Min: 0 Max: 152 Err: 399 (5,54%)
Generate Summary Results = 17646 in 440,4s = 40,1/s Avg: 15 Min: 0 Max: 2071 Err: 885 (5,02%)
Generate Summary Results + 7199 in 180,0s = 40,0/s Avg: 14 Min: 0 Max: 1797 Err: 436 (6,06%)
Generate Summary Results = 24845 in 620,4s = 40,0/s Avg: 15 Min: 0 Max: 2071 Err: 1321 (5,32%)
But I run this test inside my network.
As with any network test, there's always going to be problems, especially with latency - even if you could send exactly 6 per second, they're going to be sent sequentially (that's just how packets get sent) and may not all hit in that second, plus processing time.
Generally when performance metrics specific x per second, it's measured over a period of time. Your API may even have a buffer - so you could technically send 6 per second, but process 5 per second, with a buffer of 20, meaning it'd be fine for 20 seconds of traffic, as you'd have sent 120, which would take 120/5 = 24 seconds to process. But any more than that would overflow the buffer. So to just send exactly 6 in a second to test is insufficient.
In the thread group, you're right setting number of threads (users) to 6. Then run it looping forever (tick it or put it in a while loop) and add a listener like aggregate report and results tree. The results you can use to check the right stuff is being sent and responded to (assuming you validate the responses) and in the aggregate report, you can see how many of each activity is happening per hour (obviously multiply by 3600 for seconds, but because of this inaccuracy it's best to run it for a good length of time).
The initial load test can now be run, and as a more accurate test, you can leave it running for longer (soak test) to see if any other problems surface - buffer overflows, memory leaks, or other unexpected events.
Use the Throughput Shaping Timer
I had similar problem and here are two solutions I found:
Solution 1:
You can use Stepping Thread Group (allows to set thread number increase stages over set periods of time) with Constant Throughput Timer in it.
Throughput Timer allows you to set number of samples that thread can send per minute (e.g. if you set it to 1, the thread will only send one request per minute). Also, you can apply Throughput Timer to all threads in your Thread Group or have Timer for each thread with its own settings.
Read more about Throughput Timer here: https://www.blazemeter.com/blog/how-use-jmeters-throughput-constant-timer
Solution 2:
Use "SetUp Thread Group". You can calculate thread number and rump up time to get Threads per Second desired.
You can use Schedule Feedback Function and will also need Concurrency Thread Group
Same can Also be done by configuring "ConstantThroughputTimer" as suggested above from UI also by adding "Constant Throughput Timer" by navigating by right click on Thread Group and then click on Timer and then choose the "Constant Throughput Timer".