I use Ultimate Thread Group for performance testing in JMeter. In combination with a timer I want to scale the number of users/threads dynamically while each user has a constant throughput of e.g. 10 requests/sec.
When the Ultimate Thread Group increases the number of users, the total request rate per second will have a high peak for a short time. After the peak, the request rate is as expected.
For example, I get this trace:
Start with 40 users, 10 requests per second for each user: 40 * 10 = 400 requests/second.
Increase the number of users up to 60, I expect 60 * 10 = 600 requests/second, but I get a peak of more than 3959 requests/second at the beginning.
summary = 366401 in 00:15:16 = 400.0/s Avg: 1 Min: 0 Max: 1071 Err: 0 (0.00%)
summary + 12000 in 00:00:30 = 400.0/s Avg: 1 Min: 0 Max: 20 Err: 0 (0.00%) Active: 40 Started: 40 Finished: 0
summary = 378401 in 00:15:46 = 400.0/s Avg: 1 Min: 0 Max: 1071 Err: 0 (0.00%)
summary + 12000 in 00:00:30 = 400.0/s Avg: 1 Min: 0 Max: 44 Err: 0 (0.00%) Active: 40 Started: 40 Finished: 0
summary = 390401 in 00:16:16 = 400.0/s Avg: 1 Min: 0 Max: 1071 Err: 0 (0.00%)
summary + 31784 in 00:00:30 = 1061.9/s Avg: 4 Min: 0 Max: 414 Err: 0 (0.00%) Active: 60 Started: 60 Finished: 0
summary = 422185 in 00:16:46 = 419.7/s Avg: 1 Min: 0 Max: 1071 Err: 0 (0.00%)
summary + 118770 in 00:00:30 = 3959.0/s Avg: 5 Min: 0 Max: 493 Err: 0 (0.00%) Active: 60 Started: 60 Finished: 0
summary = 540955 in 00:17:16 = 522.2/s Avg: 2 Min: 0 Max: 1071 Err: 0 (0.00%)
summary + 79419 in 00:00:30 = 2647.3/s Avg: 10 Min: 0 Max: 1435 Err: 0 (0.00%) Active: 60 Started: 60 Finished: 0
summary = 620374 in 00:17:46 = 582.0/s Avg: 3 Min: 0 Max: 1435 Err: 0 (0.00%)
summary + 37227 in 00:00:30 = 1238.1/s Avg: 6 Min: 0 Max: 1354 Err: 0 (0.00%) Active: 60 Started: 60 Finished: 0
summary = 657601 in 00:18:16 = 600.0/s Avg: 3 Min: 0 Max: 1435 Err: 0 (0.00%)
summary + 18000 in 00:00:30 = 600.3/s Avg: 2 Min: 0 Max: 219 Err: 0 (0.00%) Active: 60 Started: 60 Finished: 0
summary = 675601 in 00:18:46 = 600.0/s Avg: 3 Min: 0 Max: 1435 Err: 0 (0.00%)
summary + 18000 in 00:00:30 = 599.9/s Avg: 1 Min: 0 Max: 46 Err: 0 (0.00%) Active: 60 Started: 60 Finished: 0
visualized it looks like this:
Is there a way to avoid this peak? (I also get this peak if I increase the number of users from 60 to 80.80 to 100, ...)
In combination with which "timer"?
I think you're using the wrong timer and the wrong thread group for your scenario, consider moving to Concurrency Thread Group and Throughput Shaping Timer, you can connect them using feedback function so the timer will be the only place which will drive the load pattern.
Jmeter non-gui mode generates real time Test Results in the below format:
summary = 1649731 in 00:10:06 = 2721.2/s Avg: 47 Min: 9 Max: 16027 Err: 112 (0.01%)
summary + 63699 in 00:00:30 = 2123.3/s Avg: 64 Min: 12 Max: 2080 Err: 4 (0.01%) Active: 192 Started: 193 Finished: 1
summary = 1713430 in 00:10:36 = 2693.0/s Avg: 47 Min: 9 Max: 16027 Err: 116 (0.01%)
summary + 62509 in 00:00:30 = 2083.6/s Avg: 66 Min: 11 Max: 2034 Err: 1 (0.00%) Active: 192 Started: 193 Finished: 1
summary = 1775939 in 00:11:06 = 2665.6/s Avg: 48 Min: 9 Max: 16027 Err: 117 (0.01%)
summary + 69708 in 00:00:30 = 2323.5/s Avg: 45 Min: 9 Max: 2074 Err: 4 (0.01%) Active: 123 Started: 193 Finished: 70
summary = 1845647 in 00:11:36 = 2650.9/s Avg: 48 Min: 9 Max: 16027 Err: 121 (0.01%)
What is summary + and summary = ?
Summary + are incremental for the latest summariser period
Summary = are cumulative for the above Summary+
summary + 63699 in 00:00:30 = 2123.3/s
summary = 1713430 in 00:10:36 = 2693.0/s
Example 63699 in 00:00:30 = 2123.3/s means that in 30seconds Jmeter have sent 63699 requests to the server with an average throughput of 2123 requests per second.
The next iteration will start from the ahead of the cumulative time of the above summary. Ex: 00:11:06 = (00:00:30 + 00:10:36)
As per Top 2 Techniques to Get JMeter Test Results in non-GUI mode
Here’s an example of what you get from the default summariser:
summary + 41 in 15.4s = 2.7/s Avg: 2234 Min: 383 Max: 6974 Err: 0 (0.00%)
summary + 57 in 21.5s = 2.6/s Avg: 2548 Min: 618 Max: 4528 Err: 0 (0.00%)
summary = 98 in 32.5s = 3.0/s Avg: 2416 Min: 383 Max: 6974 Err: 0 (0.00%)
summary + 108 in 21.8s = 5.0/s Avg: 1291 Min: 229 Max: 6317 Err: 0 (0.00%)
summary = 206 in 52.5s = 3.9/s Avg: 1827 Min: 229 Max: 6974 Err: 0 (0.00%)
The lines with summary + are incremental for the latest summariser period, the lines with summary = are cumulative. The above was with a summariser period of 20 secs, the actual periods can sometimes be longer than the specified period and the length of the very first period is somewhat random. You get the throughput statistics as well as average, min and max response times, and how many errors were detected (assuming your JMeter test plan has assertions to detect errors).
If the JMeter's summariser output doesn't seem too informative to you consider running your JMeter test using Taurus tool as a wrapper, it has console reporter which prints current metrics right in your terminal:
You can also configure Backend Listener to store results to InfluxDB and plot them using Grafana, an example dashboard can look like:
I have a test case with just one HTTP request. The number of threads and Ramp Up period is set to 30. And i planning to do the test for 600 seconds.
But looking at the JMeter output, i could see a huge number of requests generated. Why is this so? I am expecting only 60 threads in a minute to generate. Can someone suggest me how to do this?
S:\roshTests\apache-jmeter-3.1\bin>jmeter -n -t
S:\roshTests\LivePerformance\June1\PPS-Proxy\catalog\catalog.jmx -l
S:\roshTests\LivePerformance\June1\PPS-Proxy\catalog\Output\results.csv
Writing log file to: S:\roshTests\apache-jmeter-3.1\bin\jmeter.log
Creating summariser <summary>
Created the tree successfully using S:\roshTests\LivePerformance\June1\PPS-Proxy\catalog\catalog.jmx
Starting the test # Fri Jun 02 05:18:04 IDT 2017 (1496369884546)
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary + 377 in 00:00:25 = 15.3/s Avg: 766 Min: 106 Max: 3114 Err: 198 (52.52%) Active: 25 Started: 25 Finished: 0
summary + 1364 in 00:00:30 = 45.5/s Avg: 647 Min: 144 Max: 4385 Err: 1180 (86.51%) Active: 30 Started: 30 Finished: 0
summary = 1741 in 00:00:55 = 31.9/s Avg: 673 Min: 106 Max: 4385 Err: 1378 (79.15%)
summary + 985 in 00:00:30 = 32.8/s Avg: 643 Min: 88 Max: 4918 Err: 752 (76.35%) Active: 30 Started: 30 Finished: 0
summary = 12859 in 00:05:25 = 39.6/s Avg: 616 Min: 88 Max: 6149 Err: 10562 (82.14%)
summary + 984 in 00:00:30 = 32.8/s Avg: 675 Min: 113 Max: 5070 Err: 762 (77.44%) Active: 30 Started: 30 Finished: 0
summary = 11874 in 00:04:55 = 40.3/s Avg: 614 Min: 100 Max: 6149 Err: 9810 (82.62%)
summary + 1103 in 00:00:30 = 36.7/s Avg: 624 Min: 104 Max: 4935 Err: 884 (80.15%) Active: 30 Started: 30 Finished: 0
summary = 10890 in 00:04:25 = 41.2/s Avg: 609 Min: 100 Max: 6149 Err: 9048 (83.09%)
summary + 1220 in 00:00:30 = 40.6/s Avg: 584 Min: 100 Max: 6149 Err: 1001 (82.05%) Active: 30 Started: 30 Finished: 0
summary = 9787 in 00:03:55 = 41.7/s Avg: 607 Min: 100 Max: 6149 Err: 8164 (83.42%)
summary + 1336 in 00:00:30 = 44.5/s Avg: 570 Min: 106 Max: 5296 Err: 1106 (82.78%) Active: 30 Started: 30 Finished: 0
summary = 8567 in 00:03:25 = 41.9/s Avg: 610 Min: 106 Max: 5643 Err: 7163 (83.61%)
summary + 1320 in 00:00:30 = 44.0/s Avg: 587 Min: 116 Max: 5187 Err: 1111 (84.17%) Active: 30 Started: 30 Finished: 0
summary = 7231 in 00:02:55 = 41.4/s Avg: 618 Min: 106 Max: 5643 Err: 6057 (83.76%)
summary + 1273 in 00:00:30 = 42.5/s Avg: 637 Min: 106 Max: 5383 Err: 1072 (84.21%) Active: 30 Started: 30 Finished: 0
summary = 5911 in 00:02:25 = 40.9/s Avg: 625 Min: 106 Max: 5643 Err: 4946 (83.67%)
summary + 1430 in 00:00:30 = 47.6/s Avg: 591 Min: 116 Max: 5643 Err: 1229 (85.94%) Active: 30 Started: 30 Finished: 0
summary = 4638 in 00:01:55 = 40.5/s Avg: 621 Min: 106 Max: 5643 Err: 3874 (83.53%)
summary + 1467 in 00:00:30 = 48.9/s Avg: 589 Min: 119 Max: 4691 Err: 1267 (86.37%) Active: 30 Started: 30 Finished: 0
summary = 3208 in 00:01:25 = 37.9/s Avg: 635 Min: 106 Max: 4691 Err: 2645 (82.45%)
summary + 1070 in 00:00:30 = 35.7/s Avg: 651 Min: 99 Max: 7324 Err: 862 (80.56%) Active: 30 Started: 30 Finished: 0
summary = 13929 in 00:05:55 = 39.3/s Avg: 619 Min: 88 Max: 7324 Err: 11424 (82.02%)
summary + 1531 in 00:00:30 = 51.1/s Avg: 594 Min: 106 Max: 5194 Err: 1332 (87.00%) Active: 30 Started: 30 Finished: 0
summary = 15460 in 00:06:25 = 40.2/s Avg: 617 Min: 88 Max: 7324 Err: 12756 (82.51%)
I'm not sure if you understand the difference between Threads and requests. amount of Threads isn't equal to requests, what you're seeing in the output is the amount of requests made by your threads, and depending on your response times it's very well possible to get that many results.
Try setting the Loop count to 1 or 2 and you'll see that the requests will even out with your calculations.
Beside that... in your output on the far right side you can see how many Threads are started.
With your current setup, threads will start with one second delay.
Ramp up period is the time in which all treads should start.
You have loop set to infinite, so when all thread login after initial 30 seconds, you will always have 30 active users, and could never have more that that.
You have many more requests, depending on response time of your server, but threads are like users. If you set 30, it will always be 30. :-)
You are running 30 virtual users for 10 minutes. Each virtual user is executing samplers as fast as it can (the speed mostly depends on your application response time) this is why you have a lot of requests.
If you need to limit JMeter throughput to 60 requests per second only you can do it using Constant Throughput Timer:
Add Constant Throughput Timer to your Test Plan
Set "Target Troughput" to 60
Set "Calculate Troughput based on" to All active threads
This way JMeter will "pause" its threads (virtual users) to slow them down to 60 requests per minute.
Sometimes the summary didn't get update while executing JMeter script in Non GUI Mode, Please find below data.
Starting the test # Wed Jan 04 18:41:18 IST 2017 (1483535478893)
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary + 28 in 00:00:11 = 2.5/s Avg: 3376 Min: 1991 Max: 7555 Err: 0 (0.00%) Active: 37 Started: 37 Finished: 0
summary + 192 in 00:00:30 = 6.4/s Avg: 5623 Min: 812 Max: 19523 Err: 0 (0.00%) Active: 137 Started: 137 Finished: 0
summary = 220 in 00:00:41 = 5.4/s Avg: 5337 Min: 812 Max: 19523 Err: 0 (0.00%)
summary + 196 in 00:00:30 = 6.6/s Avg: 9388 Min: 813 Max: 46485 Err: 1 (0.51%) Active: 200 Started: 200 Finished: 0
summary = 416 in 00:01:11 = 5.9/s Avg: 7246 Min: 812 Max: 46485 Err: 1 (0.24%)
summary + 159 in 00:00:31 = 5.2/s Avg: 11175 Min: 796 Max: 79142 Err: 0 (0.00%) Active: 200 Started: 200 Finished: 0
summary = 575 in 00:01:42 = 5.6/s Avg: 8332 Min: 796 Max: 79142 Err: 1 (0.17%)
summary + 50 in 00:00:29 = 1.7/s Avg: 39434 Min: 820 Max: 108309 Err: 1 (2.00%) Active: 200 Started: 200 Finished: 0
summary = 625 in 00:02:11 = 4.8/s Avg: 10820 Min: 796 Max: 108309 Err: 2 (0.32%)
summary + 104 in 00:00:30 = 3.5/s Avg: 44422 Min: 846 Max: 125336 Err: 0 (0.00%) Active: 196 Started: 200 Finished: 4
summary = 729 in 00:02:41 = 4.5/s Avg: 15614 Min: 796 Max: 125336 Err: 2 (0.27%)
summary + 189 in 00:00:30 = 6.3/s Avg: 37163 Min: 5912 Max: 168988 Err: 0 (0.00%) Active: 172 Started: 200 Finished: 28
summary = 918 in 00:03:11 = 4.8/s Avg: 20051 Min: 796 Max: 168988 Err: 2 (0.22%)
summary + 208 in 00:00:30 = 6.9/s Avg: 27117 Min: 5369 Max: 180636 Err: 0 (0.00%) Active: 132 Started: 200 Finished: 68
summary = 1126 in 00:03:41 = 5.1/s Avg: 21356 Min: 796 Max: 180636 Err: 2 (0.18%)
summary + 232 in 00:00:30 = 7.7/s Avg: 34571 Min: 4706 Max: 220724 Err: 0 (0.00%) Active: 65 Started: 200 Finished: 135
summary = 1358 in 00:04:11 = 5.4/s Avg: 23614 Min: 796 Max: 220724 Err: 2 (0.15%)
summary + 171 in 00:00:30 = 5.7/s Avg: 21823 Min: 4524 Max: 220385 Err: 0 (0.00%) Active: 22 Started: 200 Finished: 178
summary = 1529 in 00:04:41 = 5.4/s Avg: 23413 Min: 796 Max: 220724 Err: 2 (0.13%)
The output is not shown after the last line, I wait up to 45 Min but logs are not getting updated.
Few hints:
Check jmeter.log file for anything suspicious
Get thread dump using jstack or pressing Win + Break (on Windows) and check threads statuses, see if there are any deadlocked or looping processes
Use VisualVM to attach to the process and inspect heap, see garbage collections, etc.
Check if anything is being added to .jtl results file
Most probably your JMeter instance isn't properly configured in terms of Java Heap size, Garbage Collection mode or your test plan has some issues.
References:
Troubleshooting Hanging or Looping Processes
9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure
I'd like to do a loading test with Jmeter.
On google compute engine, I've creared 4 VMs, one running tomcat server, one running jmeter master and two running jmeter-server.
My test plan is just a HTTP Request to the splash page(i.e. default home page, http://***.***.***.***/8080) of tomcat.
There are 2000 users in each slave, and the plan will be executed in 5 minutes.
Message on the terminal of the jmeter master are as following:
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary + 1 in 00:00:00 = 8.5/s Avg: 9 Min: 9 Max: 9 Err: 0 (0.00%) Active: 2 Started: 2 Finished: 0
summary + 300902 in 00:00:26 = 11425.9/s Avg: 75 Min: 0 Max: 1407 Err: 0 (0.00%) Active: 1745 Started: 1745 Finished: 0
summary = 300903 in 00:00:26 = 11375.0/s Avg: 75 Min: 0 Max: 1407 Err: 0 (0.00%)
summary + 321100 in 00:00:30 = 10703.3/s Avg: 243 Min: 0 Max: 7702 Err: 0 (0.00%) Active: 3727 Started: 3727 Finished: 0
summary = 622003 in 00:00:56 = 11018.1/s Avg: 162 Min: 0 Max: 7702 Err: 0 (0.00%)
summary + 314400 in 00:00:30 = 10478.6/s Avg: 372 Min: 0 Max: 31256 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 936403 in 00:01:26 = 10830.9/s Avg: 232 Min: 0 Max: 31256 Err: 0 (0.00%)
summary + 313700 in 00:00:30 = 10458.1/s Avg: 366 Min: 0 Max: 31827 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 1250103 in 00:01:56 = 10734.8/s Avg: 266 Min: 0 Max: 31827 Err: 0 (0.00%)
summary + 316200 in 00:00:30 = 10541.8/s Avg: 387 Min: 0 Max: 63556 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 1566303 in 00:02:26 = 10695.3/s Avg: 290 Min: 0 Max: 63556 Err: 0 (0.00%)
summary + 312100 in 00:00:30 = 10403.0/s Avg: 384 Min: 0 Max: 63661 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 1878403 in 00:02:56 = 10645.6/s Avg: 306 Min: 0 Max: 63661 Err: 0 (0.00%)
summary + 307900 in 00:00:30 = 10262.6/s Avg: 383 Min: 0 Max: 31917 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 2186303 in 00:03:26 = 10589.9/s Avg: 317 Min: 0 Max: 63661 Err: 0 (0.00%)
summary + 297500 in 00:00:30 = 9914.7/s Avg: 395 Min: 0 Max: 127296 Err: 2 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 2483803 in 00:03:56 = 10504.2/s Avg: 326 Min: 0 Max: 127296 Err: 2 (0.00%)
summary + 306000 in 00:00:30 = 10193.9/s Avg: 383 Min: 0 Max: 127265 Err: 20 (0.01%) Active: 4000 Started: 4000 Finished: 0
summary = 2789803 in 00:04:26 = 10469.3/s Avg: 332 Min: 0 Max: 127296 Err: 22 (0.00%)
summary + 314200 in 00:00:30 = 10483.1/s Avg: 389 Min: 0 Max: 127211 Err: 3 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 3104003 in 00:04:56 = 10470.7/s Avg: 338 Min: 0 Max: 127296 Err: 25 (0.00%)
summary + 42788 in 00:00:29 = 1478.5/s Avg: 749 Min: 0 Max: 127326 Err: 43 (0.10%) Active: 0 Started: 4000 Finished: 4000
summary = 3146791 in 00:05:25 = 9670.9/s Avg: 344 Min: 0 Max: 127326 Err: 68 (0.00%)
Tidying up remote # Wed Jan 11 03:26:58 UTC 2017 (1484105218946)
A few errors happened during the test. I looked at my log.jtl file and found they're ConnectException/Connection timed out.
Some messages are as following:
1484105011292,127254,HTTP Request,Non HTTP response code: java.net.ConnectException,Non HTTP response message: Connection timed out,jmeter-users 1-1708,text,false,,2111,0,2000,2000,0,0,127254
1484105011302,127244,HTTP Request,Non HTTP response code: java.net.ConnectException,Non HTTP response message: Connection timed out,jmeter-users 1-1897,text,false,,2111,0,2000,2000,0,0,127244
1484105138314,234,HTTP Request,200,,jmeter-users 1-80,text,true,,11369,183,2000,2000,234,0,0
1484105138279,269,HTTP Request,200,,jmeter-users 1-238,text,true,,11369,183,2000,2000,269,0,0
1484105011293,127253,HTTP Request,Non HTTP response code: java.net.ConnectException,Non HTTP response message: Connection timed out,jmeter-users 1-1565,text,false,,2111,0,2000,2000,0,0,127253
1484105138314,234,HTTP Request,200,,jmeter-users 1-1634,text,true,,11369,183,2000,2000,234,0,0
1484105138315,233,HTTP Request,200,,jmeter-users 1-1550,text,true,,11369,183,2000,2000,233,0,0
1484105138315,233,HTTP Request,200,,jmeter-users 1-1017,text,true,,11369,183,2000,2000,233,0,0
I've configured the timeout of tomcat and test plan, and set them to 300,000.
My question is why this error happened and how to configure to improve the throughput of tomcat.
EDIT(new error):
After adding a slave, I got a few following error in the log file(SocketException/Connection reset):
1484117548733,69717,HTTP Request,Non HTTP response code: java.net.SocketException,Non HTTP response message: Connection reset,jmeter-users 1-1749,text,false,,2629,0,34,34,0,0,118
I think you are overloading the tomcat server.
ConnectException: Observed when all possible request processing threads (mapped to thread count & connection count) are busy serving the existing requests, then the requests will be kept in queue (of length configured in acceptCount parameter). If the requests did not get the chance, then timeout will occur and raise connectException.
SocketException: Observed when the queue (of acceptCount) is full, then subsequent requests will be immediately rejected.
As you are generating a load of 2000 vusers (on each slave), there must be supporting threads available on the server end also.
Tomcat configuration:
maxConnections: The maximum number of connections that the server will accept and process at any given time. When this number has been reached, the server will accept, but not process, one further connection
maxThreads: The maximum number of request processing threads to be created by this Connector, which therefore determines the maximum number of simultaneous requests that can be handled. If not specified, this attribute is set to 200.
acceptCount: The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 100.
These are important parameters to be considered while tweaking tomcat for better performance.
Note: Just increasing the numbers/threads won't result in better performance, as it completely depends on the hardware capabilities. so, it is like trail and error, where you need find the best combination with optimal performance that suits the given hardware.