when I am load testing for 100 users at rampup of 2 seconds I am getting different values for same users and same ramp up time why is this happening can anyone help?
in remote server
1st time
summary = 100 in 00:00:03 = 31.7/s Avg: 1916 Min: 985 Max: 2737 Err: 0 (0.00%)
2nd time
summary + 1 in 00:00:01 = 1.5/s Avg: 260 Min: 260 Max: 260 Err: 0 (0.00%) Active: 22 Started: 22 Finished: 0
summary + 99 in 00:00:02 = 65.6/s Avg: 113 Min: 18 Max: 371 Err: 0 (0.00%) Active: 0 Started: 100 Finished: 100
summary = 100 in 00:00:02 = 45.9/s Avg: 114 Min: 18 Max: 371 Err: 0 (0.00%)
3rd time
summary + 1 in 00:00:01 = 2.0/s Avg: 92 Min: 92 Max: 92 Err: 0 (0.00%) Active: 14 Started: 14 Finished: 0
summary + 99 in 00:00:02 = 58.7/s Avg: 26 Min: 13 Max: 92 Err: 0 (0.00%) Active: 0 Started: 100 Finished: 100
summary = 100 in 00:00:02 = 45.5/s Avg: 26 Min: 13 Max: 92 Err: 0 (0.00%)
Normally tests should be repeatable which means that re-running the test should produce the same results.
Your test case is kind of specific, it looks like a spike test, for the first time response times are quite high and 2nd and 3rd execution they're going down. I can think of the following possible reasons:
First time response times are high because caches are not warmed up
Your application might have some optimization mechanisms like JIT
If you run exactly the same test it might be the case your system under test caches responses in different levels (operating system memory, database query cache, 3rd-party solutions like memcached
In order to have more repeatable results and better picture I would suggest:
increasing your test duration to i.e. 10 minutes
increasing your load gradually, i.e. change ramp-up time to 5 minutes
implementing monitoring of the system under test using i.e. JMeter PerfMon Plugin
Jmeter non-gui mode generates real time Test Results in the below format:
summary = 1649731 in 00:10:06 = 2721.2/s Avg: 47 Min: 9 Max: 16027 Err: 112 (0.01%)
summary + 63699 in 00:00:30 = 2123.3/s Avg: 64 Min: 12 Max: 2080 Err: 4 (0.01%) Active: 192 Started: 193 Finished: 1
summary = 1713430 in 00:10:36 = 2693.0/s Avg: 47 Min: 9 Max: 16027 Err: 116 (0.01%)
summary + 62509 in 00:00:30 = 2083.6/s Avg: 66 Min: 11 Max: 2034 Err: 1 (0.00%) Active: 192 Started: 193 Finished: 1
summary = 1775939 in 00:11:06 = 2665.6/s Avg: 48 Min: 9 Max: 16027 Err: 117 (0.01%)
summary + 69708 in 00:00:30 = 2323.5/s Avg: 45 Min: 9 Max: 2074 Err: 4 (0.01%) Active: 123 Started: 193 Finished: 70
summary = 1845647 in 00:11:36 = 2650.9/s Avg: 48 Min: 9 Max: 16027 Err: 121 (0.01%)
What is summary + and summary = ?
Summary + are incremental for the latest summariser period
Summary = are cumulative for the above Summary+
summary + 63699 in 00:00:30 = 2123.3/s
summary = 1713430 in 00:10:36 = 2693.0/s
Example 63699 in 00:00:30 = 2123.3/s means that in 30seconds Jmeter have sent 63699 requests to the server with an average throughput of 2123 requests per second.
The next iteration will start from the ahead of the cumulative time of the above summary. Ex: 00:11:06 = (00:00:30 + 00:10:36)
As per Top 2 Techniques to Get JMeter Test Results in non-GUI mode
Here’s an example of what you get from the default summariser:
summary + 41 in 15.4s = 2.7/s Avg: 2234 Min: 383 Max: 6974 Err: 0 (0.00%)
summary + 57 in 21.5s = 2.6/s Avg: 2548 Min: 618 Max: 4528 Err: 0 (0.00%)
summary = 98 in 32.5s = 3.0/s Avg: 2416 Min: 383 Max: 6974 Err: 0 (0.00%)
summary + 108 in 21.8s = 5.0/s Avg: 1291 Min: 229 Max: 6317 Err: 0 (0.00%)
summary = 206 in 52.5s = 3.9/s Avg: 1827 Min: 229 Max: 6974 Err: 0 (0.00%)
The lines with summary + are incremental for the latest summariser period, the lines with summary = are cumulative. The above was with a summariser period of 20 secs, the actual periods can sometimes be longer than the specified period and the length of the very first period is somewhat random. You get the throughput statistics as well as average, min and max response times, and how many errors were detected (assuming your JMeter test plan has assertions to detect errors).
If the JMeter's summariser output doesn't seem too informative to you consider running your JMeter test using Taurus tool as a wrapper, it has console reporter which prints current metrics right in your terminal:
You can also configure Backend Listener to store results to InfluxDB and plot them using Grafana, an example dashboard can look like:
As an example, when running a test plan from the command line, I get the following output (this is only a part of the output):
summary + 1030 in 00:00:06 = 172.9/s Avg: 3 Min: 1 Max: 267 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
summary + 2996 in 00:00:34 = 89.3/s Avg: 9 Min: 0 Max: 6832 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
summary = 4026 in 00:00:39 = 102.0/s Avg: 7 Min: 0 Max: 6832 Err: 0 (0.00%)
What is that first number counting? Why doesn't this seem to be documented anywhere?
First column represents the number of sample collected at regular intervals.
Here is the excerpt from Blazemeter article.
The lines with “summary +” are incremental for the latest summariser
period, the lines with “summary =” are cumulative.
By default, in jmeter.properties, you can find below settings. You can configure based on your requirements. E.g. if you want to enable summarizer log, you can set that property true.
I am not sure why it is not documented. I will need to raise a documentation enhancement request. Thanks for letting us know.
Sometimes the summary didn't get update while executing JMeter script in Non GUI Mode, Please find below data.
Starting the test # Wed Jan 04 18:41:18 IST 2017 (1483535478893)
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary + 28 in 00:00:11 = 2.5/s Avg: 3376 Min: 1991 Max: 7555 Err: 0 (0.00%) Active: 37 Started: 37 Finished: 0
summary + 192 in 00:00:30 = 6.4/s Avg: 5623 Min: 812 Max: 19523 Err: 0 (0.00%) Active: 137 Started: 137 Finished: 0
summary = 220 in 00:00:41 = 5.4/s Avg: 5337 Min: 812 Max: 19523 Err: 0 (0.00%)
summary + 196 in 00:00:30 = 6.6/s Avg: 9388 Min: 813 Max: 46485 Err: 1 (0.51%) Active: 200 Started: 200 Finished: 0
summary = 416 in 00:01:11 = 5.9/s Avg: 7246 Min: 812 Max: 46485 Err: 1 (0.24%)
summary + 159 in 00:00:31 = 5.2/s Avg: 11175 Min: 796 Max: 79142 Err: 0 (0.00%) Active: 200 Started: 200 Finished: 0
summary = 575 in 00:01:42 = 5.6/s Avg: 8332 Min: 796 Max: 79142 Err: 1 (0.17%)
summary + 50 in 00:00:29 = 1.7/s Avg: 39434 Min: 820 Max: 108309 Err: 1 (2.00%) Active: 200 Started: 200 Finished: 0
summary = 625 in 00:02:11 = 4.8/s Avg: 10820 Min: 796 Max: 108309 Err: 2 (0.32%)
summary + 104 in 00:00:30 = 3.5/s Avg: 44422 Min: 846 Max: 125336 Err: 0 (0.00%) Active: 196 Started: 200 Finished: 4
summary = 729 in 00:02:41 = 4.5/s Avg: 15614 Min: 796 Max: 125336 Err: 2 (0.27%)
summary + 189 in 00:00:30 = 6.3/s Avg: 37163 Min: 5912 Max: 168988 Err: 0 (0.00%) Active: 172 Started: 200 Finished: 28
summary = 918 in 00:03:11 = 4.8/s Avg: 20051 Min: 796 Max: 168988 Err: 2 (0.22%)
summary + 208 in 00:00:30 = 6.9/s Avg: 27117 Min: 5369 Max: 180636 Err: 0 (0.00%) Active: 132 Started: 200 Finished: 68
summary = 1126 in 00:03:41 = 5.1/s Avg: 21356 Min: 796 Max: 180636 Err: 2 (0.18%)
summary + 232 in 00:00:30 = 7.7/s Avg: 34571 Min: 4706 Max: 220724 Err: 0 (0.00%) Active: 65 Started: 200 Finished: 135
summary = 1358 in 00:04:11 = 5.4/s Avg: 23614 Min: 796 Max: 220724 Err: 2 (0.15%)
summary + 171 in 00:00:30 = 5.7/s Avg: 21823 Min: 4524 Max: 220385 Err: 0 (0.00%) Active: 22 Started: 200 Finished: 178
summary = 1529 in 00:04:41 = 5.4/s Avg: 23413 Min: 796 Max: 220724 Err: 2 (0.13%)
The output is not shown after the last line, I wait up to 45 Min but logs are not getting updated.
Few hints:
Check jmeter.log file for anything suspicious
Get thread dump using jstack or pressing Win + Break (on Windows) and check threads statuses, see if there are any deadlocked or looping processes
Use VisualVM to attach to the process and inspect heap, see garbage collections, etc.
Check if anything is being added to .jtl results file
Most probably your JMeter instance isn't properly configured in terms of Java Heap size, Garbage Collection mode or your test plan has some issues.
References:
Troubleshooting Hanging or Looping Processes
9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure
I'd like to do a loading test with Jmeter.
On google compute engine, I've creared 4 VMs, one running tomcat server, one running jmeter master and two running jmeter-server.
My test plan is just a HTTP Request to the splash page(i.e. default home page, http://***.***.***.***/8080) of tomcat.
There are 2000 users in each slave, and the plan will be executed in 5 minutes.
Message on the terminal of the jmeter master are as following:
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary + 1 in 00:00:00 = 8.5/s Avg: 9 Min: 9 Max: 9 Err: 0 (0.00%) Active: 2 Started: 2 Finished: 0
summary + 300902 in 00:00:26 = 11425.9/s Avg: 75 Min: 0 Max: 1407 Err: 0 (0.00%) Active: 1745 Started: 1745 Finished: 0
summary = 300903 in 00:00:26 = 11375.0/s Avg: 75 Min: 0 Max: 1407 Err: 0 (0.00%)
summary + 321100 in 00:00:30 = 10703.3/s Avg: 243 Min: 0 Max: 7702 Err: 0 (0.00%) Active: 3727 Started: 3727 Finished: 0
summary = 622003 in 00:00:56 = 11018.1/s Avg: 162 Min: 0 Max: 7702 Err: 0 (0.00%)
summary + 314400 in 00:00:30 = 10478.6/s Avg: 372 Min: 0 Max: 31256 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 936403 in 00:01:26 = 10830.9/s Avg: 232 Min: 0 Max: 31256 Err: 0 (0.00%)
summary + 313700 in 00:00:30 = 10458.1/s Avg: 366 Min: 0 Max: 31827 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 1250103 in 00:01:56 = 10734.8/s Avg: 266 Min: 0 Max: 31827 Err: 0 (0.00%)
summary + 316200 in 00:00:30 = 10541.8/s Avg: 387 Min: 0 Max: 63556 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 1566303 in 00:02:26 = 10695.3/s Avg: 290 Min: 0 Max: 63556 Err: 0 (0.00%)
summary + 312100 in 00:00:30 = 10403.0/s Avg: 384 Min: 0 Max: 63661 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 1878403 in 00:02:56 = 10645.6/s Avg: 306 Min: 0 Max: 63661 Err: 0 (0.00%)
summary + 307900 in 00:00:30 = 10262.6/s Avg: 383 Min: 0 Max: 31917 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 2186303 in 00:03:26 = 10589.9/s Avg: 317 Min: 0 Max: 63661 Err: 0 (0.00%)
summary + 297500 in 00:00:30 = 9914.7/s Avg: 395 Min: 0 Max: 127296 Err: 2 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 2483803 in 00:03:56 = 10504.2/s Avg: 326 Min: 0 Max: 127296 Err: 2 (0.00%)
summary + 306000 in 00:00:30 = 10193.9/s Avg: 383 Min: 0 Max: 127265 Err: 20 (0.01%) Active: 4000 Started: 4000 Finished: 0
summary = 2789803 in 00:04:26 = 10469.3/s Avg: 332 Min: 0 Max: 127296 Err: 22 (0.00%)
summary + 314200 in 00:00:30 = 10483.1/s Avg: 389 Min: 0 Max: 127211 Err: 3 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 3104003 in 00:04:56 = 10470.7/s Avg: 338 Min: 0 Max: 127296 Err: 25 (0.00%)
summary + 42788 in 00:00:29 = 1478.5/s Avg: 749 Min: 0 Max: 127326 Err: 43 (0.10%) Active: 0 Started: 4000 Finished: 4000
summary = 3146791 in 00:05:25 = 9670.9/s Avg: 344 Min: 0 Max: 127326 Err: 68 (0.00%)
Tidying up remote # Wed Jan 11 03:26:58 UTC 2017 (1484105218946)
A few errors happened during the test. I looked at my log.jtl file and found they're ConnectException/Connection timed out.
Some messages are as following:
1484105011292,127254,HTTP Request,Non HTTP response code: java.net.ConnectException,Non HTTP response message: Connection timed out,jmeter-users 1-1708,text,false,,2111,0,2000,2000,0,0,127254
1484105011302,127244,HTTP Request,Non HTTP response code: java.net.ConnectException,Non HTTP response message: Connection timed out,jmeter-users 1-1897,text,false,,2111,0,2000,2000,0,0,127244
1484105138314,234,HTTP Request,200,,jmeter-users 1-80,text,true,,11369,183,2000,2000,234,0,0
1484105138279,269,HTTP Request,200,,jmeter-users 1-238,text,true,,11369,183,2000,2000,269,0,0
1484105011293,127253,HTTP Request,Non HTTP response code: java.net.ConnectException,Non HTTP response message: Connection timed out,jmeter-users 1-1565,text,false,,2111,0,2000,2000,0,0,127253
1484105138314,234,HTTP Request,200,,jmeter-users 1-1634,text,true,,11369,183,2000,2000,234,0,0
1484105138315,233,HTTP Request,200,,jmeter-users 1-1550,text,true,,11369,183,2000,2000,233,0,0
1484105138315,233,HTTP Request,200,,jmeter-users 1-1017,text,true,,11369,183,2000,2000,233,0,0
I've configured the timeout of tomcat and test plan, and set them to 300,000.
My question is why this error happened and how to configure to improve the throughput of tomcat.
EDIT(new error):
After adding a slave, I got a few following error in the log file(SocketException/Connection reset):
1484117548733,69717,HTTP Request,Non HTTP response code: java.net.SocketException,Non HTTP response message: Connection reset,jmeter-users 1-1749,text,false,,2629,0,34,34,0,0,118
I think you are overloading the tomcat server.
ConnectException: Observed when all possible request processing threads (mapped to thread count & connection count) are busy serving the existing requests, then the requests will be kept in queue (of length configured in acceptCount parameter). If the requests did not get the chance, then timeout will occur and raise connectException.
SocketException: Observed when the queue (of acceptCount) is full, then subsequent requests will be immediately rejected.
As you are generating a load of 2000 vusers (on each slave), there must be supporting threads available on the server end also.
Tomcat configuration:
maxConnections: The maximum number of connections that the server will accept and process at any given time. When this number has been reached, the server will accept, but not process, one further connection
maxThreads: The maximum number of request processing threads to be created by this Connector, which therefore determines the maximum number of simultaneous requests that can be handled. If not specified, this attribute is set to 200.
acceptCount: The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 100.
These are important parameters to be considered while tweaking tomcat for better performance.
Note: Just increasing the numbers/threads won't result in better performance, as it completely depends on the hardware capabilities. so, it is like trail and error, where you need find the best combination with optimal performance that suits the given hardware.