Has anybody already worked with the zalopay-oss/jmeter-grpc-plugin / Apache License 2.0 GRPC Sampler for JMeter?
The following problem occurs while testing e.g. 100 quick requests which lead to 2MB payload size per response. I have no idea how to solve it. The more data was send the earlier the issue occurs.
Warning: Nashorn engine is planned to be removed from a future JDK release
Generate Summary Results + 50 in 00:00:11 = 4.6/s Avg: 188 Min: 129 Max: 1435 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
Generate Summary Results + 50 in 00:00:08 = 6.6/s Avg: 148 Min: 127 Max: 227 Err: 0 (0.00%) Active: 0 Started: 1 Finished: 1
Generate Summary Results = 100 in 00:00:18 = 5.4/s Avg: 168 Min: 127 Max: 1435 Err: 0 (0.00%)
Generate Summary Results + 1 in 00:00:01 = 0.7/s Avg: 1043 Min: 1043 Max: 1043 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid10268.hprof ...
Heap dump file created [1085958721 bytes in 2.412 secs]
Uncaught Exception java.lang.OutOfMemoryError: Java heap space in thread Thread[AWT-EventQueue-0,6,main]. See log file for details.
Generate Summary Results + 40 in 00:00:30 = 1.3/s Avg: 719 Min: 507 Max: 2895 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
Generate Summary Results = 41 in 00:00:32 = 1.3/s Avg: 727 Min: 507 Max: 2895 Err: 0 (0.00%)
Generate Summary Results + 41 in 00:00:30 = 1.4/s Avg: 681 Min: 505 Max: 1156 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
Generate Summary Results = 82 in 00:01:02 = 1.3/s Avg: 704 Min: 505 Max: 2895 Err: 0 (0.00%)
Generate Summary Results + 18 in 00:00:12 = 1.5/s Avg: 622 Min: 494 Max: 828 Err: 0 (0.00%) Active: 0 Started: 1 Finished: 1
Generate Summary Results = 100 in 00:01:14 = 1.4/s Avg: 689 Min: 494 Max: 2895 Err: 0 (0.00%)
Java heap space OOM error means that JVM isn't able to fit all the necessary objects within the define Java Heap space, default setting for the latest JMeter is 1 GB
If the default setting is not enough you can increase heap like:
Windows
set HEAP="-Xms2g -Xmx2g -XX:MaxMetaspaceSize=256m" && jmeter.bat
Unix and derivaties:
HEAP="-Xms1g -Xmx2g -XX:MaxMetaspaceSize=256m" && export HEAP && ./jmeter.sh
to make the changes permanent edit jmeter.bat or jmeter shell startup script wrappers
The above setting will increase the maximum heap size allocation twice (up to 2 GB), it may or may not be sufficient for your scenario, check whether it works and amend if necessary, just make sure that garbage collection doesn't occur to often and doesn't take too much time blocking other threads.
More information: 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure
Also looking at AWT-EventQueue it seems that you're using JMeter GUI for running the test, if it's the case - please don't, JMeter GUI is for tests development and debugging, tests are supposed to be run in command-line non-GUI mode
Jmeter non-gui mode generates real time Test Results in the below format:
summary = 1649731 in 00:10:06 = 2721.2/s Avg: 47 Min: 9 Max: 16027 Err: 112 (0.01%)
summary + 63699 in 00:00:30 = 2123.3/s Avg: 64 Min: 12 Max: 2080 Err: 4 (0.01%) Active: 192 Started: 193 Finished: 1
summary = 1713430 in 00:10:36 = 2693.0/s Avg: 47 Min: 9 Max: 16027 Err: 116 (0.01%)
summary + 62509 in 00:00:30 = 2083.6/s Avg: 66 Min: 11 Max: 2034 Err: 1 (0.00%) Active: 192 Started: 193 Finished: 1
summary = 1775939 in 00:11:06 = 2665.6/s Avg: 48 Min: 9 Max: 16027 Err: 117 (0.01%)
summary + 69708 in 00:00:30 = 2323.5/s Avg: 45 Min: 9 Max: 2074 Err: 4 (0.01%) Active: 123 Started: 193 Finished: 70
summary = 1845647 in 00:11:36 = 2650.9/s Avg: 48 Min: 9 Max: 16027 Err: 121 (0.01%)
What is summary + and summary = ?
Summary + are incremental for the latest summariser period
Summary = are cumulative for the above Summary+
summary + 63699 in 00:00:30 = 2123.3/s
summary = 1713430 in 00:10:36 = 2693.0/s
Example 63699 in 00:00:30 = 2123.3/s means that in 30seconds Jmeter have sent 63699 requests to the server with an average throughput of 2123 requests per second.
The next iteration will start from the ahead of the cumulative time of the above summary. Ex: 00:11:06 = (00:00:30 + 00:10:36)
As per Top 2 Techniques to Get JMeter Test Results in non-GUI mode
Here’s an example of what you get from the default summariser:
summary + 41 in 15.4s = 2.7/s Avg: 2234 Min: 383 Max: 6974 Err: 0 (0.00%)
summary + 57 in 21.5s = 2.6/s Avg: 2548 Min: 618 Max: 4528 Err: 0 (0.00%)
summary = 98 in 32.5s = 3.0/s Avg: 2416 Min: 383 Max: 6974 Err: 0 (0.00%)
summary + 108 in 21.8s = 5.0/s Avg: 1291 Min: 229 Max: 6317 Err: 0 (0.00%)
summary = 206 in 52.5s = 3.9/s Avg: 1827 Min: 229 Max: 6974 Err: 0 (0.00%)
The lines with summary + are incremental for the latest summariser period, the lines with summary = are cumulative. The above was with a summariser period of 20 secs, the actual periods can sometimes be longer than the specified period and the length of the very first period is somewhat random. You get the throughput statistics as well as average, min and max response times, and how many errors were detected (assuming your JMeter test plan has assertions to detect errors).
If the JMeter's summariser output doesn't seem too informative to you consider running your JMeter test using Taurus tool as a wrapper, it has console reporter which prints current metrics right in your terminal:
You can also configure Backend Listener to store results to InfluxDB and plot them using Grafana, an example dashboard can look like:
I have a test case with just one HTTP request. The number of threads and Ramp Up period is set to 30. And i planning to do the test for 600 seconds.
But looking at the JMeter output, i could see a huge number of requests generated. Why is this so? I am expecting only 60 threads in a minute to generate. Can someone suggest me how to do this?
S:\roshTests\apache-jmeter-3.1\bin>jmeter -n -t
S:\roshTests\LivePerformance\June1\PPS-Proxy\catalog\catalog.jmx -l
S:\roshTests\LivePerformance\June1\PPS-Proxy\catalog\Output\results.csv
Writing log file to: S:\roshTests\apache-jmeter-3.1\bin\jmeter.log
Creating summariser <summary>
Created the tree successfully using S:\roshTests\LivePerformance\June1\PPS-Proxy\catalog\catalog.jmx
Starting the test # Fri Jun 02 05:18:04 IDT 2017 (1496369884546)
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary + 377 in 00:00:25 = 15.3/s Avg: 766 Min: 106 Max: 3114 Err: 198 (52.52%) Active: 25 Started: 25 Finished: 0
summary + 1364 in 00:00:30 = 45.5/s Avg: 647 Min: 144 Max: 4385 Err: 1180 (86.51%) Active: 30 Started: 30 Finished: 0
summary = 1741 in 00:00:55 = 31.9/s Avg: 673 Min: 106 Max: 4385 Err: 1378 (79.15%)
summary + 985 in 00:00:30 = 32.8/s Avg: 643 Min: 88 Max: 4918 Err: 752 (76.35%) Active: 30 Started: 30 Finished: 0
summary = 12859 in 00:05:25 = 39.6/s Avg: 616 Min: 88 Max: 6149 Err: 10562 (82.14%)
summary + 984 in 00:00:30 = 32.8/s Avg: 675 Min: 113 Max: 5070 Err: 762 (77.44%) Active: 30 Started: 30 Finished: 0
summary = 11874 in 00:04:55 = 40.3/s Avg: 614 Min: 100 Max: 6149 Err: 9810 (82.62%)
summary + 1103 in 00:00:30 = 36.7/s Avg: 624 Min: 104 Max: 4935 Err: 884 (80.15%) Active: 30 Started: 30 Finished: 0
summary = 10890 in 00:04:25 = 41.2/s Avg: 609 Min: 100 Max: 6149 Err: 9048 (83.09%)
summary + 1220 in 00:00:30 = 40.6/s Avg: 584 Min: 100 Max: 6149 Err: 1001 (82.05%) Active: 30 Started: 30 Finished: 0
summary = 9787 in 00:03:55 = 41.7/s Avg: 607 Min: 100 Max: 6149 Err: 8164 (83.42%)
summary + 1336 in 00:00:30 = 44.5/s Avg: 570 Min: 106 Max: 5296 Err: 1106 (82.78%) Active: 30 Started: 30 Finished: 0
summary = 8567 in 00:03:25 = 41.9/s Avg: 610 Min: 106 Max: 5643 Err: 7163 (83.61%)
summary + 1320 in 00:00:30 = 44.0/s Avg: 587 Min: 116 Max: 5187 Err: 1111 (84.17%) Active: 30 Started: 30 Finished: 0
summary = 7231 in 00:02:55 = 41.4/s Avg: 618 Min: 106 Max: 5643 Err: 6057 (83.76%)
summary + 1273 in 00:00:30 = 42.5/s Avg: 637 Min: 106 Max: 5383 Err: 1072 (84.21%) Active: 30 Started: 30 Finished: 0
summary = 5911 in 00:02:25 = 40.9/s Avg: 625 Min: 106 Max: 5643 Err: 4946 (83.67%)
summary + 1430 in 00:00:30 = 47.6/s Avg: 591 Min: 116 Max: 5643 Err: 1229 (85.94%) Active: 30 Started: 30 Finished: 0
summary = 4638 in 00:01:55 = 40.5/s Avg: 621 Min: 106 Max: 5643 Err: 3874 (83.53%)
summary + 1467 in 00:00:30 = 48.9/s Avg: 589 Min: 119 Max: 4691 Err: 1267 (86.37%) Active: 30 Started: 30 Finished: 0
summary = 3208 in 00:01:25 = 37.9/s Avg: 635 Min: 106 Max: 4691 Err: 2645 (82.45%)
summary + 1070 in 00:00:30 = 35.7/s Avg: 651 Min: 99 Max: 7324 Err: 862 (80.56%) Active: 30 Started: 30 Finished: 0
summary = 13929 in 00:05:55 = 39.3/s Avg: 619 Min: 88 Max: 7324 Err: 11424 (82.02%)
summary + 1531 in 00:00:30 = 51.1/s Avg: 594 Min: 106 Max: 5194 Err: 1332 (87.00%) Active: 30 Started: 30 Finished: 0
summary = 15460 in 00:06:25 = 40.2/s Avg: 617 Min: 88 Max: 7324 Err: 12756 (82.51%)
I'm not sure if you understand the difference between Threads and requests. amount of Threads isn't equal to requests, what you're seeing in the output is the amount of requests made by your threads, and depending on your response times it's very well possible to get that many results.
Try setting the Loop count to 1 or 2 and you'll see that the requests will even out with your calculations.
Beside that... in your output on the far right side you can see how many Threads are started.
With your current setup, threads will start with one second delay.
Ramp up period is the time in which all treads should start.
You have loop set to infinite, so when all thread login after initial 30 seconds, you will always have 30 active users, and could never have more that that.
You have many more requests, depending on response time of your server, but threads are like users. If you set 30, it will always be 30. :-)
You are running 30 virtual users for 10 minutes. Each virtual user is executing samplers as fast as it can (the speed mostly depends on your application response time) this is why you have a lot of requests.
If you need to limit JMeter throughput to 60 requests per second only you can do it using Constant Throughput Timer:
Add Constant Throughput Timer to your Test Plan
Set "Target Troughput" to 60
Set "Calculate Troughput based on" to All active threads
This way JMeter will "pause" its threads (virtual users) to slow them down to 60 requests per minute.
As an example, when running a test plan from the command line, I get the following output (this is only a part of the output):
summary + 1030 in 00:00:06 = 172.9/s Avg: 3 Min: 1 Max: 267 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
summary + 2996 in 00:00:34 = 89.3/s Avg: 9 Min: 0 Max: 6832 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
summary = 4026 in 00:00:39 = 102.0/s Avg: 7 Min: 0 Max: 6832 Err: 0 (0.00%)
What is that first number counting? Why doesn't this seem to be documented anywhere?
First column represents the number of sample collected at regular intervals.
Here is the excerpt from Blazemeter article.
The lines with “summary +” are incremental for the latest summariser
period, the lines with “summary =” are cumulative.
By default, in jmeter.properties, you can find below settings. You can configure based on your requirements. E.g. if you want to enable summarizer log, you can set that property true.
I am not sure why it is not documented. I will need to raise a documentation enhancement request. Thanks for letting us know.
I'd like to do a loading test with Jmeter.
On google compute engine, I've creared 4 VMs, one running tomcat server, one running jmeter master and two running jmeter-server.
My test plan is just a HTTP Request to the splash page(i.e. default home page, http://***.***.***.***/8080) of tomcat.
There are 2000 users in each slave, and the plan will be executed in 5 minutes.
Message on the terminal of the jmeter master are as following:
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary + 1 in 00:00:00 = 8.5/s Avg: 9 Min: 9 Max: 9 Err: 0 (0.00%) Active: 2 Started: 2 Finished: 0
summary + 300902 in 00:00:26 = 11425.9/s Avg: 75 Min: 0 Max: 1407 Err: 0 (0.00%) Active: 1745 Started: 1745 Finished: 0
summary = 300903 in 00:00:26 = 11375.0/s Avg: 75 Min: 0 Max: 1407 Err: 0 (0.00%)
summary + 321100 in 00:00:30 = 10703.3/s Avg: 243 Min: 0 Max: 7702 Err: 0 (0.00%) Active: 3727 Started: 3727 Finished: 0
summary = 622003 in 00:00:56 = 11018.1/s Avg: 162 Min: 0 Max: 7702 Err: 0 (0.00%)
summary + 314400 in 00:00:30 = 10478.6/s Avg: 372 Min: 0 Max: 31256 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 936403 in 00:01:26 = 10830.9/s Avg: 232 Min: 0 Max: 31256 Err: 0 (0.00%)
summary + 313700 in 00:00:30 = 10458.1/s Avg: 366 Min: 0 Max: 31827 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 1250103 in 00:01:56 = 10734.8/s Avg: 266 Min: 0 Max: 31827 Err: 0 (0.00%)
summary + 316200 in 00:00:30 = 10541.8/s Avg: 387 Min: 0 Max: 63556 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 1566303 in 00:02:26 = 10695.3/s Avg: 290 Min: 0 Max: 63556 Err: 0 (0.00%)
summary + 312100 in 00:00:30 = 10403.0/s Avg: 384 Min: 0 Max: 63661 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 1878403 in 00:02:56 = 10645.6/s Avg: 306 Min: 0 Max: 63661 Err: 0 (0.00%)
summary + 307900 in 00:00:30 = 10262.6/s Avg: 383 Min: 0 Max: 31917 Err: 0 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 2186303 in 00:03:26 = 10589.9/s Avg: 317 Min: 0 Max: 63661 Err: 0 (0.00%)
summary + 297500 in 00:00:30 = 9914.7/s Avg: 395 Min: 0 Max: 127296 Err: 2 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 2483803 in 00:03:56 = 10504.2/s Avg: 326 Min: 0 Max: 127296 Err: 2 (0.00%)
summary + 306000 in 00:00:30 = 10193.9/s Avg: 383 Min: 0 Max: 127265 Err: 20 (0.01%) Active: 4000 Started: 4000 Finished: 0
summary = 2789803 in 00:04:26 = 10469.3/s Avg: 332 Min: 0 Max: 127296 Err: 22 (0.00%)
summary + 314200 in 00:00:30 = 10483.1/s Avg: 389 Min: 0 Max: 127211 Err: 3 (0.00%) Active: 4000 Started: 4000 Finished: 0
summary = 3104003 in 00:04:56 = 10470.7/s Avg: 338 Min: 0 Max: 127296 Err: 25 (0.00%)
summary + 42788 in 00:00:29 = 1478.5/s Avg: 749 Min: 0 Max: 127326 Err: 43 (0.10%) Active: 0 Started: 4000 Finished: 4000
summary = 3146791 in 00:05:25 = 9670.9/s Avg: 344 Min: 0 Max: 127326 Err: 68 (0.00%)
Tidying up remote # Wed Jan 11 03:26:58 UTC 2017 (1484105218946)
A few errors happened during the test. I looked at my log.jtl file and found they're ConnectException/Connection timed out.
Some messages are as following:
1484105011292,127254,HTTP Request,Non HTTP response code: java.net.ConnectException,Non HTTP response message: Connection timed out,jmeter-users 1-1708,text,false,,2111,0,2000,2000,0,0,127254
1484105011302,127244,HTTP Request,Non HTTP response code: java.net.ConnectException,Non HTTP response message: Connection timed out,jmeter-users 1-1897,text,false,,2111,0,2000,2000,0,0,127244
1484105138314,234,HTTP Request,200,,jmeter-users 1-80,text,true,,11369,183,2000,2000,234,0,0
1484105138279,269,HTTP Request,200,,jmeter-users 1-238,text,true,,11369,183,2000,2000,269,0,0
1484105011293,127253,HTTP Request,Non HTTP response code: java.net.ConnectException,Non HTTP response message: Connection timed out,jmeter-users 1-1565,text,false,,2111,0,2000,2000,0,0,127253
1484105138314,234,HTTP Request,200,,jmeter-users 1-1634,text,true,,11369,183,2000,2000,234,0,0
1484105138315,233,HTTP Request,200,,jmeter-users 1-1550,text,true,,11369,183,2000,2000,233,0,0
1484105138315,233,HTTP Request,200,,jmeter-users 1-1017,text,true,,11369,183,2000,2000,233,0,0
I've configured the timeout of tomcat and test plan, and set them to 300,000.
My question is why this error happened and how to configure to improve the throughput of tomcat.
EDIT(new error):
After adding a slave, I got a few following error in the log file(SocketException/Connection reset):
1484117548733,69717,HTTP Request,Non HTTP response code: java.net.SocketException,Non HTTP response message: Connection reset,jmeter-users 1-1749,text,false,,2629,0,34,34,0,0,118
I think you are overloading the tomcat server.
ConnectException: Observed when all possible request processing threads (mapped to thread count & connection count) are busy serving the existing requests, then the requests will be kept in queue (of length configured in acceptCount parameter). If the requests did not get the chance, then timeout will occur and raise connectException.
SocketException: Observed when the queue (of acceptCount) is full, then subsequent requests will be immediately rejected.
As you are generating a load of 2000 vusers (on each slave), there must be supporting threads available on the server end also.
Tomcat configuration:
maxConnections: The maximum number of connections that the server will accept and process at any given time. When this number has been reached, the server will accept, but not process, one further connection
maxThreads: The maximum number of request processing threads to be created by this Connector, which therefore determines the maximum number of simultaneous requests that can be handled. If not specified, this attribute is set to 200.
acceptCount: The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 100.
These are important parameters to be considered while tweaking tomcat for better performance.
Note: Just increasing the numbers/threads won't result in better performance, as it completely depends on the hardware capabilities. so, it is like trail and error, where you need find the best combination with optimal performance that suits the given hardware.