JMeter stops sending HTTP Requests for specific threads - jmeter

Question

I am using JMeter 2.11.
The following parameters are defined in the jmeter.bat file
set HEAP=-Xms512m -Xmx12144m
set NEW=-XX:NewSize=128m -XX:MaxNewSize=128m set SURVIVOR=-XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=50% set TENURING=-XX:MaxTenuringThreshold=2
set RMIGC=-Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000
set PERM=-XX:PermSize=64m -XX:MaxPermSize=64m
The test is launched in batch mode.
The jmeter results are stored in the jtl file in an XML Format.
We made a scenario that needs while statement.
If we remove the while statement, JMeter manages to handle 50 users concurrently.
If we add the while statement, around 80% of JMeter user threads are executed correctly (40 users are carrying out the scenario without any problem).
20% of JMeter user threads stop during a variable period : sometimes 15 minutes, sometimes 40 minutes, sometimes an hour and then the scenario keeps on with the next statement (10 users are launching requests and then stop during 15 min for example and then restart).
By tracing the activity with debug sampler, it just stops anywhere usually before or after a timer of a few seconds. It stops during 40 minutes for example and after 40 minutes, it sends again HTTP requests (the problem is that my IIS application session is timed out and all requests failed).
It seems that the more we add debug sampler, the more JMeter works correctly.
There is no log...
We tried the following :
Change JMeter.bat settings
Upgrade JMeter version
Increase timers in order the scenario to be less stressful
Nothing works. We still have the problem. I am therefore wondering if JMeter has reached its maximum capacity.
It should be noted that the injector CPU is around 60% and the memory is ok.
What I am afraid about is that 50 users is quite low... We need to handle tests with 1000 users and we cannot buy 20 machines to just handle the injectors.
If anyone has any idea concerning this problem, I would really appreciate.
Regards
Sylvie

Answer 1

I think your issue is more related to your server slowing down due to Load and the fact that you didn't set a timeout on Http Request meaning they wait until your server responds.
So first thing to do is set Connection and Response Timeout in a HTTP Request Default element.
Next thing is to check you follow best practices when load testing , see for this:
http://jmeter.apache.org/usermanual/best-practices.html
http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
Finally regarding your GC tuning, I suggest you just keep:
set HEAP=-Xms512m -Xmx12144m
set PERM=-XX:PermSize=64m -XX:MaxPermSize=64m
As a summary JMeter will be able to Load test without any problem 1000 Users and you won't need 20 machines for this :-)

Answer 2

I made a test using ThreadStackSize=4096 set in the JMeter.bat file and I still have the problem (My scenario is launched in batch mode).
the following requests are sent :
**Sent at 19/09/2014 12:06:06**
<sample t="0" lt="0" ts="1411121166434" s="true" lb="------------------------Begin Loop Page de prop App From Tree " rc="200" rm="OK" tn="Groupe d&apos;unitأ©s APM 1-9" dt="text" by="923">
**sent at 19/09/2014 12:36:16**
<sample t="0" lt="0" ts="1411122975778" s="true" lb="----------Technology " rc="200" rm="OK" tn="Groupe d&apos;unitأ©s APM 1-9" dt="text" by="923">
As you can see, there is 30 minutes between the 2 Requests
The matching JMeter View is the following:
"Begin Loop Page de prop App From Tree " is a debug sampler, it goes through a "If Create Scope" which is a "If Statement" followed by a "Loop Create Technology" where loop is set to 1 as shown below:
Followed by the debug sampler "--------------Technology" executed 30 minutes after the debug sampler "Begin Loop Page de prop App From Tree".
The timer used are Constant Timer.
It should be noted that I have the same problem in another scenario with Gaussian Timers. The timer values depend on the actions but are between 2000 ms and 6000 ms.
In the other scenario, if I remove the while statement, the scenario works... Unfortunately, I need the while statement.
In this scenario, the problem seems to come from the loop set to 1.
Concerning the VisualVM tool, I just saw that the garbage collector is done very frequently. I did not see memory problem... But I will have a look to it on the next test.
Hope it helps
Regards
Sylvie

Answer 3

In fact, the test I made above with Connect timeout set to 1000 and response timeout to 2000 was on a single thread (user).
So the socket error was probably due to a connect timeout parameter too low...
I change those parameters and set connect timeout to 60 000 (1min) and response timeout to 360 000 (6min because sometimes we have requests that do not send response and we limit them to 5 minutes, it is very rare but this was blocking the scenario).
I removed this from JMeter.bat file :
set NEW=-XX:NewSize=128m -XX:MaxNewSize=128m
set SURVIVOR=-XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=50%
set TENURING=-XX:MaxTenuringThreshold=2
set RMIGC=-Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 set PERM=-XX:PermSize=64m -XX:MaxPermSize=64m
I played my scenario in batch mode with 50 users. It appears that we do not have anymore threads that are blocked. Unfortunately we saw the following for most of our users:a request is played, the server respond with good delay (less than one second) and the next request is played an hour later which gives a 500 HTTP Error....
Example : if we have a look at the unit Group6. The following is played and written in the JTL File
<httpSample t="13" lt="13" ts="1410856270124" s="true" lb="/hopex/service.aspx?data=generationType-standard|generator-E98AEA3A4F717715" rc="200" rm="OK" tn="Groupe d&apos;unités 1-6" dt="text" by="412">
<java.net.URL>http://172.16.1.23/hopex/service.aspx?data=generationType-standard|generator-E98AEA3A4F717715</java.net.URL>
</httpSample>
**played at 16/09/2014 10:31:10**
<httpSample t="0" lt="0" ts="1410856270138" s="true" lb="/hopex/statesessionprovider.aspx" rc="200" rm="OK" tn="Groupe d&apos;unités 1-6" dt="text" by="238">
<java.net.URL>http://172.16.1.23/hopex/statesessionprovider.aspx</java.net.URL>
</httpSample>
**played at 16/09/2014 10:31:10**
<sample t="0" lt="0" ts="1410856274818" s="true" lb="Timer between steps" rc="200" rm="OK" tn="Groupe d&apos;unités 1-6" dt="text" by="1478"/>
**played at 16/09/2014 10:31:15**
<httpSample t="3" lt="3" ts="1410860493293" s="false" lb="/Hopex/service.aspx?data=generationType-standard|generator-E98AEA3A4F717715" rc="500" rm="Internal Server Error" tn="Groupe d&apos;unités 1-6" dt="text" by="298">
<java.net.URL>http://172.16.1.23/Hopex/service.aspx?data=generationType-standard|generator-E98AEA3A4F717715</java.net.URL>
</httpSample>
**played at 16/09/2014 11:41:33**
Most of the time, we have problems just after timers. Here, you can see that the last request has been played more than one hour after the previous one (which was a JMeter timer)...
Our application log shows that the last request has never been sent to the application.
So it means JMeter made a pause of more than one hour before sending the request.
It should be noted that if we remove the while statement from our scenario, it works.
It should be noted also that the errors do not apply near a while statement.
Since you were thinking that the server was overloaded, I registered Windows indicators with performance monitor.
It appears that the average CPU during the test was around 10% (probably because most of the threads stop). If I have a look at 10:31, the CPU does not go over 30%.
If I checked the memory consumption, there was 20 GB of RAM available when the problem occurred.
So, I think the server is not overloaded...
I retrieve this information from JMeter logs. It seems the problem comes from JMeter with stack overflow. I do not know how to solve this problem. I tried to change JMeter.bat parameters but we had side effects.
Here is a part of JMeter log :
2014/09/16 10:30:49 WARN - jmeter.control.GenericController: StackOverflowError detected
2014/09/16 10:30:49 WARN - jmeter.control.GenericController: StackOverflowError detected
2014/09/16 10:30:49 WARN - jmeter.control.GenericController: StackOverflowError detected
2014/09/16 10:30:51 WARN - jmeter.control.GenericController: StackOverflowError detected
2014/09/16 10:31:00 INFO - jmeter.reporters.Summariser: summary + 196 in 30s = 6.5/s Avg: 154 Min: 0 Max: 11347 Err: 0 (0.00%) Active: 50 Started: 50 Finished: 0
2014/09/16 10:31:00 INFO - jmeter.reporters.Summariser: summary = 5974 in 1103s = 5.4/s Avg: 406 Min: 0 Max: 47864 Err: 0 (0.00%)
2014/09/16 10:31:01 WARN - jmeter.control.GenericController: StackOverflowError detected
2014/09/16 10:31:32 INFO - jmeter.reporters.Summariser: summary + 154 in 32s = 4.9/s Avg: 94 Min: 0 Max: 10982 Err: 0 (0.00%) Active: 50 Started: 50 Finished: 0
2014/09/16 10:31:32 INFO - jmeter.reporters.Summariser: summary = 6128 in 1135s = 5.4/s Avg: 399 Min: 0 Max: 47864 Err: 0 (0.00%)
2014/09/16 10:31:37 WARN - jmeter.control.GenericController: StackOverflowError detected
I am on this problem since one month now and I do not know how to solve it...
If you have an idea, I would really appreciate.
Regards
Sylvie