JMeter test not stopping after duration ends in distribution mode - jmeter

Question

I'm running a very simple JMeter test with Basic Thread Group and HTTP Sampler. The duration set for execution is 10 min (600 sec). enter image description here
The test ran and stopped itself (after 10 min) successfully on my local both in JMeter GUI as well as CLI mode.
However, when I run the same test in distributed mode, the test does not stop itself and gets hanged. I've been observing this issue mostly with thread count 200 and above.
Some of the JMeter properties I have overridden:
"server.rmi.ssl.disable": "true",
"jmeterengine.nongui.maxport": JMETER_EXEC_PORT,
"jmeterengine.nongui.port": JMETER_EXEC_PORT,
"client.tries": str(3),
"client.retries_delay": str(5000),
"client.rmi.localport": CLIENT_RMI_LOCALPORT,
"server.rmi.localport": SERVER_RMI_LOCALPORT,
"server_port": SERVER_PORT,
"server.exitaftertest": "true",
"jmeterengine.stopfail.system.exit": "true",
"jmeterengine.remote.system.exit": "true",
"jmeterengine.force.system.exit": "true",
"jmeter.save.saveservice.output_format": "csv",
"jmeter.save.saveservice.autoflush": "true",
"beanshell.server.file": "./extras/startup.bsh",
"jmeter.save.saveservice.connect_time": "true",
"jpgc.repo.sendstats": "false",
Here're the JMeter CLI commands I'm using for JMeter client and server respectively:
// JMeter Client
jmeter.sh -n -f -t {testPlan} -j jmeter.log -l report.csv -LINFO -Lorg.apache.http=DEBUG -Lorg.apache.http.wire=ERROR -Ljmeter.engine=DEBUG -X -R {serverIPs}
// JMeter Server
jmeter.sh -s -Jbeanshell.server.port={beanshellServerPort}
Any help pointers to make sure the test ends after the specified
duration?
Can this be controlled/enforced by any JMeter
setting/property?
Is it something related to Basic thread group v/s
thread group plugins like Concurrency/Ultimate thread group?
Example test run:
I tried to run this test plan with 10 workers out of which 6 were successfully finished while 4 of them hanged. Please find the logs and test plan here.
Also, why does the Summariser show Active + Finished threads more than Started?

Answer 1

Looking into the log it seems that JMeter didn't finish all the threads after reporting the test completion:
2020-12-16 08:19:48,933 INFO o.a.j.t.JMeterThread: Thread finished: 10.244.11.4-Thread Group 1-86
2020-12-16 08:25:22,926 INFO o.a.j.e.RemoteJMeterEngineImpl: Shutting test ...
2020-12-16 08:25:22,927 INFO o.a.j.e.RemoteJMeterEngineImpl: ... stopped
2020-12-16 08:25:22,928 INFO o.a.j.t.JMeterThread: Stopping: 10.244.11.4-Thread Group 1-78
2020-12-16 08:25:22,928 INFO o.a.j.t.JMeterThread: Stopping: 10.244.11.4-Thread Group 1-190
You need to compare log files from the slaves which are failing with the ones which are passing, it will allow you to detect any inconsistencies.
One of possible reasons I can think of is that you allocated too little JVM Heap space, at least one controller is using default 1 GB of heap which might be not sufficient for executing your test for 200+ virtual users. Consider ramping-up this to a higher value. See 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article for more details on this and other JMeter performance and tuning tips
If the issue persists the only way of finding out the cause of why threads are stuck is taking a thread dump