Distributed testing with 2 CentOS VM machines and Jmeter - jmeter

Question

I have 2 virtual machines with CentOS. Both of them with the same version of Jmeter installed and working.
I am trying to run a distributed test, I have modified the jmeter.properties file (remote_host property and set the attribute #server.rmi.ssl.disable to true) I can make a normal ping between the machines and it seems that jmeter has access to the machines working as "slave" or load genarator but I get the following error:
[root#01 bin]# jmeter -n -t jmeter_script1.jmx -r
Creating summariser <summary>
Created the tree successfully using jmeter_script1.jmx
Configuring remote engine: 10.111.2.4
Starting remote engines
Starting the test # Tue Oct 15 14:12:17 CEST 2019 (1571141537956)
Error in rconfigure() method java.rmi.ConnectIOException: Exception creating connection to: 10.111.2.4; nested exception is:
java.net.NoRouteToHostException: No route to host (Host unreachable)
Remote engines have been started
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
-------------Update 21/10/19.
I have removed the firewall with sudo systemctl stop firewalld in both machines (master & slave) but I get a new error message:
16:48:21 INFO: Taurus CLI Tool v1.13.8
16:48:21 INFO: Starting with configs: ['test1.yml']
16:48:21 INFO: Configuring...
16:48:22 INFO: Artifacts dir: /root/taurus_container_scripts/2019-10-21_16-48-22.015788
16:48:22 WARNING: at path 'execution': 'execution' should be a list
16:48:22 INFO: Preparing...
16:48:22 WARNING: There is newer version of Taurus 1.13.9 available, consider upgrading. What's new: http://gettaurus.org/docs/Changelog/
16:48:24 INFO: Starting...
16:48:24 INFO: Waiting for results...
16:48:25 INFO: Waiting for finish...
16:48:27 WARNING: Please wait for graceful shutdown...
16:48:27 INFO: Shutting down...
16:48:27 INFO: Post-processing...
16:48:27 INFO: Test duration: 0:00:03
16:48:27 INFO: Test duration: 0:00:03
16:48:27 ERROR: Child Process Error: Empty results, most likely quick-test (JMeterExecutor) failed. Actual reason for this can be found in logs under /root/taurus_container_scripts/2019-10-21_16-48-22.015788
16:48:27 ERROR: JMeter STDOUT:
Created the tree successfully using /root/taurus_container_scripts/2019-10-21_16-48-22.015788/modified_requests.jmx
Configuring remote engine: 10.111.2.4
Starting remote engines
Starting the test # Mon Oct 21 16:48:26 CEST 2019 (1571669306099)
Error in rconfigure() method java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.lang.ClassNotFoundException: com.blazemeter.jmeter.threads.concurrency.ConcurrencyThreadGroup (no security manager: RMI class loader disabled)
Remote engines have been started
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
16:48:27 ERROR: JMeter log:
2019-10-21 16:48:26,902 ERROR o.a.j.e.ClientJMeterEngine: Error in rconfigure() method
java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.lang.ClassNotFoundException: com.blazemeter.jmeter.threads.concurrency.ConcurrencyThreadGroup (no security manager: RMI class loader disabled)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:389) ~[?:1.8.0_222]
at sun.rmi.transport.Transport$1.run(Transport.java:200) ~[?:1.8.0_222]
at sun.rmi.transport.Transport$1.run(Transport.java:197) ~[?:1.8.0_222]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222]
at sun.rmi.transport.Transport.serviceCall(Transport.java:196) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) ~[?:1.8.0_222]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) ~[?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222]
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283) ~[?:1.8.0_222]
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260) ~[?:1.8.0_222]
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) ~[?:1.8.0_222]
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227) ~[?:1.8.0_222]
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179) ~[?:1.8.0_222]
at com.sun.proxy.$Proxy19.rconfigure(Unknown Source) ~[?:?]
at org.apache.jmeter.engine.ClientJMeterEngine.runTest(ClientJMeterEngine.java:153) [ApacheJMeter_core.jar:5.1.1 r1855137]
at org.apache.jmeter.engine.DistributedRunner.start(DistributedRunner.java:132) [ApacheJMeter_core.jar:5.1.1 r1855137]
at org.apache.jmeter.engine.DistributedRunner.start(DistributedRunner.java:149) [ApacheJMeter_core.jar:5.1.1 r1855137]
at org.apache.jmeter.JMeter.runNonGui(JMeter.java:1089) [ApacheJMeter_core.jar:5.1.1 r1855137]
at org.apache.jmeter.JMeter.startNonGui(JMeter.java:991) [ApacheJMeter_core.jar:5.1.1 r1855137]
at org.apache.jmeter.JMeter.start(JMeter.java:563) [ApacheJMeter_core.jar:5.1.1 r1855137]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at org.apache.jmeter.NewDriver.main(NewDriver.java:253) [ApacheJMeter.jar:5.1.1 r1855137]
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.lang.ClassNotFoundException: com.blazemeter.jmeter.threads.concurrency.ConcurrencyThreadGroup (no security manager: RMI class loader disabled)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:348) ~[?:1.8.0_222]
at sun.rmi.transport.Transport$1.run(Transport.java:200) ~[?:1.8.0_222]
at sun.rmi.transport.Transport$1.run(Transport.java:197) ~[?:1.8.0_222]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222]
at sun.rmi.transport.Transport.serviceCall(Transport.java:196) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) ~[?:1.8.0_222]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) ~[?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222]
Caused by: java.lang.ClassNotFoundException: com.blazemeter.jmeter.threads.concurrency.ConcurrencyThreadGroup (no security manager: RMI class loader disabled)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:396) ~[?:1.8.0_222]
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:186) ~[?:1.8.0_222]
at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:637) ~[?:1.8.0_222]
at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:264) ~[?:1.8.0_222]
at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:219) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1867) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1750) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2041) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1572) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430) ~[?:1.8.0_222]
at java.util.HashMap.readObject(HashMap.java:1410) ~[?:1.8.0_222]
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2177) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2068) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1572) ~[?:1.8.0_222]
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2286) ~[?:1.8.0_222]
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:560) ~[?:1.8.0_222]
at org.apache.jorphan.collections.HashTree.readObject(HashTree.java:948) ~[jorphan.jar:5.1.1 r1855137]
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2177) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2068) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1572) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430) ~[?:1.8.0_222]
at java.util.HashMap.readObject(HashMap.java:1412) ~[?:1.8.0_222]
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2177) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2068) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1572) ~[?:1.8.0_222]
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2286) ~[?:1.8.0_222]
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:560) ~[?:1.8.0_222]
at org.apache.jorphan.collections.HashTree.readObject(HashTree.java:948) ~[jorphan.jar:5.1.1 r1855137]
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2177) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2068) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1572) ~[?:1.8.0_222]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430) ~[?:1.8.0_222]
at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:322) ~[?:1.8.0_222]
at sun.rmi.server.UnicastServerRef.unmarshalParametersUnchecked(UnicastServerRef.java:629) ~[?:1.8.0_222]
at sun.rmi.server.UnicastServerRef.unmarshalParameters(UnicastServerRef.java:617) ~[?:1.8.0_222]
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:338) ~[?:1.8.0_222]
at sun.rmi.transport.Transport$1.run(Transport.java:200) ~[?:1.8.0_222]
at sun.rmi.transport.Transport$1.run(Transport.java:197) ~[?:1.8.0_222]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222]
at sun.rmi.transport.Transport.serviceCall(Transport.java:196) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) ~[?:1.8.0_222]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) ~[?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222]
2019-10-21 16:48:26,917 INFO o.a.j.e.ClientJMeterEngine: Interrupting RMI Reaper
2019-10-21 16:48:26,917 ERROR o.a.j.u.JMeterUtils: Error in rconfigure() method java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.lang.ClassNotFoundException: com.blazemeter.jmeter.threads.concurrency.ConcurrencyThreadGroup (no security manager: RMI class loader disabled)
2019-10-21 16:48:26,918 INFO o.a.j.e.DistributedRunner: Remote engines have been started
------- Update 22-10-2019
Dmitri's comment was really helpful. He was right and the jmeter-slave machine was missing the Concurrency Thread Group. After I installed it, no more error messages, the test is starting properly and I get the start/finish info in the slave machine Jmeter Slave output.
I am using Taurus in order to execute the test and even if the test is running in the slave machines, no data is comming to the Taurus GUI. The Taurus GUI is up and running all the time even if the test was finished already in the slave machine (I need to kill it to stop it). Am I missing another pluging?

Answer 1

Most probably it's a networking or firewall issue.
Make sure to bind the JMeter slave machine to the given network interface like:
./jmeter-server -Djava.rmi.server.hostname=10.111.2.4
Make sure to allow the following ports in the firewall
the port you define as server_port, by default 1099
the port you define as server.rmi.localport
the ports you define as client.rmi.localport
Additionally you can force JVM to use IPv4 addresses by setting java.net.preferIPv4Stack property to true
More information: How to Perform Distributed Testing in JMeter