distributed load testing on aws with jmeter - amazon-ec2

Question

I have been trying to setup aws ec2 machines for load testing of my web server using jmeter but I am stuck. I have a jmeter client on my local machine and I want to set up multiple jmeter-server nodes on ec2 to do the load testing and I am, thus far, just trying to get one server node up and running. But it hasn't worked out for me yet.
I have the same jmeter running on my local machine and the server and the java version was a little different but I don't think that is the problem. Most of the people have had problems with getting the correct ip for connecting between the client and the server nodes but I, after a lot of searching, have gotten through all those problems. I am stuck at when the server node attempts to return the result and tries to connect to the client, my local machine. The server tries to connect to the external ip address of my local machine. But it throws a connection refused error, which apparently was caused by connection timeout. I guess it's some firewall issue but I tried turning off the firewall on my local machine but it still throws the same error. I am not sure how can I get past this and it's taking way too much time then it should.
Could somebody please suggest me something to solve this? Thanks!
My local machine is a Mac OS X 10.7.5 and my server nodes are on ubuntu.
This is the error that it throws:
2013/01/29 12:23:37 ERROR - jmeter.samplers.RemoteListenerWrapper: testStarted(host) java.rmi.ConnectException: Connection refused to host: xxx.xxx.xxx.10; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:128)
at org.apache.jmeter.samplers.RemoteSampleListenerImpl_Stub.testStarted(Unknown Source)
at org.apache.jmeter.samplers.RemoteListenerWrapper.testStarted(RemoteListenerWrapper.java:83)
at org.apache.jmeter.engine.StandardJMeterEngine.notifyTestListenersOfStart(StandardJMeterEngine.java:226)
at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:349)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
at java.net.Socket.connect(Socket.java:495)
at java.net.Socket.<init>(Socket.java:392)
at java.net.Socket.<init>(Socket.java:206)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 8 more

Answer 1

Well, I finally solved the problem. I ended up using ssh reverse tunnels. I am not sure if there is a better way to do this though. So, in case anyone has a similar problem, this is how I did it:
Create a reverse ssh tunnel from the server to the client. So, at client side:
ssh -Nf -R [client.rmi.localport]:localhost:[client.rmi.localport on serverside] user#server
start server and have a client.rmi.localport as well; the port at which the tunnel was created;
start the client as: ./bin/jmeter-server -Djava.rmi.server.hostname=127.0.0.1.
And that's it! You have your distributed testing ready.

Answer 2

Solution that worked for me on Linux/OSX:
1.On the client edit bin/jmeter.properties and add:
remote_hosts=127.0.0.1:55501
client.rmi.localport=55512
mode=Batch
num_sample_threshold=250
2.On the server edit bin/jmeter.properties and add:
server_port=55501
server.rmi.localhostname=127.0.0.1
server.rmi.localport=55511
3.Now connect to the server using this ssh tunel:
ssh -L 55501:127.0.0.1:55501 -L 55511:127.0.0.1:55511 -R 55512:127.0.0.1:55512 user#hostname
4.Edit jmeter-server script to start jmeter.sh
${DIRNAME}/jmeter.sh ${RMI_HOST_DEF} -Dserver_port=${SERVER_PORT:-1099} -s -j jmeter-server.log "$#"
5.Now run on the server:
bin/jmeter-server -Djava.rmi.server.hostname=127.0.0.1
6.And on the client run jmeter with gui or add -n if gui is not needed:
bin/jmeter.sh -Djava.rmi.server.hostname=127.0.0.1
or, with test plan:
bin/jmeter.sh -Djava.rmi.server.hostname=127.0.0.1 -t /path/to/test-plan.jmx

Answer 3

Looks like you have to move your jmeter-master instance (jmeter client) to EC2 instance too.
As per JMeter Distributed Testing Step-by-step:
2. check all the clients are on the same subnet;
For distributed testing to work, the systems must be on the same subnet, otherwise RMI will not be able to connect.
Looks like to be your case: jmeter-slaves are in one subnet (EC2) and jmeter-master in another (your local workstation).

Answer 4

I wrote a free, open source script to help do exactly this. I went through the same issues listed by the OP and, even though I did get things working in the end, it was never great and I wanted something to automate away the hassle.