I want to setup a 3 node cluster for JMeter on AWS EC2 instances.
I am not able to find any reference. Can anybody guide it
AWS EC2 instances don't differ from any other computers so you can follow instructions from Remote Testing guide.
Just make sure to unblock the following ports in AWS Security Groups
1099
the port you define as server.rmi.localport
the port(s) you define as client.rmi.localport
References:
Remote hosts and RMI configuration
How to Perform Distributed Testing in JMeter
You may also find JMeter ec2 Script useful
You may refer to this AWS CDK based framework for running Jmeter tests in distributed mode on AWS (FARGATE/ECS using Jmeter Docker containers)
https://github.com/pranavkakde/aws-cdk-load-test-jmeter
Related
I try to start remote testing using my computer as Client/Master and EC2 instance as Slave
I achieve all these Points:
Disable Firewall on both Master and Slave.
I have the same version of java and JMeter on both Master and Slave.
I have set all communication to be over port 4000.
My configuration for Master:
remote_hosts=10.xx.xx.xxx
server_port=4000
server.rmi.port=4000
server.rmi.localport=4000
server.rmi.ssl.disable=true
My Configuration for Slave [EC2 instance]
server_port=4000
server.rmi.port=4000
server.rmi.localport=4000
server.rmi.ssl.disable=true
Command to start JMeter server on slave [ EC2 instance]:
./jmeter-server -Gjava.rmi.server.hostname:10.xx.xx.xxx
Command to start JMeter server on Master [ My Computer]:
./jmeter-server -Gjava.rmi.server.hostname:192.xx.xx.xxx
After Running the test from the master the test started on the slave and finished.
My issue that the Client/Master didn't get any Result or Summary, it stuck and freeze on this line:
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445.
Your 10.xx.xx.xxx and 192.xx.xx.xxx are class A and class C local networks, it means that they are not accessible from anywhere else, only from their respective local networks.
So you won't be able to reach the EC2 instance internal IP from your computer and vice versa.
In order to be able to connect to the EC2 instance you need to:
Use external (public) IP address or DNS hostname
Open the port 4000 in the AWS Security Groups
In order to get results back to your computer from the EC2 machine you need to have static external IP address, you need to reach out to your ISP or network administrator to get this configured and assigned
An example of master/slave configuration with custom ports can be found in the JMeter Distributed Testing with Docker article.
More information: Remote hosts and RMI configuration
If you have only one slave machine it doesn't make sense to invest into master/slave configuration at all, just run JMeter in command-line non-GUI mode in the EC2 instance and analyze the results locally.
If you plan to use more than 1 slave - it makes sense to transfer the master to the EC2 as well, this way you will be able to use internal IP addresses
Should I run consul slaves alongside nomad slaves or inside them?
The later might not make sense at all but I'm asking it just in case.
I brought my own nomad cluster up with consul slaves running alongside nomad slaves (inside worker nodes), my deployable artifacts are docker containers (java spring applications).
The issue with my current setup is that my applications can't access consul slaves (to read configurations) (none of 0.0.0.0, localhost, worker node ip worked)
Lets say my service exposes 8080, I configured docker part (in hcl file) to use bridge as network mode. Nomad maps 8080 to 43210.
Everything is fine until my service tries to reach the consul slave to read configuration. Ideally giving nomad worker node IP as consul host to Spring should suffice. But for some reason it's not.
I'm using latest version of nomad.
I configured my nomad slaves like https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/nomad/client1.hcl
And the link below shows how I configured/ran my consul slave:
https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/server2.yml
Note: if I use static port mapping and host as the network mode for docker (in nomad) I'll be fine but then I can't deploy more than one instance of each application in each worker node (due to port conflic)
Nomad jobs listen on a specific host/port pair.
You might want to ssh into the server and run docker ps to see what host/port pair the job is listening on.
a93c5cb46a3e image-name bash 2 hours ago Up 2 hours 10.0.47.2:21435->8000/tcp, 10.0.47.2:21435->8000/udp foo-bar
Additionally, you will need to ensure that the consul nomad job is listening on port 0.0.0.0, or the specific ip of the machine. I believe that is this config value: https://www.consul.io/docs/agent/options.html#_bind
All those will need to match up in order to consul to be reachable.
More generally, I might recommend: if you're going to run consul with nomad, you might want to switch to host networking, so that you don't have to deal with the specifics of the networking within a container. Additionally, you could schedule consul as a system job so that it is automatically present on every host.
So I managed to solve the issue like this:
nomad.job.group.network.mode = host
nomad.job.group.network.port: port "http" {}
nomad.job.group.task.driver = docker
nomad.job.group.task.config.network_mode = host
nomad.job.group.task.config.ports = ["http"]
nomad.job.group.task.service.connect: connect { native = true }
nomad.job.group.task.env: SERVER_PORT= "${NOMAD_PORT_http}"
nomad.job.group.task.env: SPRING_CLOUD_CONSUL_HOST = "localhost"
nomad.job.group.task.env: SPRING_CLOUD_SERVICE_REGISTRY_AUTO_REGISTRATION_ENABLED = "false"
Running consul agent (slaves) using docker-compose alongside nomad agent (slave) with host as network mode + exposing all required ports.
Example of nomad job: https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/nomad/location-update-publisher.hcl
Example of consul agent config (docker-compose file): https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/server2.yml
Disclaimer: The LAB is part of Cluster Visualization Framework called: LiteArch Trafik which I have created as an interesting exercise to understand Nomad and Consul.
It took me long time to shift my mind from K8S to Nomad and Consul,
Integration them was one of my effort I spent in the last year.
When service resolution doesn't work, I found out it's more or less the DNS configuration on servers.
There is a section for it on Hashicorp documentation called DNS Forwarding
Hashicorp DNS Forwarding
I have created a LAB which explains how to set up Nomad and Consul.
But you can use the LAB seperately.
I created the LAB after learning the hard way how to install the cluster and how to integrate Nomad and Consul.
With the LAB you need Ubuntu Multipass installed.
You execute one script and you will get full functional Cluster locally with three servers and three nodes.
It shows you as well how to install docker and integrate the services with Consul and DNS services on Ubuntu.
After running the LAB you will get the links to Nomad, Fabio, Consul.
Hopefully it will guide you through the learning process of Nomad and Consul
LAB: LAB
Trafik:Trafik Visualizer
I have an aerospike server running on EC2 and I need to run some tests on aerospike from Codebuild. Since Codebuild container's ip is not known, I can't specify an inbound rule in the security group attached to EC2. I've tried using amazon VPC as described here.
Can anyone help me get this working?
On the server node in EC2, open port 3000-3003 to all 0.0.0.0/0 (all ips) in the custom TCP rules- that should work.
Is there a way to host Moqui on AWS? I was trying to host Moqui using a EC2 instance but couldn't figure out a way to connect them.
The Run and Deploy document on moqui.org has a section for a simple recommended deployment using ElasticBeanstalk and RDS:
https://www.moqui.org/m/docs/framework/Run+and+Deploy#AWSElasticBeanstalkandRDS
With more details about how you want to set things up on AWS the answer to how might vary from this.
For clustered setups things get more involved to get the right settings for Hazelcast AWS discovery and it is best to use an external ElasticSearch server like an AWS ElasticSearch instance and configure Moqui using environment variables to use the Java REST Client mode instead of the Embedded Node mode. Settings for the moqui-hazelcast and moqui-elasticsearch components can be seen in the MoquiConf.xml file in each component.
I would like to use one EC2 instance as a controller for 1 or more remote server instances in Amazon. I can start and run the test (on the remote side it runs) but the controller never exits. It fails with:
2015/02/12 17:34:25 ERROR - jmeter.JMeter: Error in NonGUIDriver
java.lang.IllegalStateException: Engine is busy - please try later
at org.apache.jmeter.engine.RemoteJMeterEngineImpl.rconfigure(RemoteJMeterEngineImpl.java:151)
If I run the test w/o the -R it works fine.
The same test setup works in SoftLayer so I think it is a firewall issue but I think I have put all the ports into my security group.
JMeter use Java RMI for distributed tests. This protocol is sensible for using different networks or firewalls.
For EC2 I suggest to use:
Amazon VPC for creating own virtual network for Jmeter instances.
Use ssh tunnels between Jmeter machines - this type of conenction is usually more stable and better "debuggable" then RMI