I have 3 remote VMs. All have both public and internal IP. All blong to eth0.
I have installed Hadoop 3.1.1 on them.
For master node, I wrote its internal ip as master host in its config files (core-site, yarn-site). and public ip of other VMs in its wokers file.
Fo wokers, I wrote public ip in all config files.
but it does not work and jobs stuck in accepted status waiting for container to be lunched!!!
It seems that the workers want to connect with public ip but master is listening on internal one!!!!
If i set public ip for master VM, obviously it does not work!
If i set interal ip of master in workers machine, resourcemanager shows them as CORRUPT datanodes; So there is no active datanode!!
Any Idea how to set IPs?
Thanks in advance
Related
I try to run a distributed test using JMeter, I have 2 EC2 instances
Master Public IP: 54.xxx.xx.xx
Slave Public IP: 204.xxx.xxx.xxx
I have opened all the necessary ports that were used in the configuration.
I can ping each EC2 from the other one and the ping is successful.
But when I try to start the test, the server failed and return [No route to host (Host unreachable)].
My plan is to use more than 1 slave.
Error Return From Master Server
As per NoRouteToHostException exception description:
Signals that an error occurred while attempting to connect a socket to a remote address and port. Typically, the remote host cannot be reached because of an intervening firewall, or if an intermediate router is down.
So make sure that the RMI ports which JMeter slave is listening are:
Not dynamic
Open in your operating system firewall
Open in EC2 Security Groups
More information:
Remote hosts and RMI configuration
How to Perform Distributed Testing in JMeter
Apache JMeter Distributed Testing Step-by-step
I have installed ICP within a private newtork. And each VM having public IP address. I am able to access ICP dashboard using master node private IP address (https://master-node-private-ip:8443).
But I am not able to access using master node public IP address (https://master-node-public-ip:8443). I tried with setting cluster_lb_address: in config.yaml file.
But it doesn't work.
In config.yaml file, uncomment and set both your cluster_lb_address and proxy_lb_address to the master node public IP address. In cluster/hosts file, all the IPs must have the same subnet.
Example:
# cat cluster/hosts
[master]
172.16.151.126
[worker]
172.16.151.182
172.16.155.135
[proxy]
172.16.151.126
#[management]
#4.4.4.4
#[va]
#5.5.5.5
config.yaml
------------------------------
cluster_lb_address: 9.x.x.x
proxy_lb_address: 9.x.x.x
I am setting up distributed testing in Linux machines with a master and a slave. Need your advice to overcome the below error message
"Exception Creating a connection to 192.xx.xx. xx ;nested exception is java.net.NoRouteToHostException: no route to host ( Host Unreachable )"
I did the following steps
Ensured Master and Slave has the same version of Jmeter
Added the slave machine IP in Master's Jmeter.properites
Created a keystore file in master and copied the generated
rmi_keystore.jks to slave machine bin folder
Run jmeter-server file in the slave machine ( There was an error
hence added the slave machines ip in
RMI_HOST_DEF=-Djava.rmi.server.hostname=192.xx.xx.xx)- success up
and running
Run the intended jmx file in master Run - Remote start - Slave
machine
Error Exception Creating a connection to 192.xx.xx.xx ; nested
exception is java.net.NoRouteToHostException: no route to host (Host
Unreachable )
I did check the connectivity between two machines when I ping i was able to reach each machine from one another,
Could be an issue with Firewall or port ? not sure
I have been banging my head , any pointers would be helpfull
thanks in advance
check your Slave from Master
example: nc -zvw3 192.168.1.31 1099
if you get Ncat: No route to host then disable firewall on Salve and Master:
systemctl stop firewalld
systemctl disable firewalld
I have set the following.
rpc_address to external public ip.
Listen Address : internal ip address (not the local host),
rpc_broad_cast: internal ip address
On Dev center, I am using external ip and port 9042.
Let me know, if am doing anything wrong.
Thank you snakecharmerb for trying to help me out on this.
I was able to find a solution for this myself. The actual problem was I was using Dev Center 1.4 to connect to Cassandra 3. Once I upgraded to the Dev Center 1.5, it worked like a charm with SSH Local port forwarding enabled.
These are the following settings
Listen Address : internal ip address (not the localhost),
rpc_address: internal ip address (same as above)
Steps After setting the above steps
On my terminal enabled local port forwarding
ssh -L 9042::9042 #
Start Dev Center 1.5
It worked like a charm
It's worked finally :
steps :
1. set listen_address to private IP of EC2 instance.
2. do not set any broadcast_address
3. set rpc_address to 0.0.0.0
4. set broadcast_rpc_address to public ip of EC2 instance.
I'm using the cloudera distribution of Hadoop and recently had to change the IP addresses of a few nodes in the cluster. After the change, on one of the nodes (Old IP:10.88.76.223, New IP: 10.88.69.31) the following error comes up when I try to start the data node service.
Initialization failed for block pool Block pool BP-77624948-10.88.65.174-13492342342 (storage id DS-820323624-10.88.76.223-50010-142302323234) service to hadoop-name-node-01/10.88.65.174:6666
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode: DatanodeRegistration(10.88.69.31, storageID=DS-820323624-10.88.76.223-50010-142302323234, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=cluster25;nsid=1486084428;c=0)
at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:656)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3593)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:899)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:91), I was unable to start the datanode service due to the following error:
Has anyone had success with changing the IP address of a hadoop data node and join it back to the cluster without data loss?
CHANGE HOST IP IN CLOUDERA MANAGER
Change Host IP on all node
sudo nano /etc/hosts
Edit the ip cloudera config.ini on all node if the master node ip changes
sudo nano /etc/cloudera-scm-agent/config.ini
Change IP in PostgreSQL Database
For the password Open PostgreSQL password
cat /etc/cloudera-scm-server/db.properties
Find the password lines
Example. com.cloudera.cmf.db.password=gUHHwvJdoE
Open PostgreSQL
psql -h localhost -p 7432 -U scm
Select table in PostgreSQL
select name,host_id,ip_address from hosts;
Update table IP
update hosts set ip_address = 'xxx.xxx.xxx.xxx' where host_id=x;
Exit the tool
\q
Restart the service on all node
service cloudera-scm-agent restart
Restart the service on master node
service cloudera-scm-server restart
Turns out its better to:
Decommission the server from the cluster to ensure that all blocks are replicated to other nodes in the cluster.
Remove the server from the cluster
Connect to the server and change the IP address then restart the cloudera agent
Notice that cloudera manager now shows two entries for this server. Delete the entry with the old IP and longest heartbeat time
Add the server to the required cluster and add required roles back to the server (e.g. HDFS datanode, HBASE RS, Yarn)
HDFS will read all data disks and recognize the block pool and cluster IDs, then register the datanode.
All data will be available and the process will be transparent to any client.
NOTE: If you run into name resolution errors from HDFS clients, the application has likely cached the old IP and will most likely need be restarted. Particularly Java clients that previously referenced this server e.g. HBASE clients must be restarted due to the JVM caching IPs indefinitely. Java based clients will likely throw errors relating to connectivity to the server with changed IP because they have the old IP cached until they are restarted.