I install Hadoop on a single machine with two VMs(x1 & x2), and set x1(10.42.0.70) as namenode and datanode, x2(10.42.0.71) as the only datanode. When I start Hadoop. I found that Hadoop assign x1 and x2 with the wrong IP address. x1 and x2 both have IP address 127.0.1.1. I want to know how Hadoop finds an IP address for a node from the hostname and where is the problem. Thanks for any tips.
enter image description here
enter image description here
127.0.1.1 is a loopback interface. Can you paste contents of your /etc/hosts file?
Additionally check this out: https://serverfault.com/questions/424232/how-does-hadoop-decide-what-its-nodes-hostnames-are
I had the same problem, and found out that the cause was a way how nodes connect to each other. A datanode was configured to connect to local namenode by using localhost as an address (hdfs://localhost:9000 in my case). It seems that namenode uses such connection to resolve datanode address as 127.0.1.1
Following should help, add to core-size.xml:
<property>
<name>fs.defaultFS</name>
<value>hdfs://10.42.0.70:9000</value>
</property>
Related
I am trying to get files from hadoop using webhdfs, now the webhdfs is redirecting me to the datanodes. Its returning the hostnames in address , is there a way where we can make it return ip address instead of hostnames ?
dfs.datanode.hostname
Optional. The hostname for the Datanode containing this configuration file.
Will be different for each machine. Defaults to current hostname.
In hdfs-site.xml, you can set dfs.datanode.hostname.
<property>
<name>dfs.datanode.hostname</name>
<value>IP-ADDRESS</value>
</property>
By doing this, webhdfs will return ip address of datanode.
I want to acces cloudera from a distant machine via Talend for big data. In order to do that i changed the ip of the host in cloudera by editing the file /etc/hosts and /etc/sysconfig/network.
I can acces cloudera from Talend. However the problem is that my datanode and Namenode seems to be not connected. When i check the log details of my Datanode i get the following errors :
Block pool BP-1183376682-127.0.0.1-1433878467265 (Datanode Uuid null) service to quickstart.cloudera/127.0.0.1:8022 beginning handshake with NN
Initialization failed for Block pool BP-1183376682-127.0.0.1-1433878467265 (Datanode Uuid null) service to quickstart.cloudera/127.0.0.1:8022 Datanode denied communication with namenode because the host is not in the include-list: DatanodeRegistration(127.0.0.1, datanodeUuid=5802ab81-2c28-4beb-b155-cac31567d23a, infoPort=50075, ipcPort=50020, storageInfo=lv=-56;cid=CID-83500942-6c65-4ecd-a0c2-a448be86bcb0;nsid=975591446;c=0)
the datanode still uses the wrong ip adress ( 127.0.0.1 ) even though i made the modifications in core-site.xml, hdfs-site.xml and mapred-site.xml by editing the previous ip adress by the new one.
I followed the steps given in this tutorial to do so :
https://www.youtube.com/watch?v=fSGpYHjGIRY
How can i fix this error ?
On Debian 8, the /etc/hosts will contain entry for 127.0.1.1 with your hostname you gave during Linux installation. Cloudera will use this IP-address for some of its services.
A regular HDFS will contain multiple servers with different hostnames/IP-addresses and will list those IPs as allowed. As your log says, the traffic is originating from 127.0.0.1, which is not the IP-address of your hostname.
For Cloudera single-server setup, the only way I found was to do the initial setup so, that /etc/hosts doesn't have the 127.0.1.1 entry in it.
I have a ubuntu server VM in virtual box(in Mac OSX). And I configured a Hadoop Cluster via docker: 1 master(172.17.0.3), 2 slave nodes(172.17.0.4, 172.17.0.6). After run "./sbin/start-dfs.sh" under Hadoop home folder, I found below error in datanode machine:
Datanode denied communication with namenode because hostname cannot be
resolved (ip=172.17.0.4, hostname=172.17.0.4): DatanodeRegistration(0.0.0.0,
datanodeUuid=4c613e35-35b8-41c1-a027-28589e007e78, infoPort=50075,
ipcPort=50020, storageInfo=lv=-55;cid=CID-9bac5643-1f9f-4bc0-abba-
34dba4ddaff6;nsid=1748115706;c=0)
Because docker does not support bidirectional name linking and further more, my docker version does not allow editing /etc/hosts file, So I use IP address to set name node and slaves. Following is my slaves file:
172.17.0.4
172.17.0.6
After searching on google and stackoverflow, no solution works for my problem. However I guess that Hadoop Namenode regard 172.17.0.4 as a "hostname", so it reports "hostname can not be resolved" where "hostname=172.17.0.4".
Any Suggestions?
Finally I got a solution, which proved my suppose:
1.upgrade my docker to 1.4.1, following instructions from: https://askubuntu.com/questions/472412/how-do-i-upgrade-docker.
2.write IP=>hostname mappings of master and slaves into /etc/hosts
3.use hostname instead of ip address in Hadoop slaves file.
4."run ./sbin/start-dfs.sh"
5.Done!
I'm using Hadoop MapReduce paradigm, and i need to get the NameNode IP address from the DataNode, can any one give me an idea how to do this?
Thanks.
Easiest way would be to quickly open the core-site.xml file under HADOOP_HOME/conf directory. The value of fs.default.name property will tell you the host and port where NN is running.
Delete the line 127.0.0.1 localhost in your /etc/host and put your IP and the name of all your machines. Hadoop is resolving all the IPs and names of machines on the cluster as 127.0.0.1 localhost if you leave the file as default.
I have a network with some weird (as I understand) DNS server which causes Hadoop or HBase to malfunction.
It resolves my hostname to some address my machine doesn't know about (i.e. there is no such interface).
Hadoop does work if I have following entries in /etc/hosts:
127.0.0.1 localhost
127.0.1.1 myhostname
If entry "127.0.1.1 myhostname" is not present uploading file to HDFS fails and complains that it can replicate the file only to 0 datanodes instead of 1.
But in this case HBase does not work: creating a table from HBase shell causes NotAllMetaRegionsOnlineException (caused actually by HMaster trying to bind to wrong address returned by DNS server for myhostname).
In other network, I am using following /etc/hosts:
127.0.0.1 localhost
192.168.1.1 myhostname
And both Hadoop and HBase work.
The problem is that in second network the address is dynamic and I can't list it into /etc/hosts to override result returned by weird DNS.
Hadoop is run in pseudo-distributed mode. HBase also runs on single node.
Changing behavior of DNS server is not an option.
Changing "localhost" to 127.0.0.1 in hbase/conf/regionservers doesn't change anything.
Can somebody suggest a way how can I override its behavior while retaining internet connection (I actually work at client's machine through Teamviewer). Or some way to configure HBase (or Zookeeper it is managing) not to use hostname to determine address to bind?
Luckily, I've found the workaround to this DNS server problem.
DNS server returned invalid address when queried by local hostname.
HBase by default does reverse DNS lookup on local hostname to determine where to bind.
Because the address returned by DNS server was invalid, HMaster wasn't able to bind.
Workaround:
In hbase/conf/hbase-site.xml explicitly specify interfaces that will be used for master and regionserver:
<configuration>
<property>
<name>hbase.master.dns.interface</name>
<value>lo</value>
</property>
<property>
<name>hbase.regionserver.dns.interface</name>
<value>lo</value>
</property>
</configuration>
In this case, I specified loopback interface (lo) to be used for both master and regionserver.
a simple tool I wrote to check for DNS issues:
https://github.com/sujee/hadoop-dns-checker