On hadoop 2.2.0, My datanode could't start up - hadoop

everyone, I have a little problem when I build the Hadoop Cluster
My node install CentOS 6.5,java1.7.60 and hadoop 2.2.0
I want to build a master and three slaves
I try to build it like this
But in the end of this, I try to start up my namenode and datanode
My /etc/hosts like this:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.10.10.10 master
10.10.10.11 slave1
10.10.10.12 slave2
10.10.10.13 slave3
Just like this when I type:
$ hadoop namenode -fromat:
java.net.UnknownHostException: hadoop01.hadoopcluster: hadoop01.hadoopcluster
at java.net.InetAddress.getLocalHost(InetAddress.java:1473)
at org.apache.hadoop.net.DNS.resolveLocalHostname(DNS.java:264)
at org.apache.hadoop.net.DNS.<clinit>(DNS.java:57)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:914)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
Caused by: java.net.UnknownHostException: hadoop01.hadoopcluster
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
... 8 more
14/07/01 16:50:46 WARN net.DNS: Unable to determine address of the host-falling back to "localhost" address
java.net.UnknownHostException: hadoop01.hadoopcluster: hadoop01.hadoopcluster
at java.net.InetAddress.getLocalHost(InetAddress.java:1473)
at org.apache.hadoop.net.DNS.resolveLocalHostIPAddress(DNS.java:287)
at org.apache.hadoop.net.DNS.<clinit>(DNS.java:58)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:914)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
Caused by: java.net.UnknownHostException: hadoop01.hadoopcluster
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
... 8 more
and try to issue start-dfs.sh and start-yarn.sh:
$ start-dfs.sh
14/07/01 16:55:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master]
master: namenode running as process 2395. Stop it first.
slave2: starting datanode, logging to /opt/hadoop/logs/hadoop-hadoop-datanode-hadoop03.hadopcluster.out
slave1: starting datanode, logging to /opt/hadoop/logs/hadoop-hadoop-datanode-hadoop02.hadopcluster.out
slave3: starting datanode, logging to /opt/hadoop/logs/hadoop-hadoop-datanode-hadoop04.hadopcluster.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 2564. Stop it first.
14/07/01 16:55:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/logs/yarn-hadoop-resourcemanager-hadoop01.hadoopcluster.out
slave1: starting nodemanager, logging to /opt/hadoop/logs/yarn-hadoop-nodemanager-hadoop02.hadopcluster.out
slave3: starting nodemanager, logging to /opt/hadoop/logs/yarn-hadoop-nodemanager-hadoop04.hadopcluster.out
slave2: starting nodemanager, logging to /opt/hadoop/logs/yarn-hadoop-nodemanager-hadoop03.hadopcluster.out
and type jps:
$ jps
2564 SecondaryNameNode
5591 Jps
2395 NameNode
I only look like this, didn't have DataNode, NodeManager, ResourceManger... etc,
Is it any where wrong when I setting it?
Could anyone can suggest me something, thanks!

DHCP (Dynamic Host Configuration Protocol)
it is a server deployed on an IP network and it is used to allocate IP addresses to its clients. Therefore you had to configure the DHCP on both the server and clients sides.
On the server side:
getting the package: isc-dhcp-server
editing /etc/network interfaces to configure interfaces, choose a static IP address for the DHCP server
specifying the local network interface DHCP will listen to (/etc/default/isc-dhcp-server)
editing configuration file /etc/dhcp/dhcpd.conf to choose the IP addresses that will be used by the local network, define the DNS server
On the client side:
editing /etc/network interfaces to configure interfaces
You can make sure that it is well installed and configured you can use ifconfig and ping commands.
DNS (Domain Name System)
intalling bind9
editing /etc/bind/named.conf to add hosts
Good luck.

Related

ssh connection to host port 22 connection refused

I use a VMware virtualization system. I have centos release 7 as my operating system. I installed hadoop2.7.1. After installing Hadoop I ran the command :#hdfs namenode -format, it ran successfully. But when I run the command :#./start-all.sh it gives me errors. I tried several proposals that I saw on the internet but the problem persists
[root#MASTER sbin]# ./start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
21/06/17 19:06:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [MASTER]
root#master's password:
MASTER: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-MASTER.out
localhost: ssh: connect to host localhost port 22: Connection refused
Starting secondary namenodes [0.0.0.0]
0.0.0.0: ssh: connect to host 0.0.0.0 port 22: Connection refused
21/06/17 19:06:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-MASTER.out
localhost: ssh: connect to host localhost port 22: Connection refused
Provide ssh-key less access to all your worker nodes in hosts file, even localhost. Read instruction in the Tutorial of How To Set Up SSH Keys on CentOS 7.
At last test access without password by ssh localhost and ssh [yourworkernode].
Also, run start-dfs.sh and if was successful run start-yarn.sh.

secondary namenode failed to start ssh: connect to host 0.0.0.7 port 22: Connection timed out

Starting secondary namenodes [7]
7: ssh: connect to host 0.0.0.7 port 22: Connection timed out
secondary namenode not started due to connection timeout
how to fix this on Ubuntu 20.04.1 LTS?
vivek#7:~$ start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as pradeep in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [7]
7: ssh: connect to host 0.0.0.7 port 22: Connection timed out
2020-10-15 18:00:39,116 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Starting nodemanagers
vivek#7:~$ jps
6256 NodeManager
5649 NameNode
6098 ResourceManager
6586 Jps
5788 DataNode

Hadoop 2.2 - datanode doesn't start up

I had Hadoop 2.4 this morning (see my previous 2 questions). Now I removed it and installed 2.2 as I had issues with 2.4, and also as I think 2.2 is the latest stable release. Now I followed the tutorial here:
http://codesfusion.blogspot.com/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1
I am pretty sure I did everything right but I am facing similar issues again.
When I run jps it is obvious that the data node is not starting up.
What am I doing wrong again?
hduser#test02:~$ start-dfs.sh
14/06/06 18:12:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-test02.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-test02.out
localhost: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
localhost: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hduser-secondarynamenode-test02.out
0.0.0.0: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
0.0.0.0: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/06/06 18:13:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hduser#test02:~$ jps
2201 Jps
hduser#test02:~$ jps
2213 Jps
hduser#test02:~$ start-yarn
start-yarn: command not found
hduser#test02:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-test02.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-test02.out
hduser#test02:~$ jps
2498 NodeManager
2264 ResourceManager
2766 Jps
hduser#test02:~$ jps
2784 Jps
2498 NodeManager
2264 ResourceManager
hduser#test02:~$ jps
2498 NodeManager
2264 ResourceManager
2796 Jps
hduser#test02:~$
My problem was that I took these instructions from the tutorial too literally.
Paste following between <configuration>
fs.default.name
hdfs://localhost:9000
I suspected this was wrong while doing it but still I did it.
It seemed incorrect as the core-site.xml file is in XML format.
So actually, it needs to look like this.
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
Changing it to this fixed my problem.
I had similar issues with DataNode not starting up. What I did was reformat the namenode, then restarted the cluster. Then, running jps confirmed that data node was started up.
This can be caused by placing the HDFS directory in your "home" directory (on a linux box) since upon starting up and shutting down the OS affects these folders (not exactly sure how, but to prevent this problem in the future, move the HDFS directory out of your home directory).
Please let me know if this works.

hadoop datanode dies after few seconds

I installed hadoop-2.2.0 version on vmware and when it starts, it shows all the processes working, but after some time datanode gets killed. So, I checked the logs and I found this:
2014-01-21 04:36:00,325 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-01-21 04:36:21,034 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.net.UnknownHostException: ubuntu: ubuntu
at java.net.InetAddress.getLocalHost(InetAddress.java:1402)
at org.apache.hadoop.security.SecurityUtil.getLocalHostName(SecurityUtil.java:227)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:247)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1640)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1665)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1837)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1858)
2014-01-21 04:36:21,107 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2014-01-21 04:36:21,114 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at java.net.UnknownHostException: ubuntu: ubuntu
************************************************************/
I think the problem is in your /etc/hosts file.
Make sure it is configured correctly. My /etc/hosts file looks like this:
127.0.0.1 localhost
192.168.0.1 master mypcname
192.168.0.18 slave1
192.168.0.12 slave2
192.168.0.17 slave3
for master and
127.0.0.1 localhost
192.168.0.1 master
192.168.0.18 slave1 slave1pcname
192.168.0.12 slave2
192.168.0.17 slave3
for slave1.

Hadoop pseudo distributed mode - Datanode and tasktracker not starting

I am running a Red Hat Enterprise Linux Server release 6.4 (Santiago) distribution with Hadoop 1.1.2 installed on it. I have made the required configurations to enable the pseudo distributed mode. But on trying to run hadoop, the datanode and tasktracker don't start.
I am not able to copy any files to hdfs.
[hduser#is-joshbloom-hadoop hadoop]$ hadoop dfs -put README.txt /input
Warning: $HADOOP_HOME is deprecated.
13/05/23 16:42:00 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /input could only be replicated to 0 nodes, instead of 1
Also after trying hadoop-daemon.sh start datanode I get the message:
starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-datanode-is-joshbloom-hadoop.out
same goes for tasktracker. But when I try the same command for namenode, secondarynamenode, jobtracker they are seem to be running.
namenode running as process 32933. Stop it first.
I tried the following solutions:
Reformatting namenode
Reinstalling hadoop
Installing different version of hadoop (1.0.4)
None seem to work. I have followed the same installation steps on my Mac and on amazon ubuntu VM and it works perfectly.
How can I get hadoop working?
Thanks!
*UPDATE**
Here is the log entry of namenode
2013-05-23 16:27:44,087 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = java.net.UnknownHostException: is-joshbloom-hadoop: is-joshbloom-hadoop
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.1.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
************************************************************/
2013-05-23 16:27:44,382 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-05-23 16:27:44,432 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2013-05-23 16:27:44,446 ERROR org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Error getting localhost name. Using 'localhost'...
java.net.UnknownHostException: is-joshbloom-hadoop: is-joshbloom-hadoop
at java.net.InetAddress.getLocalHost(InetAddress.java:1438)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.getHostname(MetricsSystemImpl.java:463)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSystem(MetricsSystemImpl.java:394)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:390)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:152)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:133)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1589)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1608)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1734)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1751)
Caused by: java.net.UnknownHostException: is-joshbloom-hadoop
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:866)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1258)
at java.net.InetAddress.getLocalHost(InetAddress.java:1434)
... 11 more
2013-05-23 16:27:44,453 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-05-23 16:27:44,453 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2013-05-23 16:27:44,768 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-05-23 16:27:44,914 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2013-05-23 16:27:45,212 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.net.UnknownHostException: is-joshbloom-hadoop: is-joshbloom-hadoop
at java.net.InetAddress.getLocalHost(InetAddress.java:1438)
at org.apache.hadoop.security.SecurityUtil.getLocalHostName(SecurityUtil.java:271)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:289)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:301)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1651)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1590)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1608)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1734)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1751)
Caused by: java.net.UnknownHostException: is-joshbloom-hadoop
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:866)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1258)
at java.net.InetAddress.getLocalHost(InetAddress.java:1434)
... 8 more
2013-05-23 16:27:45,228 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at java.net.UnknownHostException: is-joshbloom-hadoop: is-joshbloom-hadoop
************************************************************/
*UPDATE***
content of /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
Amend your /etc/hosts to include a hostname loopback mapping:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
127.0.1.1 is-joshbloom-hadoop
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
Your problem is your machine doesn't know how to resolve the hostname is-joshbloom-hadoop to a specific IP address. There are typically two places/methods resolution occurs - either via a DNS server or using a local hosts file (hosts file takes precedence).
The above amendment to your hosts file allows you machine to resolve the machine name is-joshbloom-hadoop to the IP address 127.0.1.1. The OS has an internal loopback address for the range 127.0.0.0/8, so you could name any address in here. On my Ubuntu laptop, it uses the 127.0.1.1 and i'm sure it changes between OS's, but my guess is by not using 127.0.0.1 you don't have to search for it in the localhost line if you change your machine name in future.
Check your core-site.xml in HADOOP_HOME/conf.It will have fs.default.name property.It should have a hostname specified in your /etc/hosts."is-joshbloom-hadoop" hostname is not there in /etc/hosts.Use localhost instead
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
</property>
</configuration>
Problem seems to be like that u have nothing in slaves file under conf/slaves.
Check your slaves file in conf/slaves.Remove everything and add localhost in that file.Remove the name and data directory mentioned in dfs.name.dir and dfs.data.dir property under hdfs-site.xml.
Format ur HDFS file system and then start ur daemons again.

Resources