Data-Node Does Not Start - hadoop

Question

I have trouble starting my Hadoop data-node. I did all the research that I could and none of the methods were helpful in solving my issue. Here's my terminal console output when I try to start it using
hadoop datanode -start
This is what happens:
root#Itanium:~/Desktop/hadoop# hadoop datanode -start
Warning: $HADOOP_HOME is deprecated.
13/09/29 22:11:42 INFO datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = Itanium/127.0.1.1
STARTUP_MSG: args = [-start]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_25
************************************************************/
Usage: java DataNode
[-rollback]
13/09/29 22:11:42 INFO datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at Itanium/127.0.1.1
************************************************************/
root#Itanium:~/Desktop/hadoop# jps
31438 SecondaryNameNode
32013 Jps
31818 TaskTracker
1146 Bootstrap
31565 JobTracker
30930 NameNode
root#Itanium:~/Desktop/hadoop#
As we can see the DataNode attempts to start but then shuts down. All the while I have been having trouble with NameNode starting up. I used to fix this by manually starting it using
start-dfs.sh
And now the problem is with DataNode. I really would appreciate all your help in resolving this issue.
And one more generic question. Why is Hadoop displaying such inconsistent behavior. I am sure I did not change any of the *-site.xml settings.

Answer 1

use this command hadoop datanode -rollback

Answer 2

I had a similar issue as well. Looking at the comment posted by Anup "seems to be an issue with namespaceIDs not matching" I was able to find a reference that showed me how to solve my issue.
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/#caveats
I took a look at the logfile on the slave nodes where the DataNodes did not start. They both had the following exception :
2014-11-05 10:26:14,289 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /scratch/hdfs/data/srinivasand: namenode namespaceID = 1296690356; datanode namespaceID = 1228298945
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
Fixing this exception solved the issue.
The fix is to either
a) delete the dfs data directory. reformat using namenode -format.
b) update the VERSION file so that the two namespace IDs match.
I was able to use option b) and the datanodes started successfully after that.
The bug report that leads to this issue is recorded at : https://issues.apache.org/jira/browse/HDFS-107

Answer 3

I ever got the same issue, it turns out that 50010 port is occupied by other application, stop the application, restart Hadoop