hadoop's datanode is not starting - hadoop

Question

I am using ubuntu 14.04 LTS Java version 8 and Hadoop 2.5.1 for installation. I followed this guide to install all the components. Sorry for not using michael noll's.
Now the problem that i face is when i do start-dfs.sh i get the following message
oroborus#Saras-Dell-System-XPS-L502X:~$ start-dfs.sh <br>
14/11/12 16:12:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<br>
Starting namenodes on [localhost]<br>
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-oroborus-namenode-Saras-Dell-System-XPS-L502X.out<br>
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-oroborus-datanode-Saras-Dell-System-XPS-L502X.out<br>
Starting secondary namenodes [0.0.0.0]<br>
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-oroborus-secondarynamenode-Saras-Dell-System-XPS-L502X.out<br>
14/11/12 16:12:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<br>
Now after running start-yarn.sh(which seems to work fine) and jps i get the following output
oroborus#Saras-Dell-System-XPS-L502X:~$ jps
9090 NodeManager
5107 JobHistoryServer
8952 ResourceManager
12442 Jps
11981 NameNode
The ideal output should have datanode in it, but it is not there. Googling and SOing a bit i found out the error is looged in the logs so here are the logs for datanode.(Only error part if you need more let me know)
2014-11-08 23:30:32,709 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
2014-11-08 23:30:33,132 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid dfs.datanode.data.dir /usr/local/hadoop_store/hdfs/datanode :
EPERM: Operation not permitted
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:226)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:642)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:472)
at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:126)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:142)
at org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:1866)
at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:1908)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1890)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1782)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1829)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2005)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2029)
2014-11-08 23:30:33,134 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/usr/local/hadoop_store/hdfs/datanode/"
at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:1917)
Now my doubt is how to make it valid.
Help is appreciated.
P.S. I tried a lot of forums, SO post none of them could give me solution to this problem. Hence the question.

Answer 1

First delete all contents from hdfs folder:
Value of <name>hadoop.tmp.dir</name>
rm -rf /usr/local/hadoop_store
Make sure that dir has right owner and permission /usr/local/hadoop_store
hduser#localhost$sudo chown hduser:hadoop -R /usr/local/hadoop_store
hduser#localhost$sudo chmod 777 -R /usr/local/hadoop_store
Format the namenode:
hduser#localhost$hadoop namenode -format
Start all processes again

Answer 2

I also got the same problem,and I fixed them by changing the owner of those working directory.Although you have permissions 777 to these two directory,framework will not be able to use it unless you change the owner to hduser.
$ sudo chown -R hduser:hadoop /usr/local/hadoop/yarn_data/hdfs/namenode
$ sudo chown -R hduser:hadoop /usr/local/hadoop/yarn_data/hdfs/datanode
After this,you again start your cluster and you should see datanode running.

Answer 3

first stop all the entities like namenode, datanode etc. (you will
be having some script or command to do that)
Format tmp directory
Go to /var/cache/hadoop-hdfs/hdfs/dfs/ and delete all the contents in the directory manually
Now format your namenode again
start all the entities then use jps command to confirm that the datanode has been started
Now run whichever application you may like or have.
Hope this helps.

Answer 4

When using cloudera, I had the same problem.
* chown hdfs:hadoop all_hdfs_directories
chown 997:996 all_hdfs_directories (if users don't exist yet)
* chmod 0755 is enough
* delete contents of directories if problem persists