Why Datanode process didn't run in wsl? - hadoop

Question

I have set up and configured my hadoop in wsl, but when I start dataNode, it did't work.
This is it's log
2020-03-24 23:47:08,788 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
2020-03-24 23:47:09,809 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-03-24 23:47:10,199 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid dfs.datanode.data.dir /mnt/d/hadoop/hadoop-2.7.1/tmp/dfs/data :
ExitCodeException exitCode=1: chmod: changing permissions of '/mnt/d/hadoop/hadoop-2.7.1/tmp/dfs/data': Operation not permitted
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:815)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:798)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:728)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:502)
at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:140)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:156)
at org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:2344)
at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2386)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2368)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2260)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2307)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2484)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2508)
2020-03-24 23:47:10,207 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/mnt/d/hadoop/hadoop-2.7.1/tmp/dfs/data/"
at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2395)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2368)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2260)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2307)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2484)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2508)
2020-03-24 23:47:10,209 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2020-03-24 23:47:10,213 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at DESKTOP-U1EOV4J.localdomain/127.0.1.1
************************************************************/
I want to let the dataNode and nameNode directory in my windows directory(such as in mnt/d/hadoop/hadoop-2.7.1/tmp) but failed.
I tried to use chmod 777 -R hadoop /mnt/d/hadoopbut it still can't start. Also I tired to delete the tmp directory and reformat namenode.
But when I change the directory to /home/hadoop/tmp(my wsl directory), dataNode can start normally.
According to the log file, I think this is a problem of authority but I don't konw why. How can I fix this?

Answer 1

I have solved this problem, this is caused by windows authority.
You can't change the file's authority under /mnt in wsl.
When you use chown or chmod, they actually don't work.
You need to use the following instructions first.
sudo umount /mnt/d
sudo mount -t drvfs D: /mnt/d -o metadata
Then you can change the file's authority in wsl.
You can get more information here: https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/