I am able to create directory using the below command but not able to create the subdir under already created dir. May I know what could be the reason. I have setup hdfs on my mac in pseudo distributed mode and trying to create these directories. Any help would be appreciated.
hadoop fs -mkdir /test/subdir
The above command doesn't create any sub directory however the below command creates a directory.
hadoop fs -mkdir test
To recursively create subdirectories inside parent directory, you have to provide -p option or else you can create one directory at a time.
hdfs dfs -mkdir -p /test/subdir
will work in your case.
Try giving it the parent creation flag.
hadoop fs -mkdir -p /test/subdir
Related
How to create a sub folder in hdfs directory which is already created
You make it like every other directory
hdfs dfs -mkdir -p /path/to/directory
With the p flag, it doesn't matter if the parent exists
I've been starting NameNode and DataNode, but when I try to use HDFS command to make a directory(in any place), it doesn't work.
Here is my command:
./hdfs dfs -mkdir -p /usr/master/datas
and I also trying to change the format of my path:
./hdfs dfs -mkdir -p "/usr/master/datas"
but I get same result.
I'm just starting to learn big-data. Can anyone tell me how to fix this issue and how debug the issue?
/usr doesn't exist on HDFS. That's a Unix directory.
The user directory in HDFS is /user.
Plus, you need to be an HDFS superuser to create HDFS folders under the root path, or at least folders not owned by the current user.
I have this:
I had also tried to edit this:
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib
as
export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_INSTALL/lib
in ~/.bashrc
But still I am getting a warning message and I'm not able to solve the problem.
Unable to create the directory
I'm using this code to create the directory for twitter analysis:
hadoop fs -mkdir hdfs://localhost:54310/home/vipal/hadoop_store/hdfs/namenode/twitter_data
Notice how hadoop fs -ls says .: No such file or directory?
First, you must create your home directory, which is /user in HDFS.
hdfs dfs -mkdir -p /user/$(whoami)
(You should also chown and chmod that directory)
Then, you can place files into a twitter_data directory.
hdfs dfs -mkdir twitter_data
hdfs dfs -put <local_files> twitter_data
(I removed hadoop_store/hdfs/namenode because that doesn't make sense)
i'm looking for a shell script which should copy directory (with files under) from HDFS to local system.
I think it is pointless to write a whole script, when you only need to write one command into terminal.
With
hadoop fs -ls /myDir/path
you can verify name and path to directory, which you want to copy and write
hadoop fs -get /myDir/path
to get file into local. You also can specify destination directory by
hadoop fs -get /myDir/path /myLocal/destDir
It copies while directory (with subdirectories) to your working directory or to specified directory. You also can get file by file (dir by dir) with
hadoop fs -get /myDir/path/*
or specific dirs or files in one command
hadoop fs -get /myDir/path/dir1 /myDir/path/dir2 .
to your directory. I tried it on my Hadoop VM and it works fine.
i have problem, how create folder in hadoop but name path folder in year,date,time.?
example:
i want path folder:
/user/hdfs/2015/10/10/0000
i try my code:
hadoop fs -mkdir /user/hdfs/2015/10/10/0000
but i have error,
No such file or directory.
How i get path folder using hadoop fs -mkdir like /user/hdfs/2015/10/10/0000.?
Thanks.
Maybe run :
hadoop fs -mkdir -p /user/hdfs/2015/10/10/0000
The -p option will create all the directories in the path as needed. See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#mkdir for more information.