How to find full path for HDFS storage in my system?
e.g. I have /user/cloudera/ folder on hdfs storage, but what is path to the "/user/cloudera"? Are there any specific commands?
HDFS dfs -ls and HDFS dfs -ls -R return only directory list, but not path.
My question is unique, because in here you don't get the HDFS path in the end.
If you are an HDFS admin, you can run:
hdfs fsck /user/cloudera -files -blocks -locations
References:
HDFS Commands Guide: fsck
hdfs file actual block paths
Related
I have created a directory and set a quota in HDFS using the following commands:
hdfs dfs -mkdir /user/hdadmin/directorio_prueba
hdfs dfsadmin -setQuota 4 /user/hdadmin/directorio_prueba
The I have put some files in it:
hdfs dfs -put /opt/bd/ejemplo1.txt /user/hdadmin/directorio_prueba
hdfs dfs -put /opt/bd/ejemplo2.txt /user/hdadmin/directorio_prueba
hdfs dfs -put /opt/bd/ejemplo3.txt /user/hdadmin/directorio_prueba
But when I tried to put the fourth file, the HDFS did not let me saying "The NameSpace quota (directories and files) of directory /user/hdadmin/directorio_prueba is exceeded: quota=4 file count=5". I only have 3 files, but it says there are 4 items (directories and files in the directory). I have also used the following command to gather more information:
hdfs dfs -count -q -h -v /user/hdadmin/directorio_prueba
So there is a hidden directory there. What is this directory? Maybe "." or ".."?
You can directly view hidden files using the command
hdfs dfs -ls /user/hdfs
Please read show hidden hdfs files
Reading the official documentation for the HDFS name quotas (https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html) I found this: 'A quota of one forces a directory to remain empty. (Yes, a directory counts against its own quota!)'.
So there is not ".." nor "." directory. It is the directory itself that counts to the quota. However the "." directory is not explicitly shown. That is why the hdfs dfs -ls /user/hdadmin/directorio_prueba command did not show me any hidden directory like ".".
In docker, I want to copy a file README.md from an existing directory /opt/ibm/labfiles to a new one /input/tmp. I try this
hdfs dfs -put /opt/ibm/labfiles/README.md input/tmp
to no effect, because there seems to be no /input folder in the root. So I try to create it:
hdfs dfs -mkdir /input
mkdir:'/input': File exists
However, when I ls, there is no input file or directory
How can I create a folder and copy the file? Thank you!!
Please try hdfs dfs -ls / if you want to see there is an input folder that exists in HDFS at the root.
You cannot cd into an HDFS directory
It's also worth mentioning that the leading slash is important. In other words,
This will try to put the file in HDFS at /user/<name>/input/tmp
hdfs dfs -put /opt/ibm/labfiles/README.md input/tmp
While this puts the file at the root of HDFS
hdfs dfs -put /opt/ibm/labfiles/README.md /input/tmp
Im trying to copy a file to hdfs using the below command. The filename is googlebooks-eng.... etc....
When I try to list the file within hdfs I don't see the filename being listed.What would be the actual filename?
hadoop-user#hadoop-desk:~/hadoop$ bin/hadoop dfs -put /home/hadoop-user/googlebooks-eng-all-1gram-20120701-0 /user/prema
hadoop-user#hadoop-desk:~/hadoop$ bin/hadoop dfs -ls /user/prema
Found 1 items
-rw-r--r-- 1 hadoop-user supergroup 192403080 2014-11-19 02:43 /user/prema
Almost all hadoop dfs utilies follows unix style. Syntax of hadoop dfs -put is
hadoop dfs -put <source_file> <destination>. Here destination can be a directory or a file. In your case /user directory exists but the directory prema doesn't exist, So when you copy files from local to hdfs prema will be used for the name of the file. googlebooks-eng-all-1gram-20120701-0 and /user/prema are same file.
If you wanted to persist the file name. You need to delete the existing file and create a new directory /user/prema before copying;
bin/hadoop dfs -rm /user/prema;
bin/hadoop dfs -mkdir /user/prema;
bin/hadoop dfs -put /home/hadoop-user/googlebooks-eng-all-1gram-20120701-0 /user/prema
Now you should be able to see the file inside the hdfs directory /user/prema
bin/hadoop dfs -rm /user/prema
Good Day,
I have added a file to HDFS via the command
hadoop fs -put query1.txt .
Now I would like to remove it but I don't have the HDFS location of the file. Is there any way to remove it
You can remove the file using this command.
hadoop fs -rmr query1.txt
By default it will store in /user/(hadoopuser) in your hdfs path.
Use the below command to see the HDFS file location
hadoop fs -ls
hadoop fs -ls /
You will see the hdfs location of your file.
To remove the file use below command
hadoop fs -rmr query1.txt
Can anyone let me know what seems to be wrong here ? hadoop dfs command seems to be OK but any following options are not recognized.
[hadoop-0.20]$bin/hadoop dfs -ls ~/wordcount/input/
ls: Cannot access /home/cloudera/wordcount/input/ : No such file or directory
hadoop fs -ls /some/path/here - will list a HDFS location, not your local linux location
try first this command
hadoop fs -ls /
then investigate step by step other folders.
if you want to copy some files from local directory to users directory on HDFS location, then just use this:
hadoop fs -mkdir /users
hadoop fs -put /some/local/file /users
for more hdfs commands see this: http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html
FS relates to a generic file system which can point to any file systems like local, HDFS, s3 etc But dfs is very specific to HDFS. So when we use FS it can perform operation with from/to local or hadoop distributed file system to destination. But specifying DFS operation relates to HDFS.