I'd like to know how to do this in Hadoop: Say I'm logged in as 'dev'. When I issue this command hadoop fs -ls /, I will automatically see the files that belong to me. Like so:
# hadoop fs -ls /
/user/dev/mydata1
/user/dev/mydata2
The reason for which is we have existing shell scripts that don't have to specify what subfolders in Hadoop to get the data. They only have to call /mydata1 and it knows it belongs to /user/dev/
Thanks in advance
use
hadoop fs -ls .
or
hadoop fs -ls
Related
To find the number of files present in a HDFS directory at any point of time using hive, I have created a hive external table. Can any one please help me in extracting the file details of directories present in HDFS as INPUT__FILE__NAME or hdfs dfs -stat is not serving my purpose and I want all the -ls into a csv file.
Working with the output of ls is not recommended, it is not made for this. That being said this is not the normal ls so perhaps there is no alternative.
You can put its output in a file like so:
hadoop fs -ls /path > output.txt
you can also use hdfs to find a table in all databases:
the path of hive databases is:
/apps/hive/warehouse/
so, by using hdfs :
hdfs dfs -find /apps/hive/warehouse/ -name t*
or
hadoop fs -ls /path
I am learning Hadoop and I have never worked on Unix before . So, I am facing a problem here . What I am doing is:
$ hadoop fs -mkdir -p /user/user_name/abcd
now I am gonna put a ready made file with name file.txt in HDFS
$ hadoop fs -put file.txt /user/user_name/abcd
The file gets stored in hdfs since it shows up on running -ls command.
Now , I want to remove this file from HDFS . How should i do this ? What command should i use?
If you run the command hadoop fs -usage you'll get a look at what commands the filesystem supports and with hadoop fs -help you'll get a more in-depth description of them.
For removing files the commands is simply -rm with -rf specified for recursively removing folders. Read the command descriptions and try them out.
i'm beginner in hadoop, when i use
Hadoop fs -ls /
And
Hadoop fs - mkdir /pathname
Every thing is ok, but i want to use my csv file in hadoop, my file is in c drive, i used -put and wget and copyfromlocal commands like these:
Hadoop fs -put c:/ path / myhadoopdir
Hadoop fs copyFromLoacl c:/...
Wget ftp://c:/...
But in two of above it errors in no such file or directory /myfilepathinc:
And for the third
Unable to resolve host address"c"
Thanks for your help
Looking at your command, it seems that there could be couple of reasons for this issue.
Hadoop fs -put c:/ path / myhadoopdir
Hadoop fs copyFromLoacl c:/...
Use hadoop fs -copyFromLocal correctly.
Check your local file permission. You have to give full access to that file.
You have to give your absolute path location both in local and in hdfs.
Hope it will work for you.
salmanbw's answer is exact. To be more clear.
Suppose your file is "c:\testfile.txt", use the command below.
And also make sure you have write permission to your directory in HDFS.
hadoop fs -copyFromLocal c:\testfile.txt /HDFSdir/testfile.txt
I created some files on my HDFS. I was able to execute the hadoop fs -ls command to view them and it worked fine. Later I was not able to view them. Hence I tried to format the namenode and add the files back in and it says the files already exits. Any idea why this is happening? How can I delete the files if I am not able to view them ?
I was able to view the files on the browser.
You appear to be logged in as the user yeshwanthvenkatesh. When you run hadoop fs -ls with no path, hadoop shows you the contents of your home directory (probably /user/yeshwanthvenkatesh) which appears to be empty. You can try running:
hadoop fs -ls /user/hadoop/scolopax/input/ebird/autogen
but since that directory belongs to the user hadoop, you likely don't have permissions. If not, try this:
sudo -u hadoop hadoop fs -ls /user/hadoop/scolopax/input/ebird/autogen
Can anyone let me know what seems to be wrong here ? hadoop dfs command seems to be OK but any following options are not recognized.
[hadoop-0.20]$bin/hadoop dfs -ls ~/wordcount/input/
ls: Cannot access /home/cloudera/wordcount/input/ : No such file or directory
hadoop fs -ls /some/path/here - will list a HDFS location, not your local linux location
try first this command
hadoop fs -ls /
then investigate step by step other folders.
if you want to copy some files from local directory to users directory on HDFS location, then just use this:
hadoop fs -mkdir /users
hadoop fs -put /some/local/file /users
for more hdfs commands see this: http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html
FS relates to a generic file system which can point to any file systems like local, HDFS, s3 etc But dfs is very specific to HDFS. So when we use FS it can perform operation with from/to local or hadoop distributed file system to destination. But specifying DFS operation relates to HDFS.