I created some files on my HDFS. I was able to execute the hadoop fs -ls command to view them and it worked fine. Later I was not able to view them. Hence I tried to format the namenode and add the files back in and it says the files already exits. Any idea why this is happening? How can I delete the files if I am not able to view them ?
I was able to view the files on the browser.
You appear to be logged in as the user yeshwanthvenkatesh. When you run hadoop fs -ls with no path, hadoop shows you the contents of your home directory (probably /user/yeshwanthvenkatesh) which appears to be empty. You can try running:
hadoop fs -ls /user/hadoop/scolopax/input/ebird/autogen
but since that directory belongs to the user hadoop, you likely don't have permissions. If not, try this:
sudo -u hadoop hadoop fs -ls /user/hadoop/scolopax/input/ebird/autogen
Related
I use Windows 8 with a cloudera-quickstart-vm-5.4.2-0 virtual box.
I downloaded a text file as words.txt into the Downloads folder.
I changed directory to Downloads and used hadoop fs -copyFromLocal words.txt
I get the no such file or directory error.
Can anyone explain me why this is happening / how to solve this issue?
Here is a screenshot of the terminal:
Someone told me this error occurs when Hadoop is in safe mode, but I have made sure that the safe mode is OFF.
It's happening because hdfs:///user/cloudera doesn't exist.
Running hdfs dfs -ls probably gives you a similar error.
Without specified destination folder, it looks for ., the current HDFS directory for the UNIX account running the command.
You must hdfs dfs -mkdir "/user/$(whoami)" before your current UNIX account can use HDFS, or you can specify an otherwise existing HDFS location to copy to
I have just installed a standalone cluster on my laptop. On running the hdfs dfs -ls command in a terminal, I get to see a list of folders. Upon searching the local file system through the File Explorer window I couldn't locate those files in my file system.
rishirich#localhost:/$ hdfs dfs -ls
Found 1 items
drwxr-xr-x - rishirich supergroup 0 2017-11-09 03:32 user
This folder named 'user' was nowhere to be seen on the local filesystem. Is it that the folder is hidden?
If so, then what terminal command should I use in order to find this folder?
If not, then how do I locate it?
You can't see the hdfs directory structure in graphical view to view it you have to use your terminal only.
hdfs dfs -ls /
and to see local file directory structure in the terminal you should try
ls <path>
cd <path>
cd use to change the directory in terminal.
In your installation of Hadoop, you had set up a core-site.xml file to establish the fs.defaultFS property. If you did not make this file://, it will not be the local filesystem.
If you set it to hdfs://, then the default locations for the namenode and datanode directories are in your local /tmp folder.
Note - those are HDFS blocks, not whole, readable files stored in HDFS.
If you want to list your local filesystem, you're welcome to use hadoop fs -ls file://
I am a beginner in hadoop. I have two doubts
1) how to access files stored in the hdfs? Is it same as using a FileReader in java.io and giving the local path or is it something else?
2) i have created a folder where i have copied the file to be stored in hdfs and the jar file of the mapreduce program. When I run the command in any directory
${HADOOP_HOME}/bin/hadoop dfs -ls
it just shows me all the files in the current dir. So does that mean all the files got added without me explicitly adding it?
Yes, it's pretty much the same. Read this post to read files from HDFS.
You should keep in mind that HDFS is different than your local file system. With hadoop dfs you access the HDFS, not the local file system. So, hadoop dfs -ls /path/in/HDFS shows you the contents of the /path/in/HDFS directory, not the local one. That's why it's the same, no matter where you run it from.
If you want to "upload" / "download" files to/from HDFS you should use the commads:
hadoop dfs -copyFromLocal /local/path /path/in/HDFS and
hadoop dfs -copyToLocal /path/in/HDFS /local/path, respectively.
I'd like to know how to do this in Hadoop: Say I'm logged in as 'dev'. When I issue this command hadoop fs -ls /, I will automatically see the files that belong to me. Like so:
# hadoop fs -ls /
/user/dev/mydata1
/user/dev/mydata2
The reason for which is we have existing shell scripts that don't have to specify what subfolders in Hadoop to get the data. They only have to call /mydata1 and it knows it belongs to /user/dev/
Thanks in advance
use
hadoop fs -ls .
or
hadoop fs -ls
I'm totally new to hadoop and just finished installing which took me 2 days...
I'm now trying with the hadoop dfs command, but i just couldn't understand it, although i've been browsing for days, i couldnt find the answer to what i want to know.
All the examples shows what the result is supposed to be, without explaining the real structure of it, so i will be happy if someone could assist me in understanding hadoop hdfs.
I've created a directory on the HDFS.
bin/hadoop fs -mkdir input
OK, i shall check on it with the ls command.
bin/hadoop fs -ls
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2012-07-30 11:08 input
OK, no problem, everything seems perfect.. BUT where is actually the HDFS data stored?
I thought it would store in the my datanode directory (/home/hadoop/datastore), which was defined in core-site.xml under hadoop.tmp.dir, but it is not there..
Then i tried to view through the WEB-UI and i found that "input" was created under "/user/hadoop/" (/user/hadoop/input).
My questions are
(1) What are the datanode directory (hadoop.tmp.dir) used for, since it doesnt store everything i processed through dfs command?
(2) Everything created with dfs command goes to /user/XXX/ , how to change the value of it?
(3) I cant see anything when i try to access through normal linux command (ls /user/hadoop). Does /user/hadoop exists logically?
I'm sorry if my questions are stupid..
a newbie struggling to understand hadoop better..
Thank you in advance.
Hdfs is not a posix file system and you have to use hadoop api to read and view this file system. That's the reason you have to do hadoop fs -ls as you are using hadoop API to read files here. Data in hdfs are stored in blocks and is stored in all datanodes. Metadata about this file system is stored on Namenode. The data files you see in the directory "/home/hadoop/datastore " are blocks stored on individual datanode.
I think you should explore more about its file system in its tutorial. Yahoo, YDN tutorial on hdfs