how do you check the directory in dfs - hadoop

I have created a directory in dfs called /foodir to test, as below:
hadoop dfs -mkdir /foodir
Can someone tell me where is this /foodir saved? How can I check the path? I need to make sure, it is not saved under localfile system /tmp because, everytime server is rebooted /tmp is deleted.
Any ideas how to check the /foodir path in the server file system?

This depends how you setup your core-site.xml and hdfs-site.xml files...
If fs.defaultFS is not set to a file:// path (the default), then your local /tmp is not touched
If your datanode and namenode data directories are not set to your local /tmp (also the default), then nothing is stored there either
You can explicitly make a HDFS path via
hdfs dfs -mkdir hdfs://namenode.fqdn:port/foodir
Otherwise, just run ls /tmp and check if there's files there you made

Related

How do I get the path to my user's home directory in HDFS

I want to see the absolute path to my home directory so that my code can pick up those files and process. But I find myself having to hdfs dfs -ls / and then explore from there until I come across my user's directory.
Effectively I want an hdfs dfs -pwd but of course this does not exist. If I can get a command to list a file on hdfs that shows it's full path, that will also work.
Safe lockdown everyone
$ hdfs getconf -confKey dfs.user.home.dir.prefix

Can't create directory with Hadoop command?

I've been starting NameNode and DataNode, but when I try to use HDFS command to make a directory(in any place), it doesn't work.
Here is my command:
./hdfs dfs -mkdir -p /usr/master/datas
and I also trying to change the format of my path:
./hdfs dfs -mkdir -p "/usr/master/datas"
but I get same result.
I'm just starting to learn big-data. Can anyone tell me how to fix this issue and how debug the issue?
/usr doesn't exist on HDFS. That's a Unix directory.
The user directory in HDFS is /user.
Plus, you need to be an HDFS superuser to create HDFS folders under the root path, or at least folders not owned by the current user.

Hadoop copyFromLocal: '.': No such file or directory

I use Windows 8 with a cloudera-quickstart-vm-5.4.2-0 virtual box.
I downloaded a text file as words.txt into the Downloads folder.
I changed directory to Downloads and used hadoop fs -copyFromLocal words.txt
I get the no such file or directory error.
Can anyone explain me why this is happening / how to solve this issue?
Here is a screenshot of the terminal:
Someone told me this error occurs when Hadoop is in safe mode, but I have made sure that the safe mode is OFF.
It's happening because hdfs:///user/cloudera doesn't exist.
Running hdfs dfs -ls probably gives you a similar error.
Without specified destination folder, it looks for ., the current HDFS directory for the UNIX account running the command.
You must hdfs dfs -mkdir "/user/$(whoami)" before your current UNIX account can use HDFS, or you can specify an otherwise existing HDFS location to copy to

hadoop dfs -ls gives list of folders not present in the local file system

I have just installed a standalone cluster on my laptop. On running the hdfs dfs -ls command in a terminal, I get to see a list of folders. Upon searching the local file system through the File Explorer window I couldn't locate those files in my file system.
rishirich#localhost:/$ hdfs dfs -ls
Found 1 items
drwxr-xr-x - rishirich supergroup 0 2017-11-09 03:32 user
This folder named 'user' was nowhere to be seen on the local filesystem. Is it that the folder is hidden?
If so, then what terminal command should I use in order to find this folder?
If not, then how do I locate it?
You can't see the hdfs directory structure in graphical view to view it you have to use your terminal only.
hdfs dfs -ls /
and to see local file directory structure in the terminal you should try
ls <path>
cd <path>
cd use to change the directory in terminal.
In your installation of Hadoop, you had set up a core-site.xml file to establish the fs.defaultFS property. If you did not make this file://, it will not be the local filesystem.
If you set it to hdfs://, then the default locations for the namenode and datanode directories are in your local /tmp folder.
Note - those are HDFS blocks, not whole, readable files stored in HDFS.
If you want to list your local filesystem, you're welcome to use hadoop fs -ls file://

How files or directories are getting stored in hadoop hdfs

I have created a file in hdfs using below command
hdfs dfs -touchz /hadoop/dir1/file1.txt
I could see the created file by using below command
hdfs dfs -ls /hadoop/dir1/
But, I could not find the location itself by using linux commands (using find or locate). I searched on internet and found following link.
How to access files in Hadoop HDFS? . It says, hdfs is virtual storage. In that case, How its taking partition which one or how much it needs to be used, where the meta data being stored
Is it taking datanode location for virtual storage which I have mentioned in hdfs-site.xml to store all the data?
I looked into datanode location and there are files available. But I could not find out anything related to my file or folder which I have created.
(I am using hadoop 2.6.0)
HDFS file system is a distributed storage system wherein the storage location is virtual and created using the disk space from all the DataNodes. While installing hadoop, you must have specified paths for dfs.namenode.name.dir and dfs.datanode.data.dir. These are the locations at which all the HDFS related files are stored on individual nodes.
While storing the data onto HDFS, it is stored as blocks of a specified size (default 128MB in Hadoop 2.X). When you use hdfs dfs commands you will see the complete files but internally HDFS stores these files as blocks. If you check the above mentioned paths on your local file system, you will see a bunch of files which correcpond to files on your HDFS. But again, you will not see them as actual files as they are split into blocks.
Check below mentioned command's output to get more details on how much space from each DataNode is used to create the virtual HDFS storage.
hdfs dfsadmin -report #Or
sudo -u hdfs hdfs dfsadmin -report
HTH
As we creating a file in local file system i.e on creating a directory in it
for ex:$/mkdir MITHUN94** it is a directory entering into that(LFS) cd MITHUN90
in that **create a new file as **$nano file1.log .
And now create a directory in** hdfs for ex: hdfs dfs -mkdir /mike90 .Here "mike90"
refers to directory name . After that creating a directory send files from LFS to hdfs. By using this command $hdfs dfs -copyFromLocal /home/gopalkrishna/file1.log
/mike90
Here '/home/gopalkrishna/file1.log' refers to pwd (present working directory)
and '/mike90' refers to directory in hdfs. By clickig $hdfs dfs -ls /mike90
the list of files .

Resources