I am trying my hands on Hadoop 1.0. I am getting Target does not exists while copying one file from local system into HDFS.
My hadoop command and its output is as follows :
shekhar#ubuntu:/host/Shekhar/Softwares/hadoop-1.0.0/bin$ hadoop dfs -put /host/Users/Shekhar/Desktop/Downloads/201112/20111201.txt .
Warning: $HADOOP_HOME is deprecated.
put: Target already exists
After observing the output, we can see that there are two blank spaces between word 'Target' and 'already'. I think there has to be something like /user/${user} between those 2 words. If I give destination path explicitly as /user/shekhar then I get following error :
shekhar#ubuntu:/host/Shekhar/Softwares/hadoop-1.0.0/bin$ hadoop dfs -put /host/Users/Shekhar/Desktop/Downloads/201112/20111201.txt /user/shekhar/data.txt
Warning: $HADOOP_HOME is deprecated.
put: java.io.FileNotFoundException: Parent path is not a directory: /user/shekhar
Output of ls command is as follows :
shekhar#ubuntu:/host/Shekhar/Softwares/hadoop-1.0.0/bin$ hadoop dfs -lsr /
Warning: $HADOOP_HOME is deprecated.
drwxr-xr-x - shekhar supergroup 0 2012-02-21 19:56 /tmp
drwxr-xr-x - shekhar supergroup 0 2012-02-21 19:56 /tmp/hadoop-shekhar
drwxr-xr-x - shekhar supergroup 0 2012-02-21 19:56 /tmp/hadoop-shekhar/mapred
drwx------ - shekhar supergroup 0 2012-02-21 19:56 /tmp/hadoop-shekhar/mapred/system
-rw------- 1 shekhar supergroup 4 2012-02-21 19:56 /tmp/hadoop-shekhar/mapred/system/jobtracker.info
drwxr-xr-x - shekhar supergroup 0 2012-02-21 19:56 /user
-rw-r--r-- 1 shekhar supergroup 6541526 2012-02-21 19:56 /user/shekhar
Please help me in copying file into HDFS. If you need any other information then please let me know.
I am trying this in Ubuntu which is installed using WUBI (Windows Installer for ubuntu).
Thanks in Advance !
The problem in the put command is the trailing .. You need to specify the full path on HDFS where you want the file to go, for ex:
hadoop fs -put /host/Users/Shekhar/Desktop/Downloads/201112/20111201.txt /whatever/20111201.txt
If the directory that you are putting the file in doesn't exist yet, you need to create it first:
hadoop fs -mkdir /whatever
The problem that you are having when you specify the path explicitly is that on your system, /user/shekar is a file, not a directory. You can see that because it has a non-0 size.
-rw-r--r-- 1 shekhar supergroup 6541526 2012-02-21 19:56 /user/shekhar
shekhar#ubuntu:/host/Shekhar/Softwares/hadoop-1.0.0/bin$ hadoop dfs -put /host/Users/Shekhar/Desktop/Downloads/201112/20111201.txt /user/shekhar/data.txt
you must make the file first!
hdfs dfs -mkdir /user/hadoop
hdfs dfs -put /home/bigdata/.password /user/hadoop/
Related
I get this error when I try the command in R > f = hdfs.file('./foo.data', 'r').
When I run # hdfs dfs -ls /user/ I get
Found 1 items
-rw-r--r-- 3 rstudio supergroup 3974305 2019-11-09 19:06 /user/rstudio
And when I try to create the directory I get
# hdfs dfs -mkdir /user/rstudio/
mkdir: `/user/rstudio': Is not a directory
Also when I run in R > hdfs.file.info('./') I get
perms isDir block replication owner group size modtime
1 rw-r--r-- FALSE 134217728 3 rstudio supergroup 3974305 51826-10-07 23:50:34
path
1 ./
Reinforcing that /user/rstudio is not a directory.
It isn't.
-rw-r--r-- would have to start with a d in order to be a directory.
You must delete the file using hdfs dfs -rm, then use the mkdir command
HDFS ls on below two directories returning No such file or directory error.
[mybox]$ hdfs dfs -ls /data/tdc/dv1/corp/base/dpp/raw/load_date=2018-05-01/ | grep Tenant
drwxr-xr-x - tdcdv1r tdcdv1c 0 2018-05-01 18:28 /data/tdc/dv1/corp/base/dpp/raw/load_date=2018-05-01/rtng_ky=Access.NBNOrder.Amend.Info.{Tenant}.Rejected.v2.event
drwxr-xr-x - tdcdv1r tdcdv1c 0 2018-05-01 15:35 /data/tdc/dv1/corp/base/dpp/raw/load_date=2018-05-01/rtng_ky=Access.NBNOrder.Amend.Info.{Tenant}.v2.event
See the error:
[mybox]$ hdfs dfs -ls /data/tdc/dv1/corp/base/dpp/raw/load_date=2018-05-01/rtng_ky=Access.NBNOrder.Amend.Info.{Tenant}.Rejected.v2.event
ls: `/data/tdc/dv1/corp/base/dpp/raw/load_date=2018-05-01/rtng_ky=Access.NBNOrder.Amend.Info.{Tenant}.Rejected.v2.event': No such file or directory
I am not able to understand. Its a directory, it should return the content but its returning error.
You just need to escape the weird characters ({ and }) in the path:
hdfs dfs -ls /data/tdc/dv1/corp/base/dpp/raw/load_date=2018-05-01/rtng_ky=Access.NBNOrder.Amend.Info.\\{Tenant\\}.Rejected.v2.event
EDIT
As in the comments said you can comment the path to avoid escape the weird characters.
This should work fine:
hdfs dfs -ls '/data/tdc/dv1/corp/base/dpp/raw/load_date=2018-05-01/rtng_ky=Access.NBNOrder.Amend.Info.{Tenant}.Rejected.v2.event'
here is my files on hdfs:
hadoop fs -ls /
Found 5 items
-rw-r--r-- 3 hadoop supergroup 25 2016-04-18 11:29 /abc.txt
drwxr-xr-x - hadoop supergroup 0 2016-04-17 11:39 /hbase
drwxr-xr-x - hadoop supergroup 0 2016-04-18 11:49 /tank
drwx------ - hadoop supergroup 0 2016-04-18 11:30 /tmp
-rw-r--r-- 3 hadoop supergroup 66 2016-04-18 11:29 /user.txt
hadoop fs -ls /tank/
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2016-04-18 11:49 /tank/zoo.har
while l am typing
hadoop fs -ls har://hdfs-master/zoo.har/
Blockquote
that got response:
ls: Invalid path for the Har Filesystem. No index file in
har://hdfs-master/zoo.har
please help me out! Thanks!
I guess there are two format to access these files or directories:
First one as following:
hadoop fs -lsr har:///tank/zoo.har/
The other:
hadoop fs -lsr har://hdfs-master/tank/zoo.har/
By the way, are you sure your host is master and the HDFS daemon is listening on default port? Cause second format means har://hdfs-host:port/path/to/somewhere.
I forgot to add my parent path to the har url，it should be har:///parent-path/har-path!
Im trying to copy a file from source to hdfs
Query: Hadoop fs -copyFromLocal (Local path) (source path)
e.g Hadoop fs -copyFromLocal C:\users\Desktop (source path)
as well e.g Hadoop fs -copyFromLocal C:\users\Desktop URI
but geting error
-copyFromLocal: Can not create a Path from a null string
Usage: hadoop fs [generic options] -copyFromLocal [-f] [-p] (localsrc) ... (dst)
try hadoop dfs -copyFromLocal file_to_be_copied hdfs://namenode:/path_to_location
fs command is being depreciated.
Note: you do not have to mention the actual HDFS path. You could also do something like
hadoop dfs -copyFromLocal file_to_be_copied /path_to_location_within_hdfs
You are not specifying the path of file.
Make sure hdfs path that you specified exists.
Finally try escaping the forward slash like:
Hadoop fs -copyFromLocal C:\\users\\Desktop\myfile.txt URI
Or use
Hadoop fs -copyFromLocal C:/users/Desktop/myfile.txt URI
Unfortunately, I am unable to comment below your reply.
The easiest way to to get the path is to go under the UI and click on "Browse the filesystem". Another option would be to hdfs dfs -ls / or hdfs dfs -lsr /
ls will list the dir/files in your root dir.
lsr will recursively read through all the sub directories in your root dir.
HDFS is just like other file systems. The path begins with / as root. The period (.) denotes current directory. I am on Linux platform. Hope the examples below would help.
name#host: ~/oracle/sql$ hdfs dfs -ls /
Found 5 items
drwxr-xr-x - myname supergroup 0 2014-09-05 03:06 /directory
drwxr-xr-x - myname supergroup 0 2014-05-10 03:34 /in
drwxr-xr-x - myname supergroup 0 2014-10-16 22:50 /system
drwxrwx--- - myname supergroup 0 2014-08-18 08:44 /tmp
drwxr-xr-x - myname supergroup 0 2014-08-18 08:41 /user
name#host: ~/oracle/sql$ hdfs dfs -copyFromLocal foo.sql /
name#host: ~/oracle/sql$ hdfs dfs -ls /
Found 6 items
drwxr-xr-x - myname supergroup 0 2014-09-05 03:06 /directory
-rw-r--r-- 3 myname supergroup 90 2015-03-24 05:33 /foo.sql
drwxr-xr-x - myname supergroup 0 2014-05-10 03:34 /in
drwxr-xr-x - myname supergroup 0 2014-10-16 22:50 /system
drwxrwx--- - myname supergroup 0 2014-08-18 08:44 /tmp
drwxr-xr-x - myname supergroup 0 2014-08-18 08:41 /user
name#name: ~/oracle/sql$ hdfs dfs -copyFromLocal foo.sql /user/myname/
name#name: ~/oracle/sql$ hdfs dfs -ls /user/myname/foo.sql
-rw-r--r-- 3 myname supergroup 90 2015-03-24 05:41 /user/myname/foo.sql
name#host: ~/oracle/sql$
I am using fs.copyFromLocalFile(local path, Hdfs dest path) in my program.
I am deleting the destination path on HDFS every time and before copying file from local machine. But after copying files from Local path, and implementing map reduce on it generates two copies of each file, hence the word count doubles.
To be clear, I have "Home/user/desktop/input/" as my local path and HDFS dest path to be "/input".
When I check the HDFS Destination path, i.e folder on which map reduce was applied this is the result
hduser#rallapalli-Lenovo-G580:~$ hdfs dfs -ls /input
14/03/30 08:30:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 4 items
-rw-r--r-- 1 hduser supergroup 62 2014-03-30 08:28 /input/1.txt
-rw-r--r-- 1 hduser supergroup 62 2014-03-30 08:28 /input/1.txt~
-rw-r--r-- 1 hduser supergroup 21 2014-03-30 08:28 /input/2.txt
-rw-r--r-- 1 hduser supergroup 21 2014-03-30 08:28 /input/2.txt~
When I provide Input as single file Home/user/desktop/input/1.txt creates no problem and only single file is copied. But mentioning the directory creates a problem
But manually placing each file in the HDFS Dest through command line creates no problem.
I am not sure If I am missing a simple logic of file system. But would be great if any one could suggest where I am going wrong.
I am using hadoop 2.2.0.
I have tried deleting the local temporary files and made sure the text files are not open. Looking for a way to avoid copiying the temporary files.
Thanks in advance.
The files /input/1.txt~ /input/2.txt~ are temporary files created by the File editor you are using in your machine.You can use Ctrl + H to see all hidden temporary files in your local directory and delete them.