I have created single node hadoop cluster in Ubuntu .
I was trying to copy file from hdfs to local fs but when i issued command
hduser#ubuntu:/usr/local/hadoop/bin$ hadoop fs -get /user/hduser/Employee /home/output/
I got a message
get: No such file or directory
How to fix this?
The general format for hadoop shell command get shown below,
hadoop fs -get <HDFS File> <local File Directory>
You have used as, hduser#ubuntu:/usr/local/hadoop/bin$ hadoop fs -get /user/hduser/Employee /home/output/ in here /user/hduser/Employee is an directory not a file.
you should do as,
hduser#ubuntu:/usr/local/hadoop/bin$ hadoop fs -get /user/hduser/Employee[/FILENAME] /home/output/
Else If you want to copy a directory(i.e folder) you can use dfs -copyToLocal,
hduser#ubuntu:/usr/local/hadoop/bin$ hadoop dfs -copyToLocal /user/hduser/Employee /home/output/
You can find Hadoop Shell Commands here.
You need to make sure that /user/hduser is a directory and not a file.I once had this problem and I tried hadoop fs -ls which showed
-rwx r-x -r-x
A directory would be drwx r-x r-x .
If this is the problem you need to remove it using -rmr /user/hduser and make it again with -mkdir.
Other options,try -copyToLocal or try downloading the file from HDFS webportal i.e. namenode_IP:50070
Related
I am currently working on a project for one of my lectures at the university. The task is to download a book from https://www.gutenberg.org/ and copy it into HDFS. I've tried using put <localSrc> <dest> but it didnt work at all.
This is how my code looks in Terminal at the moment:
[cloudera#quickstart ~]$ put <pg16328.txt> <documents>
bash: syntax error near unexpected token `<'
Any help is appreciated. Thanks in advance.
UPDATE 30.05.2017: I haved used following link https://www.cloudera.com/downloads/quickstart_vms/5-10.html to install Hadoop and did not configure anything at all. Only thing I did was to absolve the tutorial Getting started.
It should just be:
hdfs fs -copyFromLocal pg16328.txt /HDFS/path
I'm not familiar with the put command, but have you tried it without the <>s?
If you have successfully extracted and configured Hadoop, then
you should be in hadoop-home directory ( the location where you extracted and configured hadoop)
Then apply the following command
bin/hadoop dfs -put <local file location> <hdfs file location>
or
bin/hdfs dfs -put <local file location> <hdfs file location>
You can do the same with -copyFromLocal command too. Just replace -put with -copyFromLocal in above commands.
for example :
Lets say you have pg16328.txt in your Desktop directory, then the above command would be
bin/hadoop dfs -put /home/cloudera/Desktop/pg16328.txt /user/hadoop/
where /user/hadoop is a directory in hdfs
If /user/hadoop directory doesn't exists then you can create it by
bin/hadoop dfs -mkdir -f /user/hadoop
You can look at the uploaded file using webUI (namenodeIP:50070) or by using command line as
bin/hadoop dfs -ls /user/hadoop/
i'm beginner in hadoop, when i use
Hadoop fs -ls /
And
Hadoop fs - mkdir /pathname
Every thing is ok, but i want to use my csv file in hadoop, my file is in c drive, i used -put and wget and copyfromlocal commands like these:
Hadoop fs -put c:/ path / myhadoopdir
Hadoop fs copyFromLoacl c:/...
Wget ftp://c:/...
But in two of above it errors in no such file or directory /myfilepathinc:
And for the third
Unable to resolve host address"c"
Thanks for your help
Looking at your command, it seems that there could be couple of reasons for this issue.
Hadoop fs -put c:/ path / myhadoopdir
Hadoop fs copyFromLoacl c:/...
Use hadoop fs -copyFromLocal correctly.
Check your local file permission. You have to give full access to that file.
You have to give your absolute path location both in local and in hdfs.
Hope it will work for you.
salmanbw's answer is exact. To be more clear.
Suppose your file is "c:\testfile.txt", use the command below.
And also make sure you have write permission to your directory in HDFS.
hadoop fs -copyFromLocal c:\testfile.txt /HDFSdir/testfile.txt
Good Day,
I have added a file to HDFS via the command
hadoop fs -put query1.txt .
Now I would like to remove it but I don't have the HDFS location of the file. Is there any way to remove it
You can remove the file using this command.
hadoop fs -rmr query1.txt
By default it will store in /user/(hadoopuser) in your hdfs path.
Use the below command to see the HDFS file location
hadoop fs -ls
hadoop fs -ls /
You will see the hdfs location of your file.
To remove the file use below command
hadoop fs -rmr query1.txt
How to copy file from HDFS to the local file system . There is no physical location of a file under the file , not even directory . how can i moved them to my local for further validations.i am tried through winscp .
bin/hadoop fs -get /hdfs/source/path /localfs/destination/path
bin/hadoop fs -copyToLocal /hdfs/source/path /localfs/destination/path
Point your web browser to HDFS WEBUI(namenode_machine:50070), browse to the file you intend to copy, scroll down the page and click on download the file.
In Hadoop 2.0,
hdfs dfs -copyToLocal <hdfs_input_file_path> <output_path>
where,
hdfs_input_file_path maybe obtained from http://<<name_node_ip>>:50070/explorer.html
output_path is the local path of the file, where the file is to be copied to.
you may also use get in place of copyToLocal.
In order to copy files from HDFS to the local file system the following command could be run:
hadoop dfs -copyToLocal <input> <output>
<input>: the HDFS directory path (e.g /mydata) that you want to copy
<output>: the destination directory path (e.g. ~/Documents)
Update: Hadoop is deprecated in Hadoop 3
use hdfs dfs -copyToLocal <input> <output>
you can accomplish in both these ways.
1.hadoop fs -get <HDFS file path> <Local system directory path>
2.hadoop fs -copyToLocal <HDFS file path> <Local system directory path>
Ex:
My files are located in /sourcedata/mydata.txt
I want to copy file to Local file system in this path /user/ravi/mydata
hadoop fs -get /sourcedata/mydata.txt /user/ravi/mydata/
If your source "file" is split up among multiple files (maybe as the result of map-reduce) that live in the same directory tree, you can copy that to a local file with:
hadoop fs -getmerge /hdfs/source/dir_root/ local/destination
This worked for me on my VM instance of Ubuntu.
hdfs dfs -copyToLocal [hadoop directory] [local directory]
1.- Remember the name you gave to the file and instead of using hdfs dfs -put. Use 'get' instead. See below.
$hdfs dfs -get /output-fileFolderName-In-hdfs
if you are using docker you have to do the following steps:
copy the file from hdfs to namenode (hadoop fs -get output/part-r-00000 /out_text).
"/out_text" will be stored on the namenode.
copy the file from namenode to local disk by (docker cp namenode:/out_text output.txt)
output.txt will be there on your current working directory
bin/hadoop fs -put /localfs/destination/path /hdfs/source/path
Can anyone let me know what seems to be wrong here ? hadoop dfs command seems to be OK but any following options are not recognized.
[hadoop-0.20]$bin/hadoop dfs -ls ~/wordcount/input/
ls: Cannot access /home/cloudera/wordcount/input/ : No such file or directory
hadoop fs -ls /some/path/here - will list a HDFS location, not your local linux location
try first this command
hadoop fs -ls /
then investigate step by step other folders.
if you want to copy some files from local directory to users directory on HDFS location, then just use this:
hadoop fs -mkdir /users
hadoop fs -put /some/local/file /users
for more hdfs commands see this: http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html
FS relates to a generic file system which can point to any file systems like local, HDFS, s3 etc But dfs is very specific to HDFS. So when we use FS it can perform operation with from/to local or hadoop distributed file system to destination. But specifying DFS operation relates to HDFS.