I ran the following command to create a directory on the HDFS side
$HADOOP_HOME/bin/hadoop fs -mkdir 20news-bydate/
but I got a message that the directory already exist. So how can I overwrite the directory?
Thank you
You can remove directory first
$HADOOP_HOME/bin/hadoop fs -rm -R 20news-bydate/
$HADOOP_HOME/bin/hadoop fs -mkdir 20news-bydate/
How about you can try to remove first "hadoop fs -rmr" and then create....
as there are no ways pass parameter to owewrite it....
Related
I am currently working on a project for one of my lectures at the university. The task is to download a book from https://www.gutenberg.org/ and copy it into HDFS. I've tried using put <localSrc> <dest> but it didnt work at all.
This is how my code looks in Terminal at the moment:
[cloudera#quickstart ~]$ put <pg16328.txt> <documents>
bash: syntax error near unexpected token `<'
Any help is appreciated. Thanks in advance.
UPDATE 30.05.2017: I haved used following link https://www.cloudera.com/downloads/quickstart_vms/5-10.html to install Hadoop and did not configure anything at all. Only thing I did was to absolve the tutorial Getting started.
It should just be:
hdfs fs -copyFromLocal pg16328.txt /HDFS/path
I'm not familiar with the put command, but have you tried it without the <>s?
If you have successfully extracted and configured Hadoop, then
you should be in hadoop-home directory ( the location where you extracted and configured hadoop)
Then apply the following command
bin/hadoop dfs -put <local file location> <hdfs file location>
or
bin/hdfs dfs -put <local file location> <hdfs file location>
You can do the same with -copyFromLocal command too. Just replace -put with -copyFromLocal in above commands.
for example :
Lets say you have pg16328.txt in your Desktop directory, then the above command would be
bin/hadoop dfs -put /home/cloudera/Desktop/pg16328.txt /user/hadoop/
where /user/hadoop is a directory in hdfs
If /user/hadoop directory doesn't exists then you can create it by
bin/hadoop dfs -mkdir -f /user/hadoop
You can look at the uploaded file using webUI (namenodeIP:50070) or by using command line as
bin/hadoop dfs -ls /user/hadoop/
I have this:
I had also tried to edit this:
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib
as
export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_INSTALL/lib
in ~/.bashrc
But still I am getting a warning message and I'm not able to solve the problem.
Unable to create the directory
I'm using this code to create the directory for twitter analysis:
hadoop fs -mkdir hdfs://localhost:54310/home/vipal/hadoop_store/hdfs/namenode/twitter_data
Notice how hadoop fs -ls says .: No such file or directory?
First, you must create your home directory, which is /user in HDFS.
hdfs dfs -mkdir -p /user/$(whoami)
(You should also chown and chmod that directory)
Then, you can place files into a twitter_data directory.
hdfs dfs -mkdir twitter_data
hdfs dfs -put <local_files> twitter_data
(I removed hadoop_store/hdfs/namenode because that doesn't make sense)
I am using
hdfs dfs -put myfile mypath
and for some files I get
put: 'myfile': File Exists
does that mean there is a file with the same name or does that mean the same exact file (size, content) is already there?
how can I specify an -overwrite option here?
Thanks!
put: 'myfile': File Exists
Means,the file named "myfile" already exists in hdfs. You cannot have multiple files of the same name in hdfs
You can overwrite it using hadoop fs -put -f /path_to_local /path_to_hdfs
You can overwrite your file in hdfs using -f command.For example
hadoop fs -put -f <localfile> <hdfsDir>
OR
hadoop fs -copyFromLocal -f <localfile> <hdfsDir>
It worked fine for me. However -f command won't work in case of get or copyToLocal command. check this question
A file with the same name exists at the location you're trying to write to.
You can overwrite by specifying the -f flag.
Just updates to this answer, in Hadoop 3.X the command a bit different
hdfs dfs -put -f /local/to/path hdfs://localhost:9870/users/XXX/folder/folder2
I have created single node hadoop cluster in Ubuntu .
I was trying to copy file from hdfs to local fs but when i issued command
hduser#ubuntu:/usr/local/hadoop/bin$ hadoop fs -get /user/hduser/Employee /home/output/
I got a message
get: No such file or directory
How to fix this?
The general format for hadoop shell command get shown below,
hadoop fs -get <HDFS File> <local File Directory>
You have used as, hduser#ubuntu:/usr/local/hadoop/bin$ hadoop fs -get /user/hduser/Employee /home/output/ in here /user/hduser/Employee is an directory not a file.
you should do as,
hduser#ubuntu:/usr/local/hadoop/bin$ hadoop fs -get /user/hduser/Employee[/FILENAME] /home/output/
Else If you want to copy a directory(i.e folder) you can use dfs -copyToLocal,
hduser#ubuntu:/usr/local/hadoop/bin$ hadoop dfs -copyToLocal /user/hduser/Employee /home/output/
You can find Hadoop Shell Commands here.
You need to make sure that /user/hduser is a directory and not a file.I once had this problem and I tried hadoop fs -ls which showed
-rwx r-x -r-x
A directory would be drwx r-x r-x .
If this is the problem you need to remove it using -rmr /user/hduser and make it again with -mkdir.
Other options,try -copyToLocal or try downloading the file from HDFS webportal i.e. namenode_IP:50070
I tried to copy a local file from hard disk directly to Hadoop directory and got the following results. None of them work. Can anyone help me with the right syntax?
$ hadoop fs -copyFromLocal C:\\\temp\\\sample_file.txt /user/user_name/sample_file.txt
**copyFromLocal: unexpected URISyntaxException**
$ hadoop fs -copyFromLocal C://temp//sample_file.txt /user/user_name /sample_file.txt
**copyFromLocal: `//sample_file.txt': No such file or directory**
$ hadoop fs -copyFromLocal C:\temp\sample_file.txt /user/user_name/sample_file.txt
**-copyFromLocal: Can not create a Path from a null string Usage: hadoop fs [generic options] -copyFromLocal [-f] [-p] ...**
$ hadoop fs -copyFromLocal C:/temp/sample_file.txt /user/user_name/sample_file.txt
**copyFromLocal: `/temp/sample_file.txt': No such file or directory**
One problem I've encountered when doing a fs -copyFromLocal /localpath/localfile /hdfspath/hdfsfile is when the target /hdfspath doesn't exist.
I usually create the entire target hdfspath first: fs -mkdir /hdfspath, then issue the -copyFromLocal command.
Beyond that, take a look at this bug:
https://issues.apache.org/jira/browse/HADOOP-10030 which seems to explain your first error.
Perhaps your version of hadoop/hdfs doesn't have this fix.
Also according to this blog (http://www.hindoogle.com/blog/2009/12/cygwin-vs-hadoop-dfs-copy/) your first syntax seems to work for them:
$ bin/hadoop dfs -copyFromLocal C:\\cygwin\\home\\Rajat\\java\\mrh\\input input