I can use hadoop fs -put local hdfs to copy from local to HDFS. Is there a way to do recursive put? Tried -put -r and it doesnt work.
Try this command
hadoop fs -copyFromLocal docs/* /input/
Where /input/ is the path of the HDFS location where you want to store.
hadoop fs -copyFromLocal local/* /Hdfs/
Related
I want to store some .tbl files in hadoop.
I am using this command: hadoop fs -put customer.tbl
But Im getting:
Usage: java FsShell [- put <localsrc> .. <dst>]
If I do hadoop fs -cat cusomer.tbl
It appears that file does note exist.
It seems like you need to provide local-src and HDFS-dst.
Can you try to add destination?
e.g. hadoop fs -put customer.tbl .
please also try execute "ls" on the HDFS:
hadoop fs -ls
please also try execute "ls" on the HDFS using hdfs command, 'hdfs' should be found under hadoop-version-number/bin/:
hdfs dfs -ls
I try to copy file from local to hadoop file system...
I'm using single node cluster
hduser#jothinathan-VirtualBox:~$ hdfs dfs -mkdir -p /usr/hduser
hduser#jothinathan-VirtualBox:~$ hadoop fs -ls
Found 1 items
drwxr-xr-x - hduser supergroup 0 2015-03-10 18:33 sample
hduser#jothinathan-VirtualBox:~$ cd Documents
hduser#jothinathan-VirtualBox:~/Documents$ ls
file hadoopFIle.txt URICat URICat.java
hduser#jothinathan-VirtualBox:~/Documents$ cd
hduser#jothinathan-VirtualBox:~$ hadoop fs -copyFromLocal /Documents/file /usr/local/hadoop
copyFromLocal: `/usr/local/hadoop': No such file or directory
I am getting this error message, please help me with this problem.
first try this command.
hadoop fs -ls /
if it is listing out the local file system files.(not hdfs),then try
hadoop fs -ls hdfs://IP-ADDRESS-of your-machine/
now copy your file to hdfs by
hadoop fs -copyFromLocal /Documents/file hdfs://Ip-addressofyourmachine/above result path
I have installed hadoop and I can't use -copyFromLocal , it responses
[root#hadoop-master ~]# hadoop fs -copyFromLocal file.dat
copyFromLocal: `.': No such file or directory
I have tryed -mkdir dir but it responses
[root#hadoop-master ~]# hadoop fs -mkdir dir
mkdir: `dir': No such file or directory
and -ls
[root#hadoop-master ~]# hadoop fs -ls
ls: `.': No such file or directory
Only it works when I change my user
su hdfs
hadoop fs -mkdir tfm
but it fails when I try again -copyFromLocal
Any solution?
Try the following
hadoop fs -ls /
hadoop fs -lsr /
hadoop fs -mkdir /dir
hadoop fs -copyFromLocal localfilename /newlocation
HDFS is the Owner of the Hadoop file system so you are able to create directories using HDFS user refer bellow mention commands to create directory or to copy files
sudo -u hdfs hadoop fs -mkdir /path/dir
sudo -u hdfs hadoop fs -copyFromLocal localpath hdfspath
or make sure the user (from whom your are running the command) have needed permissions on the directory you are running the command.
I am getting error while copying files from local file system to hdfs,
will you please help me regarding this,
I am using this command :
hadoopd fs -put text.txt file
put and copyFromLocal command helps you to copy data from your local system to HDFS,provided you have the permission to do so.
hadoop fs -put /path/to/textfile /path/to/hdfs
OR
hadoop dfs -put /path/to/textfile /path/to/hdfs
Comming to your error:
You typed the above command as
hadoopd fs
use
hadoop dfs -put /text.txt /file
hadoop dfs -put /path/to/local/file /path/to/hdfs/file
You can use following command
hadoop fs -copyFromLocal text.txt <path_to_hdfs_directory_where_you_want_to_keep_text.txt>
Without knowing the specific error you are getting, it's difficult to answer. The other responders posted the proper syntax. However, it is not uncommon to see permission issues when attempting to copy files to HDFS.
By default the user and group are typically "hdfs" and "supergroup". Your user account likely doesn't belong to "supergroup" and will get permission denied errors. Try running the command as:
sudo -u hdfs hadoop fs -put /path/to/local/file /path/to/hdfs/file
or
sudo -u hdfs hadoop dfs -put /path/to/local/file /path/to/hdfs/file
You can get around having to do this by changing the ownership and permission of the destination directory on HDFS to be more permissive.
"DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hduser/myfile could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock". From this I thinrk your data node is not running/properly. Check that in cluster UI.Then try
hadoop dfs -put /path/file /hdfs/file (hadoop YARN)
hadoop fs -copyFromLocal /path/file /hdfs/file (hadoop1.x)
I have constructed a single-node Hadoop environment on CentOS using the Cloudera CDH repository. When I want to copy a local file to HDFS, I used the command:
sudo -u hdfs hadoop fs -put /root/MyHadoop/file1.txt /
But,the result depressed me:
put: '/root/MyHadoop/file1.txt': No such file or directory
I'm sure this file does exist.
Please help me,Thanks!
As user hdfs, do you have access rights to /root/ (in your local hdd)?. Usually you don't.
You must copy file1.txt to a place where local hdfs user has read rights before trying to copy it to HDFS.
Try:
cp /root/MyHadoop/file1.txt /tmp
chown hdfs:hdfs /tmp/file1.txt
# older versions of Hadoop
sudo -u hdfs hadoop fs -put /tmp/file1.txt /
# newer versions of Hadoop
sudo -u hdfs hdfs dfs -put /tmp/file1.txt /
--- edit:
Take a look at the cleaner roman-nikitchenko's answer bellow.
I had the same situation and here is my solution:
HADOOP_USER_NAME=hdfs hdfs fs -put /root/MyHadoop/file1.txt /
Advantages:
You don't need sudo.
You don't need actually appropriate local user 'hdfs' at all.
You don't need to copy anything or change permissions because of previous points.
try to create a dir in the HDFS by usig: $ hadoop fs -mkdir your_dir
and then put it into it $ hadoop fs -put /root/MyHadoop/file1.txt your_dir
Here is a command for writing df directly to hdfs file system in python script:
df.write.save('path', format='parquet', mode='append')
mode can be append | overwrite
If you want to put in in hdfs using shell use this command:
hdfs dfs -put /local_file_path_location /hadoop_file_path_location
You can then check on localhost:50070 UI for verification