I'm trying to copy a local file called 'afile' to the HDFS.
So I ran the following command:
'hadoop fs -copyFromLocal /home/neo/afile in'
or 'hadoop fs -put /home/neo/afile in'
However, it says:
'File /home/neo/afile does not exist'
Then I put the file 'afile' into the directory under hadoop. Now the copyFromLocal succeeded. However, the file 'in' is empty, since I run
'hadoop fs - ls', it shows
'-rw-r--r--' 1 neo supergroup 0 2015-04-06 17:45 /user/neo/in
I also tried 'hadoop fs -cat in', nothing returned.
Could someone please help?
Thanks!
Create a new file in local filesystem named test.txt in /home/neo/
Add content to test.txt: echo "sample text for wordcount" > /home/neo/test.txt
Create a new directory in hdfs using: hadoop fs -mkdir /user/neo/in/
Copy file from local directory to HDFS: hadoop fs -copyFromLocal /home/neo/test.txt /user/neo/in/test.txt
Related
I want to store some .tbl files in hadoop.
I am using this command: hadoop fs -put customer.tbl
But Im getting:
Usage: java FsShell [- put <localsrc> .. <dst>]
If I do hadoop fs -cat cusomer.tbl
It appears that file does note exist.
It seems like you need to provide local-src and HDFS-dst.
Can you try to add destination?
e.g. hadoop fs -put customer.tbl .
please also try execute "ls" on the HDFS:
hadoop fs -ls
please also try execute "ls" on the HDFS using hdfs command, 'hdfs' should be found under hadoop-version-number/bin/:
hdfs dfs -ls
I try to copy file from local to hadoop file system...
I'm using single node cluster
hduser#jothinathan-VirtualBox:~$ hdfs dfs -mkdir -p /usr/hduser
hduser#jothinathan-VirtualBox:~$ hadoop fs -ls
Found 1 items
drwxr-xr-x - hduser supergroup 0 2015-03-10 18:33 sample
hduser#jothinathan-VirtualBox:~$ cd Documents
hduser#jothinathan-VirtualBox:~/Documents$ ls
file hadoopFIle.txt URICat URICat.java
hduser#jothinathan-VirtualBox:~/Documents$ cd
hduser#jothinathan-VirtualBox:~$ hadoop fs -copyFromLocal /Documents/file /usr/local/hadoop
copyFromLocal: `/usr/local/hadoop': No such file or directory
I am getting this error message, please help me with this problem.
first try this command.
hadoop fs -ls /
if it is listing out the local file system files.(not hdfs),then try
hadoop fs -ls hdfs://IP-ADDRESS-of your-machine/
now copy your file to hdfs by
hadoop fs -copyFromLocal /Documents/file hdfs://Ip-addressofyourmachine/above result path
I have installed hadoop and I can't use -copyFromLocal , it responses
[root#hadoop-master ~]# hadoop fs -copyFromLocal file.dat
copyFromLocal: `.': No such file or directory
I have tryed -mkdir dir but it responses
[root#hadoop-master ~]# hadoop fs -mkdir dir
mkdir: `dir': No such file or directory
and -ls
[root#hadoop-master ~]# hadoop fs -ls
ls: `.': No such file or directory
Only it works when I change my user
su hdfs
hadoop fs -mkdir tfm
but it fails when I try again -copyFromLocal
Any solution?
Try the following
hadoop fs -ls /
hadoop fs -lsr /
hadoop fs -mkdir /dir
hadoop fs -copyFromLocal localfilename /newlocation
HDFS is the Owner of the Hadoop file system so you are able to create directories using HDFS user refer bellow mention commands to create directory or to copy files
sudo -u hdfs hadoop fs -mkdir /path/dir
sudo -u hdfs hadoop fs -copyFromLocal localpath hdfspath
or make sure the user (from whom your are running the command) have needed permissions on the directory you are running the command.
I am getting error while copying files from local file system to hdfs,
will you please help me regarding this,
I am using this command :
hadoopd fs -put text.txt file
put and copyFromLocal command helps you to copy data from your local system to HDFS,provided you have the permission to do so.
hadoop fs -put /path/to/textfile /path/to/hdfs
OR
hadoop dfs -put /path/to/textfile /path/to/hdfs
Comming to your error:
You typed the above command as
hadoopd fs
use
hadoop dfs -put /text.txt /file
hadoop dfs -put /path/to/local/file /path/to/hdfs/file
You can use following command
hadoop fs -copyFromLocal text.txt <path_to_hdfs_directory_where_you_want_to_keep_text.txt>
Without knowing the specific error you are getting, it's difficult to answer. The other responders posted the proper syntax. However, it is not uncommon to see permission issues when attempting to copy files to HDFS.
By default the user and group are typically "hdfs" and "supergroup". Your user account likely doesn't belong to "supergroup" and will get permission denied errors. Try running the command as:
sudo -u hdfs hadoop fs -put /path/to/local/file /path/to/hdfs/file
or
sudo -u hdfs hadoop dfs -put /path/to/local/file /path/to/hdfs/file
You can get around having to do this by changing the ownership and permission of the destination directory on HDFS to be more permissive.
"DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hduser/myfile could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock". From this I thinrk your data node is not running/properly. Check that in cluster UI.Then try
hadoop dfs -put /path/file /hdfs/file (hadoop YARN)
hadoop fs -copyFromLocal /path/file /hdfs/file (hadoop1.x)
sudo -u hdfs hadoop fs -copyFromLocal input.csv input.csv
copyFromLocal: `input.csv': No such file or directory
Can anyone tell me the exact reason why I am getting this kind of error? I gave all permissions to the input.csv file and I even changed the owner to hdfs. I am new to Hadoop and Hbase.
In this case you are trying to read the file as the hdfs user, which may not have permission to view this file. To test, do this:
sudo -u hdfs cat input.csv
If you get permission denied, you either need to change the permissions of this file so the hdfs user can read it (or if it already has read permissions, move the file to a directory that the hdfs user can read), or use a different user that has permission to access the local and the remote directories/files.
You need to make sure that user hdfs has read permission to all the parent directories of input.csv along the path.
syntax: hadoop dfs -copyFromLocal completelocalfilesystempath hdfspath
Example: Let input.csv be in localpath /usr/examples and my hdfs path where it needs to be copied is /usr/input so the command will be
hadoop dfs -copyFromLocal /usr/examples/input.csv /usr/input/
To copy files you can run it like this:
cat input.csv | sudo -u hdfs hadoop fs -put - input.csv