Store file on hadoop - hadoop

I want to store some .tbl files in hadoop.
I am using this command: hadoop fs -put customer.tbl
But Im getting:
Usage: java FsShell [- put <localsrc> .. <dst>]
If I do hadoop fs -cat cusomer.tbl
It appears that file does note exist.

It seems like you need to provide local-src and HDFS-dst.
Can you try to add destination?
e.g. hadoop fs -put customer.tbl .
please also try execute "ls" on the HDFS:
hadoop fs -ls
please also try execute "ls" on the HDFS using hdfs command, 'hdfs' should be found under hadoop-version-number/bin/:
hdfs dfs -ls

Related

Hadoop - Recursive copy/put from local to HDFS

I can use hadoop fs -put local hdfs to copy from local to HDFS. Is there a way to do recursive put? Tried -put -r and it doesnt work.
Try this command
hadoop fs -copyFromLocal docs/* /input/
Where /input/ is the path of the HDFS location where you want to store.
hadoop fs -copyFromLocal local/* /Hdfs/

How to unzip file in hadoop?

I was trying to unzip a zip file, stored in Hadoop file system, & store it back in hadoop file system. I tried following commands, but none of them worked.
hadoop fs -cat /tmp/test.zip|gzip -d|hadoop fs -put - /tmp/
hadoop fs -cat /tmp/test.zip|gzip -d|hadoop fs -put - /tmp
hadoop fs -cat /tmp/test.zip|gzip -d|hadoop put - /tmp/
hadoop fs -cat /tmp/test.zip|gzip -d|hadoop put - /tmp
I get errors like gzip: stdin has more than one entry--rest ignored, cat: Unable to write to output stream., Error: Could not find or load main class put on terminal, when I run those commands. Any help?
Edit 1: I don't have access to UI. So, only command lines are allowed. Unzip/gzip utils are installed on my hadoop machine. I'm using Hadoop 2.4.0 version.
To unzip a gzipped (or bzipped) file, I use the following
hdfs dfs -cat /data/<data.gz> | gzip -d | hdfs dfs -put - /data/
If the file sits on your local drive, then
zcat <infile> | hdfs dfs -put - /data/
I use most of the times hdfs fuse mounts for this
So you could just do
$ cd /hdfs_mount/somewhere/
$ unzip file_in_hdfs.zip
http://www.cloudera.com/content/www/en-us/documentation/archive/cdh/4-x/4-7-1/CDH4-Installation-Guide/cdh4ig_topic_28.html
Edit 1/30/16: In case if you use hdfs ACLs: In some cases fuse mounts don't adhere to hdfs ACLs, so you'll be able to do file operations that are permitted by basic unix access privileges. See https://issues.apache.org/jira/browse/HDFS-6255, comments at the bottom that I recently asked to reopen.
To stream the data through a pipe to hadoop, you need to use the hdfs command.
cat mydatafile | hdfs dfs -put - /MY/HADOOP/FILE/PATH/FILENAME.EXTENSION
gzip use -c to read data from stdin
hadoop fs -put doesnt support read the data from stdin
I tried a lots of things and would help.I cant find the zip input support of hadoop.So it left me no choice but download the hadoop file to local fs ,unzip it and upload to hdfs again.

Hadoop -copyFromLocal cant find the destination

I try to copy file from local to hadoop file system...
I'm using single node cluster
hduser#jothinathan-VirtualBox:~$ hdfs dfs -mkdir -p /usr/hduser
hduser#jothinathan-VirtualBox:~$ hadoop fs -ls
Found 1 items
drwxr-xr-x - hduser supergroup 0 2015-03-10 18:33 sample
hduser#jothinathan-VirtualBox:~$ cd Documents
hduser#jothinathan-VirtualBox:~/Documents$ ls
file hadoopFIle.txt URICat URICat.java
hduser#jothinathan-VirtualBox:~/Documents$ cd
hduser#jothinathan-VirtualBox:~$ hadoop fs -copyFromLocal /Documents/file /usr/local/hadoop
copyFromLocal: `/usr/local/hadoop': No such file or directory
I am getting this error message, please help me with this problem.
first try this command.
hadoop fs -ls /
if it is listing out the local file system files.(not hdfs),then try
hadoop fs -ls hdfs://IP-ADDRESS-of your-machine/
now copy your file to hdfs by
hadoop fs -copyFromLocal /Documents/file hdfs://Ip-addressofyourmachine/above result path

Hadoop access HDFS

I have installed hadoop and I can't use -copyFromLocal , it responses
[root#hadoop-master ~]# hadoop fs -copyFromLocal file.dat
copyFromLocal: `.': No such file or directory
I have tryed -mkdir dir but it responses
[root#hadoop-master ~]# hadoop fs -mkdir dir
mkdir: `dir': No such file or directory
and -ls
[root#hadoop-master ~]# hadoop fs -ls
ls: `.': No such file or directory
Only it works when I change my user
su hdfs
hadoop fs -mkdir tfm
but it fails when I try again -copyFromLocal
Any solution?
Try the following
hadoop fs -ls /
hadoop fs -lsr /
hadoop fs -mkdir /dir
hadoop fs -copyFromLocal localfilename /newlocation
HDFS is the Owner of the Hadoop file system so you are able to create directories using HDFS user refer bellow mention commands to create directory or to copy files
sudo -u hdfs hadoop fs -mkdir /path/dir
sudo -u hdfs hadoop fs -copyFromLocal localpath hdfspath
or make sure the user (from whom your are running the command) have needed permissions on the directory you are running the command.

I am getting errors while copying files from local to hdfs

I am getting error while copying files from local file system to hdfs,
will you please help me regarding this,
I am using this command :
hadoopd fs -put text.txt file
put and copyFromLocal command helps you to copy data from your local system to HDFS,provided you have the permission to do so.
hadoop fs -put /path/to/textfile /path/to/hdfs
OR
hadoop dfs -put /path/to/textfile /path/to/hdfs
Comming to your error:
You typed the above command as
hadoopd fs
use
hadoop dfs -put /text.txt /file
hadoop dfs -put /path/to/local/file /path/to/hdfs/file
You can use following command
hadoop fs -copyFromLocal text.txt <path_to_hdfs_directory_where_you_want_to_keep_text.txt>
Without knowing the specific error you are getting, it's difficult to answer. The other responders posted the proper syntax. However, it is not uncommon to see permission issues when attempting to copy files to HDFS.
By default the user and group are typically "hdfs" and "supergroup". Your user account likely doesn't belong to "supergroup" and will get permission denied errors. Try running the command as:
sudo -u hdfs hadoop fs -put /path/to/local/file /path/to/hdfs/file
or
sudo -u hdfs hadoop dfs -put /path/to/local/file /path/to/hdfs/file
You can get around having to do this by changing the ownership and permission of the destination directory on HDFS to be more permissive.
"DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hduser/myfile could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock". From this I thinrk your data node is not running/properly. Check that in cluster UI.Then try
hadoop dfs -put /path/file /hdfs/file (hadoop YARN)
hadoop fs -copyFromLocal /path/file /hdfs/file (hadoop1.x)

Resources