I am getting error while copying files from local file system to hdfs,
will you please help me regarding this,
I am using this command :
hadoopd fs -put text.txt file
put and copyFromLocal command helps you to copy data from your local system to HDFS,provided you have the permission to do so.
hadoop fs -put /path/to/textfile /path/to/hdfs
OR
hadoop dfs -put /path/to/textfile /path/to/hdfs
Comming to your error:
You typed the above command as
hadoopd fs
use
hadoop dfs -put /text.txt /file
hadoop dfs -put /path/to/local/file /path/to/hdfs/file
You can use following command
hadoop fs -copyFromLocal text.txt <path_to_hdfs_directory_where_you_want_to_keep_text.txt>
Without knowing the specific error you are getting, it's difficult to answer. The other responders posted the proper syntax. However, it is not uncommon to see permission issues when attempting to copy files to HDFS.
By default the user and group are typically "hdfs" and "supergroup". Your user account likely doesn't belong to "supergroup" and will get permission denied errors. Try running the command as:
sudo -u hdfs hadoop fs -put /path/to/local/file /path/to/hdfs/file
or
sudo -u hdfs hadoop dfs -put /path/to/local/file /path/to/hdfs/file
You can get around having to do this by changing the ownership and permission of the destination directory on HDFS to be more permissive.
"DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hduser/myfile could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock". From this I thinrk your data node is not running/properly. Check that in cluster UI.Then try
hadoop dfs -put /path/file /hdfs/file (hadoop YARN)
hadoop fs -copyFromLocal /path/file /hdfs/file (hadoop1.x)
Related
I'm new to Hadoop, and am trying to check what data is available in HDFS. However, the dfs command returns a response that indicates the class is deprecated, and that hdfs should be used:
-bash-4.2$ hadoop dfs -ls
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
ls: `.': No such file or directory
When I try the hdfs command, though, I get what appears to be a Java class lookup error:
-bash-4.2$ hadoop hdfs -ls
Error: Could not find or load main class hdfs
Is there something wrong with my Hadoop setup, or have others encountered this catch-22?
It is hadoop fs or hdfs dfs, then -ls
You can run hdfs dfs -ls / to check the root of HDFS, but you will get .: No such file or directory because the output of echo "hdfs:///user/$(whoami)" does not exist yet, and you need to make it using hadoop fs -mkdir -p hdfs:///user/$(whoami).
That command must be repeated for every user account that attempts to access their HDFS user directory
the problem that I am facing is that when I give this command
"hadoop fs -ls" , it throws this message , "ls: `.': No such file or directory
".
For reference Output result to my "jps" command is
18276 SecondaryNameNode
19684 Jps
17942 NameNode
18566 NodeManager
18441 ResourceManager
First you should have a data node running which stores the data otherwise you will not be able to deal with hadoop fs (File System).
Try to stall all services
$start-all-sh
$jps
Ensure that data node is running and nothing obstacles it
Then try
$hadoop fs -ls /
When you don't pass any argument to this hadoop fs -ls command, the default hdfs directory it tries to list is /user/{your_user_name}
The problem in your case could be that this hdfs directory does not exist.
Try running hadoop fs -ls /user/ to see which directories are created for which users.
You can also just create your user's hdfs default directory. Running the below command will fix your error:
hadoop fs -mkdir -p /user/$(whoami)
I want to store some .tbl files in hadoop.
I am using this command: hadoop fs -put customer.tbl
But Im getting:
Usage: java FsShell [- put <localsrc> .. <dst>]
If I do hadoop fs -cat cusomer.tbl
It appears that file does note exist.
It seems like you need to provide local-src and HDFS-dst.
Can you try to add destination?
e.g. hadoop fs -put customer.tbl .
please also try execute "ls" on the HDFS:
hadoop fs -ls
please also try execute "ls" on the HDFS using hdfs command, 'hdfs' should be found under hadoop-version-number/bin/:
hdfs dfs -ls
I can use hadoop fs -put local hdfs to copy from local to HDFS. Is there a way to do recursive put? Tried -put -r and it doesnt work.
Try this command
hadoop fs -copyFromLocal docs/* /input/
Where /input/ is the path of the HDFS location where you want to store.
hadoop fs -copyFromLocal local/* /Hdfs/
I have constructed a single-node Hadoop environment on CentOS using the Cloudera CDH repository. When I want to copy a local file to HDFS, I used the command:
sudo -u hdfs hadoop fs -put /root/MyHadoop/file1.txt /
But,the result depressed me:
put: '/root/MyHadoop/file1.txt': No such file or directory
I'm sure this file does exist.
Please help me,Thanks!
As user hdfs, do you have access rights to /root/ (in your local hdd)?. Usually you don't.
You must copy file1.txt to a place where local hdfs user has read rights before trying to copy it to HDFS.
Try:
cp /root/MyHadoop/file1.txt /tmp
chown hdfs:hdfs /tmp/file1.txt
# older versions of Hadoop
sudo -u hdfs hadoop fs -put /tmp/file1.txt /
# newer versions of Hadoop
sudo -u hdfs hdfs dfs -put /tmp/file1.txt /
--- edit:
Take a look at the cleaner roman-nikitchenko's answer bellow.
I had the same situation and here is my solution:
HADOOP_USER_NAME=hdfs hdfs fs -put /root/MyHadoop/file1.txt /
Advantages:
You don't need sudo.
You don't need actually appropriate local user 'hdfs' at all.
You don't need to copy anything or change permissions because of previous points.
try to create a dir in the HDFS by usig: $ hadoop fs -mkdir your_dir
and then put it into it $ hadoop fs -put /root/MyHadoop/file1.txt your_dir
Here is a command for writing df directly to hdfs file system in python script:
df.write.save('path', format='parquet', mode='append')
mode can be append | overwrite
If you want to put in in hdfs using shell use this command:
hdfs dfs -put /local_file_path_location /hadoop_file_path_location
You can then check on localhost:50070 UI for verification