Assume that I want to move a csv file from /home/user to /hdfs/data/adhoc/PR/02/RDO0/OUTPUT/
So :
hadoop fs mkdir -m 777 /hdfs/data/adhoc/PR/02/RDO0/OUTPUT/
hadoop fs -moveFromLocal RDO07J420.csv $OUTPUT_FILE_OCRE/MGM7J420-${OPC_DISO8601}.csv
But, I get this problem :
moveFromLocal: Permission denied: user=fs191, access=WRITE,
inode="/hdfs/data/adhoc/PR/02/RDO0/OUTPUT/MGM7J420-.csv.COPYING":RDO0-mdoPR:bfRDO0:drwxr-x---
You local user does not have write rights in hdfs.
Try
sudo -u hdfs hadoop fs -moveFromLocal RDO07J420.csv $OUTPUT_FILE_OCRE/MGM7J420-${OPC_DISO8601}.csv
hdfs is the root user and has write rights, but I suggest managing users and permissions better
http://www.informit.com/articles/article.aspx?p=2755708&seqNum=3
Related
I am trying to create a folder in HDFS from the command line with a user different from hdfs. The directory has permissions 775 for hdfs:hdfs:
$ hadoop fs -ls /
... directories ...
drwxrwxr-x - hdfs hdfs 0 2018-02-21 11:37 /data
... more directories
My user is in the group hdfs:
$ cat /etc/group
hdfs:x:nnnn:myusername
However, when I do hadoop fs -mkdir /data/foo I get:
mkdir: Permission denied: user=myusername, access=WRITE, inode="/data/foo":hdfs:hdfs:drwxrwxr-x
Does hdfs have to be my primary group for this?
I am trying to copy text file into hdfs location.
I'm facing Access issue, so I tried changing permissions.
But I'm unable to change the same facing below error:
chaithu#localhost:~$ hadoop fs -put test.txt /user
put: Permission denied: user=chaithu, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
chaithu#localhost:~$ hadoop fs -chmod 777 /user
chmod: changing permissions of '/user': Permission denied. user=chaithu is not the owner of inode=user
chaithu#localhost:~$ hadoop fs -ls /
Found 2 items
drwxrwxrwt - hdfs supergroup 0 2017-12-20 00:23 /tmp
drwxr-xr-x - hdfs supergroup 0 2017-12-20 10:24 /user
Kindly help me how can I change the rights to full read and write for all users to access the HDFS folder.
First off, you shouldn't be writing into the /user folder directly nor set 777 on it
You're going to need a user directory for your current user to even run a mapreduce job, so you need to sudo su - hdfs first to become an HDFS superuser.
Then run these to create HDFS directories for your user account
hdfs dfs -mkdir -p /user/chaithu
hdfs dfs -chown -R chaithu /user/chaithu
hdfs dfs -chmod -R 770 /user/chaithu
Then exit from the hdfs user, and chaithu can now write to its own HDFS directory.
hadoop fs -put test.txt
That alone will put the file in the current user's folder.
Or, if that's too much work for you write to /tmp instead
A lazy option is to rewrite your user account to the super user.
export HADOOP_USER_NAME=hdfs
hadoop fs -put test.txt /user
And this is why hadoop is not secure or enforce user account access by default (i.e. never do this in production)
And finally, you can always just turn permissions completely off in hdfs-site.xml (again, only useful in development phases)
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
If you observe your hdfs dfs -ls result you see that only HDFS super user have the permissions to that path.
you have two solutions here
One is to change the permissions to chaitu through root user and making chaitu as user or owner, something like this hdfs dfs -chown -R hdfs:chaitu /path then you will be able to get access to that being a owner. Other dirty way is to give hdfs dfs -chmod -R 777 /path from the root, from the security stand point this 777 is not good.
Second one is using ACLS which gives you the temporary access
Please go through this link for more understanding.
More on ACLS
This is so basic and important for you to learn, try the above suggested ones and let me know if those don’t work I can help more based on the error you get.
I am trying to create a folder in hdfs hadoop file system but it is not allowing me to create a folder using the user cloudera nor as root. What should I configure to make it to allow me to hier was my attempt:
[cloudera#quickstart ~]$ sudo hadoop fs -mkdir /solr/test_core
mkdir: Permission denied: user=root, access=WRITE, inode="/solr":solr:supergroup:drwxr-xr-x
[cloudera#quickstart ~]$ su
Password:
[root#quickstart cloudera]# hadoop fs -mkdir /solr/test_core
mkdir: Permission denied: user=root, access=WRITE,inode="/solr":solr:supergroup:drwxr-xr-x
[root#quickstart cloudera]#
Neither cloudera nor root users will have permissions to run any command on /solr
to run any command you need to change into hdfs and then issue the commands like below:
su - hdfs
hadoop fs -mkdir /solr/test_core/
exit
Found the answer:
You should use these weird command.
sudo -u hdfs hdfs dfs -mkdir /solr/test_core/
To switch user to hdfs:
sudo su - hdfs.
Then you can make directory under /solr
To switch back to cloudera user
su - cloudera
and enter the password for cloudera
I have a local VM that has Hortonworks Hadoop and hdfs installed on it. I ssh'ed into the VM from my machine and now I am trying to copy a file from my local filesystem into hdfs through following set of commands:
[root#sandbox ~]# sudo -u hdfs hadoop fs -mkdir /folder1/
[root#sandbox ~]# sudo -u hdfs hadoop fs -copyFromLocal /root/folder1/file1.txt /hdfs_folder1/
When I execute it I get following error as - copyFromLocal:/root/folder1/file1.txt': No such file or directory
I can see that file right in /root/folder1/ directory but with hdfs command its throwing above error. I also tried to cd to /root/folder1/ and then execute the command but same error comes. Why is the file not getting found when it is right there?
By running sudo -u hdfs hadoop fs..., it tries to read the file /root/folder1/file.txt as hdfs.
You can do this.
Run chmod 755 -R /root. It will change permissions on directory and file recursively. But it is not recommended to open up permission on root home directory.
Then you can run the copyFromLocal as sudo -u hdfs to copy file from local file system to hdfs.
Better practice is to create user space for root and copy files directly as root.
sudo -u hdfs hadoop fs -mkdir /user/root
sudo -u hdfs hadoop fs -chown root:root /user/root
hadoop fs -copyFromLocal
I had the same problem running a Hortonworks 4 node cluster. As mentioned, user "hdfs" doesn't have permission to the root directory. The solution is to copy the information from the root folder to something the "hdfs" user can access. In the standard Hortonworks installation this is /home/hdfs
as root run the following...
mkdir /home/hdfs/folder1
cp /root/folder1/file1.txt /home/hdfs/folder1
now change users to hdfs and run from the hdfs USER's accessible directory
su hdfs
cd /home/hdfs/folder1
now you can access files as the hdfs user
hdfs dfs -put file1.txt /hdfs_folder1
sudo -u hdfs hadoop fs -copyFromLocal input.csv input.csv
copyFromLocal: `input.csv': No such file or directory
Can anyone tell me the exact reason why I am getting this kind of error? I gave all permissions to the input.csv file and I even changed the owner to hdfs. I am new to Hadoop and Hbase.
In this case you are trying to read the file as the hdfs user, which may not have permission to view this file. To test, do this:
sudo -u hdfs cat input.csv
If you get permission denied, you either need to change the permissions of this file so the hdfs user can read it (or if it already has read permissions, move the file to a directory that the hdfs user can read), or use a different user that has permission to access the local and the remote directories/files.
You need to make sure that user hdfs has read permission to all the parent directories of input.csv along the path.
syntax: hadoop dfs -copyFromLocal completelocalfilesystempath hdfspath
Example: Let input.csv be in localpath /usr/examples and my hdfs path where it needs to be copied is /usr/input so the command will be
hadoop dfs -copyFromLocal /usr/examples/input.csv /usr/input/
To copy files you can run it like this:
cat input.csv | sudo -u hdfs hadoop fs -put - input.csv