I am currently working on the Hortonworks practice exam and I am getting errors I have not been able to troubleshoot.
During the first step the prompt asks Put the three files from the home/horton/datasets/flight delays directory on the local machine into the user/horton/flight delays directory in hdfs permission denied error. When on the node that hdfs is installed on (root#namenode). I run the simple command:
hadoop fs -copyFromLocal /home/horton/datasets/flightdelays/flight_delays1.csv /user/horton/flightdelays
This returns the error /home/horton/datasets/flightdelays/flight_delays1.csv no such file or directory
When I run the same exact command above from the command line on the local machine instead of running it after being ssh'd onto the namenode (horton#some-ip) I get a permission denied error:
permission denied user=horton access=WRITE inode='/user/horton/flightdelays":hdfs:hdfs:drwxr-xr-x
If anyone has done this practice exam before or knows what this error is and could lend any assistance it would be greatly appreciated. When researching online a lot of people are running into the same issue with the permission denied but im going to assume that on a practice exam that they set up you shouldn't be needing to use sudo for every command you run.
Again any help would be fantastic thanks!!
Try this on CLI
sudo -u hdfs hdfs -copyFromLocal /input/file/path /hdfs/path/
Try this in your command line
hadoop fs -put /localfile.txt /hdfs path
The issue is that the folder you're trying to write to has ownership and permssions of hdfs:hdfs:drwxr-xr-x meaning it is owned by the 'hdfs' user and group. Only the hdfs user has write permissions to that folder everyone else has read and execute permissions only. Thus writing to that folder as the 'horton' user will not work.
You need to run the command as hdfs like so:
sudo -u hdfs hadoop fs -copyFromLocal /home/horton/datasets/flightdelays/flight_delays1.csv /user/horton/flightdelays
Related
While trying to use the put command to add patternsToSkip file to hdfs, I get an error saying: Permission denied: user=root, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-x
In the image below, you can see the sequence of commands written along with the error:
I tried to user access as biadmin, root, and even hdfs but with no luck! (details in the image)
please help me fix this error. Thanks folks.
Reason, it is giving permission issue is because you are trying to put the file inside /user directory in hdfs since you are using 2 dots in put statement. You need to login as supergroup to access or copy file inside that particular directory.
What i would suggest is try running below commands to copy file to hdfs.
Target with one dot
hadoop fs -put patternsToSkip .
OR
Giving complete target directory path
hadoop fs -put patternsToSkip /user/<instance_name>/output
I'm following Hortonworks Hadoop tutorial: https://hortonworks.com/tutorial/manage-files-on-hdfs-via-cli-ambari-files-view/section/1/#create-a-directory-in-hdfs-upload-a-file-and-list-contents.
And I was able to create a directory in HDFS, but having a problem uploading a file from my local system to the directory.
I gave root access to read and write to the user directory with the command hdfs dfs -chmod 777 /user. And then I gave permissions to root with the following command hdfs dfs -chown root:hdfs /user/hadoop.
But for some reason when I'm trying to execute the command hdfs dfs -put sf-salaries-2011-2013.csv /user/hadoop/sf-salaries-2011-2013/sf-salaries-2011-2013.csv - I'm getting the error:
put: `/user/hadoop/sf-salaries-2011-2013/sf-salaries-2011-2013.csv': No such file or directory: `hdfs://sandbox-hdp.hortonworks.com:8020 /user/hadoop/sf-salaries-2011-2013/sf-salaries-2011-2013.csv'
Could it be the problem with port 8020? I'm following the tutorial step by step and can not figure out what I may be missing here.
Here's my terminal view:
I can see the directory created in Ambari as well (created it twice):
The first one probably has a typo (upper-case H). But as I see the directory is empty but it exists.
You are trying to put in HDFS directory which is not there in HDFS. Its not with the permissions.
hdfs dfs -ls /user/hadoop/sf-salaries-2011-2013
The above directory is not there in HDFS, you are trying to put in the sf-salaries-2011-2013 directory but the directory in HDFS is like sf-salaries-2011-1013. check out the highlights in the below picture, you can find out what you are doing wrong here.
I am trying to install hadoop (2.7) in cluster (two machines hmaster and hslave1). I installed hadoop in the folder /opt/hadoop/
I followed this tutorial but Iwhen I run the command start-dfs.sh, I got the following error about:
hmaster: starting namenode, logging to /opt/hadoop/logs/hadoop-hadoop-namenode-hmaster.out
hmaster: starting datanode, logging to /opt/hadoop/logs/hadoop-hadoop-datanode-hmaster.out
hslave1: mkdir: impossible to create the folder « /opt/hadoop\r »: Permission denied
hslave1: chown: impossible to reach « /opt/hadoop\r/logs »: no file or folder of this type
/logs/hadoop-hadoop-datanode-localhost.localdomain.out
I used the command chmod 777 for the folder hadoop in hslave but I still have this error.
Insted of using /opt/ use /usr/local/ if you get that permission issue again give the root permissions using chmod. I already configured hadoop 2.7 in 5 machines. Or else use "Sudo chown user:user /your log files directory".
Seems you have already gave master password less access to login slave.
Make sure you are logged in with username available on both servers.
(hadoop in your case, as tutorial you are following uses 'hadoop' user.)
you can edit the '/etc/sudoer' file using 'sudo' or directly type 'visudo' in the terminal and add the following permission for newly created user 'hadoop' :-
hadoop ALL = NOPASSWD: ALL
might it will resolved your issues.
I have tried the below query.
hadoop jar /home/cloudera/workspace/para.jar word.Paras examples/wordcount /home/cloudera/Desktop/words/output
map reduce is started after that its showing below error. can anyone please help on this issue.
15/11/04 10:33:57 INFO mapred.JobClient: Task Id : attempt_201511040935_0008_m_000002_0, Status : FAILED
org.apache.hadoop.security.AccessControlException: Permission denied: user=cloudera, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
Do I need to change anything config file or in cloudera manager.
The exception suggests that you are trying to write to the HDFS root directory "/" which you (user:cloudera) does not have permission to do.
Without knowing what your specific jar does:
I guess that the last argument ("/home/cloudera/Desktop/words/output") is where you wish to place the output.
I guess this is supposed to be within HDFS where /home does not exist.
Try to change this to somewhere where you can write, possibly "/user/cloudera/words/output"
There are set of default directories to be created before you start using the hadoop cluster,
do, it should show you the directories
$ hadoop fs -ls /
sample user, if you want to run as cloudera you need on hdfs
/user/cloudera -- the user running the program
/user/hadoop -- your hadoop file system user
/user/mapred -- your mapred user
/tmp -- temporary which needs to have permission hdfs chmod 1777
HTH.
The last argument that you are passing should be the output path of HDFS not the default file system.
As you are running with cloudera user, you can point to the /user/cloudera/words/output. But first you need to check whether you have cloudera in your HDFS and you have write permission by issuing the following
hadoop fs -ls /user/
Once you have it change your command to following:
hadoop jar /home/cloudera/workspace/para.jar word.Paras examples/wordcount <path_where_you_have_write_permission_in_HDFS>
I am trying to invoke Sqoop through a map reduce program using
Sqoop.runTool(arguments,_conf);
When executing, I receive the following error
Exception in thread "main" java.lang.RuntimeException: Could not create temporary directory: /tmp/sqoop-hdfs/compile/a609226c19d65f561dd7035c00d318f6; check for a directory permissions issue on /tmp.
I have set the permissions on /tmp and it's subdirectories in HDFS to 777
I can invoke the same command fine through command line using sudo -u hdfs sqoop ...
This is Cloudera's hadoop distirbution and I am running the job as hdfs user.
This probably isn't the /tmp directory in HDFS, but rather then /tmp directory on the local file system - whats the permissions on that directory (and would also explain why it works when you 'sudo' the command)
Just clean /tmp/sqoop-hdfs/compile floder it works