I am installing Hadoop in Ubuntu 18.04 following this tutorial. I have tried to solve it reading stackoverflow answers, but I am still having the issue. Here is the output. What can I do?
hadoop#victor-GL552VW:~/hadoop/sbin$ ./start-yarn.sh
Starting resourcemanager
Starting nodemanagers
localhost: hadoop#localhost: Permission denied (publickey,password).
Related
Ubuntu 16.04.1 LTS
Hadoop 3.3.1
when I run start-dfs.sh,
hadoop#ubuntu:~/hadoop/sbin$ start-dfs.sh
Starting namenodes on [ubuntu]
ubuntu: Warning: Permanently added 'ubuntu' (ECDSA) to the list of known hosts.
ubuntu: Permission denied (publickey,password).
Starting datanodes
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
localhost: Permission denied (publickey,password).
Starting secondary namenodes [ubuntu]
ubuntu: Permission denied (publickey,password).
2021-06-25 18:05:42,961 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... >using builtin-java classes where applicable
I also encountered this problem.
I resolved using the following shell commands.
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
I meet some problems in the configuration of Hadoop3.2.1 for YARN learning. And I found that there are two different conditions in user root and user host1 when I run the sbin/start-all.sh. Can you tell me how to solve it and whether it has connection with the SSH? Thank you very much.
In Root:
root#host1-virtual-machine:/home/host1/usr/hadoop-3.2.1# sbin/start-all.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [host1-virtual-machine]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2020-02-12 14:40:27,093 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
In host1 user:
host1#host1-virtual-machine:~/usr/hadoop-3.2.1$ sbin/start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as host1 in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [localhost]
localhost: host1#localhost: Permission denied (publickey,password).
Starting datanodes
localhost: host1#localhost: Permission denied (publickey,password).
Starting secondary namenodes [host1-virtual-machine]
host1-virtual-machine: host1#host1-virtual-machine: Permission denied (publickey,password).
Starting resourcemanager
Starting nodemanagers
localhost: host1#localhost: Permission denied (publickey,password).
You need to set up passwordless connection between nodes. This link might help
http://mynotesonhadoop.blogspot.com/2017/07/configuring-passwordless-ssh-from.html?m=1
when i start hadoop daemon i got following error
[hdp#localhost ~]$ start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as hdp in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [localhost]
localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting datanodes
localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting secondary namenodes [localhost.localdomain]
localhost.localdomain: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting resourcemanager
Starting nodemanagers
localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
I have Hadoop 2.9.0 on Ubuntu at
/usr/local/hadoop
But when I try start-dfs.sh
No error is shown while starting namenode
But when I type jps, only
10900 SecondaryNameNode
11047 Jps
10696 DataNode
Seams to have started, not namenode
Things tried:
=> Removed temp files and formatted namenode hadoop namenode -format
terminal:
blaze#blazian:/tmp$ start-dfs.sh
Starting namenodes on [localhost]
blaze#localhost's password:
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-blaze-namenode-blazian.out
blaze#localhost's password:
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-blaze-datanode-blazian.out
Starting secondary namenodes [0.0.0.0]
blaze#0.0.0.0's password:
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-blaze-secondarynamenode-blazian.out
blaze#blazian:/tmp$ jps
10900 SecondaryNameNode
11047 Jps
10696 DataNode
You don't have a SSH setup with localhost. Please follow these steps and you'll be able to run the namenode.
Go to your system terminal and type:
cd(It will redirect you to ~)
ssh-keygen(hit enter three times and it will create a .ssh directory in ~)
cat id_rsa.pub >> authorized_keys(it will make sure that your localhost is the trusted source and give permission to make
passwordless ssh.)
Then simply run start-all.sh and you're all set.
I am using ubuntu 14.04 LTS Java version 8 and Hadoop 2.5.1 for installation. I followed this guide to install all the components. Sorry for not using michael noll's.
Now the problem that i face is when i do start-dfs.sh i get the following message
oroborus#Saras-Dell-System-XPS-L502X:~$ start-dfs.sh <br>
14/11/12 16:12:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<br>
Starting namenodes on [localhost]<br>
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-oroborus-namenode-Saras-Dell-System-XPS-L502X.out<br>
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-oroborus-datanode-Saras-Dell-System-XPS-L502X.out<br>
Starting secondary namenodes [0.0.0.0]<br>
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-oroborus-secondarynamenode-Saras-Dell-System-XPS-L502X.out<br>
14/11/12 16:12:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<br>
Now after running start-yarn.sh(which seems to work fine) and jps i get the following output
oroborus#Saras-Dell-System-XPS-L502X:~$ jps
9090 NodeManager
5107 JobHistoryServer
8952 ResourceManager
12442 Jps
11981 NameNode
The ideal output should have datanode in it, but it is not there. Googling and SOing a bit i found out the error is looged in the logs so here are the logs for datanode.(Only error part if you need more let me know)
2014-11-08 23:30:32,709 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
2014-11-08 23:30:33,132 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid dfs.datanode.data.dir /usr/local/hadoop_store/hdfs/datanode :
EPERM: Operation not permitted
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:226)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:642)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:472)
at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:126)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:142)
at org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:1866)
at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:1908)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1890)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1782)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1829)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2005)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2029)
2014-11-08 23:30:33,134 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/usr/local/hadoop_store/hdfs/datanode/"
at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:1917)
Now my doubt is how to make it valid.
Help is appreciated.
P.S. I tried a lot of forums, SO post none of them could give me solution to this problem. Hence the question.
First delete all contents from hdfs folder:
Value of <name>hadoop.tmp.dir</name>
rm -rf /usr/local/hadoop_store
Make sure that dir has right owner and permission /usr/local/hadoop_store
hduser#localhost$sudo chown hduser:hadoop -R /usr/local/hadoop_store
hduser#localhost$sudo chmod 777 -R /usr/local/hadoop_store
Format the namenode:
hduser#localhost$hadoop namenode -format
Start all processes again
I also got the same problem,and I fixed them by changing the owner of those working directory.Although you have permissions 777 to these two directory,framework will not be able to use it unless you change the owner to hduser.
$ sudo chown -R hduser:hadoop /usr/local/hadoop/yarn_data/hdfs/namenode
$ sudo chown -R hduser:hadoop /usr/local/hadoop/yarn_data/hdfs/datanode
After this,you again start your cluster and you should see datanode running.
first stop all the entities like namenode, datanode etc. (you will
be having some script or command to do that)
Format tmp directory
Go to /var/cache/hadoop-hdfs/hdfs/dfs/ and delete all the contents in the directory manually
Now format your namenode again
start all the entities then use jps command to confirm that the datanode has been started
Now run whichever application you may like or have.
Hope this helps.
When using cloudera, I had the same problem.
* chown hdfs:hadoop all_hdfs_directories
chown 997:996 all_hdfs_directories (if users don't exist yet)
* chmod 0755 is enough
* delete contents of directories if problem persists