I am new in hadoop and I run it by below steps:
ssh-keygen
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh localhost
./start-all.sh
but I get below error:
WARNING: Attempting to start all Apache Hadoop daemons as ... in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [localhost]
localhost: Permission denied (publickey,password).
Starting datanodes
localhost: Permission denied (publickey,password).
Starting secondary namenodes [karbasi]
karbasi: Permission denied (publickey,password).
Starting resourcemanager
Starting nodemanagers
localhost: Permission denied (publickey,password).
Please help me in solving my problem.
Permission denied issue at Hadoop Installation
In the above link, you can find the reason for the issue.
The issue occurs if any mistakes occur in ssh key generation or if the hadoop installation is not extracted by same user and started by the same user.
Related
I am trying to run Hadoop on a GCP. When ever I type in the command
start-dfs.ssh && start-yarn
I get the following.......
localhost: chuckpryorjr#localhost: Permission denied (publickey).
localhost: chuckpryorjr#localhost: Permission denied (publickey).
Starting secondary namenodes [0.0.0.0]
0.0.0.0: chuckpryorjr#0.0.0.0: Permission denied (publickey).
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-ecosystem/hadoop-2.9.2/logs/yarn-chuckpryorj
r-resourcemanager-hadoopmasters.out
localhost: chuckpryorjr#localhost: Permission denied (publickey).
I don't get it. Before they used to prompt me for a password(which I didn't ever recall making), now its just outright denying me. How can make this passwordless? Also the very first time I installed hadoop on GCP and ran it..it worked fine. Sometimes I can get through to complete my work..sometimes I cant.
How can make this passwordless?
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Then update your local authorized keys file for localhost
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
And if you have other servers, you can use ssh-copy-id to place the key into those
In my case when i added my Hadoop user to sudoer it worked fine.
sudo adduser hadoop sudo
While installing Hadoop I got many errors but this one just doesn't go. No matter what I do, it keeps popping again and again. As soon as I am starting Hadoop by the command ./start-all.sh, I get the error:
localhost: rajneeshsahai#localhost: Permission denied
(publickey,password,keyboard-interactive)
Error logs:
Starting namenodes on [localhost]
localhost: rajneeshsahai#localhost: Permission denied (publickey,password,keyboard-interactive).
Starting datanodes
localhost: rajneeshsahai#localhost: Permission denied (publickey,password,keyboard-interactive).
Starting secondary namenodes [MacBook-Air.local]
MacBook-Air.local: rajneeshsahai#macbook-air.local: Permission denied (publickey,password,keyboard-interactive).
2020-05-29 18:42:06,106 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
resourcemanager is running as process 2937. Stop it first.
Starting nodemanagers
localhost: rajneeshsahai#localhost: Permission denied (publickey,password,keyboard-interactive).
I already tried the following things:
ssh-keygen -t rsa
cat ~/.ssh/id-rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
I think repeating this process has created multiple keys in my system.
sudo passwd
Configured /etc/ssh/sshd_config
(i) Changed PermitRootLogin prohibit-password to PermitRootLogin yes
(ii) Changed PasswordAuthentication no to PasswordAuthentication yes
I do have one doubt: Do I have to remove the hash tag (#) from the lines?
I am using macOS Catalina.
You can try the following:
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
On Windows WSL2 Ubuntu container you have to restart the ssh service to make it available for Hadoop. You could try to run the Hadoop in a docker container. See https://github.com/big-data-europe/docker-hadoop.
In the Ubuntu 20.04 container each time before I start Hadoop I restart the ssh service.
sudo service ssh restart
For more details see the following tutorial https://dev.to/samujjwaal/hadoop-installation-on-windows-10-using-wsl-2ck1.
I run namenode with user hduser#master, datanodes are running with user1#slave1, user1#slave2. Setting up SSh keys works fine, I can ssh remotely to my DataNode machines from master.
However, when I try to run the hadoop-daemons.sh for my datanodes it fails because it tries to ssh with the wrong user:
hduser#master:~$ hadoop-daemons.sh start datanode
hduser#slave3's password: hduser#slave1's password: hduser#slave2's password:
slave1: Permission denied (publickey,password).
slave2: Permission denied (publickey,password).
slave3: Permission denied (publickey,password).
I tried to reset the public and private key for my master and copying it to the data nodes
$ ssh-keygen -t rsa -P ""
$ ssh-copy-id -i $HOME/.ssh/id_rsa.pub user1#slave1
But gives me same error.
Does the user on the NameNode need to be the same as for the DataNodes?
Answer: After resetting the VMs and adding same user and installing Hadoop on the Data Nodes with the same user as for Name Node, it worked. So I guess the answer is yes...
I am following this tutorial (currently 2.7.2) https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html on an earlier (2.0.0alpha) version of Hadoop on Ubuntu 14.04.4.
however, running sbin/start-dfs.sh after formatting namenode gives the following error:
17/01/06 11:46:20 WARN util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
Starting namenodes on [localhost]
localhost: mkdir: cannot create directory ‘/username’: Permission denied
localhost: chown: cannot access ‘/username/username’: No such file or directory
localhost: starting namenode, logging to /username/username/hadoop-username-namenode-hostname.out
localhost: /home/username/project/hadoop/hadoop-dist/target/hadoop-2.0.0-alpha/sbin/hadoop-daemon.sh: line 149: /username/username/hadoop-username-namenode-hostname.out: No such file or directory
localhost: head: cannot open ‘/username/username/hadoop-username-namenode-hostname.out’ for reading: No such file or directory
localhost: mkdir: cannot create directory ‘/username’: Permission denied
localhost: chown: cannot access ‘/username/username’: No such file or directory
localhost: starting datanode, logging to /username/username/hadoop-username-datanode-hostname.out
localhost: /home/username/project/hadoop/hadoop-dist/target/hadoop-2.0.0-alpha/sbin/hadoop-daemon.sh: line 149: /username/username/hadoop-username-datanode-hostname.out: No such file or directory
localhost: head: cannot open ‘/username/username/hadoop-username-datanode-hostname.out’ for reading: No such file or directory
Starting secondary namenodes [0.0.0.0]
0.0.0.0: mkdir: cannot create directory ‘/username’: Permission denied
0.0.0.0: chown: cannot access ‘/username/username’: No such file or directory
0.0.0.0: starting secondarynamenode, logging to /username/username/hadoop-username-secondarynamenode-hostname.out
0.0.0.0: /home/username/project/hadoop/hadoop-dist/target/hadoop-2.0.0-alpha/sbin/hadoop-daemon.sh: line 149: /username/username/hadoop-username-secondarynamenode-hostname.out: No such file or directory
0.0.0.0: head: cannot open ‘/username/username/hadoop-username-secondarynamenode-hostname.out’ for reading: No such file or directory
I have seen many other posts about changing the owner, but I didn't create a special user for hadoop, so sshing into localhost still shows the user as my "username". Obviously then the hadoop folder is owned by me, and I actually ran chown just to make sure as well, so why is there still a permission issue here?
I have a hadoop installation on the Amazon Elastic MapReduce , whenever I try to restart the cluster I get the following error:
/stop-all.sh
no jobtracker to stop
The authenticity of host 'localhost (::1)' can't be established. RSA key fingerprint is
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
localhost: Permission denied (publickey).
no namenode to stop
localhost: Permission denied (publickey).
localhost: Permission denied (publickey).
Any idea on how to restart hadoop?
Following hack worked for me.
I have replaced "ssh" command in sbin/slaves.sh & sbin/hadoop-daemon.sh with "ssh -i ~/.ssh/keyname"
I'm using hadoop version 2.4 and this worked for me:
export HADOOP_SSH_OPTS="-i /home/hadoop/mykey.pem"
For the stop-all.sh script to work, you probably need to have the same user in all the machines as the user with which you are executing the stop-all.sh script.
Moreover, it seems you do not have a password less ssh setup from the machine you are executing stop-all.sh to rest of the machines that will spare you from manually entering the password for each machine separately. Passwords might be different for the same user for different machines, please don't forget that.