I have a hadoop installation on the Amazon Elastic MapReduce , whenever I try to restart the cluster I get the following error:
/stop-all.sh
no jobtracker to stop
The authenticity of host 'localhost (::1)' can't be established. RSA key fingerprint is
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
localhost: Permission denied (publickey).
no namenode to stop
localhost: Permission denied (publickey).
localhost: Permission denied (publickey).
Any idea on how to restart hadoop?
Following hack worked for me.
I have replaced "ssh" command in sbin/slaves.sh & sbin/hadoop-daemon.sh with "ssh -i ~/.ssh/keyname"
I'm using hadoop version 2.4 and this worked for me:
export HADOOP_SSH_OPTS="-i /home/hadoop/mykey.pem"
For the stop-all.sh script to work, you probably need to have the same user in all the machines as the user with which you are executing the stop-all.sh script.
Moreover, it seems you do not have a password less ssh setup from the machine you are executing stop-all.sh to rest of the machines that will spare you from manually entering the password for each machine separately. Passwords might be different for the same user for different machines, please don't forget that.
Related
I am trying to run Hadoop on a GCP. When ever I type in the command
start-dfs.ssh && start-yarn
I get the following.......
localhost: chuckpryorjr#localhost: Permission denied (publickey).
localhost: chuckpryorjr#localhost: Permission denied (publickey).
Starting secondary namenodes [0.0.0.0]
0.0.0.0: chuckpryorjr#0.0.0.0: Permission denied (publickey).
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-ecosystem/hadoop-2.9.2/logs/yarn-chuckpryorj
r-resourcemanager-hadoopmasters.out
localhost: chuckpryorjr#localhost: Permission denied (publickey).
I don't get it. Before they used to prompt me for a password(which I didn't ever recall making), now its just outright denying me. How can make this passwordless? Also the very first time I installed hadoop on GCP and ran it..it worked fine. Sometimes I can get through to complete my work..sometimes I cant.
How can make this passwordless?
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Then update your local authorized keys file for localhost
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
And if you have other servers, you can use ssh-copy-id to place the key into those
In my case when i added my Hadoop user to sudoer it worked fine.
sudo adduser hadoop sudo
When I try to ssh into localhost, I am prompted for password. See below
"
ssh connection to localhost:
[hadoop#mftrhel74 sbin]$ ssh localhost
hadoop#localhost's password:
Last login: Fri Aug 23 15:44:08 2019 from mah"
---The above statement means, passwordless connection is not setup----
But when I try to start Hadoop nodes as below, it doesn't prompt for password.
And the nodes are not starting, I see below message
I think it should prompt me to enter the password for the user just like as SSH connection is to be established.
[hadoop#mftrhel74 ~]$ start-dfs.sh
Starting namenodes on [mftrhel74]
mftrhel74: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting datanodes
localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting secondary namenodes [mftrhel74]
mftrhel74: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
************I DO NOT WANT A PASSWORDLESS CONNECTION*****
I suspect you are able to log in to one of the nodes with SSH, however probably you have not set up passwordless ssh between the nodes, so the steps you try to execute from the node will fail.
Here is some documentation that should explain that you need to set up passwordless ssh or otherwise install an ambari client (assuming you work on HDP).
https://ambari.apache.org/1.2.2/installing-hadoop-using-ambari/content/ambari-chap1-5-2.html
I run namenode with user hduser#master, datanodes are running with user1#slave1, user1#slave2. Setting up SSh keys works fine, I can ssh remotely to my DataNode machines from master.
However, when I try to run the hadoop-daemons.sh for my datanodes it fails because it tries to ssh with the wrong user:
hduser#master:~$ hadoop-daemons.sh start datanode
hduser#slave3's password: hduser#slave1's password: hduser#slave2's password:
slave1: Permission denied (publickey,password).
slave2: Permission denied (publickey,password).
slave3: Permission denied (publickey,password).
I tried to reset the public and private key for my master and copying it to the data nodes
$ ssh-keygen -t rsa -P ""
$ ssh-copy-id -i $HOME/.ssh/id_rsa.pub user1#slave1
But gives me same error.
Does the user on the NameNode need to be the same as for the DataNodes?
Answer: After resetting the VMs and adding same user and installing Hadoop on the Data Nodes with the same user as for Name Node, it worked. So I guess the answer is yes...
I am trying to install Hadoop on Amazon EC2 Instance CentOS 6.5.I am connected to the instance but want to make the session passwordless SSH. To do this I used the following commands:
ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub node01
I get an error saying : Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
I tried logging in as "root" as well as "ec2-user" but it shows the same error.
Could anyone help on this.
I have created a simple scriptlet to ease this process on EC2 - Ubuntu instances.
You can check it out here.
Just give the machine names and key path, you are done!
https://github.com/hshinde/pwless
I followed this Hadoop tutorial here.
My end goal is to install Hadoop and Spark, and what not on my Mac, but honestly, I'm an amateur at this thing, at best.
So, I got up to step 2, under Execution of Psuedo-Distributed Operation and then the following appears:
sudo start-dfs.sh
Password:
2014-06-10 18:42:01.200 java[6982:1303] Unable to load realm info from SCDynamicStore
14/06/10 18:42:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 17:d5:20:eb:8d:f9:24:2f:c6:46:d7:e2:f5:6a:b8:c1.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Password:
Password:
Password:
localhost: Permission denied (publickey,keyboard-interactive).
Basically, I typed in my password for user lanceguinto, and it is apparently incorrect. Of note is that I did not follow the Setup passphraseless ssh portion because I thought it was unnecessary - I can already ssh, but I'm entirely sure that code does.
During the setup, I also simply used my local user. I did not sudo su / anything.
So, how can I solve this problem? Thanks.
Check if this helps
You need to change the sshd_config file in the remote server (probably in /etc/ssh/sshd_config).
Change
PasswordAuthentication no
to
PasswordAuthentication yes
And then restart the sshd daemon.
Permission denied (publickey,keyboard-interactive)