My manager has provided me with an Amazon instance along with a ppk. Able to login; trying to install hadoop; made the needed config changes like, edited the masters and slaves file from localhost to the EC2 instance name, added needed properties to the mapred-site.xml/hdfs-site.xml/core-site.xml files, formatted the namenode into HDFS.
Now, when i run start-dfs.sh script, i get the following errors.
starting namenode, logging to /home/ubuntu/hadoop/libexec/../logs/hadoop-ubuntu-namenode-domU-12-31-39-07-60-A9.out
The authenticity of host 'XXX.amazonaws.com (some IP)' can't be established.
Are you sure you want to continue connecting (yes/no)? yes
XXX.amazonaws.com: Warning: Permanently added 'XXX.amazonaws.com,' (ECDSA) to the list of known hosts.
XXX.amazonaws.com: Permission denied (publickey).
XXX.amazonaws.com: Permission denied (publickey).
as of now, the master and slave nodes would be the same machine.
XXX is the instance name, and some IP is its IP. Masking them for security reasons.
I have absolutely no idea about using an EC2 instance, SSH etc. only need to run a simple MapReduce Program in it.
Kindly suggest.
Hadoop uses SSH to transfer information from master to slaves. It looks like your nodes are trying to talk to each other via SSH but haven't been configured to do so. In order to communicate, the Hadoop master node need passwordless SSH access to the slave nodes. Passwordless is useful so that every time you try to run a job you don't have to enter your password again for each of the slave nodes. That would be quite tedious. It looks like you'll have to set this up between the nodes before you can continue.
I would suggest you check this guide and find the section called "Configuring SSH". It lays out how to accomplish this.
Related
Another episode of the dreaded SSH fail...seems awfully common!
I'm installing Hadoop 3.0.3 on a cluster of 3 servers -- A, B, and C.
A is the Namenode, B and C are workers.
Each server can contact the other two servers using SSH
without a password. Permissions for the .ssh directory are
at 700 on each machine, and permissions for the authorized_keys
are at 600 on each machine. Good for me.
Regardless -- attempting to run start-dfs.sh or start-all.sh
throws the message:
A: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
(A -- for server A)
Any suggestions? Thanks!
Solved.
In addition to setting up SSH between
all pairs of machines -- I had to configure
SSH from each machine to itself.
I'd love to know exactly why this was necessary,
can only guess that multiple processes on the same
machine are trying to communicate...
Just created instance and deployed a cluster using bdutil. SSH works fine as I can ssh into instance using ./bdutil shell.
When I try to access directories such as Hadoop, hdfs etc., it throws an error:
Permission Denied
The terminal appears like this username#hadoop-m $ I know hadoop-m is the name of the instance. What is the username? It says my name but I don't know where it got this from or what the password is.
I am using Ubuntu to ssh into the instance.
Not a hadoop expert, I can answer a bit generally. On GCE when you ssh in gcloud creates a username from you google account name. Hadoop directories such as hadoop or hdfs are probably owned by a different user. Please try using sudo chmod to make give your username permissions to read/write the directories you need.
To elaborate on Jeff's answer, bdutil-deployed clusters set up the user hadoop as the Hadoop admin (this 'admin' user may differ on different Hadoop systems, where Hadoop admin accounts may be split into separate users hdfs, yarn, mapred, etc). Note that bdutil clusters should work without needing deal with Hadoop admin stuff for normal jobs, but if you need to access those Hadoop directories, you can either do:
sudo su hadoop
or
sudo su
to open a shell as hadoop or root, respectively. Or as Jeff mentions, you can sudo chmod to grant broader access to your own username.
I have the existing oeprating Spark cluster that was launched with spark-ec2 script. I'm trying to add new slave by following the instructions:
Stop the cluster
On AWS console "launch more like this" on one of the slaves
Start the cluster
Although the new instance is added to the same security group and I can successfully SSH to it with the same private key, spark-ec2 ... start call can't access this machine for some reason:
Running setup-slave on all cluster nodes to mount filesystems, etc...
[1] 00:59:59 [FAILURE] xxx.compute.amazonaws.com
Exited with error code 255 Stderr: Permission denied (publickey).
, obviously, followed by tons of other errors while trying to deploy Spark stuff on this instance.
The reason is that Spark Master machine doesn't have an rsync access for this new slave, but the 22 port is open...
The issue was that SSH key generated on Spark Master was not transferred to this new slave. Spark-ec2 script with start command omits this step. The solution is to use launch command with --resume options. Then the SSH key is transferred to the new slave and everything goes smooth.
Yet another solution is to add the master's public key (~/.ssh/id_rsa.pub) to the newly added slaves ~/.ssh/authorized_keys. (Got this advice on Spark mailing list)
I am trying to setup hadoop cluster in Google Compute Engine through "Launch click-to-deploy software" feature .I have created 1 master and 1 slave node and tried to start the cluster using start-all.sh script from master node and i got error "permission denied(publickey)" .
I have generated public and private keys in both slave and master nodes .
currently i logged into the master with my username, is it mandatory to login into master as "hadoop" user .If so ,what is the password for that userid .
please let me know how to overcome this problem .
The deployment creates a user hadoop which owns Hadoop-specific SSH keys which were generated dynamically at deployment time; this means since start-all.sh uses SSH under the hood, you must do the following:
sudo su hadoop
/home/hadoop/hadoop-install/bin/start-all.sh
Otherwise, your "normal" username doesn't have SSH keys properly set up so you won't be able to launch the Hadoop daemons, as you saw.
Another thing to note is that the deployment should have already started all the Hadoop daemons automatically, so you shouldn't need to manually run start-all.sh unless you're rebooting the daemons after some manual configuration updates. If the daemons weren't running after the deployment ran, you may have encountered some unexpected error during initialization.
master host ip:192.168.10.10 user:hadoopm
slaver host ip:192.168.10.11 user:slaver1
slaver host ip:192.168.10.12 user:slaver2
hadoop version 1.0
How do I configure ？
because namenode use start-all.sh script ,it will ssh to datanode use the same username(hadoopm), but the datanode（slaver） has no name （hadoopm）， so it will start fail.
and i must use diffrent username between master and slave。
so how do i configure ???
You can just create a user on each node, then all your slaves will share same user.
on debian , add users .