I am an aws newbie, and I'm trying to run Hadoop on EC2 via Cloudera's AMI. I installed the AMI, downloaded the cloudera-haddop-for-ec2-tools, and now I'm trying to configure
haddop-ec2-env.sh
It is asking for the following:
AWS_ACCOUNT_ID
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
EC2_KEYDIR
PRIVATE_KEY_PATH
when running:
./hadoop-ec2 launch-cluster my-cluster 10
i'm getting
AWS was not able to validate the provided access credentials
Firstly, I have the first 3 attributes for my own account. This is a corporate account, and I received an email with the access key id and secret access key for my email. Is it possible that my account doesn't have the proper permissions to do what is needed here. Exactly why does this script need my credentials? What does it need to do?
Secondly, where is the EC2 key dir? I've uploaded my key.pem file that amazon created for me, and hard coded that into the PRIVATE_KEY_PATH and chmod 400 on the .pem file. Is that the correct key that this script needs?
Any help is appreciated?
Sam
The cloudera ec2 tools heavily rely on the amazon ec2 api tools. Therefore, you must do the following:
1) Download amazon ec2 api tools from http://aws.amazon.com/developertools/351
2) Download cloudera ec2 tools from http://cloudera-packages.s3.amazonaws.com/cloudera-for-hadoop-on-ec2-0.3.0.tar.gz
3) Set the following env variables I am only giving Unix based examples
export EC2_HOME=<path-to-tools-from-step-1>
export $PATH=$PATH:$EC2_HOME/bin
export $PATH=$PATH:<path-to-cloudera-ec2-tools>/bin
export EC2_PRIVATE_KEY=<path-to-private-key.pem>
export EC2_CERT=<path-to-cert.pem>
4) In cloudera-ec2-tools/bin set the following variables
AWS_ACCOUNT_ID=<amazon-acct-id>
AWS_ACCESS_KEY_ID=<amazon-access-key>
AWS_SECRET_ACCESS_KEY=<amazon-secret-key>
EC2_KEYDIR=<dir-where-the-ec2-private-key-and-ec2-cert-are>
KEY_NAME=<name-of-ec2-private-key>
And then run
$ hadoop-ec2 launch-cluster my-hadoop-cluster 10
Which will create a hadoop cluster called "my-hadoop" with 10 nodes on multiple ec2 machines
Related
I have installed salt-master and minion on aws ec2 instances and configured cloud in the below formatenter image description here
and i came up with the error like permission denied(publickey) can any one suggest me a better solution
Owner of the key file must be the same user that start your services (salt-master/minion).
Otherwise OS won't allow you to read such file
The command is:
chown this_maybe_salt_user /path/to/your/key
i have a shell script which need to be installed over 100 Ubuntu instances/servers.What is the best way to install the same script on all instance without logging into each one.
You can use AWS System Manager , according to AWS Documentation :
You can send commands to tens, hundreds, or thousands of instances by
using the targets parameter (the Select Targets by Specifying a
Tag option in the Amazon EC2 console). The targets parameter accepts
a Key,Value combination based on Amazon EC2 tags that you specified
for your instances. When you execute the command, the system locates
and attempts to run the command on all instances that match the
specified tags
You can Target Instance by tag :
aws ssm send-command --document-name name --targets Key=tag:tag_name,Values=tag_value [...]
or
Targeting Instance IDs:
aws ssm send-command --document-name name --targets Key=instanceids,Values=ID1,ID2,ID3 [...]
Read the AWS Documentation for Details.
Thanks
You have several different options when trying to accomplish this task.
Like Kush mentioned, AWS System manager is great, but is a tightly coupled AWS service.
Packer - You could use Packer to create an AMI of the servers, and have the script installed on them, or just executed whatever the script is doing.
Configuration Management.
Ansible/Puppet/Chef. - These tools allow you to manage thousands of servers with only a couple of commands. My preference would be for Ansible, it is light weight, the syntax is only yaml, connects over ssh, and still allows use of placing shell scripts, if need be.
I have to do some quick benchmarking.
I am unable to my vms since neutron is not setup properly.
I can create centos vm.. but i can not log into it.
I tried adding keypair, i tried could init change root password
#cloud-config
chpasswd:
list: |
root:stackops
centos:stackops
expire: False
it does not work. I mean it did not give any errors on log console but i am not abel to login with the credentials i set.
So my question is ..where can i find a openstack centos 7 image whose password is already set ( i guess it would be a custom one)
If Neutron isn't set up correctly, you're not going to be able to do much with your OpenStack environment. However, even with broken networking, you can pass your user-data script to the instance using the --config-drive option, e.g:
nova boot --user-data /path/to/config.yaml --config-drive=true ...
There is a checkbox in the Horizon gui to use this feature as well. This attaches your configuration as a virtual CD-ROM device, which cloud-init will use rather than the network metadata service.
If I put your cloud-config into a file called user-data.yaml, and then run:
nova boot --image centos-7-cloud --user-data user-data.yaml centos
Then I can log in as the centos user using the password stackops.
I am trying to launch a Spark cluster on an EC2 that I created in a development AWS instance. I was able to successfully connect to the EC2 instance using the AWSCLI as ec2-user. I used the existing VPC and AMI to create this EC2. Unzipped the Spark files on EC2 and using the private key tried starting the cluster using the below:
export AWS_SECRET_ACCESS_KEY=xxx
export AWS_ACCESS_KEY_ID=xxx
/home/ec2-user/spark-1.2.0/ec2$ ./spark-ec2 -k test -i /home/ec2-user/identity_files/test.pem launch test-spark-cluster
Got the Error:
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
InvalidKeyPair.NotFoundThe key pair 'test' does not existxxx
I thought, this might have been due to the region issue, so I used the region and zone parameters while launching spark
/home/ec2-user/spark-1.2.0/ec2$ ./spark-ec2 -k test -i /home/ec2-user/identity_files/test.pem -r us-west-2 -z us-west-2a launch test-spark-cluster
However, when I run this, I encounter a different error:
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
VPCIdNotSpecifiedNo default VPC for this userxxx
How can I resolve this issue?
I am no expert on this area, but I would recommend setting more parameters on your script call, something like:
./spark-ec2 -k test
-i /home/ec2-user/identity_files/test.pem
-s 5
--instance-type=m3.medium
--region=eu-west-1
--spark-version=1.2.0
launch myCluster
The -s refers to the instante quantity to be created. Furthermore, you might want to check the following, pay special attention to the last one:
The key pair test exists on your account
The key pair test.pem is present on the EC2-console
The region for both key pair and instances is the same
Searching on the web I have found out that most of the errors related to key pairs not being found are caused by region mismatching.
I am running into problems when I am trying to run a mapreduce job on AWS via the command line. I have to perform a large set of steps (approx 100) that are all chained to each other. Since I am not looking forward to configuring that by hand with the AWS graphic interface, I am trying to get it done with the CLI.
However, even the most easy command does not work:
$ aws emr list-clusters
hostname 'elasticmapreduce.us-west-1.amazonaws.com' doesn't match u'us-west-1.elasticmapreduce.amazonaws.com'
On S3 my configurations seem to work fine, since this command creates the bucket without any problems:
$ aws s3 mb s3://randombigdatabucket
These are my configurations:
$ aws configure list
Name Value Type Location
---- ----- ---- --------
profile <not set> None None
access_key ****************fooo shared-credentials-file
secret_key ****************fooo shared-credentials-file
region us-west-1 config-file ~/.aws/config
I hope somebody can help me out with this one!
try to install AWS CLI v1.6.6 or later