After launched a t2.micro instance with Debian and import my AWS keys, i tried to launch a Spark cluster on Frankfurt server with this command :
spark-1.4.0-bin-hadoop2.6/ec2/spark-ec2 -k spark_frankfurt -i spark_frankfurt.pem -s 1 -t t2.micro --region=eu-central-1 --hadoop-major-version=2 launch mycluster
But it replies me the following answer :
Setting up security groups...
Searching for existing cluster mycluster in region eu-central-1...
Could not resolve AMI at: https://raw.github.com/mesos/spark-ec2/branch-1.3/ami-list/eu-central-1/hvm
In fact Frankfurt, eu-central-1, is not in the AMI list on the official EC2 repository : https://github.com/mesos/spark-ec2/tree/branch-1.4/ami-list.
Thus it's normal that it doesn't work for the moment.
Related
How I can setup and securely access a kubernetes cluster on EC2 instance from my laptop? I want it to be a single-node cluster, like running only one instance. Have tried run minikube at EC2 instance, but can't config laptop to connect to it.
So, in the result, I want to run like 10 services/pods in EC2 instance and just debug run on my dev laptop.
Thanks!
You can use KOPS (Kubernetes Ops) to Accomplish this. Its a really handy tool. There's a whole section for configuring a cluster on AWS. I use it on a couple of projects and id really recommend it. Its an easy to understand setup and straight forward.
After the cluster is up you can use kubectl proxy to proxy locally and interact with the cluster. Or use kubectl with config files to set up services and pods.
It does not create a new instance per service or pod it creates a pod on the node(s) that is already existing on the cluster.
In your case you could have a single master and a single node in whatever size that suits your needs.t.2 micro or otherwise
A command to accomplish that would look like:
kops create cluster \
--cloud aws \
--state $KOPS_STATE_STORE \
--node-count $NODE_COUNT \
--zones $ZONES \
--master-zones $MASTER_ZONES \
--node-size $NODE_SIZE \
--master-size $MASTER_SIZE \
-v $V_LOG_LEVEL \
--ssh-public-key $SSH_KEY_PATH \
--name=$CLUSTER_NAME
Where the $NODE_COUNT would be 1 thus having a single Node or EC2 Instance and another instance as the master
To connect to it locally you can also deploy the kubernetes dashboard on your cluster.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
To access Dashboard from your local workstation you must create a secure channel to your Kubernetes cluster. Run the following command:
kubectl proxy
Now you can access the Dashboard at:
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
I'm trying to create an AWS server in the region "Canada(Central)", which, according to the Amazon documentation (http://docs.aws.amazon.com/general/latest/gr/rande.html), is called ca-central-1.
The command I'm running is:
knife ec2 server create -I ami-70299b14 -f t2.nano -S my-key -i ~/.ssh/my-key.pem -ssh-user ubuntu --region ca-central-1 -Z ca-central-1a
And the error I get is:
ERROR: ArgumentError: Unknown region: "ca-central-1"
The ami I'm using is one that I've used to launch a server in the region using the online EC2 Management Console.
I created an IAM user and kay pairs in this region and have given the user all permissions on ec2 resources, and I've also created an inbound rule for ssh in the region. Is there something else I'm missing?
Unfortunately fog-aws only added this region very recently so there isn't yet support for it in knife ec2. Will hopefully be in the next ChefDK release in a few weeks. For now you can just create VMs either from the aws command line tool or the web UI, and then use knife bootstrap on them.
1: your knife.rb as
knife[:aws_access_key_id] = "AWS_ACCESS_KEY"
knife[:aws_secret_access_key] = "AWS_SECRET"
knife[:ssh_key_name] = 'my-key'
knife[:image] = 'ami-21414f36'
knife[:flavor] = 't2.micro'
knife[:region] = 'ca-central-1'
knife[:availability_zone] = 'ca-central-1a'
knife[:ebs_size] = 30
knife[:editor] = 'nano'
2: Generate Key Pair for Canada (Center)
3: Run the knife ec2 server create command.
knife ec2 server create -I ami-70299b14 -f t2.nano -ssh-user ubuntu --region ca-central-1 -Z ca-central-1a
That's worked for me
Try this command:
knife ec2 server create -N node_name -I ami-21414f36 -f t2.micro -x '.\key_ca' -P 'ec2#123' --ssh-key key --region ca-central-1 --availability-zone 'ca-central-1a' --ebs-size 30 --security-group-ids sg-75cbd50d --bootstrap-protocol winrm --winrm-transport ssl --winrm-ssl-verify-mode verify_none
I'm trying to add a EC2 host to my Rancher setup. I have seen this tutorial, however I wanted to use Docker-machine instead.
To that extend, I have done the following:
MAC:~ user1$ docker-machine create -d amazonec2 --amazonec2-vpc-id vpc-84fd6de0 --amazonec2-region eu-west-1 --amazonec2-ami ami-c5f1beb6 Rancher-node-aws-01Running pre-create checks...
Creating machine...
(Rancher-node-aws-01) Launching instance...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded
Note: the AMI ID corresponds to rancheros-v0.7.0-hvm-1.
As you can see, I cannot SSH into the RancherOS (SSH port is open on AWS). Any ideas why this is?
The trick is to use an SSH user called 'rancher'. So the full command will be:
docker-machine create -d amazonec2 --amazonec2-vpc-id vpc-84fd6de0 --amazonec2-region eu-west-1 --amazonec2-ami ami-c5f1beb6 --amazonec2-ssh-user rancher Rancher-node-aws-01
I am new to chef. I have successfully configured chef workstation and server.
So by using this below command I am able to launch only one instance:
knife ec2 server create –image ami-cc5af9a5 -i ram.pem –flavor m1.small -x root –groups chef-client -Z us-east-1a -r “role[webserver]”
By this command I can bootstrap only one node:
knife ec2 server create –image ami-a4827dc9 -i NVirginia.pem –flavor t2.micro -x root –groups RC-Corporation -Z us-east-1a -r "role[learn_chef_httpd]"
I want to launch and Bootstrap 100+ instances, so how can I customize these commands?
knife-ec2 is aimed at relatively small scale interactive usage and is not intended for this. Look at tools like CloudFormation, SparkleFormation, and Terraform.
I've a linux instance in Amazon EC2 instance. I manually installed Spark in this instance and it's working fine. Next I wanted to set up a spark cluster in Amazon.
I ran the following command in ec2 folder:
spark-ec2 -k mykey -i mykey.pem -s 1 -t t2.micro launch mycluster
which successfully launched a master and a worker node. I can ssh into the master node using ssh -i mykey.pem ec2-user#master
I've also exported the keys: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
I've a jar file (which has a simple Spark program) which I tried to submit to the master:
spark-submit --master spark://<master-ip>:7077 --deploy-mode cluster --class com.mycompany.SimpleApp ./spark.jar
But I get the following error:
Error connecting to master (akka.tcp://sparkMaster#<master>:7077).
Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster#<master>:7077
No master is available, exiting.
I'm also updated EC2 security settings for master to accept all inbound traffic:
Type: All traffic, Protocol: All, Port Range: All, Source: 0.0.0.0/0
A common beginner mistake is to assume Spark communication follows a program to master and master to workers hierarchy whereas currently it does not.
When you run spark-submit your program attaches to a driver running locally, which communicates with the master to get an allocation of workers. The driver then communicates with the workers. You can see this kind of communications between driver (not master) and workers in a number of diagrams in this slide presentation on Spark at Stanford
It is important that the computer running spark-submit be able to communicate with all of the workers, and not simply the master. While you can start an additional EC2 instance in a security zone allowing access to the master and workers or alter the security zone to include your home PC, perhaps the easiest way is to simply log on to the master and run spark-submit, pyspark or spark-shell from the master node.