How I can setup and securely access a kubernetes cluster on EC2 instance from my laptop? I want it to be a single-node cluster, like running only one instance. Have tried run minikube at EC2 instance, but can't config laptop to connect to it.
So, in the result, I want to run like 10 services/pods in EC2 instance and just debug run on my dev laptop.
Thanks!
You can use KOPS (Kubernetes Ops) to Accomplish this. Its a really handy tool. There's a whole section for configuring a cluster on AWS. I use it on a couple of projects and id really recommend it. Its an easy to understand setup and straight forward.
After the cluster is up you can use kubectl proxy to proxy locally and interact with the cluster. Or use kubectl with config files to set up services and pods.
It does not create a new instance per service or pod it creates a pod on the node(s) that is already existing on the cluster.
In your case you could have a single master and a single node in whatever size that suits your needs.t.2 micro or otherwise
A command to accomplish that would look like:
kops create cluster \
--cloud aws \
--state $KOPS_STATE_STORE \
--node-count $NODE_COUNT \
--zones $ZONES \
--master-zones $MASTER_ZONES \
--node-size $NODE_SIZE \
--master-size $MASTER_SIZE \
-v $V_LOG_LEVEL \
--ssh-public-key $SSH_KEY_PATH \
--name=$CLUSTER_NAME
Where the $NODE_COUNT would be 1 thus having a single Node or EC2 Instance and another instance as the master
To connect to it locally you can also deploy the kubernetes dashboard on your cluster.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
To access Dashboard from your local workstation you must create a secure channel to your Kubernetes cluster. Run the following command:
kubectl proxy
Now you can access the Dashboard at:
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
Related
I've used the helm chart to deploy Spark to Kubernetes in GCE. According to default configuration in values.yaml the Spark is deployed to the path /opt/spark. I've checked that Spark has deployed successfully by running kubectl --namespace=my-namespace get pods -l "release=spark". There is 1 master and 3 workers running.
However when I've tried to check Spark version by executing spark-submit --version from the Google cloud console it returned -bash: spark-submit: command not found.
I've navigated to the /opt directory and the /spark folder is missing. What should I do to be able to open Spark shell Terminal and to execute Spark commands?
You can verify by checking service
kubectl get services -n <namespace>
you can port-forward particular service and try running locally to check
kubectl port-forward svc/<service name> <external port>:<internal port or spark running port>
Locally you can try running spark terminal it will be connected to spark running on GCE instance.
If you check the helm chart document there is also options for UI you can also do same to access UI via port-forward
Access via SSH inside pod
Kubectl exec -it <spark pod name> -- /bin/bash
here you can directly run spark commands. spark-submit --version
Access UI
Access UI via port-forwarding if you have enable UI in helm chart.
kubectl port-forward svc/<spark service name> <external port>:<internal port or spark running port>
External Load balancer
This particular helm chart also creating External Load balancer you can also get External IP using
Kubectl get svc -n <namespace>
Access Shell
If want to connect via LB IP & port
./bin/spark-shell --conf spark.cassandra.connection.host=<Load balancer IP> spark.cassandra-connection.native.port=<Port>
Creating connection using port-forward
kubectl port-forward svc/<spark service name> <external(local) port>:<internal port or spark running port>
./bin/spark-shell --conf spark.cassandra.connection.host=localhost spark.cassandra-connection.native.port=<local Port>
One way would be login to pod and then run Spark commands
List the pod
kubectl --namespace=my-namespace get pods -l "release=spark"
Now, Login to the pod using following command:
kubectl exec -it <pod-id> /bin/bash
Now, you should be inside the pod and can run spark commands
spark-submit --version
Ref: https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/#getting-a-shell-to-a-container
Hope this helps.
This worked for me.
spark-shell --master k8s://localhost:32217
My spark master is a LoadBalancer exposed at localhost:32217
Is it possible to create a Docker Swarm on 1 AWS EC2? Of course this is for training purposes.
If possible, what are the steps to create a manager and 2 workers on that 1 EC2?
Can I use the public URL of the AWS to use the docker swarm commands?
And the same URL for reaching the docker services / stack items?
It is of course possible to turn Docker Swarm on in a (regular) Docker environment.
As normal, just use the following command:
$ docker swarm init
You will see the default node (as Leader). Now you can move forward using the regular 'docker stack deploy' command.
I have a new laptop and kubernetes cluster running on Google Cloud Platform. How can I access that cluster from local machine to execute kubectl commands, open dashboard etc?
That is not clearly stated in the documentation.
From your local workstation, you need to have the gcloud tool installed and properly configured to connect to the correct GCE account. Then you can run:
gcloud container clusters get-credentials [CLUSTER_NAME]
This will setup kubectl to connect to your kubernetes cluster.
Of course you'll need to install kubectl either using gcloud with:
gcloud components install kubectl
Or using specific instructions for your operating system.
Please check the following link for more details: https://cloud.google.com/kubernetes-engine/docs/quickstart
Once you have kubectl access you can deploy and access the kubernetes dashboard as described here: https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
The first thing you would need to do once you've installed Cloud SDK is ensure it is authenticated to your Google Cloud Platform account/project. To do this you need to run:
gcloud auth login
And then follow the on screen instructions.
Also you will need to install kubectl to access/control aspests of your cluster:
gcloud components install kubectl
You can also install it through native package management by following the instructions here.
Once your gcloud is authenticated to your project you can run this to ensure kubectl is pointing at your cluster and authenticated:
gcloud container clusters get-credentials CLUSTER_NAME --zone ZONE
You'll now be able to issue commands with kubectl that target the cluster you defined in the previous step.
You can access the dashboard following the instructions here.
I've a linux instance in Amazon EC2 instance. I manually installed Spark in this instance and it's working fine. Next I wanted to set up a spark cluster in Amazon.
I ran the following command in ec2 folder:
spark-ec2 -k mykey -i mykey.pem -s 1 -t t2.micro launch mycluster
which successfully launched a master and a worker node. I can ssh into the master node using ssh -i mykey.pem ec2-user#master
I've also exported the keys: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
I've a jar file (which has a simple Spark program) which I tried to submit to the master:
spark-submit --master spark://<master-ip>:7077 --deploy-mode cluster --class com.mycompany.SimpleApp ./spark.jar
But I get the following error:
Error connecting to master (akka.tcp://sparkMaster#<master>:7077).
Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster#<master>:7077
No master is available, exiting.
I'm also updated EC2 security settings for master to accept all inbound traffic:
Type: All traffic, Protocol: All, Port Range: All, Source: 0.0.0.0/0
A common beginner mistake is to assume Spark communication follows a program to master and master to workers hierarchy whereas currently it does not.
When you run spark-submit your program attaches to a driver running locally, which communicates with the master to get an allocation of workers. The driver then communicates with the workers. You can see this kind of communications between driver (not master) and workers in a number of diagrams in this slide presentation on Spark at Stanford
It is important that the computer running spark-submit be able to communicate with all of the workers, and not simply the master. While you can start an additional EC2 instance in a security zone allowing access to the master and workers or alter the security zone to include your home PC, perhaps the easiest way is to simply log on to the master and run spark-submit, pyspark or spark-shell from the master node.
Is there a procedure for adding another host into an existing cluster? I'm using EC2.
I'm thinking it could be done by using CloudFormation again:
aws cloudformation create-stack \
--template-body "$(<deis.template)" \
--stack-name deis-2 \
--parameters "$(<cloudformation.json)"
Which would need a new stack name. That adds the new host.
Or just launch a new instance with the cli?
aws ec2 run-instances --image-id ami-cfe125b8 --count 1 --instance-type m3.medium --key-name deis --security-groups sg-b7edc3c0,sg-c9edc3be
I'm guessing the host should be in both the coreos and deis security groups? And how does fleet know about the new host?
Then, we need to alter the hosts field?
deis clusters:info <cluster>
deis clusters:update <cluster> hosts=x,y,z
Anything else necessary? Is there another, easier way of doing it?
Since all we're dealing with here is CoreOS, it's completely possible to add new nodes to the cluster. The only requirement that you must do is to apply the cloud-config template that you applied to every other node in the cluster to the new instance. See https://coreos.com/docs/running-coreos/cloud-providers/ec2/ for more info.