I have a large read-only elasticsearch database running in a kubernetes cluster on Google Container Engine, and am using minikube to run a local dev instance of my app.
Is there a way I can have my app connect to the cloud elasticsearch instance so that I don't have to create a local test database with a subset of the data?
The database contains sensitive information, so can't be visible outside it's own cluster or VPC.
My fall-back is to run kubectl port-forward inside the local pod:
kubectl --cluster=<gke-database-cluster-name> --token='<token from ~/.kube/config>' port-forward elasticsearch-pod 9200
but this seems suboptimal.
I'd use a ExternalName Service like
kind: Service
apiVersion: v1
metadata:
name: elastic-db
namespace: prod
spec:
type: ExternalName
externalName: your.elastic.endpoint.com
According to the docs
An ExternalName service is a special case of service that does not have selectors. It does not define any ports or endpoints. Rather, it serves as a way to return an alias to an external service residing outside the cluster.
If you need to expose the elastic database, there are two ways of exposing applications to outside the cluster:
Creating a Service of type LoadBalancer, that would load balance the traffic for all instances of your elastic database. Once the Load Balancer is created on GKE, just add the load balancer's DNS as the value for the elastic-db ExternalName created above.
Using an Ingress controller. The Ingress controller will have an IP that is reachable from outside the cluster. Use that IP as ExternalName for the elastic-db created above.
Related
I'm using kubernetes from docker for windows and I encountered problem. I use statefulset with following part of config:
spec:
terminationGracePeriodSeconds: 300
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
In classic kubernetes this spec exposes all ports from pod on node ip, so all of them can be accessed through it. I'm trying to develop it on kubernetes from docker for windows, but it seems that I cannot access node by it's ip (like in minikube or microk8s), but docker for windows maps localhost to the cluster. So here is a problem: this config exposes all ports on node ip, which is for example 192.168.65.4, but i cannot access it from windows - I can only access cluster via localhost, but it only exposes protocol related port, for example 443. So when my service runs on port i.e. 10433, there is no access from localhost:10433 but also there is no access in general through node ip. Is there any way to configure it to work as classic kubernetes, where all ports are exposed? I know that single port can be exposed through NodePort, but it's important for me to expose all ports from the pod to imitate real kubernetes behaviour
In general, Docker host networking doesn't work on non-Linux platforms. It's accepted as a valid Docker option, but the "host" network isn't actually the physical system's network. This probably applies to the Kubernetes setup embedded in Docker Desktop as well.
It should be pretty rare to need host networking, and even more unusual in Kubernetes. Host networking disables the normal inter-container communication mechanisms. Kubernetes in particular has a complex network environment and there is usually more than one node; opting out of the network setup like this can make it all but impossible to reach your service, either from inside the cluster or outside.
Instead of host networking, you should use the normal Kubernetes networking setup. Pretty much every Deployment you create will need a matching Service, and if you set that Service to have type: nodePort then it will be accessible from outside the cluster (try both the assigned nodePort: number and the service's cluster-internal port:; it's not clear which port Docker Desktop actually uses).
For some purposes, the easiest approach is to set up a local port-forward to the service
kubectl port-forward deployment/some-deployment 8888:3000
will set up a port-forward from port 8888 on the local system to port 3000 on some pod managed by the named deployment. This forwards to a single pod (if you have multiple replicas, it targets only one of them), it's slower than a direct connection, and the port-forward will fail occasionally, but this is good enough for maintenance tasks like database migrations.
imitate real kubernetes behaviour
In the environment I work on normally, each cluster has dozens to hundreds of nodes. The nodes can't be directly accessed from outside the cluster. It's also reasonably common to configure a PodSecurityPolicy to disallow host networking since it can be viewed as a security concern.
I've deployed grafana to to an AWS EKS cluster and I want to be able to access it from a web browser, if I create a Kubernetes service type of LoadBalancer, based on the very limited AWS networking knowledge I have, I know that this maps to an elastic load balancer, I can get the name of this, go to network and security -> network interfaces and get all the interfaces associated with this, one for each EC2 instance. Presuming its the public ip address associated with each ELB network interface I need to arrange access in order to access my grafana service, and again my AWS networking knowledge is very lacking, what is the fastest and easiest way for me to make the grafana Kubernetes service accessible via my web browser.
The Easiest way to expose any app running on kubernetes is to create a ServiceType as LoadBalancer.
I myself use the same for some of the services to get the things quickly up and running.
To get the loadbalancer name I do
kubectl get svc
which will give me the loadbalancer FQDN. I then map it to a DNS.
The other way which I use is to deploy the nginx-ingress-controller.
https://kubernetes.github.io/ingress-nginx/deploy/#aws
This creates a ServiceType as LoadBalancer.
I then create the Ingress which will be mapped to the ingress controller elb.
https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minimal-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /testpath
pathType: Prefix
backend:
service:
name: test
port:
number: 80
I use this for all my apps running with one loadbalancer mapping it to one elb using the nginx-ingress-controller.
In my specific scenario, the solution was to open up the port for the LoadBalancer service via a port.
I created a custom HTTPS LoadBalancer (details) and I need my Kubernetes Workload to be exposed with this LoadBalancer. For now, if I send a request to this endpoint I get the error 502.
When I choose the Expose option in the Workload Console page, there are only TCP and UDP service types available, and a TCP LoadBalancer is created automatically.
How do I expose a Kubernetes Workload with an existing LoadBalancer? Or maybe I don't even need to do it, and requests don't work because my instances are "unhealthy"? (healthcheck)
You need to create a kubernetes ingress.
First, you need to expose the deployment from k8s, for a https choose 443 port and service type can be either: LoadBalance(external ip) or ClusterIp. (you can also test that by accesing the ip or by port forwarding).
Then you need to create the ingress.
Inside yaml file when choosing the backend, set the port and ServiceName that was configured when exposing the deployment.
For example:
- path: /some-route
backend:
serviceName: your-service-name
servicePort: 443
On gcp, when ingress is created, there will be a load balancer created for that. The backends and instance groups will be automatically build too.
Then if you want to use the already created load balancer you just need to select the backend services from the lb that was created by ingress and add them there.
Also the load balancer will work only if the health checks pass. You need to use the route that will return a 200 HTTPS response for that.
I have created a kubernetes cluster on my local machine with one master and at the moment zero workers, using kubeadm as the bootstrap tool. I am trying to get Prometheus (from the helm packet manager) and Kuberntes matrics together to the Grafana Kubernetes App, but this is not the case. The way I am setting up the monitoring is:
Open grafana-server at port 3000 and install the kuberntes app.
Install stable/prometheus from helm and using this custom YAML file I found in another guide.
Adding Prometheus data source to Grafana with IP from kubernetes Prometheus service (or pods, tried both and both works well) and use TLS Client Auth.
Starting proxy port with kubectl proxy
Filling in all information needed in the Kubernetes Grafana app and then deploy it. No errors.
All kubernetes metric shows, but no Prometheus metric.
If kubernetes proxy connection is stopped, Prometheus metric can be seen. There are no problems connecting to the Prometheus pod or service IP when kubernetes proxy is running. Does someone have any clue what I am doing wrong?
I have set up a simple Kubernetes load balancer service in front of a Node.js container, which should be exposing port 80, but I can't get a response out of it. How can I debug how the load balancer is handling requests to port 80? Are there logs I can inspect?
I have set up a load balancer service and a replication controller as described in the Kubernetes guestbook example.
The service/load balancer spec is similar to this:
{
"kind":"Service",
"apiVersion":"v1",
"metadata":{
"name":"guestbook",
"labels":{
"app":"guestbook"
}
},
"spec":{
"ports": [
{
"port":3000,
"targetPort":"http-server"
}
],
"selector":{
"app":"guestbook"
},
"type": "LoadBalancer"
}
}
As for my hosting platform, I'm using AWS and the OS is CoreOS alpha (976.0.0). Kubectl is at version 1.1.2.
Kubernetes Info
$ ~/.local/bin/kubectl --kubeconfig=/etc/kubernetes/kube.conf get pods
NAME READY STATUS RESTARTS AGE
busybox-sleep 1/1 Running 0 18m
web-s0s5w 1/1 Running 0 12h
$ ~/.local/bin/kubectl --kubeconfig=/etc/kubernetes/kube.conf get services
NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE
kubernetes 10.3.0.1 <none> 443/TCP <none> 1d
web 10.3.0.171
Here is the primary debugging document for Services:
http://kubernetes.io/docs/user-guide/debugging-services/
LoadBalancer creates an external resource. What exactly that resource is depends on your Cloud Provider - some of them don't support it at all (in this case, you might want to try NodePort instead).
Both Google and Amazon support external load balancers.
Overall, when asking these questions it's extremely helpful to know if you are running on Google Container Engine, Google Compute Engine, Amazon Web Services, Digital Ocean, Vagrant, or whatever, because the answer depends on that. Showing all your configs and all your existing Kubnernetes resources (kubectl get pods, kubectl get services) along with your Dockerfiles or which images you are using will also help.
For Google (GKE or GCE), you would verify the load balancer exists:
gcloud compute forwarding-rules list
The external load balancer will map port 80 to an arbitrary Node, but then the Kubernetes proxy will map that to an ephemeral port on the correct node that actually has a Pod with that label, then it will map to the container port. So you have to figure out which step along the way isn't working. Unfortunately all those kube-proxy and iptables jumps are quite difficult to follow, so usually I would first double check all my Pods exist and have labels that match the selector of the Service. I would double check that my container is exposing the right port, I am using the right name for the port, etc. You might want to create some other Pods that just make calls to the Service (using the environment variables or KubeDNS, see the Kubernetes service documentation if you don't know what I'm referring to) and verify it's accessible internally before debugging the load balancer.
Some other good debugging steps:
Verify that your Kubernetes Service exists:
kubectl get services
kubectl get pods
Check your logs of your pod
kubectl logs <pod name>
Check that your service is created internally by printing the environment variable for it
kubectl exec <pod name> -- printenv GUESTBOOK_SERVICE_HOST
try creating a new pod and see if the service can be reached internally through GUESTBOOK_SERVICE_HOST and GUESTBOOK_SERVICE_PORT.
kubectl describe pod <pod name>
will give the instance id of the pod, you can SSH to it and run Docker and verify your container is running, attach to it, etc. If you really want to get into the IP tables debugging, try
sudo iptables-save
The target port of the LoadBalancer needs to be the port INSIDE the container. So in my case I need to set the targetPort to 3000 instead of 80, on the LoadBalancer.
Even though on the pod itself I have already mapped port 80 to 3000.
This is very counter intuitive to me, and not mentioned in all the LoadBalancer docs.