Is there a standard way to monitor the traffic of K8 Ingress. We were trying to extract metrics like
Requests per second
HTTP errors
Response time
etc...
Specific Environment
AWS EKS
Nginx ingress
AWS Elastic search service [Store and search metrics]
Kibana and Easy Alert [Dashboard and alerting]
Solutions tried
Service mesh like Istio ( Do not want the heaviness of Istio. Service to service traffic is very less. )
Custom nginx solution (https://sysdig.com/blog/monitor-nginx-kubernetes/)
Looking for something generic to K8 . Any hints ?
The ingress-nginx project has a monitoring guide that describes how to send basic statistics to Prometheus, which can then be viewed in Grafana. If you cluster already has these tools installed, then it is just a matter of configuring prometheus to scrape nginx ingress controller pods.
First of all, there is not "standard way" to monitor a cluster. People or companies have different needs, so you should implement your best solution.
If you are managing all ingress traffic from Nginx ingress controller, then you can implement Nginx based solutions. Said that, linkerd is also a great tool to monitor and manage your network stack, especially if it's simple. There is a dashboard and you can check all your requirements. Linkerd components are also not heavy as istio.
Related
I am new to Kubernetes.
If there is any service deployed using EKS having 4 replicas A,B,C,D.
Usually loadbalancer directs requests to these replicas
But if I want that my request should go to replica A only or B only...
How can we achieve it.
Request to share some links or steps for guidance
What you could use are the Headless Services:
Sometimes you don't need load-balancing and a single Service IP. In
this case, you can create what are termed "headless" Services, by
explicitly specifying "None" for the cluster IP (.spec.clusterIP).
You can use a headless Service to interface with other service
discovery mechanisms, without being tied to Kubernetes'
implementation.
For headless Services, a cluster IP is not allocated, kube-proxy
does not handle these Services, and there is no load balancing or
proxying done by the platform for them. How DNS is automatically
configured depends on whether the Service has selectors defined:
With selectors
For headless Services that define selectors, the endpoints controller
creates Endpoints records in the API, and modifies the DNS
configuration to return records (addresses) that point directly to the
Pods backing the Service.
Without selectors
For headless Services that do not define selectors, the endpoints
controller does not create Endpoints records. However, the DNS
system looks for and configures either:
CNAME records for ExternalName-type Services.
A records for any Endpoints that share a name with the Service, for all other types.
So, a Headless service is the same as default ClusterIP service, but without load balancing or proxying and therefore allowing you to connect to a Pod directly.
You can also reference below guides for further assistance:
Building a headless service in Kubernetes
Kubernetes Headless service vs ClusterIP and traffic distribution
I'm using the official aws-alb-ingress-controller for ingress + load balancing to my services hosted in an EKS cluster.
Does this offer metrics of any kind? Preferably Prometheus metrics? To show things like volume metrics etc?
https://github.com/kubernetes-sigs/aws-alb-ingress-controller
I don't see any mention of metrics in the docs, but metrics seems like a necessary part of any production load balancer.
Each AWS ALB Ingress Controller Pod
Exposes prometheus /metrics on the same port 10254 where it responds to /healthz checks.
Both endpoints are currently served with the same mux.
This is the closest option, AFAIK:
https://github.com/prometheus/cloudwatch_exporter
The AWS ALB ingress controller is integrated with CloudWatch and provides various metrics. In CloudWatch you can do monitoring and alerting based on these metrics.
If your system uses Prometheus for this, you can any exporter to send the metrics to Prometheus. Another possible exporter would be: YACE (Yet Another CloudWatch Exporter.
Here you can find an article from the AWS open source blog on how to setup an ELB with Prometheus metrics and Graphana on top with a custom dashboard. The configuration for an ALB is pretty similar. Here you have the information on how to achieve this for an ALB.
We're planning to develop some microservices based on the play framework. They will provide rest apis and lots of them will be using akka cluster/cluster-sharding under the hood.
We would like to have an api gateway that exposes the apis of our internal services, but we're facing one big issue:
- Multiple instances of each service will be running under some ip and port.
- How will the api gateway know where the services instances are running?
- Is there maybe something load-balancer-like for play that keeps track of all running services?
Which solution(s) could possibly fill the spot for the "API Gateway"/"Load Balancer"?
The question you're asking is not really related to play framework. And there is no single answer that would solve what you need.
You could start by reading akka Service Discovery and then make your choice based what fits you more.
We're building services with akka-http and use akka-cluster but use unrelated technologies to expose and run the services.
Check out
Kong for API Gateway
Consul for DNS based service discovery
docker swarm for running containers with mesh network for load balancing
You are looking for following components,
Service Registry : The whole point of this component is to keep track of "what service are running on what addresses". This can be as simple as a simple database which keeps entries for all the running services and their instances. Generally the orchestration service is responsible to register new service instances with Service Registry. Other choice can be to have instances themselves notify the service registry about their existence.
Service Health Checker : This component is mostly responsible for doing periodic runtime checks on the registered service instances and tell service registry if any of them is not working. The service registry implementation can then either mark these instances as "inactive" till they are found to be working by Service Health Checker in future (if ever).
Service Resolution : This is the conceptual component responsible for enabling a client to somehow get to the running service instances.
The whole of above components is called Service Discovery.
In your case, you have load-balancers which can act as a form of ServiceDiscovery.
I don't think load-balancers are going to change much over time unless you require a very advanced architecture, so your API gateway can simply "know" the url's to load-balancers for all your services. So, you don't really need service registry layer.
Now, your load-balancers inherently provide a health-check and quarantine mechanism for instances. So, you don't need an extra health check layer.
So, the only piece missing is to register your instances with the load balancer. This part you will have to figure out based on what your load-balancers are and what ecosystem they live in.
If you live in AWS ecosystem and your load balancers are ELB, then you should have things sorted out in that respect.
Based on Ivan's and Sarvesh's answers we did some research and discovered the netflix OSS projects.
Eureka can be used as service locator that integrates well with the Zuul api gateway. Sadly there's not much documentation on the configuration, so we looked further...
We've now finally choosen Kubernetes as Orchestator.
Kubernetes knows about all running containers, so there's no need for an external service locator like Eureka.
Traefik is an api gateway that utilizes the kuberentes api to discover all running microservices instances and does load balancing
Akka management finds all nodes via the kubernetes api and does the bootstrapping of the cluster for us.
Currently I'm working on a real-time online game. First I implemented a go server with socket.io for handling messages between client and my game world and it works fine. Now for user data managing I need a http api for some functionality like login. I want to use awesome http/net package for that purpose. Should I serve the http server on different Port?
My next question is for deploying I want to use google container engine. Can I use pods with two ports open?
As far as I understood from your explanation, you need two ports open for two different APIs running in your application. Regarding Exposing two ports in Google Container Engine, you can read the discussion here that describes ways to expose ports in a pod.
Moreover, I invite you read this tutorial that involves deploying an API in a GKE cluster with a containerPort in a pod, Creating a Kubernetes service to allow internal cluster traffic to your pods (routing requests on an incoming port to your API targetPort), and creating an Ingress service to define what traffic is allowed into your cluster and where it goes. You can define different APIs with different targetPorts and run them on different pods. You can try it as an alternative. For more documentation on Exposing Applications using Services, you can read this GKE doc.
in kubernetes cluster about 40 pods per one environment and five dev/test/prod environment. So totaly about 200 pods. And I have more than hunderds of thousands DNS request on our Domain controllers(outside of kubernetes) to external DNS FQDNs. Do you have any best practice how to set up some DNS cache or something inside the kubernetes? Or any other way, how to reduce the DNS queries from kubernetes cluster?
THX
The default Kubernetes internal DNS doesn't support this, however you can switch to CoreDNS as the provider, which provides you a lot more flexibility.
You can for example use the CoreDNS cache plugin: https://github.com/coredns/coredns/blob/master/man/coredns-cache.7