in kubernetes cluster about 40 pods per one environment and five dev/test/prod environment. So totaly about 200 pods. And I have more than hunderds of thousands DNS request on our Domain controllers(outside of kubernetes) to external DNS FQDNs. Do you have any best practice how to set up some DNS cache or something inside the kubernetes? Or any other way, how to reduce the DNS queries from kubernetes cluster?
THX
The default Kubernetes internal DNS doesn't support this, however you can switch to CoreDNS as the provider, which provides you a lot more flexibility.
You can for example use the CoreDNS cache plugin: https://github.com/coredns/coredns/blob/master/man/coredns-cache.7
Related
Is there a standard way to monitor the traffic of K8 Ingress. We were trying to extract metrics like
Requests per second
HTTP errors
Response time
etc...
Specific Environment
AWS EKS
Nginx ingress
AWS Elastic search service [Store and search metrics]
Kibana and Easy Alert [Dashboard and alerting]
Solutions tried
Service mesh like Istio ( Do not want the heaviness of Istio. Service to service traffic is very less. )
Custom nginx solution (https://sysdig.com/blog/monitor-nginx-kubernetes/)
Looking for something generic to K8 . Any hints ?
The ingress-nginx project has a monitoring guide that describes how to send basic statistics to Prometheus, which can then be viewed in Grafana. If you cluster already has these tools installed, then it is just a matter of configuring prometheus to scrape nginx ingress controller pods.
First of all, there is not "standard way" to monitor a cluster. People or companies have different needs, so you should implement your best solution.
If you are managing all ingress traffic from Nginx ingress controller, then you can implement Nginx based solutions. Said that, linkerd is also a great tool to monitor and manage your network stack, especially if it's simple. There is a dashboard and you can check all your requirements. Linkerd components are also not heavy as istio.
We are moving a workflow of our business to Azure. I currently have two VMs as an HA pair behind an internal load balancer in the North Central US Region as my production environment. I have mirrored this architecture in the South Central US Region for disaster recovery purposes. A vendor recommended I place an Azure Traffic Manager in front of the ILBs for automatic failover, but it appears that I cannot spec ILBs as endpoints for ATM. (For clarity, all connections to these ILBs are through VPNs.)
Our current plan is to put the IPs for both ILBs in a custom-built appliance placed on-prem, and the failover would happen on that appliance. However, it would greatly simplify things if we could present a single IP to that appliance, and let the failover happen in Azure instead.
Is there an Azure product or service, or perhaps more appropriate architecture that would allow for a single IP to be presented to the customer, but allow for automatic failover across regions?
It seems that you could configure an application gateway with an internal load balancer (ILB) endpoint. In this case, you will have a private frontend IP configuration for an Application Gateway. The APPGW will be deployed in a dedicated subnet, it will exist on the same VNet with your internal backend VMs. Please note in this case you can directly add the private VMs as the backends instead of internal load balancer frontend IP address because of private APPGW itself is an internal load balancer.
Moreover, APPGW also could configure a public frontend IP configuration, if so, you can configure the APPGW public frontend IP as the endpoints of the Azure traffic manager.
Hope this could help you.
This is a micro services deployment question. How would you deploy Envoy SDS(service discovery service) so other envoy proxies can find the SDS server hosts, in order to discover other services to build the service mesh. Should I put it behind a load balancer with a DNS name( single point of failure) or just run the SDS locally in every machine so other micro services can access it? Or is there a better way of deployment that SDS cluster can be dynamically added into envoy config without a single point of failure?
Putting it behind a DNS name with a load balancer across multiple SDS servers is a good setup for reasonable availability. If SDS is down, Envoy will simple not get updated, so it's generally not the most critical failure -- new hosts and services simply won't get added to the cluster/endpoint model in Envoy.
If you want higher availability, you set up multiple clusters. If you add multiple entries to your bootstrap config, Envoy will fail over between them. You can either specify multiple DNS names or multiple IPs.
(My answer after misunderstanding the question below, for posterity)
You can start with a static config or DNS, but you'll probably want to
check out a full integration with your service discovery.
Check out Service Discovery
Integration
on LearnEnvoy.io.
Let's say I have 20 servers at Amazon AWS and I also have AWS ELB setup for these servers. I heard that HAProxy is reliable and fast TCP/HTTP Load Balancer, so question is:
do I need to have HAProxy installed in each EC2 instances while I have AWS ELB?
What is the benefit of having both ELB and Haproxy at the same time?
Thanks
There are a few scenarios where people chose their own load balancing solution like HAProxy than ELB:
Financial transactions: ELB is an opaque service. Logs are not provided. So if you are missing transactions, you won't know if ELB dropped them or not.
Doesn't work well with traffic spikes: ELBs scaling takes at least 5 minutes. If your application traffic is doubling every 5-10 minutes, it will do well. But if it is at a constant rate and you will get a spike all of a sudden, then you will have problems with ELB.
ELBs can be slower than running your own Loadbalancing: In my environment, I got 15% performance boost by using HAProxy/Nginx (for SSL termination) instead. It was roughly 30ms per call, but keep in mind I was using SSL so I use CPU power.
ELBs only do round-robin load balancing and HAProxy has a lot more.
HAProxy also has ton more configurations that ELB does not support. It depends if one needs them for their application.
In one suite of applications, I have both running. ELB->haproxy->A suite of apps. In my case the following occurs:
ELB translates HTTPS to http
HAproxy targets to the app servers based on path
The app servers run in plain old http
The upside to this is that I can move around the apps without changing their URLs
The downside is that ELB isn't a fixed IP address so if you need to point to it from an IP adress instead of a cname you can't do it.
Short answer: No you don't need HAProxy. Go with an ELB.
tldr;
Yes HAProxy is powerful and tested.
First of all, you would need to have a separate EC2 HAProxy instance (as opposed to having HAProxy installed on every EC2 instance you need to balance). In essence an ELB is equivalent to an EC2 instance loaded with some kind of load balancing software.
Second, having both ELBs and HAProxy balancing instances in your environment is a rare use case. You might come to a point that you need more fine grained access and the ability to configure more on your load balancers. It purely depends on what you're doing and what problems an ELB might be giving you. Google to read through possible use cases.
I'm using an ELB and Haproxy behind.
When a customer uses my webservices from a unique IP, ELB redirects all his requests to the same hosts. It doesn't scale. (I supposed it's a hash from the src ip or something like that).
The haproxy has another balancer algorithm.
I keep the ELB for HA (1 haproxy / availability zone). And each haproxy instance redispatchs to region zone backend servers
We have two webservers running with nginx + php5-fpm configuration (native php sessions on a memdisk)
The webservers are using different subdomains and load-balancing is somehow satisfied.
Now we want to use the same domain address for these servers and even newer ones with the exact same configuration.
Installing an nginx in front of these web servers does not sound as a good idea since there are many file uploads to these web servers and the datatraffic passing through the loadbalancer for each upload would cause unnecessary traffic. Round robin DNS solution needs sessions shared in memcache, redis or db. This is also what we do not like because we keep too much data in sessions and very fine with it in our memdisks on local.
Are there any other configurations that fit to our needs?
The data will need to go through some networking appliance or other. This can be an application loadbalancer like Nginx, a software network loadbalancer like LVS, a hardware loadbalancer or, if DNS roundrobin is used, you still need to route the traffic through a switch.
If you are not satisfied with the performance of Nginx, check out LVS or consider buying a hardware loadbalancer. We saw really good performance through LVS (Linux Virtual Server) at the webhosting company where I used to work, so there's still much you can do with software.
Do some research. Set up an Nginx or LVS loadbalancer and benchmark it. Imitate your usual traffic patterns and check how it performs.