We have an nginx conf file that is being produced with consul-template for 2000+ services, running Consul 1.6.1.
We're trying to upgrade to Consul 1.7.2 and running into an issue where consul-template DDoSes Consul node agent.
Node/client agent v1.6.1 works fine, but node/client agent v1.7.2 starts refusing connections from consul-template.
The template in question (simplified version) is:
{{- range services }}
{{- if (in .Tags "nginxqa1") }}
{{- range service .Name }}
{{ .Address }}
{{- end }}
{{- end }}
{{- end }}
Adding the following limits config entry on the node/client consul seems to fix the issue:
http_max_conns_per_client = 5000
But, are there any implications to this?
Is there a better way of doing this?
I'm seeing these messages:
2020/05/01 18:24:51.653257 [WARN] (runner) watching 2631 dependencies - watching this many dependencies could DDoS your consul cluster
Any ideas / suggestions appreciated.
The http_max_conns_per_client parameter was introduced in hashicorp/consul#7159 to address CVE-2020-7219. The default value of 200 seemed to be a reasonable default for most use cases. However, there are scenarios such as yours where it makes sense to raise this value.
The only downside to raising this is that any client which is able to communicate with the Consul API will be able to create up to 5000 connections. It is not possible to raise this limit only for a specific client. Keep that in mind as you push these changes to production.
Related
I am using istio version 1.12.
I tried to add limit configuration but it can't limit requests based on source IP.Does Istio support this?
I am using the official sample：
https://istio.io/v1.12/docs/tasks/policy-enforcement/rate-limit/#global-rate-limit
You can use remote_address key.
ConfigMap changes ratelimit service:
- key: remote_address
rate_limit:
requests_per_unit: 10
unit: second
Restart the ratelimit service.
Envoyfilter changes:
- actions:
- remote_address: {}
See also: https://github.com/neumanndaniel/kubernetes/tree/master/envoy-ratelimit
How does it work with Envoy?
Let's say I have configured an upstream cluster like this:
clusters:
-
name: "service_a_cluster"
connect_timeout: "0.25s"
type: "strict_dns"
lb_policy: "ROUND_ROBIN"
hosts:
-
socket_address:
address: "service_a"
port_value: 8786
How is my Envoy instance (ClusterManager?) going to resolve service_a?
To whom is it going to send DNS queries?
Envoy has internal mechanisms for doing resolution, and these are all available through configuration. It looks like you're using Envoy v2 apis, so the relevant high level config is in the cluster object here.
If you read that, you'll notice the hosts field references the type field. This type field tells envoy how to handle discovery/resolution. The full details of that mechanism is here.
I have an idea to implement and i need to create a cache for the pods IPs in a kubernetes cluster and then use an http request to access the cached IPs.
I'm using Golang and as i'm new in this field i would be so grateful if anyone have an idea how to implement that. I searched a lot in internet but i didn't find any simple examples to use as a start.
I started with a piece of code to get the podlist what i need is to put he podlist in a cache, like that each time a request arrives it will use the cache insead of using the kubernetes api o get the IPs.
kubeClient, err := kubernetes.NewForConfig(cfg)
if err != nil {
fmt.Printf("Error building kubernetes clientset: %v\n", err)
os.Exit(2)}
options := metav1.ListOptions{
LabelSelector: "app=hello",}
podList, _ := kubeClient.CoreV1().Pods("namespace").List(options). What i need is to create a cache for the IPs of hello pods for-example and when an http request arrive to my http server, the request will use directly the cached IPs
I appreciate your help.
Thank you in advance,
There's only one correct answer to such question: Leave it. It's a very, very bad idea. Believe me, with such solution you'll only generate more problems that you will need to solve. What about updating such cache when a Pod gets recreated and both its name and IP changes ?
Pod in kubernetes is an object of ephemeral nature and it can be destroyed and recreated in totally normal circumstances e.g. as a result of scaling down the cluster and draining the node Pods are evicted and rescheduled on a different node, with completely different names and IP addresses.
The only stable manner of accessing your Pods is via a Service that exposes them.
To minimize the latency when receiving each time a new request i need
to use the Ips from a cache instead of trying to get the ip of the pod
from the kubernetes API
It's really reinventing the wheel. Every time a Service is created (except Services without selectors), the corresponding Endpoints object is created as well. And in fact it acts exactly like the caching mechanism you need. It keeps track of all IP addresses of Pods and gets updated if a Pod gets recreated and its IP changes. This way you have a guarantee that it is always up to date. When implementing any cache mechanism you would need to call the kubernetes API anyway, to make sure that a Pod with such IP still exists and if it doesn't, what was created instead of it, with what name, with what IP address. Quite bothersome, isn't it ?
And it is not true that each time you access a Pod you need to make a call to kubernetes API to get its IP address. In fact,Service is implemented as a set of iptables rules on each node. When the request hits the virtual IP of the Service it actually falls into specific iptables chain and gets routed by the the kernel to the backend Pod. Kubernetes networking is really wide topic, so I recommend you to read about it e.g. using the resources I attached in the end, but not going into unnecessary details it's worth mentioning that each time cluster configuration changes (e.g. some Pod is recreated and gets a different IP and the respective Endpoints object, that keeps track of Pods IPs for the specific Service, changes), kube-proxy which runs as a container on every node takes care of updating the above mentioned iptables forwarding rules. If running in iptables mode (which is the most common implementation) kube-proxy configures Netfilter chains so the connection is routed directly to the backend container’s endpoint by the node’s kernel.
So API call is made only when the cluster configuration changes so that kube-proxy can update iptables rules. Normally when you're accessing the Pod via a Service, the traffic is routed to backend Pods based on current iptables rules without a need for asking kubenetes API about the IP of such Pod.
In fact, Krishna Chaurasia have already answered your question (shortly but 100% correct) by saying:
You should not access pod by their IPs. They are not persisted across
pod restarts.
and
that's not how K8s work. Requests are forwarded based on the Service
generally and they are redirected towards the matching pods based on
label/selectors. – Krishna Chaurasia 4 hours ago
I can only agree with that. And my reasons for "why" have been explained in detail above.
Additional resources:
Kubernetes Networking Demystified: A Brief Guide
iptables: How Kubernetes Services Direct Traffic to Pods
Kubernetes: Service, load balancing, kube-proxy, and iptables
Network overview on GCP docs or more specifically very nice explanation how kube-proxy works
The problem
I'm getting 403 SSL required from Spring when trying to route through my ELB to Kubernetes Nginx ingress controller.
Setup
My set up is as follows:
I've got an ELB (AWS) with ACM for my Kubernetes cluster (created by kops) which routes all requests to the
Nginx Ingress Controller which in turn routes all requests according to the rules dictated in the
Ingress that passes the traffic unto the
Service that exposes port 80 and routes in to port 8080 in the
Pods selected with labels "app=foobar" (which are described in a Deployment)
Pods are running a Spring Boot Web App v2.1.3
So basically:
https://foo.bar.com(:443) -> ingress -> http://foo.bar.svc.cluster.local:80
This works like a charm for everything. Except SprintBoot.
For some reason, I keep getting 403 - SSL required from Spring
One note to keep in mind here: my Spring application does not have anything to do with SSL. I don't want it to do anything in that nature. For this example's purposes, this should be a regular REST API requests, with the SSL termination happening outside the container.
What I tried so far
Port-forwarding to the service itself and requesting - it works fine.
Disabling CSRF in WebSecurityConfigurerAdapter
Putting ingress annotation nginx.ingress.kubernetes.io/force-ssl-redirect=true - it gives out TOO_MANY_REDIRECTS error when I try it (instead of the 403)
Putting ingress annotation nginx.ingress.kubernetes.io/ssl-redirect=true - doesn't do anything
Putting ingress annotation nginx.ingress.kubernetes.io/enable-cors: "true" - doesn't do anything
Also nginx.ingress.kubernetes.io/ssl-passthrough: "true"
Also nginx.ingress.kubernetes.io/secure-backends: "true"
Also kubernetes.io/tls-acme: "true"
I tried a whole bunch of other stuff that I can't really remember right now
How it all looks like in my cluster
Nginx ingress controller annotations look like this (I'm using the official nginx ingress controller helm chart, with very little modifications other than this thing):
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "aws_acm_certificate_arn"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
Ingress looks like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: foobar
namespace: api
spec:
rules:
- host: foo.bar.com
http:
paths:
- backend:
serviceName: foobar
servicePort: http
path: /
Service looks like this:
apiVersion: v1
kind: Service
metadata:
name: foobar
namespace: api
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
selector:
app: foobar
What I think the problem is
My hunch is that it's something with X-Forwarded headers, and Spring doing its magic behind the scenes, trying to be all smart like and deciding that I need SSL based on some headers without me explicitly asking for it. But I didn't figure it out yet.
I searched far and wide for a solution, but I couldn't find anything to ease my pain... hope you'll be able to help!
Edit
I found out that my current setup (without k8s and nginx) works fine and ELB passes X-Forwarded-Port: 443 and X-Forwarded-Proto: https, and it seems to work, but on my k8s cluster with nginx, I put in a listener client that spits out all the headers, and my headers seem to be X-Forwarded-Port: 80 and X-Forwarded-Proto: http
Thanks for all the people that helped out, I actually found the answer.
Within the code there were some validations that all requests should come from a secure source, and Nginx Ingress Controller changed these headers (X-Forwarded-Proto and X-Forwarded-Port) because SSL was terminated within ELB and handed to the ingress controller as HTTP
To fix that I did the following:
Added use-proxy-protocol: true to the config map - which passed the correct headers, but got errors regarding broken connection (which I don't really remember the actual error right now, I'll edit this answer later if there will be any requests for it)
To fix these errors I added the following the the nginx ingress controller annotations configuration:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
This made sure that all traffic will use the proxy protocol, and I also had to change the backend-protocol from HTTP to TCP.
Doing this made sure all requests routed ELB reserve their original X headers, and are passed unto Nginx Ingress Controller, which in turn passed unto my applications that require these headers to be passed.
I've made a SpringBoot application that authenticate with Gloud Storage and performs action on it. It works locally, but when I deploy it on my GKE as a Pod, it suffers some errors.
I have a VPC environment Where I have a Google Cloud Storage, and a Kubernetes Cluster that will run some Spring Boot applications that performs actions on it through com.google.cloud.storage library.
It has Istio enabled for the Cluster and also a Gateway Resource with Secure HTTPS which targets the Ingress Load Balancer as defined here:
https://istio.io/docs/tasks/traffic-management/secure-ingress/sds/
Then my pods all are being reached through a Virtual Service of this Gateway, and it's working fine since they have the Istio-Car container on it and then I can reach them from outside.
So, I have configured this application in DEV environment to get the Credentials from the ENV values:
ENV GOOGLE_APPLICATION_CREDENTIALS="/app/service-account.json"
I know it's not safe, but just wanna make sure it's authenticating. And as I can see through the logs, it is.
As my code manipulates Storages, an Object of this type is needed, I get one by doing so:
this.storage = StorageOptions.getDefaultInstance().getService();
It works fine when running locally. But when I try the same on the Api now running inside the Pod container on GKE, whenever I try to make some interaction to the Storage it returns me some errors like:
[2019-04-25T03:17:40.040Z] [org.apache.juli.logging.DirectJDKLog] [http-nio-8080-exec-1] [175] [ERROR] transactionId=d781f21a-b741-42f0-84e2-60d59b4e1f0a Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.google.cloud.storage.StorageException: Remote host closed connection during handshake] with root cause
java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
...
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:994)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:142)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1011)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:499)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:358)
... 65 common frames omitted
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
...
Looks like when I make the call from the Pod, it is expected some extra Https configuration. I don't know right.
So what I'm wondering is:
If this is some kind of Firewall Rule blocking this call from my Pod to "outside" (What is weird since they run on the same network, or at least I thought so).
If it's because of the Gateway I defined that is kind of not enabling this Pod
Or if I need to create the Storage Object using some custom HTTP configurations as can be seen on this reference:
https://googleapis.github.io/google-cloud-java/google-cloud-clients/apidocs/com/google/cloud/storage/StorageOptions.html#getDefaultHttpTransportOptions--
My knowledge of HTTPs and Secure conections is not very good, so maybe my lacking on concept on this area is making me not be able to see something obvious.
If some one have any idea on what maybe causing this, I would appreciate very much.
Solved it. It was really Istio.
I didn't know that we need a Service Entry resource to define what inbound and outbound calls OUTSIDE the mesh.
So, even that GCS is in the same project of the GKE, they are threated as completely separated services.
Just had to create it and everything worked fine:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
namespace: {{ cloud_app_namespace }}
name: external-google-api
spec:
hosts:
- "www.googleapis.com"
- "metadata.google.internal"
- "storage.googleapis.com"
location: MESH_EXTERNAL
ports:
- number: 443
name: https
protocol: HTTPS
- number: 80
name: http
protocol: HTTP
https://istio.io/docs/reference/config/networking/v1alpha3/service-entry/
EDIT
I have disabled the Istio Injection on the namespace I were deploying the applications, by simply using:
kubectl label namespace default istio-injection=disabled --overwrite
Then redeployed the application and tried a curl there, and it worked fine.
My doubt now is: I though Istio only intercept on it's gateway layer, and after that the message keeps untouched, but this is not what seems to be working. Apparently he embbed some SSL layer on the request that my application doesn't do/require.
So sould I need to change my application just to fit on the service mesh requirements?