How does it work with Envoy?
Let's say I have configured an upstream cluster like this:
clusters:
-
name: "service_a_cluster"
connect_timeout: "0.25s"
type: "strict_dns"
lb_policy: "ROUND_ROBIN"
hosts:
-
socket_address:
address: "service_a"
port_value: 8786
How is my Envoy instance (ClusterManager?) going to resolve service_a?
To whom is it going to send DNS queries?
Envoy has internal mechanisms for doing resolution, and these are all available through configuration. It looks like you're using Envoy v2 apis, so the relevant high level config is in the cluster object here.
If you read that, you'll notice the hosts field references the type field. This type field tells envoy how to handle discovery/resolution. The full details of that mechanism is here.
Related
I am using istio version 1.12.
I tried to add limit configuration but it can't limit requests based on source IP.Does Istio support this?
I am using the official sampleļ¼
https://istio.io/v1.12/docs/tasks/policy-enforcement/rate-limit/#global-rate-limit
You can use remote_address key.
ConfigMap changes ratelimit service:
- key: remote_address
rate_limit:
requests_per_unit: 10
unit: second
Restart the ratelimit service.
Envoyfilter changes:
- actions:
- remote_address: {}
See also: https://github.com/neumanndaniel/kubernetes/tree/master/envoy-ratelimit
I am working on a project where I want to configure envoy using consul as the xds server. But I don't want to use consul connect. I only want to use consul for service discovery and the xds server.
I have started the consul agent with -dev mode enabling the grpc endpoints and I have registered a service hello-service which i can see in the consul ui.
My barebone envoy config
# admin web panel
admin:
access_log_path: ./admin_logs
address:
socket_address:
address: 0.0.0.0
port_value: 9902
dynamic_resources:
cds_config:
ads: {}
ads_config:
api_type: grpc
transport_api_version: v3
grpc_services:
- google_grpc:
target_uri: http://localhost:8502
stat_prefix: grpc-xds-service
I am trying to fetch some configuration like clusters/endpoints from consul for the registered hello-service, But when i start the envoy process I am getting this warning.
[2021-07-11 19:16:05.782][11825][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:93] StreamAggregatedResources gRPC config stream closed: 13,
What I could make of this error is that the connection to the management server is failing. But In the consul logs i can see the grpc port is up.
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
Consul's xDS server is not designed to be used outside of Connect, and as such has not been tested or validated to be able to provide CDS/EDS data outside of that context.
You may want to take a look at consul-envoy-xds or envoy-control, both of which may provide the xDS functionality you're looking for without requiring you to deploy a full-blown service mesh.
The problem
I'm getting 403 SSL required from Spring when trying to route through my ELB to Kubernetes Nginx ingress controller.
Setup
My set up is as follows:
I've got an ELB (AWS) with ACM for my Kubernetes cluster (created by kops) which routes all requests to the
Nginx Ingress Controller which in turn routes all requests according to the rules dictated in the
Ingress that passes the traffic unto the
Service that exposes port 80 and routes in to port 8080 in the
Pods selected with labels "app=foobar" (which are described in a Deployment)
Pods are running a Spring Boot Web App v2.1.3
So basically:
https://foo.bar.com(:443) -> ingress -> http://foo.bar.svc.cluster.local:80
This works like a charm for everything. Except SprintBoot.
For some reason, I keep getting 403 - SSL required from Spring
One note to keep in mind here: my Spring application does not have anything to do with SSL. I don't want it to do anything in that nature. For this example's purposes, this should be a regular REST API requests, with the SSL termination happening outside the container.
What I tried so far
Port-forwarding to the service itself and requesting - it works fine.
Disabling CSRF in WebSecurityConfigurerAdapter
Putting ingress annotation nginx.ingress.kubernetes.io/force-ssl-redirect=true - it gives out TOO_MANY_REDIRECTS error when I try it (instead of the 403)
Putting ingress annotation nginx.ingress.kubernetes.io/ssl-redirect=true - doesn't do anything
Putting ingress annotation nginx.ingress.kubernetes.io/enable-cors: "true" - doesn't do anything
Also nginx.ingress.kubernetes.io/ssl-passthrough: "true"
Also nginx.ingress.kubernetes.io/secure-backends: "true"
Also kubernetes.io/tls-acme: "true"
I tried a whole bunch of other stuff that I can't really remember right now
How it all looks like in my cluster
Nginx ingress controller annotations look like this (I'm using the official nginx ingress controller helm chart, with very little modifications other than this thing):
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "aws_acm_certificate_arn"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
Ingress looks like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: foobar
namespace: api
spec:
rules:
- host: foo.bar.com
http:
paths:
- backend:
serviceName: foobar
servicePort: http
path: /
Service looks like this:
apiVersion: v1
kind: Service
metadata:
name: foobar
namespace: api
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
selector:
app: foobar
What I think the problem is
My hunch is that it's something with X-Forwarded headers, and Spring doing its magic behind the scenes, trying to be all smart like and deciding that I need SSL based on some headers without me explicitly asking for it. But I didn't figure it out yet.
I searched far and wide for a solution, but I couldn't find anything to ease my pain... hope you'll be able to help!
Edit
I found out that my current setup (without k8s and nginx) works fine and ELB passes X-Forwarded-Port: 443 and X-Forwarded-Proto: https, and it seems to work, but on my k8s cluster with nginx, I put in a listener client that spits out all the headers, and my headers seem to be X-Forwarded-Port: 80 and X-Forwarded-Proto: http
Thanks for all the people that helped out, I actually found the answer.
Within the code there were some validations that all requests should come from a secure source, and Nginx Ingress Controller changed these headers (X-Forwarded-Proto and X-Forwarded-Port) because SSL was terminated within ELB and handed to the ingress controller as HTTP
To fix that I did the following:
Added use-proxy-protocol: true to the config map - which passed the correct headers, but got errors regarding broken connection (which I don't really remember the actual error right now, I'll edit this answer later if there will be any requests for it)
To fix these errors I added the following the the nginx ingress controller annotations configuration:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
This made sure that all traffic will use the proxy protocol, and I also had to change the backend-protocol from HTTP to TCP.
Doing this made sure all requests routed ELB reserve their original X headers, and are passed unto Nginx Ingress Controller, which in turn passed unto my applications that require these headers to be passed.
First of all request goes to proxy service that i've implemented, service forwards request to pods randomly without using sessionAffinity. I want to send requests to same pod based on custom value that i've set in request parameters using post method. I've used sessionAffinity with my service yml.
Here's service yml with sessionAffinity:
apiVersion: v1
metadata:
name: abcd-service
namespace: ab-services
spec:
ports:
- name: http
protocol: TCP
port: ****
targetPort: ****
nodePort: *****
selector:
app: abcd-pod
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 600
type: NodePort
Now problem is that when request are send by multiple client's from same ip address all requests are directed to single pod and not to other replicas, causing uneven load balancing. But I don't want requests to be forwarded randomly either. i want all request's from same client or different client to be forwarded based on custom value that i set in post request and not by clientIP considering clientIP resolves to source ip of each request.
As you can read here, it currently supports only ClientIP and None values.
sessionAffinity string Supports "ClientIP" and "None". Used to maintain session affinity. Enable client IP based session affinity.
Must be ClientIP or None. Defaults to None. More info:
https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
Unfortunatelly there are no other values allowed.
I'm trying to run an elasticsearch cluster with each es-node running in its own container. These containers are deployed using ECS across several machines that may be running other unrelated containers. To avoid port conflicts each port a container exposes is assigned a random value. These random ports are consistent across all running containers of the same type. In other words, all running es-node containers map port 9300 to the same random number.
Here's the config I'm using:
network:
host: 0.0.0.0
plugin:
mandatory: cloud-aws
cluster:
name: ${ES_CLUSTER_NAME}
discovery:
type: ec2
ec2:
groups: ${ES_SECURITY_GROUP}
any_group: false
zen.ping.multicast.enabled: false
transport:
tcp.port: 9300
publish_port: ${_INSTANCE_PORT_TRANSPORT}
cloud.aws:
access_key: ${AWS_ACCESS_KEY}
secret_key: ${AWS_SECRET_KEY}
region: ${AWS_REGION}
In this case _INSTANCE_PORT_TRANSPORT is the port that 9300 is bound to on the host machine. I've confirmed that all the environment variables used above are set correctly. I'm also setting network.publish_host to the host machine's local IP via a command line arg.
When I forced _INSTANCE_PORT_TRANSPORT (and in turn transport.publish_port) to be 9300, everything worked great, but as soon as it's given a random value, nodes can no longer connect to each other. I see errors like this using logger.discovery=TRACE:
ConnectTransportException[[][10.0.xxx.xxx:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.xxx.xxx:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
It seems like the port a node binds to is the same as the port it pings while trying to connect to other nodes. Is there any way to make them different? If not, what's the point of transport.publish_port?
The way the discovery-ec2 plugin works is that it's collecting a list of IP addresses using AWS EC2 API and use this list as unicast list of nodes.
But it does not collect any information from the running cluster. Obviously the node is not yet connected!
So it does not know anything about the publish_port of other nodes.
It just adds an IP address. And that's all. Elasticsearch then is using the default port which is 9300.
So there is nothing you can do IMO to fix that in the short time.
But we can imagine adding a new feature which is close to what has been implemented for Google Compute Engine. We are using a specific metadata to get this port from the GCE APIs.
We could do the same for Azure and EC2. Do you want to open an issue so we can track the effort?