I have a question regarding how Load balancing works when you have a single node Docker Swarm combined with Spring Eureka using Spring Cloud gateway. I have successfully configured a non Eureka swarm and can see Swarm load balancing between replicas for a service:
Cloud Gateway route config
.route(r -> r.path("/api/**")
.uri("http://my-service:8081")
.id("my-service"))
If I then configure this to use Eureka I now have this:
.route(r -> r.path("/api/**")
.uri("lb://MY-SERVICE")
.id("my-service"))
I believe I'm right in assuming that the gateway will know the IP/Ports and load balance accordingly, however when a request hits an IP will swarm then also decide to load balance between the replicas?
I appreciate that Eureka may be overkill for a small single node swarm but feel it could be beneficial as the app expands and possibly becomes more distributed. Obviously I want to avoid a situation where load balancing happens twice.
I assume I could just use http instead of lb to stop the Gateway from load balancing.
The eureka discovery service will provide the api gateway all available addresses for a given service. Each service registered within eureka will have a unique (container) IP and port and if the api gateway is configured to load balance the requests then yes, each replica of the service will be used, swarm doesn't need to do anything for load balancing, since you are targeting a specific running service (task) and not a node, for example.
However, for a multi-node scenario, Docker swarm has a routing mesh functionality that basically removes the need of having a discovery service. Imagine you have multiple nodes and replicas distributed across them. With swarm's routing mesh you don't even have to know which nodes have specific services running. The api gateway can route the incoming request to literally any node, and if that node happens to lack the requested service, it will automatically balance the request to nodes that do have the task (the name given to a running service).
So, that means that the load balancer doesn't need any sort of discovery service such as Eureka in order to balance the requests to certain container's IPs or nodes, it can simply round-robin all available nodes and that's it.
As for internal requests between the services that have replicas, swarm also provides load balancing capabillities.
Related
I'm building a system where client IoT devices will be making persistent websocket connections to a single instance of a microservice. We'll call it the "hardware gateway". End devices will be connecting to one of these service instances and may migrate between services at anytime (perhaps due to a reboot or network interruption).
Other services will be pushing notifications to these hardware clients via some hardware gateway instance. I need a way to route these requests to the specific instance that is maintaining a connection to a specific IoT device. At the moment, my solution is to maintain an external KV store where I can map an IoT device's UUID to a service instance, but that puts an extra dependency on all other services to know about this KV store. Not to mention the additional latency introduced by this query.
Maybe there's some reverse proxy that allows me to dynamically update its matching criteria? I've also looked into using a message broker like RabbitMQ, but it doesn't seem to support this use case.
There's a reasonable solution in JVM land for this: Akka.
The instances form an Akka cluster. When a device makes a websocket connection, an actor is spawned to handle the interactions over the websocket. The actor registers that it is the actor interacting with the device with a cluster sharded actor keyed by the device's ID (and likely periodically reregisters with the sharded actor). As instances are deployed, etc. the cluster rebalances. An important feature of this is that the service is stateful, but the instances deploy in a way that looks to the outside world like it's stateless: requests can go to any node.
For pushing notifications to the devices, the HTTP endpoint or message-bus consumer in the service looks up the cluster sharded actor which forwards the notification to the websocket actor (you'll want to think about whether you want at-least-once or at-most-once delivery, which will govern whether there's some portion of the cluster sharded actor which should be persistent).
Background
I came from HAproxy background and recently there is a lot of hype around "Service Mesh" Architecture. Long story short, I began to learn "Envoy" and "Consul".
I develop an understanding that Envoy is a proxy software but using sidecar to abstract in-out network with "xDS" as Data Plane for the source of truth (Cluster, Route, Filter, etc). Consul is Service Discovery, Segmentation, etc. It also abstracts network and has Data Plane but Consul can't do complex Load Balancing, filter routing as Envoy does.
As Standalone, I can understand how they work and set up them since documentation relatively good. But it can quickly became a headache if I want to integrate Envoy and Consul, since documentation for both Envoy & Consul lacks specific for integration, use-cases, and best practice.
Schematic
Consider the following simple infrastructure design:
Legends:
CS: Consul Server
CA: Consul Agent
MA: Microservice A
MB: Microservice B
MC: Microservice C
EF: Envoy Front Facing / Edge Proxy
Questions
Following are my questions:
In the event of Multi-Instance Microservices, Consul (as
stand-alone) will randomize round-robin. With Envoy & Consul
Integration, How consul handle multi-instance microservice? which
software does the load balance?
Consul has Consul Server to store its data, however, it seems Envoy
does not have "Envoy Server" to store its data, so where are its
data being stored and distributed across multiple instances?
What about Envoy Cluster (Logical Group of Envoy Front Facing Proxy
and NOT Cluster of Services)? How the leader elected?
As I mentioned above, Separately, Consul and Envoy have their
sidecar/agent on each Machine. I read that when integrated, Consul
injects Envoy Sidecar, but no further information on how this works?
If Envoy uses Consul Server as "xDS", what if for example I want to
add an advanced filter so that for certain URL segment it must
forward to a certain instance?
If Envoy uses Consul Server as "xDS", what if I have another machine
and services (for some reason) not managed by Consul Server. How I
configure Envoy to add filter, cluster, etc for that machine and
services?
Thank You, I'm so excited I hope this thread can be helpful to others too.
Apologies for the late reply. I figure its better late than never. :-)
If you are only using Consul for service discovery, and directly querying it via DNS then Consul will randomize the IP addresses returned to the client. If you're querying the HTTP interface, it is up to the client to implement a load balancing strategy based on the hosts returned in the response. When you're using Consul service mesh, the load balancing function will be entirely handled by Envoy.
Consul is an xDS server. The data is stored within Consul and distributed to the agents within the cluster. See the Connect Architecture docs for more information.
Envoy clusters are similar to backend server pools. Proxies contain Clusters for each upstream service. Within each cluster, there are Endpoints which represent the individual proxy instances for the upstream services.
Consul can inject the Envoy sidecar when it is deployed on Kubernetes. It does this through a Kubernetes mutating admission webhook. See Connect Sidecar on Kubernetes: Installation and Configuration for more information.
Consul supports advanced layer 7 routing features. You can configure a service-router to route requests to different destinations by URL paths, headers, query params, etc.
Consul has an upcoming feature in version 1.8 called Terminating Gateways which may enable this use case. See the GitHub issue "Connect: Terminating (External Service) Gateways" (hashicorp/consul#6357) for more information.
I have several Kubernetes clusters. Due to the company's security issues, only A 'service in Cluster A should be allowed to access B' Service in Cluster B. Can you handle such a case with istio?
Although it is possible to control the traffic using the header information in istio's virtualservice, the http header information can be manipulated at any time, which does not satisfy the security issue.
Istio has a different federation with a single control plane or multiple control plane. you can check out below. the communication across network supported by MTLS so you can be assured it can't have tampered.
Shared control plane
https://istio.io/docs/setup/kubernetes/install/multicluster/shared-gateways/
Multiple control planes
https://istio.io/docs/setup/kubernetes/install/multicluster/gateways/
This is pretty new and under heavy development, so you can try them or simply use HTTPS communication when connecting across the network.
How do I enable a port on Google Kubernetes Engine to accept websocket connections? Is there a way of doing so other than using an ingress controller?
Web sockets are supported by Google's global load balancer, so you can use a k8s Service of type LoadBalancer to expose such a service beyond your cluster.
Do be aware that load balancers created and managed outside Kubernetes in this way will have a default connection duration of 30 seconds, which interferes with web socket operation and will cause the connection to be closed frequently. This is almost useless for web sockets to be used effectively.
Until this issue is resolved, you will either need to modify this timeout parameter manually, or (recommended) consider using an in-cluster ingress controller (e.g. nginx) which affords you more control.
As per this article in the GCP documentation, there are 4 ways that you may expose a Service to external applications.
It can be exposed with a ClusterIP, a NodePort, a (TCP/UDP) Load Balancer, or an External Name.
Is that called "clustering" of servers? When a web request is sent, does it go through the main server, and if the main server can't handle the extra load, then it forwards it to the secondary servers that can handle the load? Also, is one "server" that's up and running the application called an "instance"?
[...] Is that called "clustering" of servers?
Clustering is indeed using transparently multiple nodes that are seen as a unique entity: the cluster. Clustering allows you to scale: you can spread your load on all the nodes and, if you need more power, you can add more nodes (short version). Clustering allows you to be fault tolerant: if one node (physical or logical) goes down, other nodes can still process requests and your service remains available (short version).
When a web request is sent, does it go through the main server, and if the main server can't handle the extra load, then it forwards it to the secondary servers that can handle the load?
In general, this is the job of a dedicated component called a "load balancer" (hardware, software) that can use many algorithms to balance the request: round-robin, FIFO, LIFO, load based...
In the case of EC2, you previously had to load balance with round-robin DNS and/or HA Proxy. See Introduction to Software Load Balancing with Amazon EC2. But for some time now, Amazon has launched load balancing and auto-scaling (beta) as part of their EC2 offerings. See Elastic Load Balancing.
Also, is one "server" that's up and running the application called an "instance"?
Actually, an instance can be many things (depending of who's speaking): a machine, a virtual machine, a server (software) up and running, etc.
In the case of EC2, you might want to read Amazon EC2 Instance Types.
Here is a real example:
This specific configuration is hosted at RackSpace in their Managed Colo group.
Requests pass through a Cisco Firewall. They are then routed across a Gigabit LAN to a Cisco CSS 11501 Content Services Switch (eg Load Balancer). The Load Balancer matches the incoming content to a content rule, handles the SSL decryption if necessary, and then forwards the traffic to one of several back-end web servers.
Each 5 seconds, the load balancer requests a URL on each webserver. If the webserver fails (two times in a row, IIRC) to respond with the correct value, that server is not sent any traffic until the URL starts responding correctly.
Further behind the webservers is a MySQL master / slave configuration. Connections may be mad to the master (for transactions) or to the slaves for read only requests.
Memcached is installed on each of the webservers, with 1 GB of ram dedicated to caching. Each web application may utilize the cluster of memcache servers to cache all kinds of content.
Deployment is handled using rsync to sync specific directories on a management server out to each webserver. Apache restarts, etc.. are handled through similar scripting over ssh from the management server.
The amount of traffic that can be handled through this configuration is significant. The advantages of easy scaling and easy maintenance are great as well.
For clustering, any web request would be handled by a load balancer, which being updated as to the current loads of the server forming the cluster, sends the request to the least burdened server. As for if it's an instance.....I believe so but I'd wait for confirmation first on that.
You'd' need a very large application to be bothered with thinking about clustering and the "fun" that comes with it software and hardware wise, though. Unless you're looking to start or are already running something big, it wouldn't' be anything to worry about.
Yes, it can be required for clustering. Typically as the load goes up you might find yourself with a frontend server that does url rewriting, https if required and caching with squid say. The requests get passed on to multiple backend servers - probably using cookies to associate a session with a particular backend if necessary. You might have the database on a separate server also.
I should add that there are other reasons why you might need multiple servers, for instance there may be a requirement that the database is not on the frontend server for security reasons