I am developing several services, and use consul as the service registry. I'm able to register all of my services to the consul.
And now for the next thing to do, I need to be able to communicate from service A to service B.
Without a service registry, usually what I did was simply dispatch a client HTTP request from service A to service B.
But since now I already have service discovery in place, should I get the service B host address via consul and then dispatch a client HTTP request to the service B host address something like that? Or does the consul also provide an API gateway, so I only need to dispatch my client HTTP request from service A to the consul, and then the consul will automatically forward it to the destination?
Also if there is relevant documentation about my case, I would be very glad to take a look at it? (I can't find the relevant documentation, probably my google search keyword is wrong)
Consul supports two methods for service discovery, DNS and HTTP.
Applications can perform DNS lookups against their local Consul agent which exposes a DNS server on port 8600 (you can also configure DNS forwarding). For example, an application can issue an A record query for web.service.consul and Consul will return a list of healthy instance endpoints for the web service. SRV lookups are also supported in order to retrieve the IP and port for a given service. The DNS interface also supports querying endpoints by service tag and data center. Details can be found at Consul.io: DNS - Service Lookups.
HTTP-based service discovery can be performed by querying the /v1/health/service/:name endpoint against the local agent. The following will return a full list of healthy and unhealthy endpoints for the service nginx.
$ curl http://127.0.0.1:8500/v1/health/service/nginx
You can use the passing query parameter to restrict the output to only healthy services.
$ curl "http://127.0.0.1:8500/v1/health/service/nginx?passing"
I recommend reviewing the guide Register a Service with Consul Service Discovery for more info on registering and querying services from the catalog.
Lastly, API gateways like Traefik and Solo's Gloo support using Consul for service discovery (see Traefik's Consul Catalog Provider and Gloo's Consul Services). You could configure your services to route requests to these gateways, and allow the gateway to forward to the backend destination.
I ended up getting the list of services info from the consul, and then perform name matching on it then get the service address.
I use this endpoint to get the list of the services and it's data:
http://localhost:8500/v1/agent/services
So it's the client-side discovery I guess.
Related
In Spring security you can whitelist the incoming IP using hasIpAddress. Is it possible to have a whitelist for the incoming domain/url/servicename?
I have two services in a kubernates cluster, most of their endpoints can only be accessed with the accessed token (legacy filter can't be removed). However, I would like one service to call the other without a token for a particular endpoint (let's say service A call service B). Is it possible to specify the service name rather than the ip to be whitelisted in service B? I don't want to make the endpoint to be publicly accessible. Do I have to convert to IP like this
InetAddress ipaddress = InetAddress.getByName(service);
Is there something smart for Kubernetes services?
Maybe you can try to use the internal kubernetes dns link so you don't have to specify the ip address.
I find my server node's endpoint >/v1/agent/services returns majority of services, but not all the services, anyone knows why ?
The visibility of services will depend on which API endpoint you're using.
Consul intends for services to be registered against a Consul client agent which is running on the same host as the deployed service (using the /v1/agent/service/register endpoint). The services registered with each agent in the data center are aggregated to form the service catalog (https://www.consul.io/docs/architecture/anti-entropy#catalog).
The /v1/agent/services endpoint only returns services which have been registered against the specific agent with which you are communicating. In contrast, the /v1/catalog/services endpoint returns an aggregated list of all services which have been registered every agent across the data center. If you query this endpoint, you will receive a list of all services registered with Consul.
I don’t understand the difference between consul’s agent api and catalog api
Although the consul document has always emphasized that agent and catalog should not be confused，But there are indeed many methods that look similar, such as:
/catalog/services
/agent/services
When should I use catalog or agent（Just like the above http url）?
Which one is suitable for high frequency calls?
Consul is designed for services to be registered against a Consul client agent which is running on the same host where a service is deployed. The /v1/agent/service/ endpoints provide a way for you to interact with services which are registered with the specific Consul agent to which you are communicating, and register new services against that agent.
Each Consul agent in the data center submits its registered service information to the Consul servers. The servers aggregate this information to form the service catalog (https://www.consul.io/docs/architecture/anti-entropy#catalog). The /v1/catalog/ endpoints return that aggregated information.
I want to call out this sentence from the anti-entropy doc.
Consul treats the state of the agent as authoritative; if there are any differences between the agent and catalog view, the agent-local view will always be used.
The catalog APIs can be used to register or remove services/nodes from the catalog, but normally these operations should be performed against the client agents (using the /v1/agent/ APIs) since they are authoritative for data in Consul.
The /v1/agent/ APIs should be used for high frequency calls, and should be issued against the local Consul client agent running on the same node as the app, as opposed to communicating directly with the servers.
I am new to Kubernetes.
If there is any service deployed using EKS having 4 replicas A,B,C,D.
Usually loadbalancer directs requests to these replicas
But if I want that my request should go to replica A only or B only...
How can we achieve it.
Request to share some links or steps for guidance
What you could use are the Headless Services:
Sometimes you don't need load-balancing and a single Service IP. In
this case, you can create what are termed "headless" Services, by
explicitly specifying "None" for the cluster IP (.spec.clusterIP).
You can use a headless Service to interface with other service
discovery mechanisms, without being tied to Kubernetes'
implementation.
For headless Services, a cluster IP is not allocated, kube-proxy
does not handle these Services, and there is no load balancing or
proxying done by the platform for them. How DNS is automatically
configured depends on whether the Service has selectors defined:
With selectors
For headless Services that define selectors, the endpoints controller
creates Endpoints records in the API, and modifies the DNS
configuration to return records (addresses) that point directly to the
Pods backing the Service.
Without selectors
For headless Services that do not define selectors, the endpoints
controller does not create Endpoints records. However, the DNS
system looks for and configures either:
CNAME records for ExternalName-type Services.
A records for any Endpoints that share a name with the Service, for all other types.
So, a Headless service is the same as default ClusterIP service, but without load balancing or proxying and therefore allowing you to connect to a Pod directly.
You can also reference below guides for further assistance:
Building a headless service in Kubernetes
Kubernetes Headless service vs ClusterIP and traffic distribution
We have a consul cluster of 3 servers and registering agent services on any of them via Rest Api.
In UI, registrations of a server are visible on other servers as well. For e.g. registration on server A is visible on server B's UI (accessible by http://serverb:8500/).
However when hitting server B via Rest Api, it only shows its own registrations and do not show server A registration.
Server are started as
Server A
consul -server -ui bootstrap-expect=1 -node=ServerA -data-dir=D:\data -bind=11.223.15.78 -client=0.0.0.0 -retry-join=11.223.15.79 -retry-join=11.223.15.80
Server B
consul -server -ui bootstrap-expect=1 -node=ServerB -data-dir=D:\data -bind=11.223.15.79 -client=0.0.0.0 -retry-join=11.223.15.78 -retry-join=11.223.15.80
Server C
consul -server -ui bootstrap-expect=1 -node=ServerC -data-dir=D:\data -bind=11.223.15.80 -client=0.0.0.0 -retry-join=11.223.15.78 -retry-join=11.223.15.79
Is this an issue or am I doing something wrong?
The visibility of services will depend on which API endpoint you're using, and where you're registering your services. Consul intends for services to be registered against a Consul client agent which is running on the same host as the deployed service. The services registered with each agent in the data center are aggregated to form the service catalog (https://www.consul.io/docs/architecture/anti-entropy#catalog).
The /catalog/services endpoint returns an aggregated list of services registered with each agent across the data center. The /agent/services endpoint will only return services registered against the specific local agent with which you are communicating.
If you want clients to be able to register services across any server, you'll want to register them using the /catalog/register endpoint. You can optionally use a tool like Consul External Services Monitor to provide health checking for services, independently from the Consul servers. See https://www.hashicorp.com/blog/consul-and-external-services for more information.
If a service has been registered via the agent api only on one consul node of a cluster, you can still query the service by its service name by means of the catalog api from all server nodes:
/v1/catalog/:servicename
See https://www.consul.io/api-docs/catalog#list-nodes-for-service
Note that you need to deregister a service on the same consul node via the agent api where you have registered it via the agent api in the first place. If you just deregister it from the catalog, it will be back after a few minutes (that is at least my experience)
The Consul documentation recommends to use the agent api for registration, so I would still stick to registering via the agent api, although it makes deregistering a bit tricky.