We have a number of Spring Boot applications that register themselves with Consul (via Spring Cloud Consul). If I stop those applications via docker-compose stop myservice then they de-register themselves correctly and disappear from Consul.
If I use docker-compose kill myservice then the deregistration doesn't happen. I understand that on a UNIX system it's impossible to catch the SIGKILL event, so there's no way to force the de-registration.
What we're therefore seeing is services in Consul that never removed (marked as critical but still visible in the UI). Is there a way to force Consul to refresh what's registered, so that the dead services are removed?
Thanks
Nick
It seems, that you have to use Consul HTTP API and manually deregister unavailable services. API gives you 2 different ways to deregister some service, the first one via agent endpoint like so
curl -v -X PUT http://%CONSUL_IP%:8500/v1/agent/service/deregister/<ServiceID>
and the second via catalog. Unfortunately in both cases you have to make http-request manually.
Related
I have a service fabric cluster with multiple microservices, and I want to set up Azure Front Door, however it asks for a healthcheck endpoint in the backend but I don't know how I am supposed to set it up since the cluster doesn't have an endpoint for that.
Could anyone point me in the right direction?
You could implement a health check on your service, by introducing a watchdog service. Optionally tapping into the built-in health system of SF. It could look like this:
Create an ASP .NET Core Web API, and implement some health checks. For example, a custom check if your SF Service is alive (and well). Here's how to get started. Return 200 OK from the API, if the watched SF Service is running correctly.
Run this Web API as an SF Service. Expose it through the Load Balancer.
Use its URL as the health endpoint for your Service(s).
We're planning to develop some microservices based on the play framework. They will provide rest apis and lots of them will be using akka cluster/cluster-sharding under the hood.
We would like to have an api gateway that exposes the apis of our internal services, but we're facing one big issue:
- Multiple instances of each service will be running under some ip and port.
- How will the api gateway know where the services instances are running?
- Is there maybe something load-balancer-like for play that keeps track of all running services?
Which solution(s) could possibly fill the spot for the "API Gateway"/"Load Balancer"?
The question you're asking is not really related to play framework. And there is no single answer that would solve what you need.
You could start by reading akka Service Discovery and then make your choice based what fits you more.
We're building services with akka-http and use akka-cluster but use unrelated technologies to expose and run the services.
Check out
Kong for API Gateway
Consul for DNS based service discovery
docker swarm for running containers with mesh network for load balancing
You are looking for following components,
Service Registry : The whole point of this component is to keep track of "what service are running on what addresses". This can be as simple as a simple database which keeps entries for all the running services and their instances. Generally the orchestration service is responsible to register new service instances with Service Registry. Other choice can be to have instances themselves notify the service registry about their existence.
Service Health Checker : This component is mostly responsible for doing periodic runtime checks on the registered service instances and tell service registry if any of them is not working. The service registry implementation can then either mark these instances as "inactive" till they are found to be working by Service Health Checker in future (if ever).
Service Resolution : This is the conceptual component responsible for enabling a client to somehow get to the running service instances.
The whole of above components is called Service Discovery.
In your case, you have load-balancers which can act as a form of ServiceDiscovery.
I don't think load-balancers are going to change much over time unless you require a very advanced architecture, so your API gateway can simply "know" the url's to load-balancers for all your services. So, you don't really need service registry layer.
Now, your load-balancers inherently provide a health-check and quarantine mechanism for instances. So, you don't need an extra health check layer.
So, the only piece missing is to register your instances with the load balancer. This part you will have to figure out based on what your load-balancers are and what ecosystem they live in.
If you live in AWS ecosystem and your load balancers are ELB, then you should have things sorted out in that respect.
Based on Ivan's and Sarvesh's answers we did some research and discovered the netflix OSS projects.
Eureka can be used as service locator that integrates well with the Zuul api gateway. Sadly there's not much documentation on the configuration, so we looked further...
We've now finally choosen Kubernetes as Orchestator.
Kubernetes knows about all running containers, so there's no need for an external service locator like Eureka.
Traefik is an api gateway that utilizes the kuberentes api to discover all running microservices instances and does load balancing
Akka management finds all nodes via the kubernetes api and does the bootstrapping of the cluster for us.
I'm interested in knowing if I can use Consul to solve the following issues:
1) Can Consul be used to load balance microservices? For instance, if I put console on the server that hosts my API gateway, can it be used to monitor all microservices it has discovered and load balance if I have two of the same microservice?
2) Can Consul be used at the microservice level to spin up instances as needed? Essentially, I'd like to not use IIS and find an alternative.
3) If for whatever reason Consul monitors a microservice as offline, can it attempt to start it up again? Or force a shut down of a microservice for whatever reason?
If Consul software can't solve these issues, is there other alternatives?
Thank you.
Consul DNS can provide a simple way for you to load balance services. It's especially powerful if you combine it with Consul Prepared Queries and health checks.
Consul is best suited for monitoring services (via health checks) but you can use consul watch to trigger events if a service suddenly becomes unavailable.
Hashicorp (the company behind Consul) offers another tool called Nomad.
Unlike Consul, Nomad is designed to run services (called jobs) and restart them if necessary.
Nomad works best if you tell it where to find Consul. This enables automatic service registration for any task Nomad launches, including deregistering it if you instruct Nomad to stop running that task. Health checks are supported as well.
I have consul running on Consul v0.5.2 version & services running in Mesos. Services keep moving from 1 server to another.
Is there way to deregister services in consul that are in 'failing' state? I am able to get the list of services in failing state using this curl
curl http://localhost:8500/v1/health/state/critical
Issue that we are seeing is over a period of time in consul UI we have stale data & making the whole UI unusable
Consul by default do not deregister unhealthy services instead marks them as critical.
From Consul 0.7 there is special option (deregister_critical_service_after) that allows you to define time after unhealthy service will be deregstered
From Consul 0.7 Changelog
Automatic Service Deregistration: Added a new
deregister_critical_service_after timeout field for health checks
which will cause the service associated with that check to get
deregistered if the check is critical for longer than the timeout.
This is useful for cleanup of health checks registered natively by
applications, or in other situations where services may not always be
cleanly shutdown. GH-679
If you are usign Marathon then you can consider using allegro/marathon-consul it will deregister task when its dead
Along with what janisz said, you can also run your services in Nomad and Nomad will automatically register and deregister your services for you. See the Nomad Service Discovery docs for additional details.
I am playing a little with Docker and Consul and i have a couple of questions regarding agent-service mapping especially in docker environment. Assume i have a service name "myGreatService" being simple web nodejs helloworld application encapsulated with docker image named "myGreatServiceImage". From Consul docs i did understand that when you register a service (through HTTP or service definition file) than service is about to be "wired" to agent/consul node (the wired node can be retrieved via /v1/catalog/service/). So if a consul node is down (or node health check decided it is down) than all services "wired" to that consule node will automatically be marked as down. Am i right ?
If i run my GreatServiceImage image multiple times on a single host via docker (resulting of multiple instances of "myGreatService" service)
how many agents shall I run ?
A single per host managing all containers (all service instances) on that host? Or maybe a separate agent for each container (service instance) ?
If a health check for a service fails then the service will be marked as down and won't show up if you do a DNS query for that service
dig #localhost -p 8500 apache.service.consul
If you do a call to the api you will see that the service is still listed. This is because the service is not removed, it is just marked as down. If you would do an api call to check the health of that service it would be shown as down.
curl localhost/v1/catalog/service/apache
curl localhost/v1/health/service/apache
You can add the ?passing flag to that last call to recieve only the healthy services. (just like the dns query)
curl localhost/v1/health/service/apache?passing
If the consul agent on the host fails then all services running on that host won't show up if you query consul for the services. (either via a dns query or via the api).
As for the number of agents you should be running: Run one consul agent per host. Let your services register themselves via the api of your local consul agent. (or preconfigure all your services in the config files, but I recommend you to make this a dynamic process of self registering)