I am using consul(docker.io/bitnami/consul:1.10.2-debian-10-r0') as my register center, now I am facing a problem that the consul did not remove the dead service automatically. now my register center look like this:
is it possible to remove the dead legacy service automatically and make the register center always keep the lastest avaliable service snapshot?
Consul does not automatically remove dead services (although it will remove dead nodes after 72 hours). You will need to deregister the instances from the catalog in order to remove the unhealthy service entries.
Related
I have multiple instance of a service registered with eureka; when using FeignClient I am able to successfully contact those instances using the service name of the registered application.
But there is a "problem": if I shut down one of the instances (I have also verified that the instance has gone down correctly and it has immediately been unregistered) and then make some requests to the "gateway" that then calls the services via Feign, the load-balancer still tries for some time to contact the instance that is off, resulting in timeouts and obviously failure of the request.
How is it possible to avoid this behaviour? Is there any way to force the update of the online instances so that to avoid the timeout of the requests.
I have also tried to manually get all the online instances from the discovery-client at runtime during the application execution and the online instances list is correct (the discovery server notifies correctly about every shutdown/start of the instances almost immediately).
Why is that the FeignClient doesn't get "updated" and still calls the dead ones even if the in app discovery-client instances list has been updated?
Here you can find an example of the configuration I'm trying to use.
https://github.com/fearlessfara/feign-test
I have a consumer that reads and writes messages to a time-series database. We have multiple instances of the time series database running as a cluster on multiple physical machines.
Our plan is to deploy the consumer on Kubernetes so I can scale if I need more instance with load-balance they all point to a single time series service that is running.
Now I getting an Issue where it's come to my mind that if I have 5 instances which consume the same topic then they work individually means( they all get message payload and save like any one instance is doing )
What we want is
we want if one consumer is busy then it will go to the next free instance but not be subscribed to by all instance running. To scale or load-balance means I want like normal load-balancing application or how spring-boot app works when you scale on Kubernetes
so is there any way to make it like a load-balancing consumer and processing only one, even consume by 1st or 2nd or 3rd like normal app work as loadbanlacer?
if anyone has ideas about this, how it going to behave and what kind of output we are going to get if doing this with Kafka Spring boot application?
I have a scenario where two or more instances of the same verticle will be instantiated. I want to make sure that only one of the instances is consuming the key 'keyx' and in order to do that I check on a Service Discovery instance if a certain type of record is there and, if not, I can safely say that no one is consuming 'keyx'.
Therefore, I publish a record on the Service Discovery instance and I subscribe to 'keyx'. All the other instances will now check with the Service Discovery that some instance is already registered for 'keyx'.
If the machine with the verticle instance has any serious problem, it will get the verticle killed and the record will still be on the Service Discovery (in this case, removing the record in the stop() method would not work because this method wouldn't be called) and all the other instances created will believe that an instance is still consuming 'keyx' when it is possibly not the case.
Does someone know any viable solution for this problem?
Thanks ;)
One way is to use a datastore that has automatic expiry for keys. Then clients must periodically re-add the key to the store to continue using it, and if they fail then the key is automatically removed. Redis offers this kind of feature (https://redis.io/commands/expire).
Alternatively if you dont have this feature, you can simply store a timestamp when you set the key. If another client reads the key but the time has expired it can safely use the key.
How do I use Consul to make sure only one service is performing a task?
I've followed the examples in http://www.consul.io/ but I am not 100% sure which way to go. Should I use KV? Should I use services? Or should I use a register a service as a Health Check and make it be callable by the cluster at a given interval?
For example, imagine there are several data centers. Within every data center there are many services running. Every one of these services can send emails. These services have to check if there are any emails to be sent. If there are, then send the emails. However, I don't want the same email be sent more than once.
How would it make sure all emails are sent and none was sent more than once?
I could do this using other technologies, but I am trying to implement this using Consul.
This is exactly the use case for Consul Distributed Locks
For example, let's say you have three servers in different AWS availability zones for fail over. Each one is launched with:
consul lock -verbose lock-name ./run_server.sh
Consul agent will only run the ./run_server.sh command on which ever server acquires the lock first. If ./run_server.sh fails on the server with the lock Consul agent will release the lock and another node which acquires it first will execute ./run_server.sh. This way you get fail over and only one server running at a time. If you registered your Consul health checks properly you'll be able to see that the server on the first node failed and you can repair and restart the consul lock ... on that node and it will block until it can acquire the lock.
Currently, Distributed Locking can only happen within a single Consul Datacenter. But, since it is up to you to decide what a Consul Servers make up a Datacenter, you should be able to solve your issue. If you want locking across Federated Consul Datacenters you'll have to wait for it, since it's a roadmap item.
First Point:
The question is how to use Consul to solve a specific problem. However, Consul cannot solve that specific problem because of intrinsic limitations in the nature of a gossip protocol.
When one datacenter cannot talk to another you cannot safely determine if the problem is the network or the affected datacenter.
The usual solution is to define what happens when one DC cannot talk to another one. For example, if we have 3 datacenters (DC1, DC2, and DC3) we can determine that whenever one DC cannot talk to the other 2 DCs then it will stop updating the database.
If DC1 cannot talk to DC2 and DC3 then DC1 will stop updating the database, and the system will assume DC2 and DC3 are still online.
Let's imagine that DC2 and DC3 are still online and they can talk to each other, then we have quorum to continue running the system.
When DC1 comes online again it will play catch up with the database.
Where can Consul help here? It can communicate between DCs and check if they are online... but so can ICMP.
Take a look at the comments. Did this answer your question? Not really. But I don't think the question has an answer.
Second point: The question is "How to use Consul in leader election?" It would have been better to ask how does Consul elect a new leader. Or "Given the documentation in Consul.io, can you give me an example on how to determine the leader using Consul".
If that is what you really want, then the question was already answered: How does a Consul agent know it is the leader of a cluster?
I am creating a camel project which polls the local directory and pushes the files to a FTP location. I want to apply failover mechanism to my design. If want another fuse instance to be ready, if the current node fails.
Without any failover, two fuse instances can together poll the files. But I wanted the second node to poll the files, when the first node fails.
Is this scenario possible if I use FuseFabric ? I dont want anyother product choice, I have this product with me. But I want to know whether I can achieve this using Fabric ?
I am sure, this is possible in Web service endpoints. Not sure about file based endpoints.
Yes there is a master component you can use, then only one is active at a time. And if the node fails, the another node is elected as the master.
Just put "master:someNameHere:" in front of any Camel endpoint in a route from, when you use Fuse Fabric. Where "someNameHere" is a logical name for the group.
For example
from("master:foo:ftp://bla-bla")
.to("someWhere");