I have a set of microservices using springboot rest. These microservices will be deployed in a autoscaled and load balanced environment. One of these services is responsible for managing the system's configuration. When other microservices startup, they obtain the configuration from this service. If and when the configuration is updated, I need to inform all currently running microservices instances to update their cached configuration.
I am considering using RabbitMQ with a fanout exchange. In this solution, each instance at startup will create its queue and bind that queue to the exchange. When there is a configuration change, the configuration service will publish an update to all queues currently bound to that exchange.
However, as service instances are deleted, I cannot figure out how would I delete the queue specific to that instance. I googled but could not find a complete working example of a solution.
Any help or advise?
The idea and solution is correct. What you just miss that those queues, created by your consumer services could be declared as auto-delete=true: https://www.rabbitmq.com/queues.html. As long as your service is UP, the queue is there as well. You stop your service, its consumers are stopped and unsubscribed. At the moment the last consumer is unsubscribed the queue is deleted from the broker.
On the other hand I would suggest to look into Spring Cloud Bus project which really is aimed for tasks like this: https://spring.io/projects/spring-cloud-bus.
Related
We are using microservice architecture in our project. We deploy each service to a cluster by using Kubernetes. Services are developed by using Java programming language and Spring Boot framework.Three replicas exist for each service. Services communicate with each other using only events. RabbitMQ is used as a message queue. One of the services is used to send an email. The details of an email are provided by another service with an event. When a SendingEmail event is published by a service, three replicas of email service consume the event and the same email is sent three times.
How can I prevent that sending emails by other two services?
I think it depends on how you work with Rabbit MQ.
You can configure the rabbit mq with one queue for these events and make spring boot applications that represent the sending servers to be "Competing" Consumers.
If you configure it like this, only one replica will get an event at a time and only if it fails to process it the message will return to the queue and will become available to other consumers.
From what you've described all of them are getting the message, so it works like a pub-sub (which is also a possible way of work with rabbit mq, its just not good in this case).
I am using rabbitmq in spring boot application. I am using aws ecs for deployment. Now suppose multiple instance is running of my service. and rabbitmq listening for order create is registered with direct exchange.. So what happen when order is placed? will my both the instance of service will get same message? If yes, How to prevent duplicate message on those 2 listeners?
If the service creates multiple Listeners/Consumers for same queue on a direct exchange below mechanism is applicable:
By default, RabbitMQ will send each message to the next consumer, in sequence. On average every consumer will get the same number of messages. This way of distributing messages is called round-robin.
Best Tutorial for this topic: https://www.rabbitmq.com/tutorials/tutorial-two-java.html
I am writing a service with Spring and I am using Spring AMQP in order to connect to Rabbitmq.
I have two rabbitmq clusters, one is only for publishing messages(the messages are sent to the other cluster via the federation plugin) and the other cluster is for declaring queues that end users will consume from.
The nodes sit behind aws lb, each cluster has a lb.
I am using CachingConnectionFactory and RabbitTemplate,RabbitAdmin in my code and I want to have connections to all the nodes so I can use them.
For the cluster that will contain the queues I added to the config the queue-master-locator=random so new queues will be declared in all the nodes in the cluster even if my service does not have a connection to them.
With the cluster that publishes messages I have more of a problem because I need a direct connection in my service to each of the nodes so I will be able to separate the load between the nodes.
So my problem is, how do I create connections in my service to all the nodes in the cluster so they will all be used for declaring queues and sending messages?
Now, after I will have some sort of solution to this issue, the next issue will be what happens when a new node is added to the cluster? How can I create a connection to it and start using it as well?
I am using Rabbitmq - 3.7.9, Spring - 2.0.5, Spring AMQP - 2.0.5
Thanks alot!
There is currently no mechanism to do anything like that.
By default, Spring AMQP opens only one connection (optionally two, one for publishing, one for consuming).
Even when using CacheMode.CONNECTION, you'll get a new connection for each consumer (and connections will be created and cached on demand for producers), you won't get any control as to which node it connects to; that's a function of the LB.
The framework does provide the LocalizedQueueConnectionFactory which will try to consume from the node that hosts a queue, but it won't work with a load balancer in place.
In general, however, such optimization is rarely needed.
Are you trying to solve an actual problem you are experiencing now, or something that you perceive that might be a problem?
It is generally best not to perform premature optimization.
I want to use Spring Batch remote partitioning to handle large workloads on the cloud, and spin up/shutdown VMs on demand.
However, when configuring the slave steps, I'm using the StepExecutionRequestHandler to handle the step requests from a JMS queue. Right now the application just hangs. How can I shut down the application after the queue is depleted?
How can I shut down the application after the queue is depleted?
In a remote partitioning setup, workers are listeners on a queue on which StepExecutionRequests are coming. The question is how to know, from the listener point of view, that the queue is depleted? This is a tricky design problem. There are some known solutions like the "End-Of-Stream" message or "Poison" record but those are tricky too since you have to make sure all listeners get one such message.
If you are using Spring Cloud Task to launch your workers, you can use the DeployerPartitionHandler which provides an elegant way to dynamically create workers on demand up to a maximum configurable number. You can find more details about it here: https://docs.spring.io/spring-cloud-task/docs/2.0.0.RELEASE/reference/htmlsingle/#batch-partitioning and an example in this github repo: https://github.com/mminella/scaling-demos/blob/master/partitioned-demo/src/main/java/io/spring/batch/partitiondemo/configuration/BatchConfiguration.java#L75
The ice on the cake is that this is based on Spring Cloud Deployer which means you can use it on any cloud provider that implements the SCD SPI. Here is how to do it for:
Kubernetes: https://docs.spring.io/spring-cloud-task/docs/2.0.0.RELEASE/reference/htmlsingle/#_notes_on_developing_a_batch_partitioned_application_for_the_kubernetes_platform
cloud foundry: https://docs.spring.io/spring-cloud-task/docs/2.0.0.RELEASE/reference/htmlsingle/#_notes_on_developing_a_batch_partitioned_application_for_the_cloud_foundry_platform
I have messages coming in from Kafka. So I am planning to write a listener and "onMessage". I want to process it and push it in to solr.
So my question is more architectural, like I have worked on web apps all my career, so in big data how to deploy the spring kafka listener, so I can process thousands of messages a second.
How do I make my spring code use multiple nodes to distribute the
load?
I am planning to write a SpringBoot application to run in
a tomcat container.
If you use the same group id for all instances, different partitions will be assigned to different consumers (instances of your application).
So, be sure that you specified enough partitions in the topic you are going to consume.