I am looking for a structure or solution that can support spring boot microservices with multiple instances, ActiveMQ Artemis and Apache Camel.
For example:
I have an ActiveMQ Artemis instance and a Spring Boot JMS consumer with instance A (on machine A) and instance B (on machine B).
Both instances (A,B) are up, but by default the instance A is the master consumer, I mean must consume the JMS message and only in case of it's down or it throw some exceptions, the instance B start consuming messages and when A is OK then it take the ball.
Nb: Instance A and B of the Spring Boot microservice are on different machine and in my case i don't have any container like docker etc...
Have you any approach to solve this issue.
I think the closest you could get to the functionality you want is by using the "exclusive queue" feature. Both consumers A & B can be active at the same time, but the broker will only send messages to one of them. If the consumer which the broker has chosen goes away for whatever reason then the broker will choose another consumer.
Related
We are setting up a cluster of 3 live nodes of Artemis 2.16.0.
They discover each other and form a cluster.
<cluster-connections>
<cluster-connection name="local-cluster">
<connector-ref>node0-connector</connector-ref>
<discovery-group-ref discovery-group-name="dg-group1"/>
</cluster-connection>
</cluster-connections>
We often have a scenario where a Spring Boot application (2-3 pods/instances of the same microservice) produces a lot of messages (let's say 1M) to a queue A.
Another Spring Boot application is consuming queue A. It uses Spring Boot Artemis autoconfiguration to connect to the cluster. It consists of 3 instances so that the consumption rate can be higher.
spring.artemis.broker-url=(tcp://localhost:61616,tcp://localhost:61617,tcp://localhost:61618)?ha=true
spring.artemis.pool.enabled=true
spring.artemis.pool.max-connections=1
The above autoconfiguration will often lead to a scenario that 2 consumers pods are connected to broker B1 and one consumer pod will be connected to broker B2. (no connection towards broker B3)
Let's assume that the producer is much faster than the consumer pods. What we observe is that the producer will start producing and messages will be delivered (ON_DEMAND) on both the brokers that have active consumer connections.
Soon (because the producer is much faster than the consumer) we observe that a "backlog" is built and more and more messages wait to be consumed on the two brokers. The producer application has completed producing (one million messages) and 300k are in B1 queue and 600K are in B2 broker queue. (100K are already consumed).
A bit later we observe that the messages from B1 queue are consumed completely and there are still (a lot of) messages to be found in B2. This is logical as there were 2 pods consuming from B1 and 1 pod consuming from B2.
At this point we would like to see a behaviour where the two consumers (currently and so far connected to B1) automatically consume messages from B2.
But what we observe is that neither the clients (spring boot consumers) nor the cluster do something about it and eventually the third pod will have to continue processing the messages by itself (the 2 pods that worked hard on consuming messages from B1 now become idle).
According to the Artemis documentation the redistribution will not take place if there is at least one connection active on the broker.
This leads to a waste of resources and lower consumption rate. Is the only solution to programmatically discover such situations and reconnect?
It seems more error prone to put the responsibility for smarter consumption (for example try to reconnect to force Artemis redistribute) in the client and I would like to ask if you know of a way to overcome this problem. Either by achieving redistribution in Artemis cluster or an alternative configuration in the Spring Boot application (client).
While doing some load tests with the ActiveMQ Artemis broker and my Spring Boot application I am getting into performance issues.
What I am doing is, sending e.g. 12,000 messages per second to the broker with JMSeter and the application receives them and saves them to a DB. That works fine. But when I extend my application by a filter mechanism, which forwards events after saving to DB, back to the broker using jmsTemplate.send(destination, messageCreator) it goes very slow.
I first used ActiveMQ 5.x and there this mechanism works fine. There you could configure the ActiveMQConnectionFactory with setAsyncSend(true) to tune performance. For the ActiveMQ Artemis ConnectionFactory implementation there is no such a possibility. Is there another way to tune performance like in ActiveMQ 5.x?
I am using Apache ActiveMQ Artemis 2.16.0 (but also tried 2.15.0), artemis-jms-client 2.6.4, and Spring Boot 1.5.16.RELEASE.
The first thing to note is that you need to be very careful when using Spring's JmsTemplate to send messages as it employs a well-known anti-pattern that can really kill performance. It will actually create a new JMS connection, session, and producer for every message it sends. I recommend you use a connection pool like this one which is based on the ActiveMQ 5.x connection pool implementation but now supports JMS 2. For additional details about the danger of using JmsTemplate see the ActiveMQ documentation. This is also discussed in an article from Pivotal (i.e. the "owners" of Spring).
The second point here is that you can tune if persistent JMS messages are sent synchronously or not using the blockOnDurableSend URL property, e.g.:
tcp://localhost:61616?blockOnDurableSend=false
This will ensure that persistent JMS messages are sent asynchronously. This is discussed further in the ActiveMQ Artemis documentation.
We have configured our ActiveMQ message broker as a Spring Boot project and there's another Spring Boot application (let's call it service-A) that has a listener configured to listen to some topics using #JmsListener annotation. It's a Spring Cloud microservice appilcation.
The problem:
It is possible that service-A can have multiple instances running.
If we have 2 instances running, then any message coming on topic gets listened to twice.
How can we avoid every instance listening to the topic?
We want to make sure that the topic is listened to only once no matte the number of service-A instances.
Is it possible to run the microservice in a cluster mode or something similar? I also checked out ActiveMQ virtual destinations but not too sure if that's the solution to the problem.
We have also thought of an approach where we can decide who's the leader node from the multiple instances, but that's the last resort and we are looking for a cleaner approach.
Any useful pointers, references are welcome.
What you really want is a shared topic subscription which was added in JMS 2. Unfortunately ActiveMQ 5.x doesn't support JMS 2. However, ActiveMQ Artemis does.
ActiveMQ Artemis is the next generation broker from ActiveMQ. It supports most of the same features as ActiveMQ 5.x (including full support for OpenWire clients) as well as many other features that 5.x doesn't support (e.g. JMS 2, shared-nothing high-availability using replication, last-value queues, ring queues, metrics plugins for integration with tools like Prometheus, duplicate message detection, etc.). Furthermore, ActiveMQ Artemis is built on a high-performance, non-blocking core which means scalability is much better as well.
I am writing a service with Spring and I am using Spring AMQP in order to connect to Rabbitmq.
I have two rabbitmq clusters, one is only for publishing messages(the messages are sent to the other cluster via the federation plugin) and the other cluster is for declaring queues that end users will consume from.
The nodes sit behind aws lb, each cluster has a lb.
I am using CachingConnectionFactory and RabbitTemplate,RabbitAdmin in my code and I want to have connections to all the nodes so I can use them.
For the cluster that will contain the queues I added to the config the queue-master-locator=random so new queues will be declared in all the nodes in the cluster even if my service does not have a connection to them.
With the cluster that publishes messages I have more of a problem because I need a direct connection in my service to each of the nodes so I will be able to separate the load between the nodes.
So my problem is, how do I create connections in my service to all the nodes in the cluster so they will all be used for declaring queues and sending messages?
Now, after I will have some sort of solution to this issue, the next issue will be what happens when a new node is added to the cluster? How can I create a connection to it and start using it as well?
I am using Rabbitmq - 3.7.9, Spring - 2.0.5, Spring AMQP - 2.0.5
Thanks alot!
There is currently no mechanism to do anything like that.
By default, Spring AMQP opens only one connection (optionally two, one for publishing, one for consuming).
Even when using CacheMode.CONNECTION, you'll get a new connection for each consumer (and connections will be created and cached on demand for producers), you won't get any control as to which node it connects to; that's a function of the LB.
The framework does provide the LocalizedQueueConnectionFactory which will try to consume from the node that hosts a queue, but it won't work with a load balancer in place.
In general, however, such optimization is rarely needed.
Are you trying to solve an actual problem you are experiencing now, or something that you perceive that might be a problem?
It is generally best not to perform premature optimization.
Our JMS Listener application connects to an ActiveMQ network of brokers through a load balancer, which we are told distributes connections amongst brokers in a round-robin fashion. Our spring boot application is creating a connection via the load balancer, which in turn feeds the connection to one of the brokers amongst the network of brokers. If a message is published to the brokers then it would be a lot quicker if the message was on the broker that the JMS listener connection lived on. However, the likelihood of that occurring is slim unless we can distribute the connections across the brokers.
I've tried increasing the concurrency in the DefaultJmsListenerContainerFactory, but that didn't do the trick. I was thinking about somehow extending the AbstractJmsListenerContainerFactory, and somehow create a Map of DefaultMessageListenerContainer instances but it looks like the createListenerContainer will only return an instance of whatever is parameterized in the AbstractJmsListenerContainerFactory and we cannot parameterize it with an instance of Map.
We are using Spring Boot 1.5.14.RELEASE.
== UPDATE ==
I've been playing around with the classes above, and it seems like it is inherent in Spring JMS that a Jms Listener be associated with a Single Message Listener Container, which in turn is associated with a single (potentially shared) connection.
For any folks that have JMS Application Listeners that are connecting to a load balanced network of brokers, are you creating a single connection that is connecting to a single broker, and if so, do you experience significant performance degradation as a result of the network of brokers having to move any inbound messages to a broker with consumers?