How to implement multiple applications to consume from the same topic at the same time? - jms

I have the following applications:
Publisher Server --> Publish {name} to topic name
Consumer Server A --> Consumes topic name, prints "Hello, " + name
Consumer Server B --> Consumes topic name, prints "Good bye, " + name
I'm using Spring Boot 2.7 with Spring JMS and ActiveMQ.
The expected result:
For each name published, both consumer servers consume the data, so the expected result for each published message should be (IE publish name John and then Jane):
Hello, John
Good bye, John
Hello, Jane
Good bye, Jane
The actual result:
Hello, John
Good bye, Jane
Meaning, the two consumers are alternating to consume from the topic.
The code:
Each server has the same application.yml configuration to connect with ActiveMQ server.
Publisher Server - publisher code:
#RequiredArgsConstructor
#Component
public class Publisher {
private final JmsTemplate jmsTemplate;
public void publish(final String name) {
jmsTemplate.convertAndSend("name-topic", name);
}
}
Consumer Server A - Subscriber code
#RequiredArgsConstructor
#Component
public class SubscriberA {
#JmsListener(destination = "name-topic")
void welcome(final String name) {
System.out.println("Hello, " + name);
}
}
Consumer ServerB - Subscriber code
#RequiredArgsConstructor
#Component
public class SubscriberB {
#JmsListener(destination = "name-topic")
void welcome(final String name) {
System.out.println("Good bye, " + name);
}
}
I tried to use the different properties of the #JmsListener like the concurrency property, with no luck.

I found the solution.
By default, Spring JMS configuration is set to work with Queues instead of Topics.
Creating the necessary configuration beans and setting the .setPubSubDomain(true); fixed the problem.
Changes I needed to implement:
Add JMS configuration:
#RequiredArgsConstructor
#Configuration
#EnableJms
public class JmsConfiguration {
private final ConnectionFactory connectionFactory;
#Bean
public JmsTemplate jmsTemplate() {
// To publish messages as topics, setPubSubDomain to true
final JmsTemplate jmsTemplate = new JmsTemplate();
jmsTemplate.setPubSubDomain(true);
jmsTemplate.setConnectionFactory(connectionFactory);
return jmsTemplate;
}
#Bean
public DefaultJmsListenerContainerFactory defaultJmsListenerContainerFactory() {
// To subscribe to topics instead of queues, setPubSubDomain to true
final DefaultJmsListenerContainerFactory defaultJmsListenerContainerFactory = new DefaultJmsListenerContainerFactory();
defaultJmsListenerContainerFactory.setConnectionFactory(connectionFactory);
defaultJmsListenerContainerFactory.setSessionTransacted(true);
defaultJmsListenerContainerFactory.setPubSubDomain(true);
return defaultJmsListenerContainerFactory;
}
}
Change the #JmsListener(s), adding the created ContainerFactory
#RequiredArgsConstructor
#Component
public class SubscriberA {
#JmsListener(destination = "name-topic", containerFactory = "defaultJmsListenerContainerFactory")
void welcome(final String name) {
System.out.println("Hello, " + name);
}
}
#RequiredArgsConstructor
#Component
public class SubscriberB {
#JmsListener(destination = "name-topic", containerFactory = "defaultJmsListenerContainerFactory")
void welcome(final String name) {
System.out.println("Good bye, " + name);
}
}

Related

Spring RabbitMQ: one #RabbitListener multiple queues with the priority of the specified queue

I have 2 queues - PRIORITY_QUEUE and SIMPLE_QUEUE. The SIMPLE_QUEUE is large, the number of messages in it reaches several thousand, the PRIORITY_QUEUE can be empty and only sometimes receive a few messages. I need to ensure that messages from the SIMPLE_QUEUE are not read while there are messages in the PRIORITY_QUEUE, and if there are messages in the priority queue, reading the SIMPLE_QUEUE is blocked/paused. When the PRIORITY_QUEUE has received a message, we must pause reading from the SIMPLE_QUEUE.
prefetch property: we read messages by one
spring.rabbitmq.listener.simple.prefetch=1
listener example
#RabbitListener(queues = {priorityQueue , simpleQueue})
public void processMyQueue(String message) {
// if priorityQueue is not empty we shouldn't consume messages from simpleQueue
}
rabbit configuration
#Slf4j
#Configuration
#PropertySource({"...,..."})
#RequiredArgsConstructor
public class RabbitConfiguration {
#Value("${spring.rabbitmq.host}")
private String queueHost;
#Value("${spring.rabbitmq.username}")
private String username;
#Value("${spring.rabbitmq.password}")
private String password;
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory cachingConnectionFactory = new
CachingConnectionFactory(queueHost);
cachingConnectionFactory.setUsername(username);
cachingConnectionFactory.setPassword(password);
return cachingConnectionFactory;
}
#Bean
public AmqpAdmin amqpAdmin() {
return new RabbitAdmin(connectionFactory());
}
#Bean
public RabbitTemplate rabbitTemplate() {
return new RabbitTemplate(connectionFactory());
}
#Bean(value = "simpleQueue")
public Queue simpleQueue() {
return new Queue("simpleQueue");
}
#Bean(name = "priorityQueue")
public Queue priorityQueue() {
return new Queue("priorityQueue");
}
}
You can't do it with a single listener; use two listeners and stop the simple listener container when the priority listener receives any message; then start the simple queue listener when the priority listener goes idle.
You can use a ListenerContainerIdleEvent to detect an idle listener.
https://docs.spring.io/spring-amqp/docs/current/reference/html/#consumer-events
Use the RabbitListenerEndpointRegistry to stop/start containers.

Using manual commit with multiple message consumer

I'm very new with Kafka.
Using spring-boot kafka, I developed a publisher and a consumer using one Message object and manual ack. My code uses spring annotation. That's works perfectly.
Now, when I connect to production brokers, this one not send one Message but a list of message.
My listener method has the following signature:
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload Message message, Acknowledgment ack)
so I can acknowledge each Message. Good.
But now it's seems I must replace it with
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload List<Message> messages, Acknowledgment ack)
Even following the documentation it seems that I should use
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload List<Message> messages, Acknowledgment ack, Consumer<?,?> consumer)
Should I set batchmode to true ?
Now the question is : how can I acknowledge each message when this one has been completely handled?
Many many thanks for your help
Something like this one can help you either if you do want to manually commit offset.
If you do not want it then switch setAckMode to other value.
Here's this thing done the spring-way.
CoreAutoConfiguration class:
#Configuration
#Import({KafkaAutoConfiguration.class})
public class CoreAutoConfiguration {
#Bean("batchKafkaListenerContainerFactory")
public ConcurrentKafkaListenerContainerFactory<?, ?> batchKafkaListenerContainerFactory(ConcurrentKafkaListenerContainerFactoryConfigurer configurer, ConsumerFactory<Object, Object> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory);
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
factory.setBatchListener(true);
return factory;
}
}
Then there goes your Config class:
#Configuration
#Import({
CoreAutoConfiguration.class,
KafkaAutoConfiguration.class,
})
#EnableKafka
#EnableRetry
public class Config {
}
Finally the consumer:
#KafkaListener(
topics = "MessagesTopic",
containerFactory = "batchKafkaListenerContainerFactory"
)
public void dataReceived(#Payload List<String> payload) throws RuntimeException {
yourService.processIncomingData(payload);
}
And lastly, the properties:
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=helloworld
spring.kafka.listener.type=batch
spring.kafka.consumer.enable-auto-commit=false
# this is size of incoming list if broker has this many entries, can be lower eventually
spring.kafka.consumer.max-poll-records=100
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer

AWS SQS (queue) with Spring Boot - performance issues

I have a service that reads all messages from AWS SQS.
#Slf4j
#Configuration
#EnableJms
public class JmsConfig {
private SQSConnectionFactory connectionFactory;
public JmsConfig(
#Value("${amazon.sqs.accessKey}") String awsAccessKey,
#Value("${amazon.sqs.secretKey}") String awsSecretKey,
#Value("${amazon.sqs.region}") String awsRegion,
#Value("${amazon.sqs.endpoint}") String awsEndpoint) {
connectionFactory = new SQSConnectionFactory(
new ProviderConfiguration(),
AmazonSQSClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(
new BasicAWSCredentials(awsAccessKey, awsSecretKey)))
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(awsEndpoint, awsRegion))
.build());
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory =
new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(this.connectionFactory);
factory.setDestinationResolver(new DynamicDestinationResolver());
factory.setConcurrency("3-10");
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
factory.setReceiveTimeout(2000L); //??????????
return factory;
}
#Bean
public JmsTemplate defaultJmsTemplate() {
return new JmsTemplate(this.connectionFactory);
}
I've heard about long polling so I wonder how I could use it in my case. I wonder how this listener works - I do not want to create unnecessary calls to the AWS SQS.
My listener that reads messages and converts them to the Object and saves on Redis db:
#JmsListener(destination = "${amazon.sqs.destination}")
public void receive(String requestJSON) throws JMSException {
log.info("Received");
try {
Trace trace = Trace.fromJSON(requestJSON);
traceRepository.save(trace);
(...)
I'd like to know your opinions - what is the best approach to minimalize unnecessary calls to SQS to get messages.
Maybe shoud I use for example
factory.setReceiveTimeout(2000L);
Unfortunately there is too little information in Internet about it
Thanks,
Matthew

How to read pending messages from an ActiveMQ queue in Spring Boot

I like to read pending (not acknowledged) messages in a ActiveMQ queue using Spring boot. How to do that?
So far I can read a message the moment it is send to the queue:
#JmsListener(destination = "LOCAL.TEST",
containerFactory = "myJmsListenerContainerFactory")
public void receiveMessage(final Message jsonMessage) throws JMSException {
String messageData = null;
// jsonMessage.acknowledge(); // dont consume message (for testing)
LOGGER.info("=== Received message {}", jsonMessage);
}
using a standard configuration for the mq-connection:
#Bean
public ActiveMQConnectionFactory getActiveMQConnectionFactory() {
ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory();
activeMQConnectionFactory.setBrokerURL(BROKER_URL + ":" + BROKER_PORT);
return activeMQConnectionFactory;
}
and a standard ListenerContainerFactory:
#Bean
public DefaultJmsListenerContainerFactory myJmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(getActiveMQConnectionFactory());
factory.setConcurrency("1-1");
return factory;
}
But this just loggs a message if I manually send one using
#Autowired
private JmsTemplate jmsTemplate;
public void send(String destination, String message) {
LOGGER.info("sending message='{}' to destination='{}'", message, destination);
jmsTemplate.convertAndSend(destination, message);
}
with the standard template
#Bean
public JmsTemplate jmsTemplate() {
JmsTemplate template = new JmsTemplate();
template.setConnectionFactory(getActiveMQConnectionFactory());
return template;
}
I cannot read messages sent earlier that are still in the Queue (since I didn't .acknowledge() them)...
JMS supports "browsing" messages which appears to be the functionality you want. You should therefore change your Spring application to use a QueueBrowser instead of actually consuming the messages.
Messages won't be resent if not acknowledged. They are not returned to the queue until the session is closed or the connection lost, for example by stopping (and restarting) the listener container created by the factory.
You can access the container using the JmsListenerEndpointRegistry bean (or stop/start the entire registry which will stop/start all of its containers).
To read all pending messages, you can do like this
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61616?jms.redeliveryPolicy.maximumRedeliveries=1");
Connection connection = connectionFactory.createConnection("admin", "admin");
connection.start();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination destination = session.createQueue("listenerQueue");
MessageConsumer consumer = session.createConsumer(destination);
QueueBrowser browser = session.createBrowser((Queue) destination);
Enumeration elems = browser.getEnumeration();
while (elems.hasMoreElements()) {
Message message = (Message) consumer.receive();
if (message instanceof TextMessage) {
TextMessage textMessage = (TextMessage) message;
System.out.println("Incoming Message: '" + textMessage.getText() + "'");
message.acknowledge();
}
}
connection.close();
Step by step implementation of Spring boot ActiveMQ. Lets write some code to make it more clear. This will help to read all pending messages in current session only.
Add these dependencies in pom.xml file.
<!-- Dependencies to setup JMS and active mq environment -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-activemq</artifactId>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-broker</artifactId>
</dependency>
Add #EnableJms into your main controller where your main() method exists.
Create connection factory by adding these 2 methods in application controller only.
#Bean
public JmsListenerContainerFactory<?> myFactory(
ConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
logger.info("configuring jms connection factory....");
// anonymous class
factory.setErrorHandler(
new ErrorHandler() {
#Override
public void handleError(Throwable t) {
logger.error("An error has occurred in the transaction", t);
}
});
// lambda function
factory.setErrorHandler(t -> logger.info("An error has occurred in the transaction"));
configurer.configure(factory, connectionFactory);
return factory;
}
// Serialize message content to json using TextMessage
#Bean
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
Mention credentials in in application.yml file as
spring.activemq.user=admin
spring.activemq.password=admin
spring.activemq.broker-url=tcp://localhost:61616?jms.redeliveryPolicy.maximumRedeliveries=1
Autowire jmsTemplate in any spring bean class.
#Autowired
private JmsTemplate jmsTemplate;
Now it is time to send message to a queue.
jmsTemplate.convertAndSend("anyQueueName", "value1");
jmsTemplate.convertAndSend("anyQueueName", "value2");
...
Add a jmslistener. This method will be called automatically by JMS when any message will be pushed to queue.
#JmsListener(destination ="anyQueueName", containerFactory = "myFactory")
public void receiveMessage(String user) {
System.out.println("Received <" + user + ">");
}
Manually you can read the messages available in queue:-
import javax.jms.TextMessage;
import javax.jms.QueueBrowser;
import javax.jms.Session;
import javax.jms.TextMessage;
public void readMessageFromQueue(){
jmsTemplate.browse("anyQueueName", new BrowserCallback<TextMessage>() {
#Override
public TextMessage doInJms(Session session, QueueBrowser browser) throws JMSException {
Enumeration<TextMessage> messages = browser.getEnumeration();
while (messages.hasMoreElements()) {
System.out.println("message found : -"+ messages.nextElement().getText());
}
}
});
}
Output :-
message found :- value1
message found :- value2
-Happy Coding

RabbitListener annotation queue name by ConfigurationProperties

I have configured my rabbit properties via application.yaml and spring configurationProperties.
Thus, when I configure exchanges, queues and bindings, I can use the getters of my properties
#Bean Binding binding(Queue queue, TopicExchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(properties.getQueue());
}
#Bean Queue queue() {
return new Queue(properties.getQueue(), true);
}
#Bean TopicExchange exchange() {
return new TopicExchange(properties.getExchange());
}
However, when I configure a #RabbitListener to log the messages on from the queue, I have to use the full properties name like
#RabbitListener(queues = "${some.long.path.to.the.queue.name}")
public void onMessage(
final Message message, final Channel channel) throws Exception {
log.info("receiving message: {}#{}", message, channel);
}
I want to avoid this error prone hard coded String and refer to the configurationProperties bean like:
#RabbitListener(queues = "${properties.getQueue()}")
I had a similar issue once with #EventListener where using a bean reference "#bean.method()" helped, but it does not work here, the bean expression is just interpreted as queue name, which fails because a queue namde "#bean...." does not exist.
Is it possible to use ConfigurationProperty-Beans for RabbitListener queue configuration?
Something like this worked for me where I just used the Bean and SpEL.
#Autowired
Queue queue;
#RabbitListener(queues = "#{queue.getName()}")
I was finally able to accomplish what we both desired to do by taking what #David Diehl suggested, using the bean and SpEL; however, using MyRabbitProperties itself instead. I removed the #EnableConfigurationProperties(MyRabbitProperties.class) in the config class, and registered the bean the standard way:
#Configuration
//#EnableConfigurationProperties(RabbitProperties.class)
#EnableRabbit
public class RabbitConfig {
//private final MyRabbitProperties myRabbitProperties;
//#Autowired
//public RabbitConfig(MyRabbitProperties myRabbitProperties) {
//this.myRabbitProperties = myRabbitProperties;
//}
#Bean
public TopicExchange myExchange(MyRabbitProperties myRabbitProperties) {
return new TopicExchange(myRabbitProperties.getExchange());
}
#Bean
public Queue myQueueBean(MyRabbitProperties myRabbitProperties) {
return new Queue(myRabbitProperties.getQueue(), true);
}
#Bean
public Binding binding(Queue myQueueBean, TopicExchange myExchange, MyRabbitProperties myRabbitProperties) {
return BindingBuilder.bind(myQueueBean).to(myExchange).with(myRabbitProperties.getRoutingKey());
}
#Bean
public MyRabbitProperties myRabbitProperties() {
return new MyRabbitProperties();
}
}
From there, you can access the get method for that field:
#Component
public class RabbitQueueListenerClass {
#RabbitListener(queues = "#{myRabbitProperties.getQueue()}")
public void processMessage(Message message) {
}
}
#RabbitListener(queues = "#{myQueue.name}")
Listener:
#RabbitListener(queues = "${queueName}")
application.properties:
queueName=myQueue

Resources