How to set Durable Subscriber in DefaultMessageListenerContainer in spring? - jms

Producer of the message is not sending message as persistent and when i am trying to consume the message through MessageListener, and any exception(runtime) occurs, it retries for specific number of times (default is 6 from AMQ side) and message get lost.
Reason is that since producer is not setting the Delivery mode as Persistent, after certain number of retry attempt, DLQ is not being created and message does not move to DLQ. Due to this , i lost the message.
My Code is like this :-
#Configuration
#PropertySource("classpath:application.properties")
public class ActiveMqJmsConfig {
#Autowired
private AbcMessageListener abcMessageListener;
public DefaultMessageListenerContainer purchaseMsgListenerforAMQ(
#Qualifier("AMQConnectionFactory") ConnectionFactory amqConFactory) {
LOG.info("Message listener for purchases from AMQ : Starting");
DefaultMessageListenerContainer defaultMessageListenerContainer =
new DefaultMessageListenerContainer();
defaultMessageListenerContainer.setConnectionFactory(amqConFactory);
defaultMessageListenerContainer.setMaxConcurrentConsumers(4);
defaultMessageListenerContainer
.setDestinationName(purchaseReceivingQueueName);
defaultMessageListenerContainer
.setMessageListener(abcMessageListener);
defaultMessageListenerContainer.setSessionTransacted(true);
return defaultMessageListenerContainer;
}
#Bean
#Qualifier(value = "AMQConnectionFactory")
public ConnectionFactory activeMQConnectionFactory() {
ActiveMQConnectionFactory amqConnectionFactory =
new ActiveMQConnectionFactory();
amqConnectionFactory
.setBrokerURL(System.getProperty(tcp://localhost:61616));
amqConnectionFactory
.setUserName(System.getProperty(admin));
amqConnectionFactory
.setPassword(System.getProperty(admin));
return amqConnectionFactory;
}
}
#Component
public class AbcMessageListener implements MessageListener {
#Override
public void onMessage(Message msg) {
//CODE implementation
}
}
Problem :- By setting the client-id at connection level (Connection.setclientid("String")), we can subscribe as durable subscriber even though message is not persistent. By doing this, if application throws runtime exception , after a certain number of retry attempt, DLQ will be created for the Queue and message be moved to DLQ.
But in DefaultMessageListenerContainer, connection is not exposed to client. it is maintained by Class itself as a pool, i guess.
How can i achieve the durable subscription in DefaultMessageListenerContainer?

You can set the client id on the container instead:
/**
* Specify the JMS client ID for a shared Connection created and used
* by this container.
* <p>Note that client IDs need to be unique among all active Connections
* of the underlying JMS provider. Furthermore, a client ID can only be
* assigned if the original ConnectionFactory hasn't already assigned one.
* #see javax.jms.Connection#setClientID
* #see #setConnectionFactory
*/
public void setClientId(#Nullable String clientId) {
this.clientId = clientId;
}
and
/**
* Set the name of a durable subscription to create. This method switches
* to pub-sub domain mode and activates subscription durability as well.
* <p>The durable subscription name needs to be unique within this client's
* JMS client id. Default is the class name of the specified message listener.
* <p>Note: Only 1 concurrent consumer (which is the default of this
* message listener container) is allowed for each durable subscription,
* except for a shared durable subscription (which requires JMS 2.0).
* #see #setPubSubDomain
* #see #setSubscriptionDurable
* #see #setSubscriptionShared
* #see #setClientId
* #see #setMessageListener
*/
public void setDurableSubscriptionName(#Nullable String durableSubscriptionName) {
this.subscriptionName = durableSubscriptionName;
this.subscriptionDurable = (durableSubscriptionName != null);
}

Related

Caused by: java.net.SocketException: Connection reset by peer: socket write error

I'm trying to connect to rabbitMQ over SSL using Spring Boot 2.7.4 and java 11.0.14 I was following this example here:
I have added the following configurations:
properties file:
# RabbitMQ Server configuration file.
rabbit.username=admin
rabbit.password=admin
rabbit.host=localhost
rabbit.port=5671
rabbit.ssl=TLSv1.2
rabbit.keystore.name=client_key.p12
rabbit.keystore.password=rabbitstore
rabbit.truststore=server_store.jks
rabbit.truststore.password=rabbitstore
client_key.p12 and server_store.jks are in my classpath.
Configuration Class:
#Configuration
#PropertySource("classpath:rabbit.properties")
public class RabbitConfiguration {
/**
* Default sample channel name to respond for requests from clients.
*/
public static final String DEFAULT_QUEUE = "sample_queue";
/**
* Environment properties file from rabbitmq configuration.
*/
#Autowired
private Environment env;
/**
* Establish a connection to a rabbit mq server.
* #return Rabbit connection factory for rabbitmq access.
* #throws IOException If wrong parameters are used for connection.
*/
#Bean
public RabbitConnectionFactoryBean connectionFactoryBean() throws IOException {
RabbitConnectionFactoryBean connectionFactoryBean = new RabbitConnectionFactoryBean();
connectionFactoryBean.setHost(Objects.requireNonNull(env.getProperty("rabbit.host")));
connectionFactoryBean.setPort(Integer.parseInt(Objects.requireNonNull(env.getProperty("rabbit.port"))));
connectionFactoryBean.setUsername(Objects.requireNonNull(env.getProperty("rabbit.username")));
connectionFactoryBean.setPassword(Objects.requireNonNull(env.getProperty("rabbit.password")));
// SSL-Configuration if set
if(env.getProperty("rabbit.ssl") != null) {
connectionFactoryBean.setUseSSL(true);
connectionFactoryBean.setSslAlgorithm(Objects.requireNonNull(env.getProperty("rabbit.ssl")));
// This information should be stored safely !!!
connectionFactoryBean.setKeyStore(Objects.requireNonNull(env.getProperty("rabbit.keystore.name")));
connectionFactoryBean.setKeyStorePassphrase(Objects.requireNonNull(env.getProperty("rabbit.keystore.password")));
connectionFactoryBean.setTrustStore(Objects.requireNonNull(env.getProperty("rabbit.truststore")));
connectionFactoryBean.setTrustStorePassphrase(Objects.requireNonNull(env.getProperty("rabbit.truststore.password")));
}
return connectionFactoryBean;
}
/**
* Connection factory which established a rabbitmq connection used from a connection factory
* #param connectionFactoryBean Connection factory bean to create connection.
* #return A connection factory to create connections.
* #throws Exception If wrong parameters are used for connection.
*/
#Bean(name = "GEO_RABBIT_CONNECTION")
public ConnectionFactory connectionFactory(RabbitConnectionFactoryBean connectionFactoryBean) throws Exception {
return new CachingConnectionFactory(Objects.requireNonNull(connectionFactoryBean.getObject()));
}
/**
* Queue initialization from rabbitmq to listen a queue.
* #return An queue to listen for listen receiver.
*/
#Bean
public Queue queue() {
// Create an new queue to handle incoming responds
return new Queue(DEFAULT_QUEUE, false, false, false, null);
}
/**
* Generates a simple message listener container.
* #param connectionFactory Established connection to rabbitmq server.
* #param listenerAdapter Listener event adapter to listen for messages.
* #return A simple message container for listening for requests.
*/
#Bean
public SimpleMessageListenerContainer container(ConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(DEFAULT_QUEUE);
container.setMessageListener(listenerAdapter);
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
return container;
}
/**
* Message listener adapter to generate a message listener.
* #param deviceMonitoringReceiver Device receive to for listening.
* #return A message listener adapter to receive messages.
*/
#Bean
public MessageListenerAdapter listenerAdapter(DeviceMonitoringReceiver deviceMonitoringReceiver) {
return new MessageListenerAdapter(deviceMonitoringReceiver, "receiveMessage");
}
}
Also I have updated rabbitMQ configurations:
[
{rabbit, [
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile, "D:\\tls-gen\\basic\\result\\ca_certificate.pem"},
{certfile, "D:\\tls-gen\\basic\\result\\server_seliiwvdec53152_certificate.pem"},
{keyfile, "D:\\tls-gen\basic\\result\\server_seliiwvdec53152_key.pem"},
{verify, verify_peer},
{fail_if_no_peer_cert, true}]}
]}
].
But the application is not starting and throwing
Caused by: java.net.SocketException: Connection reset by peer: socket write error
I resolved the issue by adding this to the configurations:
ssl_options.password = xxx
It's mentioned in the official documentation it's optional I don't know why. But whatever the issue is now resolved.

Consumer thread state when no messages are available in the given queue in rabbitmq

I am just curious about the consumer state when no data is published to queue where the consumer is linked
#Component
public class ABC {
#RabbitListener(queues = RabbitQueueName,containerFactory = RABBIT_LISTENER_CONTAINER_FACTORY, id =SomeId)
public void onMessage(String message) {
try {
//business logic
} catch (Exception e) {
//some logger
}
}
}
Any link which explains this is more useful
Thanks in Advance
See ListenerContainerIdleEvent:
/**
* An event that is emitted when a container is idle if the container
* is configured to do so.
*
* #author Gary Russell
* #since 1.6
*
*/
#SuppressWarnings("serial")
public class ListenerContainerIdleEvent extends AmqpEvent {
Docs are here: https://docs.spring.io/spring-amqp/docs/current/reference/html/#idle-containers
It depends on the container type.
With the SimpleMessageListenerContainer (default), the amqp-client pushes new deliveries into a BlockingQueue; the listener thread polls that queue for new messages according to the receiveTimeout (default 1 second). If you turn on TRACE logging, you will see that polling.
With the DirectMessageListenerContainer, the listener is called directly by the amqp-client thread whenever a new message arrives.

Multithreaded Executor channel to speed up the consumer process

I have a message producer which produces around 15 messages/second
The consumer is a spring integration project which consumes from the Message Queue and does a lot of processing. Currently it is single threaded and not able to match with the rate at which the producer are sending the messages. hence the queue depth keeps on increasing
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(Jms.container(this.emsConnectionFactory, this.emsQueue).get()))
.wireTap(FLTAWARE_WIRE_TAP_CHNL)// push raw fa data
.filter(ingFilter, "filterMessageOnEvent").transform(eventHandler, "parseEvent")
.aggregate(a -> a.correlationStrategy(corStrgy, "getCorrelationKey").releaseStrategy(g -> {
boolean eonExists = g.getMessages().stream()
.anyMatch(eon -> ((FlightModel) eon.getPayload()).getEstGmtOnDtm() != null);
if (eonExists) {
boolean einExists = g.getMessages().stream()
.anyMatch(ein -> ((FlightModel) ein.getPayload()).getEstGmtInDtm() != null);
if (einExists) {
return true;
}
}
return false;
}).messageStore(this.messageStore)).channel("AggregatorEventChannel").get();
is it possible to use executor channel to process this in a multithreaded environment and speed up the consumer process
If yes, please suggest how can i achieve - To ensure ordering of the messages I need to assign the messages of same type (based on the id of the message) to the same thread of the executor channel.
[UPDATED CODE]
I have created the below executor channels
public static final MessageChannel SKW_DEFAULT_CHANNEL = MessageChannels
.executor(ASQ_DEFAULT_CHANNEL_NAME, Executors.newFixedThreadPool(1)).get();
public static final MessageChannel RPA_DEFAULT_CHANNEL = MessageChannels
.executor(ASH_DEFAULT_CHANNEL_NAME, Executors.newFixedThreadPool(1)).get();
Now from the main message flow I redirected to a custom router which forwards the message to Executor channel as shown below -
#Bean
public IntegrationFlow baseEventFlow1() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(Jms.container(this.emsConnectionFactory, this.emsQueue).get()))
.wireTap(FLTAWARE_WIRE_TAP_CHNL)// push raw fa data
.filter(ingFilter, "filterMessageOnEvent").route(route()).get();
}
public AbstractMessageRouter router() {
return new AbstractMessageRouter() {
#Override
protected Collection<MessageChannel> determineTargetChannels(Message<?> message) {
if (message.getPayload().toString().contains("\"id\":\"RPA")) {
return Collections.singletonList(RPA_DEFAULT_CHANNEL);
} else if (message.getPayload().toString().contains("\"id\":\"SKW")) {
return Collections.singletonList(SKW_DEFAULT_CHANNEL);
} else {
return Collections.singletonList(new NullChannel());
}
}
};
}
I will have individual consumer flow for the corresponding executor channel.
Please correct my understaning
[UPDATED]
#Bean
#BridgeTo("uaxDefaultChannel")
public MessageChannel ucaDefaultChannel() {
return MessageChannels.executor(UCA_DEFAULT_CHANNEL_NAME, Executors.newFixedThreadPool(1)).get();
}
#Bean
#BridgeTo("uaDefaultChannel")
public MessageChannel ualDefaultChannel() {
return MessageChannels.executor(UAL_DEFAULT_CHANNEL_NAME, Executors.newFixedThreadPool(1)).get();
}
#Bean
public IntegrationFlow uaEventFlow() {
return IntegrationFlows.from("uaDefaultChannel").wireTap(UA_WIRE_TAP_CHNL)
.transform(eventHandler, "parseEvent")
}
So BridgeTo on the executor channel will forward the messages
hence the queue depth keeps on increasing
Since it looks like your queue is somewhere on JMS broker that is really OK to have such a behavior. That's exactly for what messaging systems have been designed - to distinguish producer and consumer and deal with messages in a destination whenever it is possible.
if you want to increase a polling from JMS, you can consider to have a concurrency option on the JMS container:
/**
* The concurrency to use.
* #param concurrency the concurrency.
* #return current {#link JmsDefaultListenerContainerSpec}.
* #see DefaultMessageListenerContainer#setConcurrency(String)
*/
public JmsDefaultListenerContainerSpec concurrency(String concurrency) {
this.target.setConcurrency(concurrency);
return this;
}
/**
* The concurrent consumers number to use.
* #param concurrentConsumers the concurrent consumers count.
* #return current {#link JmsDefaultListenerContainerSpec}.
* #see DefaultMessageListenerContainer#setConcurrentConsumers(int)
*/
public JmsDefaultListenerContainerSpec concurrentConsumers(int concurrentConsumers) {
this.target.setConcurrentConsumers(concurrentConsumers);
return this;
}
/**
* The max for concurrent consumers number to use.
* #param maxConcurrentConsumers the max concurrent consumers count.
* #return current {#link JmsDefaultListenerContainerSpec}.
* #see DefaultMessageListenerContainer#setMaxConcurrentConsumers(int)
*/
public JmsDefaultListenerContainerSpec maxConcurrentConsumers(int maxConcurrentConsumers) {
this.target.setMaxConcurrentConsumers(maxConcurrentConsumers);
return this;
}
See more info the Docs: https://docs.spring.io/spring/docs/5.2.3.RELEASE/spring-framework-reference/integration.html#jms-receiving
But that won't allow you to "asign messages to the specific thread". There is just like no way to partition in JMS.
We can do that with Spring Integration using router according your "based on the id of the message" and particular ExecutorChannel instances configured with a singled-threaded Executor. Every ExecutorChannel is going to be its dedicated executor with only single thread. This way you will ensure an order for messages with the same partition key and you'll process them in parallel. All the ExecutorChannel can have the same subscriber or bridge to the same channel for processing.
However you need to keep in mind that when you are leaving JMS listener thread, you finish JMS transaction and you fail to process a message in that separate thread you may lose a message.

Set message priority using Spring Boot, JMS and ActiveMq

I'm trying to send messages with different priority with the JmsTemplate using Spring Boot and ActiveMQ and it's not working.
I tried this :
MessageCreator mc = session -> {
TextMessage tm = session.createTextMessage("hello");
tm.setJMSPriority(6);
return tm;
};
jmsTemplate.send((Queue) () -> "box", mc);
The priority inside the ActiveMQ broker is still 4 (default value).
The only way I found to actually change the priority for a message is by change the priority at the JmsTemplate level.
jmsTemplate.setPriority(3);
The problem here is, now, all messages sent after that will have the priority 3.
I know I can reset the JmsTemplate priority after each send, but it's not "clean" and what about concurrency?
How can I set the priority for each message and get the message with the highest priority using #JmsListener?
I just ran into the same issue.
I tested your point about setting the priority of the jmsTemplate and your assumption is correct. It is not handled properly with concurrency.
The solution I found that works (albeit not ideal) is to extend JmsTemplate and override the doSend method to copy the JmsPriority from the message to the producer. This isnt ideal, extending the class make break over spring boot releases (I've tested this on 2.1.7) and there are some additional steps to register the new JmsTemplate. But it does work and I've tested it under load.
Steps.....
Create a new class that extends JmsTemplate overriding the doSend method to copy the priority from the message
import java.io.Serializable;
import javax.jms.ConnectionFactory;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageProducer;
import org.springframework.jms.core.JmsTemplate;
public class RcsJmsTemplate extends JmsTemplate implements Serializable {
public RcsJmsTemplate() {
}
public RcsJmsTemplate(ConnectionFactory connectionFactory) {
super(connectionFactory);
}
/**
* Actually send the given JMS message.
*
* AF: EXTENDED TO COPY THE PRIORITY FROM THE MESSAGE TO THE PRODUCER
*
* #param producer the JMS MessageProducer to send with
* #param message the JMS Message to send
* #throws JMSException if thrown by JMS API methods
*/
#Override
protected void doSend(MessageProducer producer, Message message) throws JMSException {
if (getDeliveryDelay() >= 0) {
producer.setDeliveryDelay(getDeliveryDelay());
}
producer.send(message, getDeliveryMode(), message.getJMSPriority(), getTimeToLive());
}
}
Add a bean (to your App.java, or appropriate config class) You may not need to pass through a message converter (Im using Jackson in my project) There may also be other configurations you need to apply to the new JmsTemplate.
#Bean
public JmsTemplate jmsTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) {
RcsJmsTemplate rcsJmsTemplate = new RcsJmsTemplate(connectionFactory);
rcsJmsTemplate.setMessageConverter(messageConverter);
return rcsJmsTemplate;
}
Then as in your question set the JmsPriority attribute of the message. Your using a MessageCreator but in my project Im using a message post processoe
public void convertAndSendWithPriority(JmsTemplate jmsTemplate, String destination, Object message, int priority) {
jmsTemplate.convertAndSend(destination, message, (Message jmsMessage) -> {
jmsMessage.setJMSPriority(priority);
return jmsMessage;
});
}
For completeness you should add the property:
spring.jms.template.qos-enabled=true
Thats it. Hope it helps
(Actually I hope someone comes up with a better answer)
Thanks

How to configure JmsListener on ActiveMQ for autoscaling using Qpid Sender

I have a kubernetes cluster with an activeMQ Artemis Queue and I am using hpa for autoscaling of micro services. The messages are send via QpidSender and received via JMSListener.
Messaging works, but I am not able to configure the Queue/Listener in a way, that autoscaling works as expacted.
This is my Qpid sender
public static void send(String avroMessage, String task) throws JMSException, NamingException {
Connection connection = createConnection();
connection.start();
Session session = createSession(connection);
MessageProducer messageProducer = createProducer(session);
TextMessage message = session.createTextMessage(avroMessage);
message.setStringProperty("task", task);
messageProducer.send(
message,
DeliveryMode.NON_PERSISTENT,
Message.DEFAULT_PRIORITY,
Message.DEFAULT_TIME_TO_LIVE);
connection.close();
}
private static MessageProducer createProducer(Session session) throws JMSException {
Destination producerDestination =
session.createQueue("queue?consumer.prefetchSize=1&heartbeat='10000'");
return session.createProducer(producerDestination);
}
private static Session createSession(Connection connection) throws JMSException {
return connection.createSession(Session.AUTO_ACKNOWLEDGE);
}
private static Connection createConnection() throws NamingException, JMSException {
Hashtable<Object, Object> env = new Hashtable<>();
env.put(Context.INITIAL_CONTEXT_FACTORY, "org.apache.qpid.jms.jndi.JmsInitialContextFactory");
env.put("connectionfactory.factoryLookup", amqUrl);
Context context = new javax.naming.InitialContext(env);
ConnectionFactory connectionFactory = (ConnectionFactory) context.lookup("factoryLookup");
PooledConnectionFactory pooledConnectionFactory = new PooledConnectionFactory();
pooledConnectionFactory.setConnectionFactory(connectionFactory);
pooledConnectionFactory.setMaxConnections(10);
return connectionFactory.createConnection(amqUsername, amqPassword);
}
This is my Listener config
#Bean
public JmsConnectionFactory jmsConnection() {
JmsConnectionFactory jmsConnection = new JmsConnectionFactory();
jmsConnection.setRemoteURI(this.amqUrl);
jmsConnection.setUsername(this.amqUsername);
jmsConnection.setPassword(this.amqPassword);
return jmsConnection;
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(jmsConnection());
return factory;
}
And here is my Listener
#JmsListener(
destination = "queue?consumer.prefetchSize=1&heartbeat='10000'",
selector = "task = 'myTask'"
)
public void receiveMsg(Message message) throws IOException, JMSException {
message.acknowledge();
doStuff();
}
I send the message like this
QpidSender.send(avroMessage, "myTask");
This setting works. I can send different messages and as soon than there are more then 2, the second instance of my service starts and consumes the message. If later the message count is below 2, the service is terminated.
The problem is: I don't want the message to be acknowledged before the doStuff(). Because if something goes wrong or if the service is terminated before finishing doStuff(), the message is lost (right?).
But if I reorder it to
doStuff();
message.acknowledge();
the second instance can not receive a message from the broker, as long as the first service is still in doStuff() and hasn't acknowledged the message.
How do I configure this in a way, that more than one instance can consume a message from the queue, but the message isn't lost, if the service gets terminated or something else fails on doStuff()?
Use factory.setSessionTransacted(true).
See the javadocs for DefaultMessageListenerContainer:
* <p><b>It is strongly recommended to either set {#link #setSessionTransacted
* "sessionTransacted"} to "true" or specify an external {#link #setTransactionManager
* "transactionManager"}.</b> See the {#link AbstractMessageListenerContainer}
* javadoc for details on acknowledge modes and native transaction options, as
* well as the {#link AbstractPollingMessageListenerContainer} javadoc for details
* on configuring an external transaction manager. Note that for the default
* "AUTO_ACKNOWLEDGE" mode, this container applies automatic message acknowledgment
* before listener execution, with no redelivery in case of an exception.

Resources