I'm currently trying to build a messaging application using JMS Listener and IBM MQ, and I need to ensure that I can run two instances of the same listener at the same time. However, I want to make sure that the second instance waits until the first instance has fully processed and acknowledged the message.
I'm using Spring Boot for my application, JMS Listener and IBM MQ.
Below is the my config class which is annotated with #component and #EnableJMS
public class jmsconfig{
#Bean
public MQConnectionfactory getConnectionFactory(){
MQConnectionFactory connectionFactory = new MQConnectionFactory();
connectionFactory.setQueueManagerName("AA");
connectionFactory.setExpirationTimeOut(3600000);
connectionFactory.setPerformExpiration(true);
connectionFactory.setPerformValidation(true);
connectionFactory.setPerformOptimalSizeCheck(false);
connectionFactory.setValidationTimeOut(180000);
connectionFactory.setMinIdle(3);
connectionFactory.setMaxIdle(5);
reurn connectionFactory ;
}
#Bean(name="jmsListenerContainerFactory")
public JmsListenerContainerFactory jmsListenerContainerFactory(MQConnectionfactory mqConnectionfactory){
DefaultJmsListenerContainerFactory containerFactory = new DefaultJmsListenerContainerFactory();
containerFactory.setConnectionFactor(mqConnectionfactory);
containerFactory.setSessionTransacted(true);
containerFactory.setSessionAcknowledgementMode(Session.CLIENT_ACKNOWLEDE)
reurn containerFactory ;
}
}
Listener code:
#Component
public class Receiver {
public sttaic int count = 0;
#JmsListener(destination = "${inwardQueueName}", containerFactory = "jmsListenerContainerFactory")
public void receiveMessage(javax.jms.Message message) throws javax.jms.JMSException {
String messagetxt = "";
OrderObject order = null;
if (message instanceof javax.jms.TextMessage) {
messagetxt = ((TextMessage)message).getText();
OrderObject order = //code to covert messagetxt to object
System.out.println("Message pciked up with Order Id : " +order.getOrderId)
TimeUnit.SECONDS.sleep(15);
count++;
if(count<=4){
throw new Exception("Exception occurred");
}
}
TimeUnit.SECONDS.sleep(15);
System.out.println("Message Acknowledged for orderId: " +order.getOrderId)
message.acknowledge();
}
}
I start the first instance and the push first message to MQ with orderId 1. I can see the below output on the console. It prints the below statement and then waits for 15 seconds
Message picked up with Order Id : 1
Immediately, I start the second instance and the push second message to MQ with orderId 2. I can see the below output on the console.
First Instance picks first message with orderId 2
Message Acknowledged for orderId: 2
After some time, when the first instance completes(since there is a wait of 15 sec), I can see the below output.
Message pciked up with Order Id : 1
Message pciked up with Order Id : 1
Message pciked up with Order Id : 1
Message Acknowledged for orderId: 1
With the above output, the Second instance is picking up second messages from the MQ in parallel and processing them while the first instance is still processing the first message.
Can anyone help with what is wrong with the above implementation?
Related
Working versions in the app
IBM AllClient version : 'com.ibm.mq:com.ibm.mq.allclient:9.1.1.0'
org.springframework:spring-jms : 4.3.9.RELEASE
javax.jms:javax.jms-api : 2.0.1
My requirement is that in case of the failure of a message processing due to say, consumer not being available (eg. DB is unavailable), the message remains in the queue or put back on the queue (if that is even possible). This is because the order of the messages is important, messages have to be consumed in the same order that they are received. The Java app is single-threaded.
I have tried the following
#Override
public void onMessage(Message message)
{
try{
if(message instanceOf Textmessage)
{
}
:
:
throw new Exception("Test");// Just to test the retry
}
catch(Exception ex)
{
try
{
int temp = message.getIntProperty("JMSXDeliveryCount");
throw new RuntimeException("Redlivery attempted ");
// At this point, I am expecting JMS to put the message back into the queue.
// But it is actually put into the Bakout queue.
}
catch(JMSException ef)
{
String temp = ef.getMessage();
}
}
}
I have set this in my spring.xml for the jmsContainer bean.
<property name="sessionTransacted" value="true" />
What is wrong with the code above ?
And if putting the message back in the queue is not practical, how can one browse the message, process it and, if successful, pull the message (so it is consumed and no longer on the queue) ? Is this scenario supported in IBM provider for JMS?
The IBM MQ Local queue has BOTHRESH(1).
To preserve message ordering, one approach might be to stop the message listener temporarily as part of your rollback strategy. Looking at the Spring Boot doc for DefaultMessageListenerContainer there is a stop(Runnable callback) method. I've experimented with using this in a rollback as follows.
To ensure my Listener is single threaded, on my DefaultJmsListenerContainerFactory I set containerFactory.setConcurrency("1").
In my Listener, I set an id
#JmsListener(destination = "DEV.QUEUE.2", containerFactory = "listenerTwoFactory", concurrency="1", id="listenerTwo")
And retrieve the DefaultMessageListenerContainer instance.
JmsListenerEndpointRegistry reg = context.getBean(JmsListenerEndpointRegistry.class);
DefaultMessageListenerContainer mlc = (DefaultMessageListenerContainer) reg.getListenerContainer("listenerTwo");
For testing, I check JMSXDeliveryCount and throw an exception to rollback.
retryCount = Integer.parseInt(msg.getStringProperty("JMSXDeliveryCount"));
if (retryCount < 5) {
throw new Exception("Rollback test "+retryCount);
}
In the Listener's catch processing, I call stop(Runnable callback) on the DefaultMessageListenerContainer instance and pass in a new class ContainerTimedRestart as defined below.
//catch processing here and decide to rollback
mlc.stop(new ContainerTimedRestart(mlc,delay));
System.out.println("#### "+getClass().getName()+" Unable to process message.");
throw new Exception();
ContainerTimedRestart extends Runnable and DefaultMessageListenerContainer is responsible for invoking the run() method when the stop call completes.
public class ContainerTimedRestart implements Runnable {
//Container instance to restart.
private DefaultMessageListenerContainer theMlc;
//Default delay before restart in mills.
private long theDelay = 5000L;
//Basic constructor for testing.
public ContainerTimedRestart(DefaultMessageListenerContainer mlc, long delay) {
theMlc = mlc;
theDelay = delay;
}
public void run(){
//Validate container instance.
try {
System.out.println("#### "+getClass().getName()+"Waiting for "+theDelay+" millis.");
Thread.sleep(theDelay);
System.out.println("#### "+getClass().getName()+"Restarting container.");
theMlc.start();
System.out.println("#### "+getClass().getName()+"Container started!");
} catch (InterruptedException ie) {
ie.printStackTrace();
//Further checks and ensure container is in correct state.
//Report errors.
}
}
I loaded my queue with three messages with payloads "a", "b", and "c" respectively and started the listener.
Checking DEV.QUEUE.2 on my queue manager I see IPPROCS(1) confirming only one application handle has the queue open. The messages are processed in order after each is rolled five times and with a 5 second delay between rollback attempts.
IBM MQ classes for JMS has poison message handling built in. This handling is based on the QLOCAL setting BOTHRESH, this stands for Backout Threshold. Each IBM MQ message has a "header" called the MQMD (MQ Message Descriptor). One of the fields in the MQMD is BackoutCount. The default value of BackoutCount on a new message is 0. Each time a message rolled back to the queue this count is incremented by 1. A rollback can be either from a specific call to rollback(), or due to the application being disconnected from MQ before commit() is called (due to a network issue for example or the application crashing).
Poison message handling is disabled if you set BOTHRESH(0).
If BOTHRESH is >= 1, then poison message handling is enabled and when IBM MQ classes for JMS reads a message from a queue it will check if the BackoutCount is >= to the BOTHRESH. If the message is eligible for poison message handling then it will be moved to the queue specified in the BOQNAME attribute, if this attribute is empty or the application does not have access to PUT to this queue for some reason, it will instead attempt to put the message to the queue specified in the queue managers DEADQ attribute, if it can't put to either of these locations it will be rolled back to the queue.
You can find more detailed information on IBM MQ classes for JMS poison message handling in the IBM MQ v9.1 Knowledge Center page Developing applications>Developing JMS and Java applications>Using IBM MQ classes for JMS>Writing IBM MQ classes for JMS applications>Handling poison messages in IBM MQ classes for JMS
In Spring JMS you can define your own container. One container is created for one Jms Destination. We should run a single-threaded JMS listener to maintain the message ordering, to make this work set the concurrency to 1.
We can design our container to return null once it encounters errors, post-failure all receive calls should return null so that no messages are polled from the destination till the destination is active once again. We can maintain an active state using a timestamp, that could be simple milliseconds. A sample JMS config should be sufficient to add backoff. You can add small sleep instead of continuously returning null from receiveMessage method, for example, sleep for 10 seconds before making the next call, this will save some CPU resources.
#Configuration
#EnableJms
public class JmsConfig {
#Bean
public JmsListenerContainerFactory<?> jmsContainerFactory(ConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory() {
#Override
protected DefaultMessageListenerContainer createContainerInstance() {
return new DefaultMessageListenerContainer() {
private long deactivatedTill = 0;
#Override
protected Message receiveMessage(MessageConsumer consumer) throws JMSException {
if (deactivatedTill < System.currentTimeMillis()) {
return receiveFromConsumer(consumer, getReceiveTimeout());
}
logger.info("Disabled due to failure :(");
return null;
}
#Override
protected void doInvokeListener(MessageListener listener, Message message)
throws JMSException {
try {
super.doInvokeListener(listener, message);
} catch (Exception e) {
handleException(message);
throw e;
}
}
private long getDelay(int retryCount) {
if (retryCount <= 1) {
return 20;
}
return (long) (20 * Math.pow(2, retryCount));
}
private void handleException(Message msg) throws JMSException {
if (msg.propertyExists("JMSXDeliveryCount")) {
int retryCount = msg.getIntProperty("JMSXDeliveryCount");
deactivatedTill = System.currentTimeMillis() + getDelay(retryCount);
}
}
#Override
protected void doInvokeListener(SessionAwareMessageListener listener, Session session,
Message message)
throws JMSException {
try {
super.doInvokeListener(listener, session, message);
} catch (Exception e) {
handleException(message);
throw e;
}
}
};
}
};
// This provides all boot's default to this factory, including the message converter
configurer.configure(factory, connectionFactory);
// You could still override some of Boot's default if necessary.
return factory;
}
}
I have two microservices, one for collecting XML files from internal FTP server ,transforming it to DTO objects and then publishing them as bytes in RabbitMQ and the other for deserializing the incoming bytes from RabbitMQ to DTO objects, mapping them to JPA entities and persisiting them to database.
I'd like configure RabbitMQ broker between these two microservices like below:
1) for microservice that collect XML files, I edited in application.properties as below:
spring.cloud.stream.bindings.output.destination=TOPIC
spring.cloud.stream.bindings.output.group=proactive-policy
2) for microservice that persist incoming DTO onjects, I configured in application.properties as following:
spring.cloud.stream.bindings.input.destination=TOPIC
spring.cloud.stream.bindings.input.group=proactive-policy
For receiving incoming bytes from RabbitMQ I'm using second microservice as sink:
#EnableJpaAuditing
#EnableBinding(Sink.class)
#SpringBootApplication(scanBasePackages = { "org.proactive.policy.data.cache" })
#RefreshScope
public class ProactivePolicyDataCacheApplication {
private static Logger logger = LoggerFactory.getLogger(ProactivePolicyDataCacheApplication.class);
#Autowired
PolicyService policyService;
public static void main(String[] args) {
SpringApplication.run(ProactivePolicyDataCacheApplication.class, args);
}
#StreamListener(Sink.INPUT)
public void input(Message<byte[]> message) throws Exception {
if (Objects.isNull(message) || Objects.isNull(message.getPayload())) {
logger.error("the message is null ");
throw new IllegalArgumentException("`message` and `message.payload` cannot be null");
}
byte[] data = message.getPayload();
if (data.length == 0) {
logger.warn("Received empty message");
return;
}
logger.info("Got data from policy-collector = " + new String(data, "UTF-8"));
PolicyListDto policyListDto = (PolicyListDto) SerializationUtils.deserialize(data);
logger.info("Policies.xml from policy-collector = " + policyListDto.getPolicy().toString());
policyService.save(policyListDto);
}
}
But when I open RabbitMQ console for looking at exchanges I didn't receive any thing in Queue TOPIC.proactive-policy But the incoming messages are received in another Queue that I haven't configured it named FTPSTREAM.proactive-policy-collector
Is there any suggestion for resolving this issue
Couple of points:
1. There is no such thing as 'group' for the output binding. Consumer Group is a consumer property. Here is the fragment of the javadocs.
/**
* Unique name that the binding belongs to (applies to consumers only). Multiple
* consumers within the same group share the subscription. A null or empty String
* value indicates an anonymous group that is not shared.
* #see org.springframework.cloud.stream.binder.Binder#bindConsumer(java.lang.String,
* java.lang.String, java.lang.Object,
* org.springframework.cloud.stream.binder.ConsumerProperties)
*/
private String group;
2. The name 'FTPSTREAM.proactive-policy-collector' is definitely not something that is generated by the spring-cloud-stream, so consider looking into your configuration and see what have you missed.
It tells me that you have some consumer that has its 'destination' named FTPSTREAM and its 'group' proactive-policy-collector. It also tells me that your producer sends messages to the FTPSTREAM exchange.
I have a issue with Receive message back from Listener to publisher. I am getting
**AmqpReplyTimeoutException **. Below is the code of Publisher from where i am publishing to queue.
for(CsvWrapperPojo item : items){
resultList.addAll(item.getDbResultList());
for(CSVPojo pojo :item.getQueueRequestList()){
sampleResponseMessageRabbitConverterFuture= asyncRabbitTemplate.convertSendAndReceive("spring-boot-rabbitmq-Interactive.async_Solve_InteractiveMsg", "Interactive_RequestQueue", pojo);
//CSVPojo res =(CSVPojo)rabbitTemplate.convertSendAndReceive("spring-boot-rabbitmq-Interactive.async_Solve_InteractiveMsg", "Interactive_RequestQueue", pojo);
System.out.println("heyyyyyy:" + sampleResponseMessageRabbitConverterFuture.get().getLatitute());
//resultList.add(res);
//resultList.add(sampleResponseMessageRabbitConverterFuture.get());
}
}
By using it i am able to publish to queue, i have subscriber code below.
#EnableRabbit
public class ListenerQueueSubscriber {
#RabbitHandler
#RabbitListener(containerFactory = "simpleMessageListenerContainerFactory", queues ="Interactive_RequestQueue")
public void subscribeToRequestQueue(#Payload CSVPojo sampleRequestMessage, Message message) throws InterruptedException {
System.out.println("inside listener");
sampleRequestMessage.setResult("Hello");
Thread.sleep(120000);
System.out.println("After sleep:" +sampleRequestMessage.getLongitude());
//return sampleRequestMessage;
}
}
By using above subscriber able to listen message and i am appending "Hello and put sleep for 2 minutes and after that i have to receive the message back to publisher from where i have published . But unfortunately not receiving the message with Hello appended getting **AmqpReplyTimeoutException **. Can please help to achieve this behavior.
Thanks in advance!!!!
I am able to stop the consuming and restart the consuming but the problem is that when I am restarting the consuming, I am able to process the already published message but when I publish the new messages those are not able to process.
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Consumer;
#Component
public class RabbitMqueue implements Consumer {
int count = 0;
#RabbitListener(queues="dataQueue")
public void receivedData(#Payload Event msg, Channel channel,
#Header(AmqpHeaders.CONSUMER_TAG) String tag) throws IOException,
InterruptedException {
count++;
System.out.println("\n Message recieved from the Dataqueue is " + msg);
//Canceling consuming working fine.
if(count == 1) {
channel.basicCancel(tag);
System.out.println("Consumer is cancle");
}
count++;
System.out.println("\n count is " + count + "\n");
Thread.sleep(5000);
//restarting consumer. able to process already consumed messages
//but not able to see the newly published messages to the queue I mean
//newly published message is moving from ready to unack state but nothing
//happening on the consumer side.
if(count == 2) {
channel.basicConsume("dataQueue", this);
System.out.println("Consumer is started ");
}
}
}
You must not do this channel.basicCancel(tag).
The channel/consumer are managed by Spring; the only thing you should do with the consumer argument is ack or nack messages (and even that is rarely needed - it's better to let the container do the acks).
To stop/start the consumer, use the endpoint registry as described in the documentation.
Containers created for annotations are not registered with the application context. You can obtain a collection of all containers by invoking getListenerContainers() on the RabbitListenerEndpointRegistry bean. You can then iterate over this collection, for example, to stop/start all containers or invoke the Lifecycle methods on the registry itself which will invoke the operations on each container.
e.g. registry.stop() will stop all the listeners.
You can also get a reference to an individual container using its id, using getListenerContainer(String id); for example registry.getListenerContainer("multi") for the container created by the snippet above.
If your are using AMQP/Rabbit, you can try one of these:
1) Prevent starting at startup in code:
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
//
//autoStartup = false, prevents handling messages immedeatly. You need to start each listener itselve.
//
factory.setAutoStartup(false);
factory.setMessageConverter(new Jackson2JsonMessageConverter());
return factory;
}
2) Prevent starting at startup in in app.yml/props:
rabbitmq.listener.auto-startup: false
rabbitmq.listener.simple.auto-startup: false
3) Start/stop individual listeners
give your #RabbitListener a id:
#RabbitListener(queues = "myQ", id = "myQ")
...
and :
#Autowired
private RabbitListenerEndpointRegistry rabbitListenerEndpointRegistry;
MessageListenerContainer listener =
rabbitListenerEndpointRegistry.getListenerContainer("myQ");
...
listener.start();
...
listener.stop();
I have created several spring 4.1.6 #JmsListener methods for receiving messages from queues. I would like to get the list of listener names for admin purpose:
#JmsListener(
destination = "${jms.destination.name}"
, containerFactory = "myJmsContainerFactory"
, id ="myListener")
public void receiveMessage(String message) {
System.out.println("Received <" + message + ">");
}
and I want to display the id/name of the listeners for admin purposes. I can stop and start the listeners with the JmsListenerEndpointRegistry, but can't see how to get the name of the specific listeners.
JmsListenerEndpointRegistry registry = context.getBean(org.springframework.jms.config.JmsListenerEndpointRegistry.class);
Collection<MessageListenerContainer> listeners = registry.getListenerContainers();
MessageListenerContainer mlc = registry.getListenerContainer("myListener");
System.out.println("Running" + mlc.isRunning());
mlc.stop();
In debug the Collection can be seen as a Collections$UnmodifiableCollection with a LinkedHashMap that has the listener id ,myListener in this case, as the key value of the LinkedHashMap
This question was quite old, but I solved it this way:
Set<String> listenerContainerIds = registry.getListenerContainerIds();
for (String id : listenerContainerIds) {
MessageListenerContainer listenerContainer = registry.getListenerContainer(id);
...
}
I got the reference to the MessageListenerContainer und via getListenerContainerIds, I got the name of the ID to the reference.
Hope this helps...