Working versions in the app
IBM AllClient version : 'com.ibm.mq:com.ibm.mq.allclient:9.1.1.0'
org.springframework:spring-jms : 4.3.9.RELEASE
javax.jms:javax.jms-api : 2.0.1
My requirement is that in case of the failure of a message processing due to say, consumer not being available (eg. DB is unavailable), the message remains in the queue or put back on the queue (if that is even possible). This is because the order of the messages is important, messages have to be consumed in the same order that they are received. The Java app is single-threaded.
I have tried the following
#Override
public void onMessage(Message message)
{
try{
if(message instanceOf Textmessage)
{
}
:
:
throw new Exception("Test");// Just to test the retry
}
catch(Exception ex)
{
try
{
int temp = message.getIntProperty("JMSXDeliveryCount");
throw new RuntimeException("Redlivery attempted ");
// At this point, I am expecting JMS to put the message back into the queue.
// But it is actually put into the Bakout queue.
}
catch(JMSException ef)
{
String temp = ef.getMessage();
}
}
}
I have set this in my spring.xml for the jmsContainer bean.
<property name="sessionTransacted" value="true" />
What is wrong with the code above ?
And if putting the message back in the queue is not practical, how can one browse the message, process it and, if successful, pull the message (so it is consumed and no longer on the queue) ? Is this scenario supported in IBM provider for JMS?
The IBM MQ Local queue has BOTHRESH(1).
To preserve message ordering, one approach might be to stop the message listener temporarily as part of your rollback strategy. Looking at the Spring Boot doc for DefaultMessageListenerContainer there is a stop(Runnable callback) method. I've experimented with using this in a rollback as follows.
To ensure my Listener is single threaded, on my DefaultJmsListenerContainerFactory I set containerFactory.setConcurrency("1").
In my Listener, I set an id
#JmsListener(destination = "DEV.QUEUE.2", containerFactory = "listenerTwoFactory", concurrency="1", id="listenerTwo")
And retrieve the DefaultMessageListenerContainer instance.
JmsListenerEndpointRegistry reg = context.getBean(JmsListenerEndpointRegistry.class);
DefaultMessageListenerContainer mlc = (DefaultMessageListenerContainer) reg.getListenerContainer("listenerTwo");
For testing, I check JMSXDeliveryCount and throw an exception to rollback.
retryCount = Integer.parseInt(msg.getStringProperty("JMSXDeliveryCount"));
if (retryCount < 5) {
throw new Exception("Rollback test "+retryCount);
}
In the Listener's catch processing, I call stop(Runnable callback) on the DefaultMessageListenerContainer instance and pass in a new class ContainerTimedRestart as defined below.
//catch processing here and decide to rollback
mlc.stop(new ContainerTimedRestart(mlc,delay));
System.out.println("#### "+getClass().getName()+" Unable to process message.");
throw new Exception();
ContainerTimedRestart extends Runnable and DefaultMessageListenerContainer is responsible for invoking the run() method when the stop call completes.
public class ContainerTimedRestart implements Runnable {
//Container instance to restart.
private DefaultMessageListenerContainer theMlc;
//Default delay before restart in mills.
private long theDelay = 5000L;
//Basic constructor for testing.
public ContainerTimedRestart(DefaultMessageListenerContainer mlc, long delay) {
theMlc = mlc;
theDelay = delay;
}
public void run(){
//Validate container instance.
try {
System.out.println("#### "+getClass().getName()+"Waiting for "+theDelay+" millis.");
Thread.sleep(theDelay);
System.out.println("#### "+getClass().getName()+"Restarting container.");
theMlc.start();
System.out.println("#### "+getClass().getName()+"Container started!");
} catch (InterruptedException ie) {
ie.printStackTrace();
//Further checks and ensure container is in correct state.
//Report errors.
}
}
I loaded my queue with three messages with payloads "a", "b", and "c" respectively and started the listener.
Checking DEV.QUEUE.2 on my queue manager I see IPPROCS(1) confirming only one application handle has the queue open. The messages are processed in order after each is rolled five times and with a 5 second delay between rollback attempts.
IBM MQ classes for JMS has poison message handling built in. This handling is based on the QLOCAL setting BOTHRESH, this stands for Backout Threshold. Each IBM MQ message has a "header" called the MQMD (MQ Message Descriptor). One of the fields in the MQMD is BackoutCount. The default value of BackoutCount on a new message is 0. Each time a message rolled back to the queue this count is incremented by 1. A rollback can be either from a specific call to rollback(), or due to the application being disconnected from MQ before commit() is called (due to a network issue for example or the application crashing).
Poison message handling is disabled if you set BOTHRESH(0).
If BOTHRESH is >= 1, then poison message handling is enabled and when IBM MQ classes for JMS reads a message from a queue it will check if the BackoutCount is >= to the BOTHRESH. If the message is eligible for poison message handling then it will be moved to the queue specified in the BOQNAME attribute, if this attribute is empty or the application does not have access to PUT to this queue for some reason, it will instead attempt to put the message to the queue specified in the queue managers DEADQ attribute, if it can't put to either of these locations it will be rolled back to the queue.
You can find more detailed information on IBM MQ classes for JMS poison message handling in the IBM MQ v9.1 Knowledge Center page Developing applications>Developing JMS and Java applications>Using IBM MQ classes for JMS>Writing IBM MQ classes for JMS applications>Handling poison messages in IBM MQ classes for JMS
In Spring JMS you can define your own container. One container is created for one Jms Destination. We should run a single-threaded JMS listener to maintain the message ordering, to make this work set the concurrency to 1.
We can design our container to return null once it encounters errors, post-failure all receive calls should return null so that no messages are polled from the destination till the destination is active once again. We can maintain an active state using a timestamp, that could be simple milliseconds. A sample JMS config should be sufficient to add backoff. You can add small sleep instead of continuously returning null from receiveMessage method, for example, sleep for 10 seconds before making the next call, this will save some CPU resources.
#Configuration
#EnableJms
public class JmsConfig {
#Bean
public JmsListenerContainerFactory<?> jmsContainerFactory(ConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory() {
#Override
protected DefaultMessageListenerContainer createContainerInstance() {
return new DefaultMessageListenerContainer() {
private long deactivatedTill = 0;
#Override
protected Message receiveMessage(MessageConsumer consumer) throws JMSException {
if (deactivatedTill < System.currentTimeMillis()) {
return receiveFromConsumer(consumer, getReceiveTimeout());
}
logger.info("Disabled due to failure :(");
return null;
}
#Override
protected void doInvokeListener(MessageListener listener, Message message)
throws JMSException {
try {
super.doInvokeListener(listener, message);
} catch (Exception e) {
handleException(message);
throw e;
}
}
private long getDelay(int retryCount) {
if (retryCount <= 1) {
return 20;
}
return (long) (20 * Math.pow(2, retryCount));
}
private void handleException(Message msg) throws JMSException {
if (msg.propertyExists("JMSXDeliveryCount")) {
int retryCount = msg.getIntProperty("JMSXDeliveryCount");
deactivatedTill = System.currentTimeMillis() + getDelay(retryCount);
}
}
#Override
protected void doInvokeListener(SessionAwareMessageListener listener, Session session,
Message message)
throws JMSException {
try {
super.doInvokeListener(listener, session, message);
} catch (Exception e) {
handleException(message);
throw e;
}
}
};
}
};
// This provides all boot's default to this factory, including the message converter
configurer.configure(factory, connectionFactory);
// You could still override some of Boot's default if necessary.
return factory;
}
}
I am using IBM MQ Classes for JMS (IBM MQ Version 8.0.0.4) with auto-reconnection configured. According to the docs, reconnection happens implicitly. I would like to issue a simple log statement in the event of reconnection. For that reason, I need to somehow get informed, when that happens.
In the IBM docs on page Application Recovery I stumbled over section "Detecting failover" where it says:
Reconnection aware: Register an MQCBT_EVENT_HANDLER event handler with
the queue manager. The event handler is posted with MQRC_RECONNECTING
when the client starts to try to reconnect to the server, and
MQRC_RECONNECTED after a successful reconnection. You can then run a
routine to reestablish a predictable state so that the client
application is able to continue processing.
Unfortunately, I did not find a code example for Java/JMS that demonstrates how and where to register such an event handler. I don't know if that is even supported in my case. Could anyone provide me to the right direction or even provide a code sample? Thank you very much.
Question Update from February 5, 2020:
Added the following code example created by myself, after having received Sashi's initial answer from January 27, 2020.
public static void main(String[] args) {
Connection connection = null;
Session session = null;
Object destination = null;
MessageProducer producer = null;
try {
JmsFactoryFactory jmsFactoryFactory = JmsFactoryFactory.getInstance(WMQConstants.WMQ_PROVIDER);
JmsConnectionFactory cf = jmsFactoryFactory.createConnectionFactory();
cf.setStringProperty(WMQConstants.WMQ_HOST_NAME, HOST);
cf.setIntProperty(WMQConstants.WMQ_PORT, PORT);
cf.setStringProperty(WMQConstants.WMQ_CHANNEL, CHANNEL);
cf.setIntProperty(WMQConstants.WMQ_CONNECTION_MODE, WMQConstants.WMQ_CM_CLIENT);
cf.setStringProperty(WMQConstants.WMQ_QUEUE_MANAGER, QM_NAME);
cf.setIntProperty(WMQConstants.WMQ_CLIENT_RECONNECT_OPTIONS, WMQConstants.WMQ_CLIENT_RECONNECT);
cf.setIntProperty(WMQConstants.WMQ_CLIENT_RECONNECT_TIMEOUT, RECONNECT_TIMEOUT);
connection = cf.createConnection();
connection.setExceptionListener(new MQExceptionListener());
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
destination = session.createQueue(QUEUE);
producer = session.createProducer((Destination)destination);
connection.start();
} catch (JMSException ex) {
LOGGER.error(ex.toString());
}
}
public class MQExceptionListener implements ExceptionListener {
public void onException(JMSException e) {
System.out.println(e);
if(e.getLinkedException() != null)
System.out.println(e.getLinkedException());
}
}
This is what I get in the logs:
ERROR [Main.main:57] (main) com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager '<hostname>' with connection mode 'Client' and host name '<hostname>(<port>)'.
Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.
ERROR [Main.main:61] (main) Inner exceptions:
ERROR [Main.main:65] (main) com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2538' ('MQRC_HOST_NOT_AVAILABLE').
ERROR [Main.main:65] (main) com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '<hostname>(<port>)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2538;AMQ9204: Connection to host '<hostname>/<ip>:<port>' rejected. [1=java.net.ConnectException[Connection refused: connect],3=<hostname>/<ip>:<port>,4=TCP,5=Socket.connect]],3=<hostname>(<port>),5=RemoteTCPConnection.bindAndConnectSocket]
ERROR [Main.main:65] (main) com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '<hostname>/<ip>:<port>' rejected. [1=java.net.ConnectException[Connection refused: connect],3=<hostname>/<ip>:<port>,4=TCP,5=Socket.connect]
ERROR [Main.main:65] (main) java.net.ConnectException: Connection refused: connect
Question Update from February 11, 2020:
I've added this additions based on feedback received by Sashi on February 5, 2020.
I have tried to build a minimal application that connects to an IBM MQ instance. Here's the code:
Application.java
public class Application {
private static final Logger LOGGER = LoggerFactory.getLogger(Application.class);
public static void main(String[] args) {
new Application().run();
}
private void run() {
MQWriter writer = new MQWriter();
int i = 1;
while (true) {
String message = "Hello Testing " + i;
LOGGER.info("Sending message {} to MQ server...", message);
writer.write(message);
i++;
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
MQConnectionDetails.java
public class MQConnectionDetails {
public static final String HOST = "XXX.XXX.XXX.XXX";
public static final int PORT = 1414;
public static final String QM_NAME = "QM1";
public static final String CHANNEL = "DEV.APP.SVRCONN";
public static final String QUEUE = "DEV.QUEUE.1";
public static final int RECONNECT_TIMEOUT = 60; // 1 minute
}
MQWriter.java
public class MQWriter {
private static final Logger LOGGER = LoggerFactory.getLogger(MQWriter.class);
private Connection connection = null;
private Session session = null;
private Object destination = null;
private MessageProducer producer = null;
public MQWriter() {
try {
JmsFactoryFactory jff = JmsFactoryFactory.getInstance(WMQConstants.WMQ_PROVIDER);
JmsConnectionFactory jcf = jff.createConnectionFactory();
jcf.setStringProperty(WMQConstants.WMQ_HOST_NAME, MQConnectionDetails.HOST);
jcf.setIntProperty(WMQConstants.WMQ_PORT, MQConnectionDetails.PORT);
jcf.setStringProperty(WMQConstants.WMQ_CHANNEL, MQConnectionDetails.CHANNEL);
jcf.setIntProperty(WMQConstants.WMQ_CONNECTION_MODE, WMQConstants.WMQ_CM_CLIENT);
jcf.setStringProperty(WMQConstants.WMQ_QUEUE_MANAGER, MQConnectionDetails.QM_NAME);
jcf.setIntProperty(WMQConstants.WMQ_CLIENT_RECONNECT_OPTIONS, WMQConstants.WMQ_CLIENT_RECONNECT);
jcf.setIntProperty(WMQConstants.WMQ_CLIENT_RECONNECT_TIMEOUT, MQConnectionDetails.RECONNECT_TIMEOUT);
LOGGER.info("Initializing connection to write queue {} on {}:{}...",
MQConnectionDetails.QUEUE,
MQConnectionDetails.HOST,
MQConnectionDetails.PORT);
connection = jcf.createConnection();
connection.setExceptionListener(new MQExceptionListener());
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
destination = session.createQueue(MQConnectionDetails.QUEUE);
producer = session.createProducer((Destination)destination);
connection.start();
} catch (JMSException ex) {
LOGGER.error("Error initializing connection to write queue", ex);
}
}
public void write(String message) {
try {
TextMessage textMessage = session.createTextMessage(message);
producer.send(textMessage);
} catch (Exception ex) {
LOGGER.error("Error sending message to write queue", ex);
}
}
}
MQExceptionListener.java
public class MQExceptionListener implements ExceptionListener {
private static final Logger LOGGER = LoggerFactory.getLogger(MQExceptionListener.class);
public void onException(JMSException ex) {
LOGGER.error("=====");
LOGGER.error(ex.toString());
if (ex.getLinkedException() != null) {
LOGGER.error(ex.getLinkedException().toString());
}
LOGGER.error("=====");
}
}
The test scenario I ran is like this:
Make sure, IBM MQ is available on TCP port 1414 (IBM MQ Docker container running on Amazon EC2).
Run the application above (Application.java) and make sure it sends messages to the queue.
Change firewall config on Amazon EC2 security groups by changing port from 1414 to 1415, which makes IBM MQ unavailable to the client.
This is what I've observed:
Only after 90 seconds of inactivity, the client started throwing exceptions. I do not understand, because my RECONNECT_TIMEOUT was set to 60 seconds, so 30 seconds off here.
MQExceptionListener is invoked only once (the first time).
There are no reason codes 2544(MQRC_RECONNECTING) only 2009(MQRC_CONNECTION_BROKEN) is present. Why is that?
Here is a summary of the exceptions that got thrown:
Exceptions on console:
2020-02-11 09:50:16,155 INFO [Application.run:21] (main) Sending message Hello Testing 13 to MQ server...
2020-02-11 09:50:17,285 INFO [Application.run:21] (main) Sending message Hello Testing 14 to MQ server...
2020-02-11 09:50:18,413 INFO [Application.run:21] (main) Sending message Hello Testing 15 to MQ server...
2020-02-11 09:50:19,555 INFO [Application.run:21] (main) Sending message Hello Testing 16 to MQ server...
2020-02-11 09:51:45,966 ERROR [MQExceptionListener.onException:14] (JMSCCThreadPoolWorker-6) =====
2020-02-11 09:51:45,966 ERROR [MQExceptionListener.onException:15] (JMSCCThreadPoolWorker-6) com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred.
An error has occurred with the IBM MQ JMS connection.
Use the linked exception to determine the cause of this error.
2020-02-11 09:51:45,966 ERROR [MQExceptionListener.onException:17] (JMSCCThreadPoolWorker-6) com.ibm.mq.MQException: MQ delivered an asynchronous event with completion code '2', and reason '2009'.
2020-02-11 09:51:45,966 ERROR [MQExceptionListener.onException:19] (JMSCCThreadPoolWorker-6) =====
2020-02-11 09:51:45,967 ERROR [MQWriter.write:52] (main) Error sending message to write queue
com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ2007: Failed to send a message to destination 'DEV.QUEUE.1'.
JMS attempted to perform an MQPUT or MQPUT1; however IBM MQ reported an error.
Use the linked exception to determine the cause of this error.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:595)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.checkJmqiCallSuccess(WMQMessageProducer.java:1288)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.checkJmqiCallSuccess(WMQMessageProducer.java:1245)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.access$800(WMQMessageProducer.java:76)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer$SpiIdentifiedProducerShadow.sendInternal(WMQMessageProducer.java:906)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer$ProducerShadow.send(WMQMessageProducer.java:566)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.send(WMQMessageProducer.java:1428)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.sendMessage(JmsMessageProducerImpl.java:855)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.synchronousSendInternal(JmsMessageProducerImpl.java:2055)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.sendInternal(JmsMessageProducerImpl.java:1993)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.send(JmsMessageProducerImpl.java:1486)
at com.ibm.mq.jms.MQMessageProducer.send(MQMessageProducer.java:293)
at org.example.MQWriter.write(MQWriter.java:50)
at org.example.Application.run(Application.java:22)
at org.example.Application.main(Application.java:13)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
... 14 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009
at com.ibm.mq.jmqi.remote.api.RemoteHconn$ReconnectionState.recordFailure(RemoteHconn.java:4931)
at com.ibm.mq.jmqi.remote.api.RemoteHconn.setReconnectionFailureInner(RemoteHconn.java:2650)
at com.ibm.mq.jmqi.remote.api.RemoteParentHconn.setReconnectionFailure(RemoteParentHconn.java:152)
at com.ibm.mq.jmqi.remote.impl.RemoteReconnectThread.bestHconn(RemoteReconnectThread.java:265)
at com.ibm.mq.jmqi.remote.impl.RemoteReconnectThread.run(RemoteReconnectThread.java:115)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:319)
at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:343)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312)
at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1227)
2020-02-11 09:51:46,969 INFO [Application.run:21] (main) Sending message Hello Testing 17 to MQ server...
2020-02-11 09:51:46,972 ERROR [MQWriter.write:52] (main) Error sending message to write queue
com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ2007: Failed to send a message to destination 'DEV.QUEUE.1'.
JMS attempted to perform an MQPUT or MQPUT1; however IBM MQ reported an error.
Use the linked exception to determine the cause of this error.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:595)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.checkJmqiCallSuccess(WMQMessageProducer.java:1288)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.checkJmqiCallSuccess(WMQMessageProducer.java:1245)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.access$800(WMQMessageProducer.java:76)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer$SpiIdentifiedProducerShadow.sendInternal(WMQMessageProducer.java:906)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer$ProducerShadow.send(WMQMessageProducer.java:566)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.send(WMQMessageProducer.java:1428)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.sendMessage(JmsMessageProducerImpl.java:855)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.synchronousSendInternal(JmsMessageProducerImpl.java:2055)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.sendInternal(JmsMessageProducerImpl.java:1993)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.send(JmsMessageProducerImpl.java:1486)
at com.ibm.mq.jms.MQMessageProducer.send(MQMessageProducer.java:293)
at org.example.MQWriter.write(MQWriter.java:50)
at org.example.Application.run(Application.java:22)
at org.example.Application.main(Application.java:13)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
... 14 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009
at com.ibm.mq.jmqi.remote.api.RemoteHconn$ReconnectionState.recordFailure(RemoteHconn.java:4931)
at com.ibm.mq.jmqi.remote.api.RemoteHconn.setReconnectionFailureInner(RemoteHconn.java:2650)
at com.ibm.mq.jmqi.remote.api.RemoteParentHconn.setReconnectionFailure(RemoteParentHconn.java:152)
at com.ibm.mq.jmqi.remote.impl.RemoteReconnectThread.bestHconn(RemoteReconnectThread.java:265)
at com.ibm.mq.jmqi.remote.impl.RemoteReconnectThread.run(RemoteReconnectThread.java:115)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:319)
at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:343)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312)
at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1227)
Question Update from February 12, 2020
Added this sample and findings based on JoshMc's answer from February 11, 2020. My comments on this sample:
I am now using the MQ* classes and set reconnect options as suggested.
Reconnect is still not happening though
MQWriter2.java
public class MQWriter2 {
private static final Logger LOGGER = LoggerFactory.getLogger(MQWriter2.class);
private Connection connection = null;
private Session session = null;
private Queue destination = null;
private MessageProducer producer = null;
public MQWriter2() {
try {
MQConnectionFactory factory = new MQConnectionFactory();
factory.setTransportType(WMQConstants.WMQ_CM_CLIENT);
factory.setConnectionNameList("XXX.XXX.XXX.XXX(1414)");
factory.setQueueManager(MQConnectionDetails.QM_NAME);
factory.setChannel(MQConnectionDetails.CHANNEL);
factory.setClientReconnectOptions(WMQConstants.WMQ_CLIENT_RECONNECT);
factory.setClientReconnectTimeout(MQConnectionDetails.RECONNECT_TIMEOUT);
LOGGER.info("Initializing connection to write queue {} on {}:{}...",
MQConnectionDetails.QUEUE,
MQConnectionDetails.HOST,
MQConnectionDetails.PORT);
connection = factory.createConnection();
connection.setExceptionListener(new MQExceptionListener());
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
destination = session.createQueue(MQConnectionDetails.QUEUE);
producer = session.createProducer(destination);
connection.start();
} catch (JMSException ex) {
LOGGER.error("Error initializing connection to write queue", ex);
}
}
public void write(String message) {
try {
TextMessage textMessage = session.createTextMessage(message);
producer.send(textMessage);
} catch (Exception ex) {
LOGGER.error("Error sending message to write queue", ex);
}
}
}
Console Output
2020-02-12 08:39:11,628 INFO [MQWriter2.<init>:29] (main) Initializing connection to write queue DEV.QUEUE.1 on 54.161.121.207:1414...
2020-02-12 08:39:14,552 INFO [Application.run:19] (main) Sending message Hello Testing 1 to MQ server...
2020-02-12 08:39:15,710 INFO [Application.run:19] (main) Sending message Hello Testing 2 to MQ server...
2020-02-12 08:39:16,841 INFO [Application.run:19] (main) Sending message Hello Testing 3 to MQ server...
...
2020-02-12 08:39:41,973 INFO [Application.run:19] (main) Sending message Hello Testing 25 to MQ server...
2020-02-12 08:41:27,314 ERROR [MQExceptionListener.onException:14] (JMSCCThreadPoolWorker-10) =====
2020-02-12 08:41:27,314 ERROR [MQExceptionListener.onException:15] (JMSCCThreadPoolWorker-10) com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred.
An error has occurred with the IBM MQ JMS connection.
Use the linked exception to determine the cause of this error.
2020-02-12 08:41:27,314 ERROR [MQWriter2.write:49] (main) Error sending message to write queue
com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ2007: Failed to send a message to destination 'DEV.QUEUE.1'.
JMS attempted to perform an MQPUT or MQPUT1; however IBM MQ reported an error.
Use the linked exception to determine the cause of this error.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:595)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.checkJmqiCallSuccess(WMQMessageProducer.java:1288)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.checkJmqiCallSuccess(WMQMessageProducer.java:1245)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.access$800(WMQMessageProducer.java:76)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer$SpiIdentifiedProducerShadow.sendInternal(WMQMessageProducer.java:906)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer$ProducerShadow.send(WMQMessageProducer.java:566)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.send(WMQMessageProducer.java:1428)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.sendMessage(JmsMessageProducerImpl.java:855)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.synchronousSendInternal(JmsMessageProducerImpl.java:2055)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.sendInternal(JmsMessageProducerImpl.java:1993)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.send(JmsMessageProducerImpl.java:1486)
at com.ibm.mq.jms.MQMessageProducer.send(MQMessageProducer.java:293)
at org.example.MQWriter2.write(MQWriter2.java:47)
at org.example.Application.run(Application.java:20)
at org.example.Application.main(Application.java:11)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
... 14 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009
at com.ibm.mq.jmqi.remote.api.RemoteHconn$ReconnectionState.recordFailure(RemoteHconn.java:4931)
at com.ibm.mq.jmqi.remote.api.RemoteHconn.setReconnectionFailureInner(RemoteHconn.java:2650)
at com.ibm.mq.jmqi.remote.api.RemoteParentHconn.setReconnectionFailure(RemoteParentHconn.java:152)
at com.ibm.mq.jmqi.remote.impl.RemoteReconnectThread.bestHconn(RemoteReconnectThread.java:265)
at com.ibm.mq.jmqi.remote.impl.RemoteReconnectThread.run(RemoteReconnectThread.java:115)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:319)
at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:343)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312)
at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1227)
You can set an ExceptionListener on connection object after creating a connection. The onException method of the ExceptionListener gets invoked when reconnection attempts are made. Here is an example:
ExceptionListener exceptionListener = new ExceptionListener(){
#Override
public void onException(JMSException e) {
System.out.println(e);
if(e.getLinkedException() != null)
System.out.println(e.getLinkedException());
}
};
MQQueueConnection connection = (MQQueueConnection) cf.createQueueConnection();
connection.setExceptionListener(exceptionListener);
I faced the same issue as the topic starter. After spending hours sifting information available in the internet, speaking to colleagues and pulling hair in an attempt to make the reconnection work, I gave up and made a decision to work around the problem by emulating that incomprehensible reconnection functionality. I hope it will help other people struggling with IBM MQ. I wrote the class that, basically, does 2 things:
Repeatedly tries to connect to IBM MQ with an increasing interval between attempts.
After connecting, sets an error handler that is fired by IBM MQ when something happens to the connection (using the same logic for reconnection).
First, here's the class itself:
package com.raiks.mqclient;
import javax.jms.Destination;
import javax.jms.JMSConsumer;
import javax.jms.JMSContext;
import javax.jms.JMSException;
import com.ibm.msg.client.jms.JmsConnectionFactory;
import com.ibm.msg.client.wmq.WMQConstants;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.raiks.mqclient.IbmMqMessageListener;
/**
* This class implements the reconnection logic for JMS brokers that don't support it
* In particular, it does it for IBM MQ with its incomprehensible reconnection algorithm
* It's expected that each connection manager receives a separate connection factory
* and a message listener - it's not guaranteed for those to be thread safe
*/
public final class IbmMqJmsConnectionManager {
private static final Logger LOGGER = LoggerFactory.getLogger(IbmMqJmsConnectionManager.class);
private static final int INITIAL_RECONNECTION_DELAY_MS = 6000;
private static final int MAX_RECONNECTION_DELAY_MS = 60000;
private static final String QUEUE_PREIX = "queue:///";
private final String connectorName;
private final JmsConnectionFactory connectionFactory;
private final String queueName;
private final IbmMqMessageListener messageListener;
private final int initialReconnectionDelayMs;
private final int maxReconnectionDelayMs;
public IbmMqJmsConnectionManager(
String connectorName,
JmsConnectionFactory connectionFactory,
String queueName,
IbmMqMessageListener messageListener,
int initialReconnectionDelayMs,
int maxReconnectionDelayMs
) {
this.connectorName = connectorName;
this.connectionFactory = connectionFactory;
this.queueName = queueName;
this.messageListener = messageListener;
this.initialReconnectionDelayMs = initialReconnectionDelayMs;
this.maxReconnectionDelayMs = maxReconnectionDelayMs;
}
/**
* Attempts to connect to a JMS broker and makes continuous retries with an increasing interval if fails
* When the maximum interval is reached, issues an error and keeps on trying
* Sets the exception listener (a callback) in the created JMSContext which calls this method when the
* connection with the broker goes down due to network issue or intentional connection termination
*/
public void connectToBrokerWithRetries() {
String connectionDetails = formatConnectionDetails();
LOGGER.info("Attempting to connect to JMS broker '{}'. Connection details = {}", connectorName, connectionDetails);
JMSContext context = null;
int sleepingTimeMs = INITIAL_RECONNECTION_DELAY_MS;
int accumulatedSleepingTimeMs = 0;
// Try to reconnect until we succeed. IMPORTANT! This is a blocking loop that never ends so it must be run in a separate thread
while (context == null) {
try {
context = connectionFactory.createContext(JMSContext.AUTO_ACKNOWLEDGE);
LOGGER.info("Successfully connected to the JMS broker '{}'. Connection details = {}", connectorName, connectionDetails);
boolean hadUnsuccessfulConnectionAttempts = accumulatedSleepingTimeMs > 0;
if (hadUnsuccessfulConnectionAttempts) {
LOGGER.warn(
"Before this successful attempt, I spent {} ms repeatedly trying to connect to '{}'. Please check the broker's health. Connection details = {}",
accumulatedSleepingTimeMs, connectorName, connectionDetails
);
}
Destination destination = context.createQueue(QUEUE_PREIX + queueName);
JMSConsumer jmsConsumer = context.createConsumer(destination);
jmsConsumer.setMessageListener(messageListener);
LOGGER.info("Successfully connected to the queue '{}' at '{}'. Connection details = {}", queueName, connectorName, connectionDetails);
// Sets a callback that will be invoked when something happens with a connection to a broker
context.setExceptionListener(
jmsException -> {
LOGGER.warn("Something bad happened to JMS broker connection to '{}'. I will try to reconnect. Connection details = {}", connectorName, connectionDetails);
connectToBrokerWithRetries();
}
);
} catch (Exception e) {
LOGGER.warn(
"Failed to create a JMS context for '{}'. I will wait for {} ms and then make a reconnection attempt. Connection details = {}",
connectorName, sleepingTimeMs, connectionDetails, e
);
context = null;
try {
Thread.sleep(sleepingTimeMs);
accumulatedSleepingTimeMs += sleepingTimeMs;
int doubledSleepingTime = sleepingTimeMs * 2;
// We double the sleeping time on each subsequent attempt until we hit the limit
// Then we just keep on reconnecting forever using the limit value
boolean nextReconnectionDelayWillExceedMaxDelay = doubledSleepingTime >= MAX_RECONNECTION_DELAY_MS;
if (nextReconnectionDelayWillExceedMaxDelay) {
sleepingTimeMs = MAX_RECONNECTION_DELAY_MS;
LOGGER.error(
"Repeatedly failed to create a JMS context for {} ms. I will keep on trying every {} ms but please check the broker availability. Connection details = {}",
accumulatedSleepingTimeMs, sleepingTimeMs, connectionDetails
);
} else {
sleepingTimeMs = doubledSleepingTime;
}
} catch (InterruptedException ex) {
throw new RuntimeException(ex);
}
}
}
}
private String formatConnectionDetails() {
String connectionDetails = "[]";
try {
connectionDetails = String.format(
"[ host = %s, port = %d, queueManager = %s, channel = %s, user = %s ]",
connectionFactory.getStringProperty(WMQConstants.WMQ_HOST_NAME),
connectionFactory.getIntProperty(WMQConstants.WMQ_PORT),
connectionFactory.getStringProperty(WMQConstants.WMQ_QUEUE_MANAGER),
connectionFactory.getStringProperty(WMQConstants.WMQ_CHANNEL),
connectionFactory.getStringProperty(WMQConstants.USERID)
);
} catch (Exception e) {
LOGGER.warn("Failed to get the connection details. This is not critical, but the details will be unavailable");
}
return connectionDetails;
}
}
And here's how you use it:
LOGGER.info("Starting the initial connection thread");
Thread cftInitialConnectionThread = new Thread(cftConnectionManager::connectToBrokerWithRetries);
cftInitialConnectionThread.start();
Check this piece of Code, for me is working with IBM WMQ 9.2.3, using a 3 node IBM WMQ Multi Instance with Pacemaker on CentOS 8.
package com.ibm.mq.samples.jms;
import javax.jms.Destination;
import javax.jms.JMSConsumer;
import javax.jms.JMSContext;
import javax.jms.JMSException;
import javax.jms.JMSProducer;
import javax.jms.TextMessage;
import com.ibm.msg.client.jms.JmsConnectionFactory;
import com.ibm.msg.client.jms.JmsFactoryFactory;
import com.ibm.msg.client.wmq.WMQConstants;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
public class JmsPutGet {
// System exit status value (assume unset value to be 1)
private static int status = 1;
// Create variables for the connection to MQ
private static final String HOST = "192.168.49.140"; // Host name or IP address
private static final int PORT = 10200; // Listener port for your queue manager
private static final String CHANNEL = "CHANNEL1"; // Channel name
private static final String QMGR = "HAQM1"; // Queue manager name
private static final String APP_USER = ""; // User name that application uses to connect to MQ
private static final String APP_PASSWORD = ""; // Password that the application uses to connect to MQ
private static final String QUEUE_NAME = "SOURCE"; // Queue that the application uses to put and get messages to and from
private static final int RECONNECT_TIMEOUT = 60; // 1 minute
private static JMSContext context = null;
private static Destination destination = null;
public static void main(String[] args) {
// Variables
JMSProducer producer = null;
JMSConsumer consumer = null;
LocalDateTime now = null;
try {
setupResources();
long uniqueNumber = System.currentTimeMillis() % 1000;
TextMessage message = context.createTextMessage("Your lucky number today is " + uniqueNumber);
DateTimeFormatter dtf = DateTimeFormatter.ofPattern("uuuu/MM/dd HH:mm:ss");
for(int i=0; i>=0; i++){
producer = context.createProducer();
producer.send(destination, message);
//System.out.println("Sent message:\n " + i + " " + message);
System.out.println("\nMensaje enviado:\n " + i );
now = LocalDateTime.now();
System.out.println(dtf.format(now));
consumer = context.createConsumer(destination); // autoclosable
String receivedMessage = consumer.receiveBody(String.class, 15000); // in ms or 15 seconds
//System.out.println("\nReceived message:\n " + i + " " + receivedMessage);
System.out.println("\nMensaje recibido:\n " + i );
now = LocalDateTime.now();
System.out.println(dtf.format(now));
Thread.sleep(1000);
}
context.close();
recordSuccess();
} catch (Exception ex) {
recordFailure(ex);
System.out.println("DETECTING ERROR... RECONNECTING");
setupResources();
}
} // end main()
private static void setupResources() {
boolean connected = false;
while (!connected) {
try {
// Create a connection factory
JmsFactoryFactory ff = JmsFactoryFactory.getInstance(WMQConstants.WMQ_PROVIDER);
JmsConnectionFactory cf = ff.createConnectionFactory();
// Set the properties
cf.setStringProperty(WMQConstants.WMQ_HOST_NAME, HOST);
cf.setIntProperty(WMQConstants.WMQ_PORT, PORT);
//cf.setStringProperty(WMQConstants.WMQ_CONNECTION_NAME_LIST, "192.168.49.140(10200),192.168.49.131(10200),192.168.49.132(10200)");
cf.setStringProperty(WMQConstants.WMQ_CHANNEL, CHANNEL);
cf.setIntProperty(WMQConstants.WMQ_CONNECTION_MODE, WMQConstants.WMQ_CM_CLIENT);
cf.setStringProperty(WMQConstants.WMQ_QUEUE_MANAGER, QMGR);
cf.setStringProperty(WMQConstants.WMQ_APPLICATIONNAME, "JmsPutGet (JMS)");
cf.setBooleanProperty(WMQConstants.USER_AUTHENTICATION_MQCSP, true);
cf.setStringProperty(WMQConstants.USERID, APP_USER);
cf.setStringProperty(WMQConstants.PASSWORD, APP_PASSWORD);
cf.setIntProperty(WMQConstants.WMQ_CLIENT_RECONNECT_TIMEOUT, RECONNECT_TIMEOUT);
cf.setIntProperty(WMQConstants.WMQ_CLIENT_RECONNECT_OPTIONS, WMQConstants.WMQ_CLIENT_RECONNECT);
//cf.setStringProperty(WMQConstants.WMQ_SSL_CIPHER_SUITE, "*TLS12");
// Create JMS objects
context = cf.createContext();
destination = context.createQueue("queue:///" + QUEUE_NAME);
// no exception? then we connected ok
connected = true;
System.out.println("CONNECTED");
}
catch (JMSException je) {
// sleep and then have another attempt
System.out.println("RECONNECTING");
try {Thread.sleep(30*1000);} catch (InterruptedException ie) {}
}
}
}
private static void recordSuccess() {
System.out.println("SUCCESS");
status = 0;
return;
}
private static void recordFailure(Exception ex) {
if (ex != null) {
if (ex instanceof JMSException) {
processJMSException((JMSException) ex);
} else {
System.out.println(ex);
}
}
System.out.println("FAILURE");
status = -1;
return;
}
private static void processJMSException(JMSException jmsex) {
System.out.println(jmsex);
Throwable innerException = jmsex.getLinkedException();
if (innerException != null) {
System.out.println("Inner exception(s):");
}
while (innerException != null) {
System.out.println(innerException);
innerException = innerException.getCause();
}
return;
}
}
I have added a link in github for any reference https://github.com/fintecheando/IBMMQSample
A few points here that may clear things up for you.
The following line sets the amount of time that MQ will attempt to reconnect to the queue manager once it notices the connection is lost.
jcf.setIntProperty(WMQConstants.WMQ_CLIENT_RECONNECT_TIMEOUT, MQConnectionDetails.RECONNECT_TIMEOUT);
How long it takes the client to notice the connection is broken depends on the type of failure, but in the situation you describe it will be based on the HBINT setting of the SVRCONN channel on the queue manager.
A Heart Beat is sent every HBINT seconds when no other normal traffic is passing over the channel. The TIMEOUT of the channel is based on the HBINT, if the HBINT is less than 60 seconds then the TIMEOUT is twice the HBINT, if the HBINT is 60 seconds or larger, the TIMEOUT is the HBINT plus 60 seconds. The TIMEOUT is based on the last time traffic or a heart beat was sent, not when you changed firewall config, although in this case it appears you were sending a message every seconds so this should have been close.
Based on the logs I'm not sure it is picking up the RECONNECT option as I would expect when the reconnect timeout expired that you would get either of these errors instead of MQRC_CONNECTION_BROKEN:
MQRC_RECONNECT_FAILED
MQRC_RECONNECT_TIMED_OUT
In samples I have, I set it this way, maybe try this instead of the way you are currently setting it:
jcf.setClientReconnectOptions(WMQConstants.WMQ_CLIENT_RECONNECT);
jcf.setClientReconnectTimeout(MQConnectionDetails.RECONNECT_TIMEOUT);
Based on what I see I would guess your HBINT is set to 45 seconds on the SVRCONN channel, and the connection is just timing out with MQRC_CONNECTION_BROKEN at 90 seconds and never attempting to reconnect.
I have a java program I run to write messages to Mid-Tier IBM MQ's to test functionality before attaching our main programs to them. The write method looks like the following below:
private static void sendSingleMessage(ConnectionFactory connectionFactory,
String[] messages, String destination) throws Exception {
Connection connection = null;
try {
connection = connectionFactory.createConnection();
for (String payload : messages) {
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue(destination);
MessageProducer producer = session.createProducer(queue);
Message msg = session.createTextMessage(payload);
System.out.println("Sending text '" + payload + "'");
producer.send(msg);
session.close();
System.out.println("Message sent");
}
} finally {
if (connection != null) {
connection.close();
}
}
}
The connectionFactory is setup before this method executes, but within that method I set the MQConncetionFactory properties(host,port,channel,queuemanager, etc...) This send method works and I can see the queue depth increasing on my IBM MQ Explorer when I call it from my main method.
When I run a similar readSingleMessage method, the code gets stuck on the consumer.receive() and never finishes executing. See below:
private static void readSingleMessage(ConnectionFactory connectionFactory,
String[] messages, String destination) throws Exception {
Connection connection = null;
try {
connection = connectionFactory.createConnection();
for (String payload : messages) {
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue(destination);
MessageConsumer consumer = session.createConsumer(queue);
System.out.println("Recieving text '" + payload + "'");
consumer.receive();
session.close();
System.out.println("Received message");
}
} finally {
if (connection != null) {
connection.close();
}
}
}
Is there anyway I can further debug this, or find why I am able to write to the queue, but unable to read a message off of it?
You have to start the JMS Connection by calling the start() method on it. You cannot receive any messages until the connection is started. This is noted in the JMS Specification and Javadoc.
As an aside, if you use the JMS 2.0 "simplified" API and create a JMSContext object (an object which is essentially a combined Connection and Session) you do not need to call start to receive messages. A consumer crated from it can be used to receive messages without being explicitly started.
I am using Spring Cloud AWS (1.0.1.RELEASE) with Spring Boot to run a SQS consumer. The application runs fine, but when it looses network connection (for instance if I switch my WIFI off on my laptop when it runs on it), I see errors on the console and the application never recovers. It just hangs there and does not reconnect after the network becomes available. I have to kill it and bring it up. How do I force it to recover by itself?
// Spring Boot entry point:
public static void main(String[] args) {
SpringApplication.run(MyConsumerConfiguration.class, args);
}
// Message Listener (A different class)
#MessageMapping(value = "myLogicalQueueName" )
public void receive(MyPOJO object) {
}
The error I see at console:
Exception in thread "simpleMessageListenerContainer-1" com.amazonaws.AmazonClientException: Unable to execute HTTP request: sqs.us-east-1.amazonaws.com
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:473)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:297)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2422)
at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1130)
at com.amazonaws.services.sqs.AmazonSQSAsyncClient$23.call(AmazonSQSAsyncClient.java:1678)
at com.amazonaws.services.sqs.AmazonSQSAsyncClient$23.call(AmazonSQSAsyncClient.java:1676)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745
I just figured out the problem why SQS is not able to reconnect after network connection lost.
Actually seems to be a problem in current Spring AWS implementation of org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.java
private class AsynchronousMessageListener implements Runnable {
private final QueueAttributes queueAttributes;
private final String logicalQueueName;
private AsynchronousMessageListener(String logicalQueueName, QueueAttributes queueAttributes) {
this.logicalQueueName = logicalQueueName;
this.queueAttributes = queueAttributes;
}
#Override
public void run() {
while (isRunning()) {
ReceiveMessageResult receiveMessageResult = getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest());
CountDownLatch messageBatchLatch = new CountDownLatch(receiveMessageResult.getMessages().size());
for (Message message : receiveMessageResult.getMessages()) {
if (isRunning()) {
MessageExecutor messageExecutor = new MessageExecutor(this.logicalQueueName, message, this.queueAttributes);
getTaskExecutor().execute(new SignalExecutingRunnable(messageBatchLatch, messageExecutor));
} else {
break;
}
}
try {
messageBatchLatch.await();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
}
Above code spins up a new thread which does the polling to SQS queue to grab messages. Once network connection is dropped getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest()) throws UnknownHostException, which is not handled in the code and causes thread termination.
So when network connection is established later on, there is no thread polling the queue to retrieve the data.
I have already raised a issue with Spring for this. Following is the link: https://github.com/spring-cloud/spring-cloud-aws/issues/82
Hope this explains it all.
I am new to JMS and have an issue connecting to a remote JMS queue from my standalone client. Any hints on resolving this issue would be highly appreciated.
Right now I have a JavaFX standalone application that runs on multiple clients and a glassfish server 3.1.2.2 running on a remote Unix machine. I am having a hard time pushing messages from my standalone app on to the queue that is residing on the server.
Client Mc: Windows PC (No server installed)
Remote Mc: Unix (GlassFish 3.1.2.2 installed)
JMS resources on the server:
JMS Destination Resource
JNDI Name: jms/ReferralQueue
Physical Destination Name: ReferralQueue
Resource Type: javax.jms.Queue
JMS Connection Factory
Pool Name: jms/ReferralConnectionFactory
JNDI Name: jms/ReferralConnectionFactory
Resource Type: javax.jms.QueueConnectionFactory
JMS Service Type: Embedded
JMS Message Store Type: File
Client Side Code to connect to the server:
jms.properties:
org.omg.CORBA.ORBInitialHost=UNIX MC URL
org.omg.CORBA.ORBInitialPort=7676
Service Locator design to implement resource caching
public class JMSServiceLocator {
private static JMSServiceLocator singletonService = null;
private static QueueConnectionFactory qFactory;
private static Queue queue;
private InitialContext context;
private static Properties properties = new Properties();
private Map cache;
static {
try {
singletonService = new JMSServiceLocator();
} catch (Exception e) {
//error handling
}
}
private JMSServiceLocator() {
try {
loadProperties();
context = new InitialContext(properties);
cache = Collections.synchronizedMap(new HashMap());
} catch (Exception e) {
//error handling
}
}
public static JMSServiceLocator getInstance() {
return singletonService;
}
public QueueConnectionFactory getQueueConnectionFactory() {
String qConnFactoryName = "jms/ReferralConnectionFactory";
qFactory = null;
try {
System.out.println("/********************Comment after Testing*****************************/");
Hashtable env = context.getEnvironment();
System.out.println("**env.size::" + env.size());
Enumeration names = env.keys();
while (names.hasMoreElements()) {
String str = (String) names.nextElement();
System.out.println("**" + str + "=" + env.get(str));
}
System.out.println("/**********************************************************************/");
if (cache.containsKey(qConnFactoryName)) {
qFactory = (QueueConnectionFactory) cache.get(qConnFactoryName);
} else {
qFactory = (QueueConnectionFactory) context.lookup(qConnFactoryName);
cache.put(qConnFactoryName, qFactory);
}
} catch (Exception e) {
//error handling
}
return qFactory;
}
public Queue getQueue() {
String queueName = "jms/ReferralQueue";
queue = null;
try {
if (cache.containsKey(queueName)) {
queue = (Queue) cache.get(queueName);
} else {
queue = (Queue) context.lookup(queueName);
cache.put(queueName, queue);
}
} catch (Exception e) {
//error handling
}
return queue;
}
private static void loadProperties() {
//Load jms properties
}
}
Eventually sending message to the server:
JMSServiceLocator jmsLocator = JMSServiceLocator.getInstance();
QueueConnectionFactory qConnFactory = jmsLocator.getQueueConnectionFactory();
qConnection = qConnFactory.createQueueConnection();
session = qConnection.createSession(false, ession.AUTO_ACKNOWLEDGE);
queue = jmsLocator.getQueue();
// Push and publish the message
messageProducer = session.createProducer(queue);
textMessage = session.createTextMessage();
textMessage.setText(message);
messageProducer.send(textMessage);
Hmmm... Now I observe a strange behavior...
I created a new GlassFish 3.1.2.2 server instance on the client machine with no jndi, no connection factories, and no jms queues what so ever.
I have started this server instance and executed the standalone client application. Strangely, everything works fine and the message is directly being pushed to the remote queue.
Did any one come across this kind of issue? I am suspecting that probably the application is loading the dependent GlassFish jars in the classpath only when a server instance (could be any random instance, totally unrelated) is started.
I have the following jars in my standalone application classpath:
*C:\Program Files\glassfish-3.1.2.2\glassfish\lib\gf-client.jar
*C:\Program Files\glassfish-3.1.2.2\glassfish\lib\appserv-rt.jar
*C:\Program Files\glassfish-3.1.2.2\glassfish\lib\install\applications\jmsra\imqbroker.jar
*C:\Program Files\glassfish-3.1.2.2\glassfish\lib\install\applications\jmsra\imqjmsra.jar
I have also posted this on Oracle JMS and GlassFish forums and haven't got a solution. Any help on this issue would be highly appreciated.
Thanks.
I think you found out by now what the problem was:
the JMS client jars were missing on the client (Client Mc: Windows PC (No server installed)).
You don't need a full Glassfish installation on the clients but only the JMS client jars (gf-client.jar) plus all the other jars referenced by gf-client.jar.