JMS commit fails - spring-boot

We have an application that uses JMS with Tibco messaging server. It’s implemented with Spring Boot.
Our sessions are transacted sessions and the acknowledge mode is set to auto.
We can receive the messages sent to the queue but for some reason that commit() method fails.
This is the error:
2018-07-13 15:50:35.858 WARN 2576 --- [enerContainer-8] o.s.j.l.DefaultMessageListenerContainer : Execution of JMS message listener failed, and no ErrorHandler has been set.
javax.jms.TransactionRolledBackException: Commit failed
2018-07-13 15:50:35.914 WARN 2576 --- [enerContainer-8] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'queue' - trying to recover. Cause: Commit failed; nested exception is javax.jms.JMSException: operation='ack' cause='Illegal state' queue='queue'
I should mention two things:
I see the commit() failure and stack trace, only when I run in debug mode. If I let the war to just run, I don’t see any trace of commit failure in the logs. I can tell that we received the messages, but can’t tell whether the commit() failed or not.
If we use the default JMS implementations of Spring Boot which is ActiveMQ, then the commit() does not fail whether its in debug mode or not, it only happens with Tibco.
Does anybody have any idea that why the commit() fails? When the commit fails, the transaction is rolled back and the messages won’t be cleared from the queue.
Why I see this error only in debug mode? OR: it indeed, does happen when I run the war but just not logged?
Below please find the complete stack trace:
15:50:35.858 [DefaultMessageListenerContainer-8] WARN o.s.j.l.DefaultMessageListenerContainer - Execution of JMS message listener failed, and no ErrorHandler has been set.
javax.jms.TransactionRolledBackException: Commit failed
at com.tibco.tibjms.Tibjmsx.buildException(Tibjmsx.java:596)
at com.tibco.tibjms.TibjmsxSessionImp._confirmTransacted(TibjmsxSessionImp.java:3251)
at com.tibco.tibjms.TibjmsxSessionImp._confirm(TibjmsxSessionImp.java:3643)
at com.tibco.tibjms.TibjmsxSessionImp._commit(TibjmsxSessionImp.java:2898)
at com.tibco.tibjms.TibjmsxSessionImp.commit(TibjmsxSessionImp.java:4860)
at org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:218)
at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:776)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:680)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1189)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1179)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1076)
at java.lang.Thread.run(Unknown Source)
2018-07-13 15:50:35.914 WARN 2576 --- [enerContainer-8] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'queue' - trying to recover. Cause: Commit failed; nested exception is javax.jms.JMSException: operation='ack' cause='Illegal state' queue='queue'
2018-07-13 15:50:36.543 INFO 2576 --- [enerContainer-8] o.s.j.l.DefaultMessageListenerContainer : Successfully refreshed JMS Connection
15:50:36.543 [DefaultMessageListenerContainer-8] INFO o.s.j.l.DefaultMessageListenerContainer - Successfully refreshed JMS Connection

This EMS exception happens when some of your messages within that transaction expired in the meantime. Just tried this one myself.
When you run outside of debugging mode, things seem to be quick enough. But when you are in debugging mode, your time before you commit() is much longer. So if your producer's time-to-live expires in the meantime, the commit assumes that something is wrong (EMS knows that there should be e.g. 5 messages in this transaction, but one of them already expired) and throws this exception.
To verify your producer's time-to-live, see the last option of your MessageProducer.send() call or msgProducer.setTimeToLive() and note, this is a time in millseconds, not seconds. Another way to check if this is the issue, is to run in debug mode and be 'really quick' with continuing your thread. That might change your behaviour.

Related

Error when Spring Cloud Kinesis app is stopped

I have a Spring Boot app with the Spring Cloud binder for Kinesis. Everything works fine, but when I stop the app I get the following exception:
2020-11-23 18:40:36.292 INFO --- [extShutdownHook] a.i.k.KinesisMessageDrivenChannelAdapter : stopped KinesisMessageDrivenChannelAdapter{shardOffsets=[KinesisShardOffset{iteratorType=TRIM_HORIZON, sequenceNumber='null', timestamp=null, stream='product-master-data-updated-event-stream', shard='shardId-000000000000', reset=false}], consumerGroup='marketing-campaign'}
2020-11-23 18:40:36.299 ERROR --- [s-shard-locks-1] a.i.k.KinesisMessageDrivenChannelAdapter : ShardConsumerManager Thread [org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumerManager#71bd846d] has been interrupted
java.lang.InterruptedException: sleep interrupted
at java.base/java.lang.Thread.sleep(Native Method)
at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter.sleep(KinesisMessageDrivenChannelAdapter.java:668)
at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter.access$1400(KinesisMessageDrivenChannelAdapter.java:100)
at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumerManager.run(KinesisMessageDrivenChannelAdapter.java:1431)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
I'm using
implementation 'org.springframework.cloud:spring-cloud-stream-binder-kinesis:2.0.3.RELEASE'
springBoot: "2.2.7.RELEASE",
How can I stop the app gracefully? An exception on every deployment doesn't look nice.
The exception is just logged under ERROR level. You can move it for now to the FATAL logging level for that a.i.k.KinesisMessageDrivenChannelAdapter category.
We just need to fix this KinesisMessageDrivenChannelAdapter.ShardConsumerManager to avoid exception logging and rethrowing when the channel adapter is already in a stopped state.
Feel free to raise a GH issue and we will see what and how we can fix over there.
The thread interruption in that logic is correct behavior and should not effect anything in your application.
UPDATE
Related GH issue: https://github.com/spring-projects/spring-integration-aws/issues/187

How to safely resume cache operation on client side after Hazelcast restart?

Whenever I restart hazelcast server, without restarting client in spring boot. I'm getting following error :
03-01-2018 16:44:17.966 [http-nio-8080-exec-7] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet].log - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.hazelcast.client.HazelcastClientNotActiveException: Partition does not have owner. partitionId : 203] with root cause
java.io.IOException: Partition does not have owner. partitionId : 203
at com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.invokeOnPartitionOwner(ClientSmartInvocationServiceImpl.java:43)
at com.hazelcast.client.spi.impl.ClientInvocation.invokeOnSelection(ClientInvocation.java:142)
at com.hazelcast.client.spi.impl.ClientInvocation.invoke(ClientInvocation.java:122)
at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:152)
at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:147)
at com.hazelcast.client.proxy.ClientMapProxy.getInternal(ClientMapProxy.java:245)
at com.hazelcast.client.proxy.ClientMapProxy.get(ClientMapProxy.java:240)
at com.hazelcast.spring.cache.HazelcastCache.lookup(HazelcastCache.java:139)
at com.hazelcast.spring.cache.HazelcastCache.get(HazelcastCache.java:57)
at org.springframework.cache.interceptor.AbstractCacheInvoker.doGet(AbstractCacheInvoker.java:71)
If I enabled hot-restart, the issue is solved. But is there a way to resume client application without restarting it and hot-restart is disabled ?
Hazelcast client tries to reconnect to the cluster if the connection drops. It uses ClientNetworkConfig.connectionAttemptLimit and ClientNetworkConfig.connectionAttemptPeriod elements to configure how frequently it will try. connectionAttemptLimit defines the number of attempts on a disconnection and connectionAttemptPeriod defines the period between two retries in ms. Please see the usage example below:
ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig().setConnectionAttemptLimit(5);
clientConfig.getNetworkConfig().setConnectionAttemptPeriod(5000);
Starting with Hazelcast 3.9, you can use reconnect-mode property to configure how the client will reconnect to the cluster after it disconnects. It has three options:
The option OFF disables the reconnection.
ON enables reconnection in a blocking manner where all the waiting invocations will be blocked until a cluster connection is established or failed.
The option ASYNC enables reconnection in a non-blocking manner where all the waiting invocations will receive a HazelcastClientOfflineException.
Its default value is ON. You can see a configuration example below:
ClientConfig clientConfig = new ClientConfig();
clientConfig.getConnectionStrategyConfig()
.setReconnectMode(ClientConnectionStrategyConfig.ReconnectMode.ON);
By using these configuration elements, you can resume your client without restarting it.

Spring JMS listener-container listener setup timeout

I recently upgraded to spring 4.1 from 3.1 and the jms:listener-container doesn't seem to wait for a timeout before retrying to setup listeners, as it used to in 3.1. I've tried recovery-interval="10000" and it's still trying around 3 times a second.
Here's the section of my spring config:
<jms:listener-container recovery-interval="10000" connection-factory="connectionFactory">
<jms:listener destination="${myQueueDestination}" ref="myListener" />
<!-- many others -->
</jms:listener-container>
My logs on 4.1 look like that (see times):
11/09 11:56:33 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] Setup of JMS message listener invoker failed f or destination 'MYQUEUE.NAME' - trying to recover. Cause: MQJMS2008: failed to open MQ queue ''.; nested exception is com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2085'.
11/09 11:56:33 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] Setup of JMS message listener invoker failed for destination 'MYQUEUE.NAME' - trying to recover. Cause: MQJMS2008: failed to open MQ queue ''.; nested exception is com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2085'.
11/09 11:56:34 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] Setup of JMS message listener invoker failed f (...)
And 3.1 looks like this (see 5 second delay):
11/09 12:14:32 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] Setup of JMS message listener invoker failed for destination 'MYQUEUE.NAME' - trying to recover. Cause: MQJMS2008: failed to open MQ queue ''.; nested exception is com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2085'.
11/09 12:14:37 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] Setup of JMS message listener invoker failed for destination 'MYQUEUE.NAME' - trying to recover. Cause: MQJMS2008: failed to open MQ queue ''.; nested exception is com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2085'.
11/09 12:14:43 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] Setup of JMS message listener invoker failed for destination 'MYQUEUE.NAME' - trying to recover. Cause: MQJMS2008: failed to open MQ queue ''.; nested exception is com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2085'.
When the queues are back up, it's all OK. Are these timeout settings available for configuration?
Regards
This definitely looks like a regression to me and something I just happened to find out yesterday as well. I have created this issue to track that problem.

Spring JMS CachingConnectionFactory doesn't seem to reconnect on exception

The JMS connection didn't get reconnected when this exception happened:
2014-08-16 10:51:36,724 WARN [TIBCO EMS TCPLink Reader (Server-12486338)] connection.CachingConnectionFactory - Encountered a JMSException - resetting the underlying JMS Connection
javax.jms.JMSException: Connection has been terminated
at com.tibco.tibjms.Tibjmsx.buildException(Tibjmsx.java:502)
at com.tibco.tibjms.TibjmsConnection._onDisconnected(TibjmsConnection.java:2286)
at com.tibco.tibjms.TibjmsConnection$ServerLinkEventHandler.onEventDisconnected(TibjmsConnection.java:341)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:320)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:247)
2014-08-16 10:51:40,286 WARN [TIBCO EMS TCPLink Reader (Server-12494013)] connection.CachingConnectionFactory - Encountered a JMSException - resetting the underlying JMS Connection
javax.jms.JMSException: Connection has been terminated
at com.tibco.tibjms.Tibjmsx.buildException(Tibjmsx.java:502)
at com.tibco.tibjms.TibjmsConnection._onDisconnected(TibjmsConnection.java:2286)
at com.tibco.tibjms.TibjmsConnection$ServerLinkEventHandler.onEventDisconnected(TibjmsConnection.java:341)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:320)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:247)
The exact same error happened a few seconds apart and I wonder if Spring tried to reconnect and gave up after two tries.
I checked Spring source code and didn't find reconnection logic in onException() method in either CachingConnectionFactory or its parent class SingleConnectionFactory. We're using spring-jms 3.1.2 but I looked at the latest Spring source code as well.
Thanks for your help.
I had similar issue where DefaultMessageListenerContainer is unable to reconnect once an exception happens(when TIBCO primary server stops and switch over to secondary TIBCO server).
I was using SingleConnectionFactory and it was causing the issue.
Fetching the connectionFactory directly from Websphere using java:comp/env/jms/TIBCOConnectionFactory resolved this issue.

mqjbnd05 error when deploying app on websphere

I have a fresh install of Wesphere 6.1 Fixpack 23. I have an app deployed that requires an MQSeries JMS Queue. I set up an MQSeries provider-based request and reply queue and an MQSeries provider-based queue connection factory. When the deployed app tries to access the queue, I receive the following error.
Any assistance would be appreciated. Thanks!
[5/28/09 10:33:42:538 EDT] 00000033 ServletWrappe E SRVE0068E: Uncaught exception thrown in one of the service methods of the servlet: espaapp. Exception thrown : org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.UnsatisfiedLinkError: mqjbnd05 (Not found in java.library.path)
Caused by: java.lang.UnsatisfiedLinkError: mqjbnd05 (Not found in java.library.path)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:953)
at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:922)
at java.lang.System.loadLibrary(System.java:451)
at com.ibm.mq.MQSESSION.loadLib(MQSESSION.java:1028)
at com.ibm.mq.server.MQSESSION$1.run(MQSESSION.java:246)
at java.security.AccessController.doPrivileged(AccessController.java:192)
at com.ibm.mq.server.MQSESSION.(MQSESSION.java:243)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:177)
at com.ibm.mq.MQSESSIONServer.getMQSESSION(MQSESSIONServer.java:68)
at com.ibm.mq.MQSESSION.getSession(MQSESSION.java:508)
at com.ibm.mq.MQManagedConnectionJ11.(MQManagedConnectionJ11.java:213)
at com.ibm.mq.MQBindingsManagedConnectionFactoryJ11._createManagedConnection(MQBindingsManagedConnectionFactoryJ11.java:186)
at com.ibm.mq.MQBindingsManagedConnectionFactoryJ11.createManagedConnection(MQBindingsManagedConnectionFactoryJ11.java:225)
at com.ibm.mq.StoredManagedConnection.(StoredManagedConnection.java:84)
at com.ibm.mq.MQSimpleConnectionManager.allocateConnection(MQSimpleConnectionManager.java:173)
at com.ibm.mq.MQQueueManagerFactory.obtainBaseMQQueueManager(MQQueueManagerFactory.java:795)
at com.ibm.mq.MQQueueManagerFactory.procure(MQQueueManagerFactory.java:709)
at com.ibm.mq.MQQueueManagerFactory.constructQueueManager(MQQueueManagerFactory.java:664)
at com.ibm.mq.MQQueueManagerFactory.createQueueManager(MQQueueManagerFactory.java:160)
at com.ibm.mq.MQQueueManager.(MQQueueManager.java:550)
at com.ibm.mq.MQSPIQueueManager.(MQSPIQueueManager.java:62)
at com.ibm.mq.jms.MQConnection.createQM(MQConnection.java:2427)
at com.ibm.mq.jms.MQConnection.createQMXA(MQConnection.java:1806)
at com.ibm.mq.jms.MQQueueConnection.(MQQueueConnection.java:105)
at com.ibm.mq.jms.MQQueueConnection.(MQQueueConnection.java:66)
at com.ibm.mq.jms.MQXAQueueConnection.(MQXAQueueConnection.java:59)
at com.ibm.mq.jms.MQXAQueueConnectionFactory.createXAQueueConnection(MQXAQueueConnectionFactory.java:82)
at com.ibm.ejs.jms.JMSManagedQueueConnection.createConnection(JMSManagedQueueConnection.java:123)
at com.ibm.ejs.jms.JMSManagedConnection.(JMSManagedConnection.java:315)
at com.ibm.ejs.jms.JMSManagedQueueConnection.(JMSManagedQueueConnection.java:71)
... More
Does this help?
java.lang.UnsatisfiedLinkError occurs when connecting to a queue manager
Also, within the JMS -> Queue Connection Factories section, select your Queue Connection Factory and check if your "Transport Type" is set to 'BINDINGS' or 'CLIENT'
I swapped mine to CLIENT and that seemed to help a lot.
Such error often happens as a result of passing null to port, host or QManager to connection factory. Try to check all parameters during execution. Normally MQ does not require mqjbnd05 library.
Try to find this file
libmqjbnd05.so
Add that to the LIBPATH for your JVM and try again.
GO to WebSphere Admin console. Environment -> WebSphere variables. Look for MQ_INSTALL_ROOT and modify its value to your MQ installation directory [MQ Home].

Resources