We were trying to send MAP (java.util) using JMS template , but at the client side we are unable to read the message and during the server start up , spring boot #JMSListener complains with the following message , this works fine with WAS 7 but doesnt with WAS Liberty 8.5.5 , iam also including the code snippet and also the different message format's that we are receiving for the different versions of server . Did any thing changed in WAS Liberty ?
WAS LIBERTY - 7 -
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:116) ~[spring-messaging-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
2019-07-05T13:07:14.357-05:00 [APP/PROC/WEB/0] [OUT] at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:114) ~[spring-jms-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
2019-07-05T13:07:14.357-05:00 [APP/PROC/WEB/0] [OUT] ... 10 common frames omitted
2019-07-05T13:07:14.393-05:00 [APP/PROC/WEB/0] [OUT] 2019-07-05 18:07:14.370 WARN 20 --- [enerContainer-1] o.s.j.l.DefaultMessageListenerContainer : Execution of JMS message listener failed, and no ErrorHandler has been set.
2019-07-05T13:07:14.394-05:00 [APP/PROC/WEB/0] [OUT] org.springframework.jms.listener.adapter.ListenerExecutionFailedException: Listener method 'public void com.aa.amps.base.task.finalize.BaseBowUpdateListener.receiveMessage(javax.jms.MapMessage)' threw exception; nested exception is org.springframework.jms.support.converter.MessageConversionException: Could not find type id property [_type] on message [ID:414d51204d454d5154433120202020205ccc4b2725acdc4c] from destination [null]
message is MAP
jmsTemplate.convertAndSend(destination, message, new MessagePostProcessor() {
#Override
public Message postProcessMessage(Message message) throws JMSException {
message.setJMSReplyTo(null);
LOGGER.info("In BaseBowUpdateSender - send() message to AMPS-BASE QUEUE : {} ", message);
return message;
}
});
WAS - LIBERTY message format
JMSMessage class: jms_text
JMSType: null
JMSDeliveryMode: 2
JMSDeliveryDelay: 0
JMSDeliveryTime: 0
JMSExpiration: 0
JMSPriority: 4
JMSMessageID: ID:414d51204d454d5154433120202020205ccc4b272634ce4d
JMSTimestamp: 1562784513140
JMSCorrelationID: null
JMSDestination: null
JMSReplyTo: null
JMSRedelivered: true
JMSXAppID: jar
JMSXDeliveryCount: 193
JMSXUserID: apmqstc
JMS_IBM_Character_Set: UTF-8
JMS_IBM_Encoding: 273
JMS_IBM_Format: MQSTR
JMS_IBM_MsgType: 8
JMS_IBM_PutApplType: 28
JMS_IBM_PutDate: 20190710
JMS_IBM_PutTime: 18483314
<map><elt name="trackType">03</elt><elt name="lastDefferDate">07/10/2019</elt><elt name="packageDesc ...
Expected message below
JMSMessage class: jms_map
JMSType: null
JMSDeliveryMode: 2
JMSDeliveryDelay: 0
JMSDeliveryTime: 0
JMSExpiration: 0
JMSPriority: 4
JMSMessageID: ID:414d51204d454d5154433120202020205ccc4b272634404a
JMSTimestamp: 1562783439854
JMSCorrelationID: null
JMSDestination: queue://- INFO IS THERE
JMSReplyTo: null
JMSRedelivered: false
JMSXAppID: WebSphere MQ Client for Java
JMSXDeliveryCount: 1
JMSXUserID: apmqstc
JMS_IBM_Character_Set: UTF-8
JMS_IBM_Encoding: 273
JMS_IBM_Format: MQSTR
JMS_IBM_MsgType: 8
JMS_IBM_PutApplType: 28
JMS_IBM_PutDate: 20190710
JMS_IBM_PutTime: 18303986
WebSphere MQ messages are being put by my application(JMS application) to a queue. These messages are to be picked up by a jms application. I have misunderstood targetClientId option available thinking that the targetClientId needed to be mentioned as "mq" if the messages where we write is an MQ but its not the case .We need to set targetClientId as MQ only if we the messages are processed by non-jms application. As i was setting the targetClientId as 'MQ' its stripping the RFH headers needed for the JMS application to process the message and parsing fails at the consuming application .
<jmsQueue id="MYAPP_QUEUE" jndiName="jms/test/testq">
<properties.wmqJms
baseQueueName="MYAPP.TEST.A"
persistence="PERS"
targetClient="MQ"/> -- removed this
</jmsQueue>
We are using Spring in our Web application.
I want to concurrently listen the messages from MQ using using Spring's DefaultMessageListenerContainer(DMLC).
In websphere I didnt change any connection configurations.
Issue:
When DMLC listner tried to receive few(6 messages) messages concurrently from Queue, Websphere Application Server logged FFDC incident logs.
Can anyone help me If I missing the connection configurations?
Version:
Server: Websphere Application Server 8.5
MessageListener:
<bean id="jmsContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="connectionFactory"/>
<property name="destinationName" ref="queue"/>
<property name="messageListener" ref="myMessageListener"/>
<property name ="concurrentConsumers" value ="5"/>
<property name ="maxConcurrentConsumers" value ="20"/>
<property name="idleTaskExecutionLimit" value="10" />
<property name="idleConsumerLimit" value="5" />
<property name="recoveryInterval" value="60000" />
</bean>
QueueConnectionfactory - Connection pools:
Logs in Server logs:
[8/1/17 12:47:54:071 ] FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource JMS$ConnectionFactory$JMSManagedConnection#0.
[8/1/17 12:43:44:131 ] FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on WebSphere/path/path/logFileName.txt Max connections reached 869
[8/1/17 12:43:44:131 ] FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on WebSphere/path/path/logFileName.txt com.ibm.ejs.jms.JMSConnectionHandle.createSession 1051
[8/1/17 12:44:44:132 ] SystemOut O 12:44:44.132 [jmsContainer-11] WARN o.s.j.l.DefaultMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'queue:' - trying to recover. Cause: Failed to create session; nested exception is com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: CWTE_NORMAL_J2CA1009
[8/1/17 12:44:45:153 ] FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on WebSphere/path/path/logFileName.txt com.ibm.ejs.jms.JMSMessageConsumerHandle.receive 264
[8/1/17 12:44:55:304 ] SystemOut O 12:44:55.304 [jmsContainer-11] INFO o.s.j.l.DefaultMessageListenerContainer - Successfully refreshed JMS Connection
[8/1/17 12:47:55:310 ] FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource JMS$ConnectionFactory$JMSManagedConnection#0
FFDC log incident:
[8/1/17 13:41:03:046] FFDC Exception:com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException SourceId:Max connections reached ProbeId:869 Reporter:com.ibm.ejs.j2c.PoolManager#aad4897
com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: CWTE_NORMAL_J2CA1009
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1782)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3816)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3092)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1548)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:1031)
at com.ibm.ejs.jms.JMSConnectionHandle.createSession(JMSConnectionHandle.java:1434)
at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
at org.springframework.jms.listener.DefaultMessageListenerContainer.access$12(DefaultMessageListenerContainer.java:1)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerContainer.java:1091)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1070)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1063)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:960)
at java.lang.Thread.run(Thread.java:863)
==> Performing default dump from com.ibm.ejs.j2c.DiagnosticModuleForJ2C :Tue Aug 01 13:41:03 2017
Maximum number of connections has been reached, and the connection request has been waiting longer than:ConnectionWaitTime. Two possible solutions : increase the max number of connections, or increase the:ConnectionWaitTime.:
Maximum Connections = :10
Current number of connections = :10
Connection Wait Timout = :180
If the current number of connections is not greater than or equal to max connections, :there has been a WebSphere internal error.:
This = BEGIN:com.ibm.ejs.j2c.PoolManager#aad4897
com.ibm.ejs.j2c.PoolManager::tc BEGIN:com.ibm.ejs.ras.TraceComponent#330b9221
com.ibm.ejs.ras.TraceElement::ivLevel:10
com.ibm.ejs.ras.TraceElement::ivName:com.ibm.ejs.j2c.PoolManager
com.ibm.ejs.ras.TraceElement::ivDebugEnabled:false
com.ibm.ejs.ras.TraceElement::ivEventEnabled:false
com.ibm.ejs.ras.TraceElement::ivEntryEnabled:false
com.ibm.ejs.ras.TraceElement::ivDetailEnabled:false
com.ibm.ejs.ras.TraceElement::ivConfigEnabled:false
com.ibm.ejs.ras.TraceElement::ivInfoEnabled:true
com.ibm.ejs.ras.TraceElement::ivServiceEnabled:true
com.ibm.ejs.ras.TraceElement::ivWarningEnabled:true
com.ibm.ejs.ras.TraceElement::ivErrorEnabled:true
com.ibm.ejs.ras.TraceElement::ivFatalEnabled:true
com.ibm.ejs.ras.TraceComponent::defaultMessageFile:com.ibm.ejs.resources.seriousMessages
com.ibm.ejs.ras.TraceComponent::EXTENSION_NAME_DPID:DiagnosticProvider
com.ibm.ejs.ras.TraceComponent::ivDumpEnabled:false
com.ibm.ejs.ras.TraceComponent::ivResourceBundleName:com.ibm.ws.j2c.resources.J2CAMessages
com.ibm.ejs.ras.TraceComponent::ivLogger:com.ibm.ws.logging.WsLogger#8273a12c
com.ibm.ejs.ras.TraceComponent::ivDiagnosticProviderID:null
com.ibm.ejs.ras.TraceComponent::anyTracingEnabled:null
END:com.ibm.ejs.ras.TraceComponent#330b9221
[8/1/17 13:42:03:108 ] FFDC Exception:com.ibm.msg.client.jms.DetailedIllegalStateException SourceId:com.ibm.ejs.jms.JMSMessageConsumerHandle.receive ProbeId:264 Reporter:com.ibm.ejs.jms.JMSMessageConsumerHandle#ab2dfd8d
com.ibm.msg.client.jms.DetailedIllegalStateException: JMSCC0032: This message consumer is closed.
An application called a method that must not be used after the message consumer is closed.
Ensure that the message consumer is not closed before calling the method.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:86)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:58)
at java.lang.reflect.Constructor.newInstance(Constructor.java:542)
at com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:314)
at com.ibm.msg.client.commonservices.nls.NLSServices.createException(NLSServices.java:228)
at com.ibm.msg.client.jms.internal.JmsErrorUtils.createException(JmsErrorUtils.java:104)
at com.ibm.msg.client.jms.internal.State.checkNotClosed(State.java:145)
at com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receive(JmsMessageConsumerImpl.java:413)
at com.ibm.mq.jms.MQMessageConsumer.receive(MQMessageConsumer.java:212)
at com.ibm.ejs.jms.JMSMessageConsumerHandle.receive(JMSMessageConsumerHandle.java:553)
at com.ibm.ejs.jms.JMSMessageConsumerHandle.receive(JMSMessageConsumerHandle.java:501)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:431)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:311)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:264)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1071)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1063)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:960)
at java.lang.Thread.run(Thread.java:863)
==> Performing default dump from com.ibm.ejs.jms.JMSDiagnosticModule :Tue Aug 01 13:42:03 2017
+Data for directive [defaultdefault] obtained.:
==> Dump complete for com.ibm.ejs.jms.JMSDiagnosticModule :Tue Aug 01 13:42:03 CEST 201
[8/1/17 13:50:13:445] FFDC Exception:javax.jms.IllegalStateException SourceId:com.ibm.ejs.jms.JMSConnectionHandle.createSession ProbeId:1044 Reporter:com.ibm.ejs.jms.JMSConnectionHandle#e8e11802
javax.jms.IllegalStateException: Connection closed
at com.ibm.ejs.jms.JMSConnectionHandle.checkOpen(JMSConnectionHandle.java:837)
at com.ibm.ejs.jms.JMSConnectionHandle.createSession(JMSConnectionHandle.java:1430)
at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
at org.springframework.jms.listener.DefaultMessageListenerContainer.access$12(DefaultMessageListenerContainer.java:1)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerContainer.java:1091)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1070)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1063)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:960)
at java.lang.Thread.run(Thread.java:863)
8/2/17 15:30:31:874 CEST] FFDC Exception:javax.jms.IllegalStateException SourceId:com.ibm.ejs.jms.JMSConnectionHandle.stop ProbeId:477 Reporter:com.ibm.ejs.jms.JMSConnectionHandle#5c5db824
javax.jms.IllegalStateException: Method stop not permitted
at com.ibm.ejs.jms.JMSCMUtils.methodNotPermittedException(JMSCMUtils.java:244)
at com.ibm.ejs.jms.JMSConnectionHandle.checkRestrictedMethod(JMSConnectionHandle.java:854)
at com.ibm.ejs.jms.JMSConnectionHandle.stop(JMSConnectionHandle.java:768)
at org.springframework.jms.listener.AbstractJmsListeningContainer.stopSharedConnection(AbstractJmsListeningContainer.java:460)
at org.springframework.jms.listener.DefaultMessageListenerContainer.stopSharedConnection(DefaultMessageListenerContainer.java:791)
at org.springframework.jms.listener.AbstractJmsListeningContainer.doStop(AbstractJmsListeningContainer.java:326)
at org.springframework.jms.listener.AbstractJmsListeningContainer.stop(AbstractJmsListeningContainer.java:302)
at org.springframework.jms.listener.DefaultMessageListenerContainer.stop(DefaultMessageListenerContainer.java:582)
at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:223)
at org.springframework.context.support.DefaultLifecycleProcessor.access$2(DefaultLifecycleProcessor.java:207)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:356)
at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:196)
at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:113)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1045)
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1000)
at com.bnppf.ssc.imb.sf.fwk.mia.context.CloseableGenericWebApplicationContext.close(CloseableGenericWebApplicationContext.java:34)
at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:548)
at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:142)
at com.ibm.ws.webcontainer.webapp.WebApp.notifyServletContextDestroyed(WebApp.java:1765)
at com.ibm.ws.webcontainer.webapp.WebApp.destroy(WebApp.java:3131)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.destroy(WebAppImpl.java:1551)
at com.ibm.ws.container.AbstractContainer.destroy(AbstractContainer.java:75)
at com.ibm.ws.webcontainer.webapp.WebGroup.destroy(WebGroup.java:228)
at com.ibm.ws.webcontainer.webapp.WebGroup.removeWebApplication(WebGroup.java:269)
at com.ibm.ws.webcontainer.VirtualHost.removeWebApplication(VirtualHost.java:296)
at com.ibm.ws.webcontainer.VirtualHostImpl.removeWebApplication(VirtualHostImpl.java:211)
at com.ibm.ws.webcontainer.WSWebContainer.removeWebApplication(WSWebContainer.java:820)
at com.ibm.ws.webcontainer.component.WebContainerImpl.uninstall(WebContainerImpl.java:458)
at com.ibm.ws.webcontainer.component.WebContainerImpl.stop(WebContainerImpl.java:729)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.stop(ApplicationMgrImpl.java:1218)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStop(DeployedApplicationImpl.java:1375)
at com.ibm.ws.runtime.component.DeployedModuleImpl.stop(DeployedModuleImpl.java:671)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.stop(DeployedApplicationImpl.java:1149)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.stopApplication(ApplicationMgrImpl.java:952)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.stopApplicationImpl(ApplicationMgrImpl.java:1488)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.stop(ApplicationMgrImpl.java:2249)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.stop(CompositionUnitMgrImpl.java:492)
at com.ibm.ws.runtime.component.CompositionUnitImpl.stop(CompositionUnitImpl.java:141)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$2.run(CompositionUnitMgrImpl.java:795)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5387)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5603)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.stopCompositionUnit(CompositionUnitMgrImpl.java:830)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.stopCompositionUnit(CompositionUnitMgrImpl.java:736)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.stopApplication(ApplicationMgrImpl.java:1465)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:88)
at sun.reflect.GeneratedMethodAccessor133.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:292)
at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1261)
at java.security.AccessController.doPrivileged(AccessController.java:311)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:88)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1255)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1093)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:832)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:814)
at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1335)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:1228)
at com.ibm.ws.management.connector.AdminServiceDelegator.invoke(AdminServiceDelegator.java:181)
at com.ibm.ws.management.connector.ipc.CallRouter.route(CallRouter.java:247)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink.doWork(IPCConnectorInboundLink.java:360)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink$IPCConnectorReadCallback.complete(IPCConnectorInboundLink.java:602)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
Looks like your current Spring configuration is draining the whole connection pool. So either configuration is not correct or you are leaking connections somewhere.
==> Performing default dump from com.ibm.ejs.j2c.DiagnosticModuleForJ2C :Tue Aug 01 13:41:03 2017
Maximum number of connections has been reached, and the connection request has been waiting longer than:ConnectionWaitTime. Two possible solutions : increase the max number of connections, or increase the:ConnectionWaitTime.:
Maximum Connections = :10
Current number of connections = :10
Connection Wait Timout = :180
so either fix the configuration - maybe start with the simpler one and remove:
<property name ="maxConcurrentConsumers" value ="20"/>
<property name="idleTaskExecutionLimit" value="10" />
<property name="idleConsumerLimit" value="5" />
or just use normal Java EE MDB for reciving messages instead of this Spring.
I'm seeing a bunch of messages on the dead letter queue and I don't understand what causes this.
I'm using MQ Explorer to browse such messages. Here's what I see in the Dead-Letter Header:
This doesn't tell me what the real cause of the problem is. How can I find out ?
I've read this article from IBM and it tells that the reason is likely a badly formatted message. In what way badly formatted?
(note: I'm in control of both producer and consumer)
you miss the forest for the trees :-)
The cause is in the field 'Reason' .
MQRC_BACKOUT_THRESHOLD_REACHED
This is described here in Knowledge Center
MQRC_BACKOUT_THRESHOLD_REACHED (0x93A; 2362)
Cause
The message has reached the Backout Threshold defined on the QLOCAL, but no Backout
Queue is defined. On platforms where you cannot define the Backout
Queue, the message has reached the JMS-defined backout threshold of
20.
Action
If this is not wanted, define the Backout Queue for the relevant QLOCAL. Also look for the cause of the multiple backouts.
As I expected the MQRC_BACKOUT_THRESHOLD_REACHED reason is really just a knock-on effect. To find the real reason you would need to look into the logs on either the consumer-side or the producer-side depending on what you see in the Put application name field in the screenshot from the dead-letter header (above).
I've now learned that MQ Classes for JMS produces log files in current dir named mqjms.log.x. By looking at this I can see the true reason for the problem:
July 19, 2016 4:48:33 PM CEST[Queue Service thread] com.ibm.msg.client.wmq.common.internal.messages.WMQReceiveMarshal
A received message could not be correctly parsed.
EXPLANATION:
The message with messageID = '414D512064657620202020202020202012048D5720064E04' and correlationID = '310000000000000000000000000000000000000000000000' could not be correctly parsed. The last successful data read from the message was at position '192' in buffer '0000: 5246 4820 0000 0002 0000 00c0 0000 0111 RFH ............
0010: 0000 04b8 4d51 5354 5220 2020 0000 0000 ....MQSTR ....
0020: 0000 04b8 0000 0020 3c6d 6364 3e3c 4d73 ....... <mcd><Ms
0030: 643e 6a6d 735f 7465 7874 3c2f 4d73 643e d>jms_text</Msd>
0040: 3c2f 6d63 643e 2020 0000 0074 3c6a 6d73 </mcd> ...t<jms
0050: 3e3c 4473 743e 7175 6575 653a 2f2f 2f6d ><Dst>queue:///m
0060: 7971 7565 7565 3c2f 4473 743e 3c54 6d73 yqueue</Dst><Tms
0070: 3e31 3436 3839 3339 3731 3338 3234 3c2f >1468939713824</
0080: 546d 733e 3c45 7870 3e31 3436 3839 3339 Tms><Exp>1468939
0090: 3734 3338 3234 3c2f 4578 703e 3c43 6964 743824</Exp><Cid
00a0: 3e49 443a 3331 3c2f 4369 643e 3c44 6c76 >ID:31</Cid><Dlv
00b0: 3e31 3c2f 446c 763e 3c2f 6a6d 733e 2020 >1</Dlv></jms>
00c0: 3c64 6174 614d 7367 2073 656e 7454 696d <mymessage .....
.................
' with exception '
Message : java.lang.Exception
Class : class java.lang.Exception
Stack : com.ibm.msg.client.wmq.internal.WMQConsumerShadow.getMsg(WMQConsumerShadow.java:1900)
: com.ibm.msg.client.wmq.internal.WMQSyncConsumerShadow.receiveInternal(WMQSyncConsumerShadow.java:231)
: com.ibm.msg.client.wmq.internal.WMQConsumerShadow.receive(WMQConsumerShadow.java:1471)
: com.ibm.msg.client.wmq.internal.WMQMessageConsumer.receive(WMQMessageConsumer.java:659)
: com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receiveInboundMessage(JmsMessageConsumerImpl.java:1036)
: com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receive(JmsMessageConsumerImpl.java:461)
: com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receive(JmsMessageConsumerImpl.java:495)
: com.ibm.mq.jms.MQMessageConsumer.receive(MQMessageConsumer.java:209)
: org.mycorp.client.base.connection.QueueService.recvMessages(QueueService.java:129)
: org.mycorp.client.base.connection.QueueService.run(QueueService.java:92)
: java.lang.Thread.run(Thread.java:745)
' with MQMD 'version:2(0x2) report:0(0x0) msgType:8(0x8) expiry:300(0x12c) feedback:0(0x0) encoding:273(0x111) codedCharSetId:1208(0x4b8) format:'MQHRF2 ' priority:4(0x4) persistence:0(0x0) msgId:414D512064657620202020202020202012048D5720064E04 correlId:310000000000000000000000000000000000000000000000 backoutCount:0(0x0) replyToQ:' ' replyToQMgr:'dev ' userIdentifier:'peter ' accountingToken:160105150000008D3439C9CC13CC025B66F34BE903000000000000000000000B applIdentityData:' ' putApplType:28(0x1c) putApplName:'Carrefour Server ' putDate:'20160719' putTime:'14483382' applOriginData:' ' groupId:000000000000000000000000000000000000000000000000 msgSeqNumber:1(0x1) physicalMsgOffset:0(0x0) msgFlags:0(0x0) originalLength:-1(0xffffffff) '
EXPLANATION:
null
ACTION:
null
So there you have it. Somehow I've managed to create a JMS message which is invalid and without the producer-side noticing that there's a problem.
I'll need to figure that out but that'll be a topic for another post.
In short the answer to the question is: The backout is just a knock-on effect. The real reason is - as the doc from IBM says - a badly formatted message. The only method to figure that out is to look into whatever logs are dumped by either the message producer or (more likely) the message consumer. In my case - since no remote queues are involved - my message consumer is not an intermediary Queue Manager but my actual destination application. That's where I found the log.
I'm configuring Spring Integration to use cleanSession=false on one of my channels.
<bean id="clientFactory" class="org.springframework.integration.mqtt.core.DefaultMqttPahoClientFactory">
<property name="cleanSession" value="false" />
</bean>
<int-mqtt:message-driven-channel-adapter id="mqttLiveDataInbound"
client-id="client1"
url="${mqtt.broker.url}"
qos="1"
topics="liveData"
client-factory="clientFactory"
channel="channelLiveData"/>
Reason being I want to be able to receive message while my application is offline. When my application restarts, I want it to receive whatever QoS>0 message that were published during my absence.
Now I noticed something weird : my application doesn't pick up the missing QoS>0 messages after a downtime.
I've logged a simple scenario where
Spring Integration starts
A QoS1 message is sent to the topic and received by Spring Integration
Spring Integration exits
A QoS1 message is sent to the topic
Spring Integration starts
Spring Integration does not receive the QoS1 message that was sent while it was offline.
The reason being the following (as can be seen from the logs below) :
spring integration exits
it unsubscribes from the topic
it disconnects the client
This is essentially telling the broker that this client is no longer interested in these messages. When my app is down, the broker is no longer persisting these QoS>0 messages for me.
When my app starts up again, it fails to receive the QoS>0 messages that were published while it was down.
1448917620: New connection from 127.0.0.1 on port 1883.
1448917620: New client connected from 127.0.0.1 as client1 (c0, k60).
1448917620: Sending CONNACK to client1 (0, 0)
1448917620: Received SUBSCRIBE from client1
1448917620: liveData (QoS 1)
1448917620: Sending SUBACK to client1
1448917632: New connection from ::1 on port 1883.
1448917632: New client connected from ::1 as mosqpub/25936-MacBook-P (c1, k60, u'system').
1448917632: Sending CONNACK to mosqpub/25936-MacBook-P (0, 0)
1448917632: Received PUBLISH from mosqpub/25936-MacBook-P (d0, q1, r0, m1, 'liveData', ... (68 bytes))
1448917632: Sending PUBACK to mosqpub/25936-MacBook-P (Mid: 1)
1448917632: Sending PUBLISH to client1 (d0, q1, r0, m1, 'liveData', ... (68 bytes))
1448917632: Received DISCONNECT from mosqpub/25936-MacBook-P
1448917632: Client mosqpub/25936-MacBook-P disconnected.
1448917633: Received PUBACK from client1 (Mid: 1)
1448917643: Received UNSUBSCRIBE from client1
1448917643: liveData
1448917643: Received DISCONNECT from client1
1448917643: Client client1 disconnected.
1448917648: New connection from ::1 on port 1883.
1448917648: New client connected from ::1 as mosqpub/25945-MacBook-P (c1, k60, u'system').
1448917648: Sending CONNACK to mosqpub/25945-MacBook-P (0, 0)
1448917648: Received PUBLISH from mosqpub/25945-MacBook-P (d0, q1, r0, m1, 'liveData', ... (68 bytes))
1448917648: Sending PUBACK to mosqpub/25945-MacBook-P (Mid: 1)
1448917648: Received DISCONNECT from mosqpub/25945-MacBook-P
1448917648: Client mosqpub/25945-MacBook-P disconnected.
1448917665: New connection from 127.0.0.1 on port 1883.
1448917665: Client client1 disconnected.
1448917665: New client connected from 127.0.0.1 as client1 (c0, k60).
1448917665: Sending CONNACK to client1 (1, 0)
1448917665: Received SUBSCRIBE from client1
1448917665: liveData (QoS 1)
1448917665: Sending SUBACK to client1
I ran this scenario using the mosquitto client tools, and there exiting the mosquitto subscriber disconnects the clients but does not unsubscribe from the topic
1448917534: New connection from ::1 on port 1883.
1448917534: New client connected from ::1 as client1 (c0, k60).
1448917534: Sending CONNACK to client1 (0, 0)
1448917534: Received SUBSCRIBE from client1
1448917534: liveData (QoS 1)
1448917534: Sending SUBACK to client1
1448917550: New connection from ::1 on port 1883.
1448917550: New client connected from ::1 as mosqpub/25879-MacBook-P (c1, k60, u'system').
1448917550: Sending CONNACK to mosqpub/25879-MacBook-P (0, 0)
1448917550: Received PUBLISH from mosqpub/25879-MacBook-P (d0, q1, r0, m1, 'liveData', ... (68 bytes))
1448917550: Sending PUBACK to mosqpub/25879-MacBook-P (Mid: 1)
1448917550: Sending PUBLISH to client1 (d0, q1, r0, m1, 'liveData', ... (68 bytes))
1448917550: Received DISCONNECT from mosqpub/25879-MacBook-P
1448917550: Client mosqpub/25879-MacBook-P disconnected.
1448917550: Received PUBACK from client1 (Mid: 1)
1448917553: Socket error on client client1, disconnecting.
1448917554: New connection from ::1 on port 1883.
1448917554: New client connected from ::1 as mosqpub/25884-MacBook-P (c1, k60, u'system').
1448917554: Sending CONNACK to mosqpub/25884-MacBook-P (0, 0)
1448917554: Received PUBLISH from mosqpub/25884-MacBook-P (d0, q1, r0, m1, 'liveData', ... (68 bytes))
1448917554: Sending PUBACK to mosqpub/25884-MacBook-P (Mid: 1)
1448917554: Received DISCONNECT from mosqpub/25884-MacBook-P
1448917555: Client mosqpub/25884-MacBook-P disconnected.
1448917556: New connection from ::1 on port 1883.
1448917556: Client client1 disconnected.
1448917556: New client connected from ::1 as client1 (c0, k60).
1448917556: Sending CONNACK to client1 (0, 0)
1448917556: Sending PUBLISH to client1 (d0, q1, r0, m2, 'liveData', ... (68 bytes))
1448917556: Received SUBSCRIBE from client1
1448917556: liveData (QoS 1)
1448917556: Sending SUBACK to client1
1448917556: Received PUBACK from client1 (Mid: 2)
Any idea how to deal with this situation ?
EDIT :
When implementing the workaround as proposed in the accepted answer, I'm getting the following error. My Spring context is loaded from a webapp. I've tried putting the IgnoreUnsubscribePahoClientFactory in a seperate JAR (same level as spring-integration / paho) as well as in the webapp classes itself.
2015-12-02 15:47:43,703 ERROR org.springframework.integration.handler.LoggingHandler - org.springframework.aop.framework.AopConfigException: Could not generate CGLIB subclass of class [class org.eclipse.paho.client.mqttv3.MqttAsyncClient]: Common causes of this problem include using a final class or a non-visible class; nested exception is org.springframework.cglib.core.CodeGenerationException: java.lang.reflect.InvocationTargetException-->null
at org.springframework.aop.framework.CglibAopProxy.getProxy(CglibAopProxy.java:206)
at org.springframework.aop.framework.ProxyFactoryBean.getProxy(ProxyFactoryBean.java:368)
at org.springframework.aop.framework.ProxyFactoryBean.getSingletonInstance(ProxyFactoryBean.java:322)
at org.springframework.aop.framework.ProxyFactoryBean.getObject(ProxyFactoryBean.java:246)
at com.ecs.vdm.rest.integration.IgnoreUnsubscribePahoClientFactory.proxy(IgnoreUnsubscribePahoClientFactory.java:62)
at com.ecs.vdm.rest.integration.IgnoreUnsubscribePahoClientFactory.getAsyncClientInstance(IgnoreUnsubscribePahoClientFactory.java:43)
at org.springframework.integration.mqtt.inbound.MqttPahoMessageDrivenChannelAdapter.connectAndSubscribe(MqttPahoMessageDrivenChannelAdapter.java:216)
at org.springframework.integration.mqtt.inbound.MqttPahoMessageDrivenChannelAdapter.access$300(MqttPahoMessageDrivenChannelAdapter.java:45)
at org.springframework.integration.mqtt.inbound.MqttPahoMessageDrivenChannelAdapter$1.run(MqttPahoMessageDrivenChannelAdapter.java:272)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.cglib.core.CodeGenerationException: java.lang.reflect.InvocationTargetException-->null
at org.springframework.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:237)
at org.springframework.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
at org.springframework.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
at org.springframework.aop.framework.ObjenesisCglibAopProxy.createProxyClassAndInstance(ObjenesisCglibAopProxy.java:57)
at org.springframework.aop.framework.CglibAopProxy.getProxy(CglibAopProxy.java:202)
... 16 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:384)
at org.springframework.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:219)
... 20 more
Caused by: java.lang.SecurityException: class "org.eclipse.paho.client.mqttv3.MqttAsyncClient$$EnhancerBySpringCGLIB$$d14754a9_4603"'s signer information does not match signer information of other classes in the same package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
... 25 more
It's a bug - it unconditionally unsubscribes during the stop().
I don't see a simple work around; but I have a couple of ideas; I'll post here if/when I have something.
In the meantime, please open a JIRA Issue.
EDIT
Gist Here
It's a bit of a sledgehammer, but it should work for you; it effectively ignores the call to unsubscribe on the client. It could be made a little more sophisticated, to only ignore when the QOS is > 0, but that would be quite a bit more involved.
If you're already using the DefaultMqttPahoClientFactory just change the bean class to this one. If you're not currently using a factory, declare it as a bean and provide it to the adapter using the client-factory attribute.
We'll fix it properly in an upcoming release.
**0002bc2d SibMessage W [:] CWSJY0003W: JMSCC3036: An exception has been delivered to the
connections exception listener: '
Message : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has o
ccurred. An error has occurred with the WebSphere MQ JMS connection. Use the linked exception to determine the cause of this e
rror.
Class : class com.ibm.msg.client.jms.DetailedJMSException
Stack : com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:608)
: com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)
: com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:851)
: com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.callEventHandler(RemoteAsyncConsume.java:1
023)
: com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.driveEventsEH(RemoteAsyncConsume.java:1381
)
: com.ibm.mq.jmqi.remote.internal.RemoteDispatchThread.run(RemoteDispatchThread.java:310)
: com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:209)
: com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.j
ava:100)
: com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:224)
: com.ibm.ws.wmqcsi.workqueue.WorkQueueManagerImpl$WorkQueueRunnable.run(WorkQueueManagerImpl.j
ava:648)
: java.lang.Thread.run(Thread.java:738)
Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED'
) reason '2009' ('MQRC_CONNECTION_BROKEN').
Class : class com.ibm.mq.MQException
Stack : com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:223)
: com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:851)
: com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.callEventHandler(RemoteAsyncConsume.java:1
023)
: com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.driveEventsEH(RemoteAsyncConsume.java:1381)
: com.ibm.mq.jmqi.remote.internal.RemoteDispatchThread.run(RemoteDispatchThread.java:310)
: com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:209)
: com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.j
ava:100)
: com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:224)
: com.ibm.ws.wmqcsi.workqueue.WorkQueueManagerImpl$WorkQueueRunnable.run(WorkQueueManagerImpl.j
ava:648)
: java.lang.Thread.run(Thread.java:738)
'.
Please can anybody explain what this exception means and why has this caused CPU starvation**
Did you review your own logfile?
JMSWMQ1107: A problem with this connection has occurred. An error has
occurred with the WebSphere MQ JMS connection. Use the linked
exception to determine the cause of this error.
And further down it says:
java.lang.Thread.run(Thread.java:738) Caused by [1] --> Message :
com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with
compcode '2' ('MQCC_FAILED' ) reason '2009'
('MQRC_CONNECTION_BROKEN').
Did you look up MQ reason code 2009 in the MQ Information Center? Basically, it means the connection to the queue manager was lost and you need to reconnect.
and why has this caused CPU starvation
Any CPU starvation you are seeing has nothing to do with error but rather your code is not properly handling the error and is probably in a loop racing around and around in circles doing nothing.