We are using spring boot to run our queue polling program.
The queue is being polled about every 2 mins, and every 2 mins the session is closed, then refreshed.
The connection is a shared connection from the external tomcat, this connection is shared with a dozen other applications.
2018-11-20 11:59:21.263 WARN [serviceRequestAdapter.container-3] org.springframework.jms.listener.DefaultMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'NPP.SERVICE_REQUEST' -
trying to recover. Cause: JMS-131: Session is closed
2018-11-20 11:59:21.265 INFO [serviceRequestAdapter.container-3] org.springframework.jms.listener.DefaultMessageListenerContainer -
Successfully refreshed JMS Connection
2018-11-20 12:01:21.781 WARN [serviceRequestAdapter.container-4] org.springframework.jms.listener.DefaultMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'NPP.SERVICE_REQUEST' -
trying to recover. Cause: JMS-131: Session is closed
2018-11-20 12:01:21.823 INFO [serviceRequestAdapter.container-4] org.springframework.jms.listener.DefaultMessageListenerContainer -
Successfully refreshed JMS Connection
This doesn't actually appear to be affecting functionality, as messages posted get consumed and processed.
Is this actually a problem, if so how do I fix it?
If it isn't a problem how do I hide these messages without reducing my log level to error?
our jms-context.xml
<context:annotation-config/>
<tx:annotation-driven/>
<int:message-history/>
<int:channel id="jms-inbound"/>
<int:channel id="voucher-create-inbound"/>
<int:channel id="voucher-update-inbound"/>
<int:channel id="default-inbound"/>
<orcl:aq-jms-connection-factory id="connectionFactory"
connection-factory-type="QUEUE_CONNECTION"
use-local-data-source-transaction="true"/>
<int:recipient-list-router input-channel="jms-inbound" default-output-channel="default-inbound"
id="action-type-router">
<int:recipient channel="voucher-create-inbound"
selector-expression="headers.actionType == 'CREATE VOUCHER'"/>
<int:recipient channel="voucher-update-inbound"
selector-expression="headers.actionType == 'UPDATE VOUCHER'"/>
</int:recipient-list-router>
<int-jms:message-driven-channel-adapter
id="serviceRequestAdapter"
channel="jms-inbound"
cache-level="3"
connection-factory="connectionFactory"
destination-name="${oracle.rqst-q-name}"/>
<int:service-activator id="createVoucherActivator"
input-channel="voucher-create-inbound"
requires-reply="false"
method="onMessage">
<beans:bean class="VoucherRequestConsumer"/>
</int:service-activator>
<int:service-activator id="updateVoucherActivator"
input-channel="voucher-update-inbound"
requires-reply="false"
method="onMessage">
<beans:bean class="VoucherRequestConsumer"/>
</int:service-activator>
<beans:bean id="defaultRequestConsumer"
class="DefaultRequestConsumer"/>
<int:service-activator id="defaultActivator"
input-channel="default-inbound"
requires-reply="false"
ref="defaultRequestConsumer"
method="onMessage">
</int:service-activator>
</beans:beans>
I am bit puzzled . . . what are you asking? I mean I don't see a question.
Are you just looking for confirmation that it's ok?
In any event, consider this doc - https://docs.spring.io/spring-data/jdbc/old-docs/2.0.0.M1/reference/html/orcl.streamsaq.html, specifically Section 4.3 Configuring the Connection Factory to use the same local transaction as your data access code. which specifically talks about implications on JMS Session when use-local-data-source-transaction is set to true.
Related
Summary - Message loss during systematic shutdown of application.
I have a application written in spring integration and I am consuming requests from external systems using 'jms:message-driven-channel-adapter'. This is the configuration of the channel adapter -
<jms:message-driven-channel-adapter id="InboundAdapter" destination="InQueue"
connection-factory="connectionFactory"
channel="responseChannel"
error-channel="errorChannel"
acknowledge="transacted"
receive-timeout="50000"
auto-startup="true"/>
my response channel looks like this -
<int:chain id="processorChain" input-channel="responseChannel">
<int:service-activator method="doProcess" ref="inputProcessor" />
<int:router id="inputRouter" method="route">
<int:mapping value="REQUIRED" channel="builderChannel"/>
<int:mapping value="NOT_REQUIRED" channel="TerminateChannel"/>
<bean id="Router" class="xxx.xxx.Router">
</bean>
</int:router>
</int:chain>
Now when i perform 'kill -9 pid', then I see that the message is rolled back to the queue and it is all good. But when I am performing 'kill pid', than the message is lost somewhere. I have enabled teh jms logs and I can see that JMS listener is waiting for the message in transit to complete before closing the JMS consumer but still I don't the message rolling back to my queue. This is the logs snippet that I see in my logs
trace: 15.11.18 15:42:46.619 [Thread-10] DEBUG org.springframework.jms.listener.DefaultMessageListenerContainer - Waiting for shutdown of message listener invokers
trace: 15.11.18 15:42:46.619 [Thread-10] DEBUG org.springframework.jms.listener.DefaultMessageListenerContainer - Still waiting for shutdown of 1 message listener invokers (iteration 0)
After this logs, it is calling the service activators which are defined in the response channel.
Can someone please shed some light on above behavior, I was expecting that when we issue kill command than all messages which are in transit should be rolled back to the queue, instead of this, it is trying to call the activators defined with in channel and after calling last component which is router, it just finishes.
Any help on this topic will be really helpful!!
After some more debugging and banging my head with the system I found the exact scenario where it is happening. Let me put it thru to see if it makes sense. When a systematic shutdown is triggered, JMS is waiting for the messages in transit to finish before stopping the application, till this point everything is good. Currently this chain is being executed -
<int:chain input-channel="inputchain">
<int:transformer id="xxx" method="transform">
<bean class="xxx" />
</int:transformer>
<int:service-activator id="xxx" method="doProcess">
<bean class="xx">
<constructor-arg ref="xxx"/>
</bean>
</int:service-activator>
<int:service-activator id="xxx" ref="rulesProcessor" method="doProcess"/>
<int:service-activator id="xxx" ref="xxx" method="doProcess"/>
<!-- Existing Flow Continues -->
<int:router id="xxxRequiredRouter" method="xxxRequired">
<int:mapping value="Required" channel="firstChannel"/>
<int:mapping value="NotRequired" channel="secondChannel"/>
<bean id="xxxRouter" class="xxx.Router" />
</bean>
</int:router>
</int:chain>
<int:chain input-channel="secondChannel">
some logic
</int:chain>
So the thread finishes this chain and exits gracefully. It doesn't call my next chain 'secondChannel'. My transaction boundary doesn't finish on this chain 'inputchain', it finishes at the end of secondChannel. So I have not comnmitted this transction to database as transaction boundary is at the end of next chain, so this is not available in DB and application is thinking that chain execution is finished so it is complete, so it is not rolled back to teh queue as well. So in teh end I don't have this message in my database and it is not in queue.
So is this the case that only the chain which is being executed when shutdown was triggered will finish and it will not delegate processing to subsequent chains?
A SIGTERM kill will wait for the thread to complete its work.
Eventually the container will interrupt it, but that will only help if it's doing something that's interruptible.
I need to connect from a web application running on a WebSphere AS 6.1 application server to a remote WebSphere MQ on z/OS queue. On WebSphere AS, I configured both QueueConnectionFactory and Queue (an object containing a part of the remote queue data), with most of the settings set to their default values - I just needed to set queue name, channel, host, port, and transport type which is CLIENT. I inject them in the following Spring 3.2 configuration using JNDI lookup:
<jee:jndi-lookup id="destination" jndi-name="MyMQQueue" expected-type="javax.jms.Queue" />
<jee:jndi-lookup id="targetConnectionFactory" jndi-name="MyMQQCF" expected-type="javax.jms.QueueConnectionFactory" />
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate"
p:connectionFactory-ref="targetConnectionFactory"
p:defaultDestination-ref="destination" />
<bean id="simpleMessageListener" class="my.own.SimpleMessageListener"/>
<bean id="msgListenerContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="targetConnectionFactory" />
<property name="destination" ref="destination" />
<property name="messageListener" ref="simpleMessageListener" />
<property name="taskExecutor" ref="managedThreadsTaskExecutor" />
<property name="receiveTimeout" value="5000" />
<property name="recoveryInterval" value="5000" />
</bean>
<bean id="managedThreadsTaskExecutor" class="org.springframework.scheduling.commonj.WorkManagerTaskExecutor">
<property name="workManagerName" value="wm/default" />
</bean>
JmsTemplate sends and receives (synchronously) messages correctly. DefaultMessageListenerContainer, an asynchronous message receiver, reads some (previously sent) messages off the MQ queue during WebSphere AS start, but chokes soon afterwards, and begins to throw repeatedly "connection closed" exception. On each such occasion it notifies me that
DefaultMessag W org.springframework.jms.listener.DefaultMessageListenerContainer handleListenerSetupFailure Setup of JMS message listener invoker failed for destination 'queue://myqueue' - trying to recover. Cause: Connection closed
DefaultMessag I org.springframework.jms.listener.DefaultMessageListenerContainer refreshConnectionUntilSuccessful Successfully refreshed JMS Connection
but stops taking messages off the queue.
Digging a bit into Spring code, I found that setting on DefaultMessageListenerContainer
<property name="cacheLevel" value="0"/>
solves the problem, in the sense that the messages are now being read off the queue every time I send them. However, looking at the TCP traffic to WebSphere MQ I find that MQCLOSE/MQOPEN commands are sent to it repeatedly as in:
Wireshark captured traffic
which probably means that the connection gets continuously closed and reopened.
Can anybody suggest what might be the cause for caching not working properly, and whether there is perhaps a relatively simple way to modify Spring code (extending DefaultMessageListenerContainer, for example), or perhaps set some property on MQ queue connection factory/queue, to get it working?
EDIT:
Searching further the internet, I have found the following link
http://forum.spring.io/forum/spring-projects/integration/jms/89532-defaultmessagelistenercontainer-cachingconnectionfactory-tomcat-and-websphere-mq
which seems to describe a similar problem occurring on Tomcat. The solution there is to set a certain exceptionListener on DefaultMessageListenerContainer. However, trying to do this on WebSphere throws the exception "javax.jms.IllegalStateException: Method setExceptionListener not permitted". The underlying cause seems to be that J2EE 1.4 spec forbids calling setExceptionListener on JMS connections.
https://www.ibm.com/developerworks/library/j-getmess/j-getmess-pdf.pdf
It seems that setting
<property name="cacheLevel" value="0"/>
on DefaultMessageListenerContainer is actually the correct solution.
I mislead myself by interpreting MQCLOSE/MQOPEN I saw on Wireshark captured TCP traffic in this case, as the heavyweight connection opening.
First, the newly created Connection Factory on the administrative console WebSphere AS 6.1 has by default a JMS connection pool (max size 10). By debugging the base class of DefaultMessageListenerContainer, AbstractPollingMessageListenerContainer, (specifically the method
protected boolean doReceiveAndExecute(
Object invoker, Session session, MessageConsumer consumer, TransactionStatus status)
)
one sees that neither the call to create a connection, neither the call to create a session from connection generate TCP traffic, and TCP traffic is generated only by creating a consumer (considered to be a "lightweight operation" if I understand correctly), trying to receive a message from the queue, and closing the consumer.
So it seems that the connection is taken from the respective pool, and also the session is somehow "cached".
So instead of caching by Spring, the caching appears to be done here by the application server.
Context: a Spring 4.0.6 app running inside JBoss EAP 6.2. Part of the application is a JMS queue and on the receiving side, its messages need to be processed in parallel, otherwise many messages take too long.
Spring's JMS listener is configured with a concurrency of 10 and delegates to an executor pool with 10 threads (first question: is this relationship correct?)
<bean id="executor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="10" />
<property name="maxPoolSize" value="10" />
<property name="queueCapacity" value="20" />
</bean>
<jms:listener-container destination-resolver="jndiDestinationResolver" destination-type="queue" acknowledge="auto" connection-factory="jmsConnectionFactory" concurrency="10" task-executor="executor">
<jms:listener destination="java:jboss/exported/jms/queue/myQueue" ref="myMessageHandler"/>
</jms:listener-container>
Under load, the following error appears frequently:
10:32:47,457 ERROR [org.hornetq.ra] (org.springframework.jms.listener.DefaultMessageListenerContainer#0-112) HQ154002: Could not create session: javax.jms.IllegalStateException: Only allowed one session per connection. See the J2EE spec, e.g. J2EE1.4 Section 6.6
at org.hornetq.ra.HornetQRASessionFactoryImpl.allocateConnection(HornetQRASessionFactoryImpl.java:811)
at org.hornetq.ra.HornetQRASessionFactoryImpl.createSession(HornetQRASessionFactoryImpl.java:465)
at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:197) [spring-jms-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer.access$1400(DefaultMessageListenerContainer.java:119) [spring-jms-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerContainer.java:1122) [spring-jms-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101) [spring-jms-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1094) [spring-jms-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:991) [spring-jms-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]
Now, I understand the JMS spec prohibits that. It's not clear how Spring gets in the position to hit the limitation. Particularly, I checked the DefaultMesageListenerContainer source which has the following fragment:
1120 if (this.session == null && getCacheLevel() >= CACHE_SESSION) {
1121 updateRecoveryMarker();
1122 this.session = createSession(getSharedConnection());
1123 }
so Spring appears to attempt to reuse a connection (see line 1122). Is there something missing from the configuration that makes it do this?
I stumbled upon the same issue while trying to use the Generic JMS RA.
What made it work correctly in the end was to configure the cache level on the listener container, setting it to "NONE", to avoid creating a new session on an existing connection.
Adding cache="none" to your jms:listener-container should do the trick for your configuration.
I have a production application that seems to be losing messages intermittently. One example that I have identified is the sending system logs 3 messages that are sent milliseconds apart. The receiving system logs that it got message 1 and message 3...but not message 2. The queue's depth is 0. I see no errors in the logs of the application that reads the messages to indicate something bad happened.
I have verified that there are no other "rogue" clients creating a race condition for reading messages.
Here is my configuration; we have a primary and secondary/failover queue...
<bean id="primaryProficiencyInboundQueue" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueManagerName" value="${lsm.primary.outbound.manager}"/>
<property name="baseQueueName" value="${lsm.proficiency.inbound.queue}"/>
</bean>
<bean id="secondaryProficiencyInboundQueue" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueManagerName" value="${lsm.secondary.outbound.manager}"/>
<property name="baseQueueName" value="${lsm.proficiency.inbound.queue}"/>
</bean>
<int:channel id="inboundProficiencyChannel">
<int:interceptors>
<int:wire-tap channel="logger" />
</int:interceptors>
</int:channel>
<!-- adapter that connects the inbound proficiency queues to the inboundProficiencyChannel -->
<jms:message-driven-channel-adapter id="primaryProficiencyInboundAdapter"
connection-factory="primaryConnectionFactory"
destination="primaryProficiencyInboundQueue"
channel="inboundProficiencyChannel" />
<jms:message-driven-channel-adapter id="secondaryProficiencyInboundAdapter"
connection-factory="secondaryConnectionFactory"
destination="secondaryProficiencyInboundQueue"
channel="inboundProficiencyChannel" />
<int:service-activator id="proficiencyInboundActivator"
input-channel="inboundProficiencyChannel">
<bean class="com.myactivator.LSMInboundProficiencyActivator" />
</int:service-activator>
Is it possible that something is failing silently?
One other noteworthy thing is that there are two production servers that are both configured as above....meaning that there are two servers that have JMS adapters configured to read the messages. I mentioned above that I see message 1 and message 3 being read. In that case, I see server one process message one and server two process mesage two...so it seems that having two servers configured like above has no negative impact. Any other thoughts on what could be happening or how I can debug the case of the missing/dropped messages?
Version information:
Spring - 3.0.5.RELEASE
Spring Integration - 2.0.3.RELEASE
Nothing springs to mind that would cause lost messages; I don't recall that ever being reported; but 2.0.3 is quite ancient (3rd birthday coming up soon). We don't support 2.0.x any more but you should at least update to 2.0.6 which is the last release of 2.0.x. But preferably move up to the latest 3.0.0 release (or at least 2.2.6).
I am working on a POC which does the following
Uses a message driven channel adapter to recieve message in a transaction
Calls the Service Activator which uses a handler to insert the message recieved from the adapter to DB and also post message to outbound channel.
Now, if the DB insert of the message fails i want the JMS message returned back to the queue so that it can be re-tried later.
With my below configuration it doesnt seems to work.(i.e. even if there is a failure while inserting into the database the message is removed from the queue.
Any pointers or sample configuration would be helpful.
<integration:channel id="jmsInChannel">
<integration:queue/>
</integration:channel>
<int-jms:message-driven-channel-adapter id="jmsIn"
transaction-manager="transactionManager"
connection-factory="sConnectionFactory"
destination-name="emsQueue"
acknowledge="client" channel="jmsInChannel"
extract-payload="false"/>
<integration:service-activator input-channel="jmsInChannel"
output-channel="fileNamesChannel" ref="handler" method="process" />
<bean id="handler" class="com.irebalpoc.integration.MessageProcessor">
<property name="jobHashTable" ref="jobsMapping" />
</bean>
Set acknowledge="transacted" and, I presume the transactionManager is a JDBC (or JTA) transaction manager.
You also need to remove <queue/> from JmsInChannel so that the database transaction occurs on the same thread.
Spring will synchronize the database transaction with the JMS transaction.
However, read http://www.javaworld.com/javaworld/jw-01-2009/jw-01-spring-transactions.html for the implications.
If you can't make your service idempotent, you may need to look at an XA transaction manager.