I am using Mule 2.2.6, with the following setup:
2 servers, each with an instance of ActiveMQ
1 server running Mule 2.2.6
Now, I have a service that picks up a message from one ActiveMQ, and places it on the other. But, this is leading to problems with lost messages when the target ActiveMQ is down. Then, Mule will pick up a message from the source ActiveMQ, but run into this exception upon trying to place it on the destination AtiveMQ:
ERROR [org.mule.DefaultExceptionStrategy] Caught exception in Exception Strategy: No JMS Connection
java.lang.IllegalStateException: No JMS Connection
at org.mule.transport.jms.JmsMessageDispatcher.doDispatch(JmsMessageDispatcher.java:81)
at org.mule.transport.AbstractMessageDispatcher.dispatch(AbstractMessageDispatcher.java:105)
at org.mule.transport.AbstractConnector$DispatchWorker.doRun(AbstractConnector.java:2561)
at org.mule.work.AbstractMuleEventWork.run(AbstractMuleEventWork.java:41)
at org.mule.work.WorkerContext.run(WorkerContext.java:310)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
at java.lang.Thread.run(Thread.java:662)
Upon this happening, the message is just lost, which is way below what is acceptable.
When examining the mule configuration, I noticed that there were no jms-transactions associaited with the service that moves messages from one ActiveMQ to the other, but after adding a transaction, the problem remained the same.
Configuration:
ActiveMQ connectors:
<jms:activemq-connector name="jmsConnectorOuter" specification="1.1"
persistentDelivery="true" disableTemporaryReplyToDestinations="true" honorQosHeaders="true"
numberOfConsumers="1" connectionFactory-ref="activeMqConnectionFactoryOuter" maxRedelivery="-1">
<ee:retry-forever-policy frequency="5000" asynchronous="false" />
</jms:activemq-connector>
<jms:activemq-connector name="jmsConnector" specification="1.1"
persistentDelivery="true" disableTemporaryReplyToDestinations="true" honorQosHeaders="true"
numberOfConsumers="1" connectionFactory-ref="activeMqConnectionFactory" maxRedelivery="-1">
<ee:retry-forever-policy frequency="5000" asynchronous="false" />
</jms:activemq-connector>
Jms-endpoints:
<endpoint name="queue.destination" address="jms://queue.destination" connector-ref="jmsConnector" />
<endpoint name="queue.source" address="jms://queue.source" connector-ref="jmsConnectorOuter" />
Service:
<service name="OuterActiveMQService">
<inbound>
<jms:inbound-endpoint ref="queue.source">
<jms:transaction action="ALWAYS_BEGIN" timeout="60000" />
</jms:inbound-endpoint>
</inbound>
<outbound>
<pass-through-router>
<jms:outbound-endpoint ref="queue.destination" />
</pass-through-router>
</outbound>
</service>
What can be done to make sure that messages aren't lost when the ActiveMQ connection to the destination is lost?
You need to add:
<jms:transaction action="ALWAYS_JOIN" />
in the outbound JMS endpoint to make it join the transaction.
Related
We are using spring boot to run our queue polling program.
The queue is being polled about every 2 mins, and every 2 mins the session is closed, then refreshed.
The connection is a shared connection from the external tomcat, this connection is shared with a dozen other applications.
2018-11-20 11:59:21.263 WARN [serviceRequestAdapter.container-3] org.springframework.jms.listener.DefaultMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'NPP.SERVICE_REQUEST' -
trying to recover. Cause: JMS-131: Session is closed
2018-11-20 11:59:21.265 INFO [serviceRequestAdapter.container-3] org.springframework.jms.listener.DefaultMessageListenerContainer -
Successfully refreshed JMS Connection
2018-11-20 12:01:21.781 WARN [serviceRequestAdapter.container-4] org.springframework.jms.listener.DefaultMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'NPP.SERVICE_REQUEST' -
trying to recover. Cause: JMS-131: Session is closed
2018-11-20 12:01:21.823 INFO [serviceRequestAdapter.container-4] org.springframework.jms.listener.DefaultMessageListenerContainer -
Successfully refreshed JMS Connection
This doesn't actually appear to be affecting functionality, as messages posted get consumed and processed.
Is this actually a problem, if so how do I fix it?
If it isn't a problem how do I hide these messages without reducing my log level to error?
our jms-context.xml
<context:annotation-config/>
<tx:annotation-driven/>
<int:message-history/>
<int:channel id="jms-inbound"/>
<int:channel id="voucher-create-inbound"/>
<int:channel id="voucher-update-inbound"/>
<int:channel id="default-inbound"/>
<orcl:aq-jms-connection-factory id="connectionFactory"
connection-factory-type="QUEUE_CONNECTION"
use-local-data-source-transaction="true"/>
<int:recipient-list-router input-channel="jms-inbound" default-output-channel="default-inbound"
id="action-type-router">
<int:recipient channel="voucher-create-inbound"
selector-expression="headers.actionType == 'CREATE VOUCHER'"/>
<int:recipient channel="voucher-update-inbound"
selector-expression="headers.actionType == 'UPDATE VOUCHER'"/>
</int:recipient-list-router>
<int-jms:message-driven-channel-adapter
id="serviceRequestAdapter"
channel="jms-inbound"
cache-level="3"
connection-factory="connectionFactory"
destination-name="${oracle.rqst-q-name}"/>
<int:service-activator id="createVoucherActivator"
input-channel="voucher-create-inbound"
requires-reply="false"
method="onMessage">
<beans:bean class="VoucherRequestConsumer"/>
</int:service-activator>
<int:service-activator id="updateVoucherActivator"
input-channel="voucher-update-inbound"
requires-reply="false"
method="onMessage">
<beans:bean class="VoucherRequestConsumer"/>
</int:service-activator>
<beans:bean id="defaultRequestConsumer"
class="DefaultRequestConsumer"/>
<int:service-activator id="defaultActivator"
input-channel="default-inbound"
requires-reply="false"
ref="defaultRequestConsumer"
method="onMessage">
</int:service-activator>
</beans:beans>
I am bit puzzled . . . what are you asking? I mean I don't see a question.
Are you just looking for confirmation that it's ok?
In any event, consider this doc - https://docs.spring.io/spring-data/jdbc/old-docs/2.0.0.M1/reference/html/orcl.streamsaq.html, specifically Section 4.3 Configuring the Connection Factory to use the same local transaction as your data access code. which specifically talks about implications on JMS Session when use-local-data-source-transaction is set to true.
Could anyone point me to some reference on setting up DLQ on WLS liberty profile version 16.0.0.2 using Liberty embedded JMS messaging provider?
I have a queue configured with a spring jms listener and when the message listener throws a RuntimeException, the message should go to deal letter queue after few retries.
Thanks.
With the wasJmsServer feature the term is the exception destination. This can be configured as an attribute on the queue object. This references the name of another queue. As an example:
<messagingEngine>
<queue id="dlq" />
<queue id="myQueue" exceptionDestination="dlq" />
</messagingEngine>
there is a default name for exception destination which is _SYSTEM.Exception.Destination, so if you have this:
<messagingEngine>
<queue id="_SYSTEM.Exception.Destination" />
<queue id="myQueue" />
</messagingEngine>
then you should have all the 'bad' messages on the default queue. By default messages will only be sent to the exception destination if delivery failed 5 times. This can be overridden with the maxRedeliveryCount setting:
<messagingEngine>
<queue id="_SYSTEM.Exception.Destination" />
<queue id="myQueue" maxRedeliveryCount="2"/>
</messagingEngine>
The configuration for this is available in the Knowledge Center for WebSphere Liberty.
I need to connect from a web application running on a WebSphere AS 6.1 application server to a remote WebSphere MQ on z/OS queue. On WebSphere AS, I configured both QueueConnectionFactory and Queue (an object containing a part of the remote queue data), with most of the settings set to their default values - I just needed to set queue name, channel, host, port, and transport type which is CLIENT. I inject them in the following Spring 3.2 configuration using JNDI lookup:
<jee:jndi-lookup id="destination" jndi-name="MyMQQueue" expected-type="javax.jms.Queue" />
<jee:jndi-lookup id="targetConnectionFactory" jndi-name="MyMQQCF" expected-type="javax.jms.QueueConnectionFactory" />
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate"
p:connectionFactory-ref="targetConnectionFactory"
p:defaultDestination-ref="destination" />
<bean id="simpleMessageListener" class="my.own.SimpleMessageListener"/>
<bean id="msgListenerContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="targetConnectionFactory" />
<property name="destination" ref="destination" />
<property name="messageListener" ref="simpleMessageListener" />
<property name="taskExecutor" ref="managedThreadsTaskExecutor" />
<property name="receiveTimeout" value="5000" />
<property name="recoveryInterval" value="5000" />
</bean>
<bean id="managedThreadsTaskExecutor" class="org.springframework.scheduling.commonj.WorkManagerTaskExecutor">
<property name="workManagerName" value="wm/default" />
</bean>
JmsTemplate sends and receives (synchronously) messages correctly. DefaultMessageListenerContainer, an asynchronous message receiver, reads some (previously sent) messages off the MQ queue during WebSphere AS start, but chokes soon afterwards, and begins to throw repeatedly "connection closed" exception. On each such occasion it notifies me that
DefaultMessag W org.springframework.jms.listener.DefaultMessageListenerContainer handleListenerSetupFailure Setup of JMS message listener invoker failed for destination 'queue://myqueue' - trying to recover. Cause: Connection closed
DefaultMessag I org.springframework.jms.listener.DefaultMessageListenerContainer refreshConnectionUntilSuccessful Successfully refreshed JMS Connection
but stops taking messages off the queue.
Digging a bit into Spring code, I found that setting on DefaultMessageListenerContainer
<property name="cacheLevel" value="0"/>
solves the problem, in the sense that the messages are now being read off the queue every time I send them. However, looking at the TCP traffic to WebSphere MQ I find that MQCLOSE/MQOPEN commands are sent to it repeatedly as in:
Wireshark captured traffic
which probably means that the connection gets continuously closed and reopened.
Can anybody suggest what might be the cause for caching not working properly, and whether there is perhaps a relatively simple way to modify Spring code (extending DefaultMessageListenerContainer, for example), or perhaps set some property on MQ queue connection factory/queue, to get it working?
EDIT:
Searching further the internet, I have found the following link
http://forum.spring.io/forum/spring-projects/integration/jms/89532-defaultmessagelistenercontainer-cachingconnectionfactory-tomcat-and-websphere-mq
which seems to describe a similar problem occurring on Tomcat. The solution there is to set a certain exceptionListener on DefaultMessageListenerContainer. However, trying to do this on WebSphere throws the exception "javax.jms.IllegalStateException: Method setExceptionListener not permitted". The underlying cause seems to be that J2EE 1.4 spec forbids calling setExceptionListener on JMS connections.
https://www.ibm.com/developerworks/library/j-getmess/j-getmess-pdf.pdf
It seems that setting
<property name="cacheLevel" value="0"/>
on DefaultMessageListenerContainer is actually the correct solution.
I mislead myself by interpreting MQCLOSE/MQOPEN I saw on Wireshark captured TCP traffic in this case, as the heavyweight connection opening.
First, the newly created Connection Factory on the administrative console WebSphere AS 6.1 has by default a JMS connection pool (max size 10). By debugging the base class of DefaultMessageListenerContainer, AbstractPollingMessageListenerContainer, (specifically the method
protected boolean doReceiveAndExecute(
Object invoker, Session session, MessageConsumer consumer, TransactionStatus status)
)
one sees that neither the call to create a connection, neither the call to create a session from connection generate TCP traffic, and TCP traffic is generated only by creating a consumer (considered to be a "lightweight operation" if I understand correctly), trying to receive a message from the queue, and closing the consumer.
So it seems that the connection is taken from the respective pool, and also the session is somehow "cached".
So instead of caching by Spring, the caching appears to be done here by the application server.
When using mule flows as listed below, we are getting duplicate ackMessages on activeMQ. That is, the count of acknowledge messages is double the dispatch Messages.
<flow name="dequeueFlow" initialState="stopped">
<jms:inbound-endpoint queue="my.mq.queueName"
connector-ref="MyDequeueJmsConnector">
<jms:transaction action="ALWAYS_BEGIN"/>
</jms:inbound-endpoint>
<processor ref="MyRequestProcessor" />
</flow>
<jms:activemq-connector name="MyDequeueJmsConnector"
specification="1.1"
connectionFactory-ref="AmqConnectionFactory"
disableTemporaryReplyToDestinations="true"
persistentDelivery="true"
cacheJmsSessions="false"
numberOfConsumers="10"
acknowledgementMode="AUTO_ACKNOWLEDGE"
maxRedelivery="1">
</jms:activemq-connector>
In JmsConnector, the acknowledgementMode is set: acknowledgementMode="AUTO_ACKNOWLEDGE".
Instead of flows, if we are using default JMS connections, not seeing the issue. Any pointers on how this can be avoided?
Software Versions:
Mule Enterprise 3.4.1
ActiveMQ 5.9.0
A spring quartz process runs every 15 minutes in my project i.e 96 times a day. This fetch certain records from database and POST it on a REST service (running on JBoss 7). These records are in general 50 to 100 in count.
On REST service there is jms event publisher that publishes this message on a topic. There are two consumers on this topic.
That process message and sends push notification messages on mobile
Talk to third party (generally takes 4 to 5 second to complete the call)
Since it is topic both consumers receive all messages but they filter them out based on some property, so few messages are processed by one and rest by another consumer.
My problem is; which is being observed recently since a week time; that consumer #1 receives response from APNS as invalid token multiple times; token is used to send push notification to mobile; after some time this consumer stops and do not respond at all while second one keeps running.
Below are configurations:
<amq:broker id="broker" useJmx="false" persistent="false">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://localhost:0"/>
</amq:transportConnectors>
</amq:broker>
<!-- ActiveMQ Destination -->
<amq:topic id="topicName" physicalName="topicPhysicalName"/>
<!-- JMS ConnectionFactory to use, configuring the embedded broker using XML -->
<amq:connectionFactory id="jmsFactory" brokerURL="vm://localhost"/>
<!-- JMS Producer Configuration -->
<bean id="jmsProducerConnectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory"
depends-on="broker"
p:targetConnectionFactory-ref="jmsFactory"/>
<!-- JMS Templates-->
<bean id="jmsTemplate"
class="org.springframework.jms.core.JmsTemplate"
p:connectionFactory-ref="jmsProducerConnectionFactory"/>
<!-- Publisher-->
<bean name="jmsEventPublisher"
class="com.jhi.mhm.services.event.jms.publisher.JMSEventPublisher">
<property name="jmsTemplate" ref="jmsTemplate"/>
<property name="topic">
<map>
<entry key="keyname" value-ref="topicName"/>
</map>
</property>
</bean>
<!-- JMS Consumer Configuration -->
<bean name="consumer2" class="Consumer2"/>
<bean name="consumer1" class="Consumer1"/>
<bean id="jmsConsumerConnectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory"
depends-on="broker"
p:targetConnectionFactory-ref="jmsFactory"/>
<jms:listener-container container-type="default"
connection-factory="jmsConsumerConnectionFactory"
acknowledge="auto"
destination-type="topic">
<jms:listener destination="topicPhysicalName" ref="consumer1"/>
<jms:listener destination="topicPhysicalName" ref="consumer2"/>
</jms:listener-container>
I search another posted questions but could not find anything related.
Your thoughts would be really helpful.
shailu - I went through similar problem. What we did is upgrade the version of MQ. Although this did not solved the problem completely as MQ shows random behavior and at the end we simply merged our endpoint and call destination as per biz logic