Spring Message Listener - Connection Refresh - spring

We have implemented a message listener with spring and the container configurations looks like below
<bean id="customContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="concurrentConsumers" value="${jms.customContainer.concurrentconsumers}" />
<property name="maxConcurrentConsumers" value="${jms.customContainer.maxconcurrentconsumers}" />
<property name="errorHandler" ref="errorHandler" />
<property name="connectionFactory" ref="jmsQueueConnectionFactory" />
<property name="destination" ref="listenerQueue" />
<property name="messageListener" ref="customContainerListener" />
<property name="receiveTimeout" value="1" />
<property name="sessionTransacted" value="true" />
<property name="transactionManager" ref="txManager" />
</bean>
Its deployed in Jboss and the server log shows the below connection refresh message frequently.
org.springframework.jms.listener.DefaultMessageListenerContainer] (customContainer-3849) Successfully refreshed JMS Connection
It's been reported a high MSU/MIPS utilization at mainframe side where the WebSphere MQ is hosted. Could this be a reason to cause frequent MQ GET calls and increase in utilization?
We haven't specified the message Reply To explicitly, will this be causing a reply to attempt to queue every time? Also the receive timeout is set to 1 ms, does that mean it will be actively polling the queue in every 1 ms? Could someone please explain the concept?

The DefaultMessageListenerContainer implements polling of the queue. This means that it issues a GET with a wait of the receiveTimeout. If you have a receiveTimeout of 1ms, that means that every GET call will return after 1ms if no message is on the queue and another GET call will be made. This would appear to the queue manager as back to back GET calls.
Two options:
You can increase receiveTimeout to a higher number which will cause get with wait of the specified value, for example, get with wait of 60000 ms would only do a get every 1 minute or after a message hits the queue. You still get messages off the queue as soon as they are PUT to the queue.
A slightly more efficient option if possible is to switch to the SimpleMessageListenerContainer which will use a native JMS async listener. For IBM MQ this is implemented with the MQ callback functionality. This means the listener registers the callback to the queue manager and the queue manager notifies the listener when new messages land on the queue, so there is no polling.

Please see https://marketaylor.synology.me/?p=668
Note that the 'contributions' to the CPU consumption - which ultimately impacts the MSU charges are driven by several things that should be controlled. The two biggest are:
The number of connects to/disconnects from the z/OS queue manager. These verbs are the most costly in terms of CPU use in MQ, and should be used judiciously on z/OS because of the cost model. A client application written to connect once and do many MQ actions is going to impact the CPU consumption, but that tends to be predictable and is as they say 'a cost of doing business.' A client application that connects, opens a queue, puts or gets a single message, then disconnects many, many times will drive up costs quickly ( we have seen instances of a 40% uplift in MSU charges in a single month). The latter is called a poorly behaved application.
The use of Temporary Dynamic queues. TD queues require a significant amount of overhead within the queue manager, as the queue infrastructure has to be built up for each new definition and then torn down when the queue is closed. This overhead can be avoided by using statically defined queues and retrieving messages via the correlation ID or message ID.
Note that while both these issues are not unique to MQ on z/OS it is the licensing/charging model that makes all the difference.
Good luck!

Related

Using DefaultMessageListenerContainer to connect to WebSphere MQ throws "Connection closed" repeatedly

I need to connect from a web application running on a WebSphere AS 6.1 application server to a remote WebSphere MQ on z/OS queue. On WebSphere AS, I configured both QueueConnectionFactory and Queue (an object containing a part of the remote queue data), with most of the settings set to their default values - I just needed to set queue name, channel, host, port, and transport type which is CLIENT. I inject them in the following Spring 3.2 configuration using JNDI lookup:
<jee:jndi-lookup id="destination" jndi-name="MyMQQueue" expected-type="javax.jms.Queue" />
<jee:jndi-lookup id="targetConnectionFactory" jndi-name="MyMQQCF" expected-type="javax.jms.QueueConnectionFactory" />
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate"
p:connectionFactory-ref="targetConnectionFactory"
p:defaultDestination-ref="destination" />
<bean id="simpleMessageListener" class="my.own.SimpleMessageListener"/>
<bean id="msgListenerContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="targetConnectionFactory" />
<property name="destination" ref="destination" />
<property name="messageListener" ref="simpleMessageListener" />
<property name="taskExecutor" ref="managedThreadsTaskExecutor" />
<property name="receiveTimeout" value="5000" />
<property name="recoveryInterval" value="5000" />
</bean>
<bean id="managedThreadsTaskExecutor" class="org.springframework.scheduling.commonj.WorkManagerTaskExecutor">
<property name="workManagerName" value="wm/default" />
</bean>
JmsTemplate sends and receives (synchronously) messages correctly. DefaultMessageListenerContainer, an asynchronous message receiver, reads some (previously sent) messages off the MQ queue during WebSphere AS start, but chokes soon afterwards, and begins to throw repeatedly "connection closed" exception. On each such occasion it notifies me that
DefaultMessag W org.springframework.jms.listener.DefaultMessageListenerContainer handleListenerSetupFailure Setup of JMS message listener invoker failed for destination 'queue://myqueue' - trying to recover. Cause: Connection closed
DefaultMessag I org.springframework.jms.listener.DefaultMessageListenerContainer refreshConnectionUntilSuccessful Successfully refreshed JMS Connection
but stops taking messages off the queue.
Digging a bit into Spring code, I found that setting on DefaultMessageListenerContainer
<property name="cacheLevel" value="0"/>
solves the problem, in the sense that the messages are now being read off the queue every time I send them. However, looking at the TCP traffic to WebSphere MQ I find that MQCLOSE/MQOPEN commands are sent to it repeatedly as in:
Wireshark captured traffic
which probably means that the connection gets continuously closed and reopened.
Can anybody suggest what might be the cause for caching not working properly, and whether there is perhaps a relatively simple way to modify Spring code (extending DefaultMessageListenerContainer, for example), or perhaps set some property on MQ queue connection factory/queue, to get it working?
EDIT:
Searching further the internet, I have found the following link
http://forum.spring.io/forum/spring-projects/integration/jms/89532-defaultmessagelistenercontainer-cachingconnectionfactory-tomcat-and-websphere-mq
which seems to describe a similar problem occurring on Tomcat. The solution there is to set a certain exceptionListener on DefaultMessageListenerContainer. However, trying to do this on WebSphere throws the exception "javax.jms.IllegalStateException: Method setExceptionListener not permitted". The underlying cause seems to be that J2EE 1.4 spec forbids calling setExceptionListener on JMS connections.
https://www.ibm.com/developerworks/library/j-getmess/j-getmess-pdf.pdf
It seems that setting
<property name="cacheLevel" value="0"/>
on DefaultMessageListenerContainer is actually the correct solution.
I mislead myself by interpreting MQCLOSE/MQOPEN I saw on Wireshark captured TCP traffic in this case, as the heavyweight connection opening.
First, the newly created Connection Factory on the administrative console WebSphere AS 6.1 has by default a JMS connection pool (max size 10). By debugging the base class of DefaultMessageListenerContainer, AbstractPollingMessageListenerContainer, (specifically the method
protected boolean doReceiveAndExecute(
Object invoker, Session session, MessageConsumer consumer, TransactionStatus status)
)
one sees that neither the call to create a connection, neither the call to create a session from connection generate TCP traffic, and TCP traffic is generated only by creating a consumer (considered to be a "lightweight operation" if I understand correctly), trying to receive a message from the queue, and closing the consumer.
So it seems that the connection is taken from the respective pool, and also the session is somehow "cached".
So instead of caching by Spring, the caching appears to be done here by the application server.

How do I configure an Spring message listener (MDP) to have one instance across a cluster

I have a spring message listener configured with
<bean id="processListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="concurrentConsumers" value="1"/>
<property name="clientId" value="process-execute"/>
<property name="connectionFactory" ref="topicConnectionFactory"/>
<property name="destination" ref="processExecuteQueue"/>
<property name="messageListener" ref="processExecuteListener"/>
</bean>
This is running on a cluster with 2 nodes. I see that it's creating 1 consumer per node rather than 1 per cluster. They're both configured with the above xml so they have the same clientId. Yet, when 2 notifications are posted to the queue, both of the listeners are running, each gets a notification, and both execute in parallel. This is a problem because the notifications need to be handled sequentially.
I can't seem to find out how to make it have only one message listener per cluster rather than per node.
I solved the problem by having the jms queue block the next consumer until the previous returned. This is a feature in the weblogic server I'm using called Unit of Order. The documentation says you just need to enable it on the queue (I used hash). However, I found that I needed to enable it on the connection factory as well and set a default name. Now I see an MDP per node but 1 waits for 2 to complete before processing and vice versa. Not the solution I intended but it's working nontheless. While oracle specific, it's actually slightly better than a single MDP solution.
Note: I did not set the unit of order name in the spring jmstemplate producer as I do not know if that's possible. I have weblogic setting a default name when none is provided by the producer.

activemq slow consumers block producer although producerFlowControl is false

I have activemq used in my system and what i see is the following message:
TopicSubscription: consumer=...: Pending message cursor [org.apache.activemq.broker.region.cursors.VMPendingMessageCursor#1684f89c] is full, temp usage (0%) or memory usage (100%) limit reached, blocking message add() pending the release of resources.
This is because if i understand correct my consumer is slow while my producer is fast. The result out of it is that eventually my producer is blocked untill consumer reads the message and frees some memory. What i whant is that my producer is not blocked and also when memory is full old messages are being discurded.
Given my understanding of what i have read the following configuration should do the trick (messageEvictionStrategy, pendingMessageLimitStrategy) but it is not working for me and i cannot figure out why.
I have specified low memoryusage limit low (35Mb) to make issue apear faster for testing reasons, but the case is that i need it eventually when the problem apears for activemq to just drop old messages.
I have found one non satisfactory solution of setting in ActiveMQConnectionFactory useAsyncSend=true and specifying sendTimeout. This makes producer not blocked but in this way the newest message is dropped and not the olderst one.
Finally, i am talking for non durable topics.
Any help guys would pe perfect. Below i have activemq configuration
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false" memoryLimit="35 Mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
<messageEvictionStrategy>
<oldestMessageEvictionStrategy/>
</messageEvictionStrategy>
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="10"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="35 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="5000 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
activemq version 5.7.0
i use spring template to send messages:
<bean class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="pooledJmsConnectionFactory"/>
<property name="timeToLive" value="100"/>
</bean>
I transmit javax.jms.ObjectMessage, relativelly small in size.
I found the problem in customer premisses I have many toppics in my application but managed to reproduce it loccally sending from 1 thread, non-stop messages continiusly always to the same topic. The message send was just a small string.
I have only one producer and problem seems to appear when i have 1 (or more) slow consumer(s) -but one slow consumer is enough-. if no slow consumer exists, problem does not appear.
I do not think it makes any difference but i use
<transportConnectors>
<transportConnector name="openwire" uri="nio://0.0.0.0:33029?wireFormat.maxInactivityDuration=60000&wireFormat.maxInactivityDurationInitalDelay=60000"/>
</transportConnectors>
How can I recreate this? How many producers/consumers are attached to this topic? Is it only one topic?
Your settings look okay, but you don't need to set memoryLimit=35mb on the policy. It kinda doesn't make sense to set it the same as the overall system usage. The idea is that memory limit for all topics combined would equal the system memory limit. So for example, if you have two topics, each would use 35MB (2 * 35 == 70MB) and that would exceed the overall system memory settings. I don't think this is the specific cause of what you're seeing, but something to keep in mind.
Also what version of ActiveMQ is this? If you have some tests already written that can produce this, let me know.
It turns out that when using JmsTemplate in order to have asynchronous sent and then messages that cannot be delivered we need to enable explicitQosEnabled and set deliveryMode=1 (non persistent).
Also on client side, the consumer needs to have a smaller prefetch limit
server
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="pooledJmsConnectionFactory"/>
<property name="explicitQosEnabled" value="true"/>
<property name="deliveryMode" value="1"/>
</bean>
client
<jms:listener-container .. prefetch="1000">
...
</jms:listener-container>
dont ask me why... but this seems to have solved my problem.
Basically non 100% needed but if someone can explain this to me would be perfect

No new consumers on activemq queue after a while

Since a month we have a reoccurring issue with activemq and spring. After a some time (between a day and a week) we have no more consumers and no new ones get started and the queue starts to fill up.
This setup ran for over a year, without any issues and as far as we can see nothing relevant has been changed.
An other queue we use also started to show the same behavior, but less frequent.
from the activemq webconsole ( as you can see lots of pending messages and no consumers)
Name ↑ Number Of Pending Messages Number Of Consumers Messages Enqueued Messages Dequeued Views Operations
queue.readresult 19595 0 40747 76651 Browse Active Consumers
contents of our bundle-context.xml
<!-- JMS -->
<bean id="jmsConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" destroy-method="stop">
<property name="maxConnections" value="5" />
<property name="maximumActive" value="5" />
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL">
<value>tcp://localhost:61616</value>
</property>
</bean>
</property>
</bean>
<bean id="ResultMessageConverter" class="com.bla.ResultMessageConverter" />
<jms:listener-container connection-factory="jmsConnectionFactory" destination-resolver="jmsDestinationResolver"
concurrency="2" prefetch="10" acknowledge="auto" cache="none" message-converter="ResultMessageConverter">
<jms:listener destination="queue.readresult" ref="ReaderRequestManager" method="handleReaderResult" />
</jms:listener-container>
There are no exception in any of the logs. Does anyone knows of a reason why after a while no new consumers could be started.
Thanks
I've run into issues before where "consumers stop consuming," but haven't seen consumers stop existing. You may be running out of memory and/or connections in the pool. Do you have to restart ActiveMQ to fix the problem or just your application?
Here are a couple suggestions:
Set the queue prefetch to 0
Add "useKeepAlive=false" on the connection string
Increase the memory limits for the queues
I can see no obvious reason in the config provided why it should fail. So you need to resort to classic trouble shooting.
Try to set logging to debug and recreate the issue. Then you should be able to see more and you might be able to detect the root cause of it.
Also, check out the JMS exception listener and try to attach it your implementation of it to get a grasp of the real problem.
http://docs.oracle.com/javaee/6/api/javax/jms/ExceptionListener.html

ServiceMix, Apache ActiveMQ, Camel sending "done" for consuming messages

The issue I have is this:
using service mix and camel routing I am sending a JSON message via activeMQ to consumer.
The issue is that the time that this consumer takes to process this message is X so it is possible the consumer get stopped or crashed during the consuming of the message. In this case the message will be half consumer and will be already deleted from the queue because well it was delivered.
Is it possible to make the queue to not remove messages when they are consumed but instead to wait for some confirmation from the consumer that the processing of this message is done before deleting it?
In a typical importing files from filesystem you will remove the file or rename it to done only at the end ones the file is fully processed and a transaction is fully committed. So how in the ESB world we can say keep the message till I finish and I tell you to remove it.
i am using spring jms:listener-container and jms:listeners for consuming this messages currently.
Your problem is what JMS transactions solves every day.
Some notes from ActiveMQ about it here
You could easily use local transactions in JMS, and setup a listener container like this (note true on sessionTransacted):
<bean id="myListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="concurrentConsumers" value="1" />
<property name="connectionFactory" ref="jmsConnectionFactory" />
<property name="destination" ref="myQueue" />
<property name="messageListener" ref="myConsumerBean" />
<property name="sessionTransacted" value="true" />
</bean>
Then you have a transacted session. The message will be rolled back to the queue if the message listener fails to consume the message. The transaction will not commit(=message removed from queue) unless the "onMessage" method in the message listener bean returns successfully (no exceptions thrown). I guess this is what you want

Resources