I am using ActiveMQ 5.13.4 and I have set up a network-of-brokers with two brokers A and B that are connected via a networkConnector. Each broker has a consumer connected to it (A' and B'). A' and B' consume messages from the same distributed queue (e.g. requests-queue). Each consumer is a java application, represented by a Spring DefaultJmsListenerContainerFactory having the setting concurrency: 5-10. Basically, each consumer has max 10 threads that can consume simultaneously messages from the queue.
In broker configuration (activemq.xml) I have set the prefetch size to 1, assuming that there are not many messages and they need a lot of processing time.
<policyEntry queue=">" producerFlowControl="true" optimizedDispatch="true" queuePrefetch="1" enableAudit="false">
The desired behavior is when I send 100 messages to A it starts processing 10 messages simultaneously (one for each consumer thread) and forwards another 10 to B. When B finishes processing the first 10 messages, it would get an extra 10. This would continue until all the messages are processed.
The actual behavior is that when I send 100 messages to A, it sends 10 messages, one per each thread, to the local consumer A' and forwards the remaining 90 to B'. The consumers on A' finish processing the messages and they wait while 90 messages get processed on broker B by its consumers.
Question: How can I configure the network in order to balance messages based on consumer availability?
The full activemq.xml configuration for one of the brokers is the following. The other one has the same configuration only a different name and IP in the networkConnector.
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<!-- Allows accessing the server log -->
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="broker-jms-tux-01-qa-gnd"
persistent="false"
dataDirectory="${activemq.data}"
cacheTempDestinations="false"
advisorySupport="true"
useJmx="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true">
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see:
http://activemq.apache.org/slow-consumer-handling.html
-->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
<pendingSubscriberPolicy>
<vmCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" optimizedDispatch="true" queuePrefetch="1" enableAudit="false">
<!--
Allow messages to be replayed back when the broker to which they were forwarded does not have any more consumers.
See section "Stuck Messages" from http://activemq.apache.org/networks-of-brokers.html
-->
<networkBridgeFilterFactory>
<!--
N.B.: When using replayWhenNoConsumers=true for versions < 5.9, it is necessary to also disable the cursors duplicate detection using enableAudit=false as the cursor could mark the replayed messages as duplicates (depending on the time window between playing and replaying these messages over the network bridge). The problem is fully explained here: http://tmielke.blogspot.de/2012/03/i-have-messages-on-queue-but-they-dont.html
-->
<conditionalNetworkBridgeFilterFactory replayWhenNoConsumers="true"/>
</networkBridgeFilterFactory>
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
<deadLetterStrategy>
<sharedDeadLetterStrategy expiration="300000"/>
</deadLetterStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see:
http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="true" connectorPort="1099"/>
</managementContext>
<!-- Network of brokers -->
<networkConnectors>
<networkConnector
name="broker-jms-tux-01-qa-gnd->broker-jms-tux-02-qa-gnd"
uri="static:(tcp://10.83.16.22:61616)"
conduitSubscriptions="false"
dynamicOnly="true"
networkTTL="2"
alwaysSyncSend="true"
decreaseNetworkConsumerPriority="true"
duplex="false"/>
</networkConnectors>
<!--
The systemUsage controls the maximum amount of space the broker will
use before disabling caching and/or slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="80"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="10 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="4 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see:
http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 400MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=3200000000"/>
</transportConnectors>
<!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook"/>
</shutdownHooks>
</broker>
<!--
Enable web consoles, REST and Ajax APIs and demos
The web consoles requires by default login, you can disable this in the jetty.xml file
Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
-->
<import resource="jetty.xml"/>
</beans>
Thanks you in advance!
**Update 25/10/2016: **
I have come across the following statement in: http://activemq.apache.org/networks-of-brokers.html
The conditionalNetworkBridgeFilterFactory factory allows a rate limit
to be specified for a destination, such that network consumer can be
throttled. Prefetch for a network consumer is largely negated by the
fact that a network consumer relays a message typically acks very
quickly so even with a low prefetch and decreased priority a network
consumer can starve a modestly quick local consumer. Throttling
provides a remedy for this.
Question: It is still not clear for me, why when the local consumer A' finishes processing the initial 10 messages, broker A does not receive any of the 80 messages remaining on B (because 10 were dispatched to the local consumer of broker B)?
I have tried setting a higher networkTTL, but it still does not work.
Related
There is one request queue and the reply queues are created by the client server instances and pinned to each instance rather than using temporary queues.
The use case needs to get an inbound jms message and then send that message to an asynchronous process. Once the async reply messsage is received from the service I need to take those results and reply back to the original message's jmsReplyTo. The Jms gateway would not work in this instance AFAIK>
I am using a jms message driven channel adapter for the message in with a series of channels and service activators to handle the out of process calls and async replies. I am trying to use the DynamicDestinationResolver to no avail. Additionally I have tried to set the outbound destination address programatically but could not figure out a good way to do this.
This seems like a common pattern but I could not find a good example for a completely disconnected async request response. Disconnected meaning that the usual async jms request reply did not seem to fit the need.
Context Config:
<!-- Input from Amq -->
<amq:queue id="requestQueue" physicalName="${request.queue}" />
<int-jms:message-driven-channel-adapter id="jmsIn"
connection-factory="jmsConnectionFactory"
destination="requestQueue"
channel="queueRequestChannel" concurrent-consumers="5" />
<int:channel id="queueRequestChannel" />
<int:service-activator input-channel="queueRequestChannel" ref="switchMessageHandler" method="processSwitchMessage"
output-channel="cardNetworkOutChannel"/>
<!-- Output to Card Network-->
<int:channel id="cardNetworkOutChannel" />
<!--<int:service-activator input-channel="cardNetworkOutChannel" ref="cardNetworkHandler" method="send8583Message" />-->
<!-- Simply used to mock the card network by transforming a SwithMessage to a SwitchMessageResponse * Not needed for target solution -->
<int:transformer id="requestResponseTransformer" ref="nettyCardNetworkClientMock" input-channel="cardNetworkOutChannel"
method="process" output-channel="cardNetworkInChannel"/>
<!-- Input from Card Network -->
<int:channel id="cardNetworkInChannel" />
<int:service-activator input-channel="cardNetworkInChannel" ref="switchMessageHandler" method="sendSwitchMessage"
output-channel="queueReplyChannel"/>
<int:channel id="queueReplyChannel"/>
<int-jms:outbound-channel-adapter
destination-resolver="simpleDestinationResolver" connection-factory="jmsConnectionFactory"
channel="queueReplyChannel" destination-expression="headers.jms_replyTo" />
I just updated the jms sample app to make the server side use independent adapters instead of the inbound gateway and it works just fine...
<!-- <jms:inbound-gateway id="jmsin" -->
<!-- request-destination="requestQueue" -->
<!-- request-channel="demoChannel"/> -->
<channel id="demoChannel"/>
<jms:message-driven-channel-adapter destination="requestQueue" channel="demoChannel" />
<service-activator input-channel="demoChannel" ref="demoBean" output-channel="reply" />
<channel id="reply" />
<jms:outbound-channel-adapter channel="reply" destination-expression="headers['jms_replyTo']" />
Turn on DEBUG logging - we put out lots of useful stuff.
EDIT
I just made it async by...
<channel id="reply">
<queue/>
</channel>
<jms:outbound-channel-adapter channel="reply" destination-expression="headers['jms_replyTo']">
<poller fixed-delay="3000"/>
</jms:outbound-channel-adapter>
EDIT2
Depending on what you are using on the client side, many clients require the inbound message id to be used as the correlation id. (This is true for the outbound gateway by default, with named reply queue, unless you provide a correlation-key).
So, to set up the correlationId, you can use a header enricher; I just tested this...
<chain input-channel="reply">
<header-enricher>
<header name="jms_correlationId" expression="headers['jms_messageId']" />
</header-enricher>
<jms:outbound-channel-adapter destination-expression="headers['jms_replyTo']"/>
<poller fixed-delay="1000" />
</chain>
This is not the issue if the client side is setting the correlation id header itself.
So I have used Spring integration to link JMS and ActiveMQ as under:-
<amq:broker useJmx="false" persistent="false">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://localhost:0" />
</amq:transportConnectors>
</amq:broker>
<bean id="amqConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" p:brokerURL="tcp://localhost:61616"/>
<bean id="destination" class="org.apache.activemq.command.ActiveMQQueue" c:name="destination"/>
<bean id="testTemplate" class="org.springframework.jms.core.JmsTemplate" depends-on="amqConnectionFactory" scope="prototype"
p:connectionFactory-ref="amqConnectionFactory"
p:pubSubDomain="false"
p:defaultDestination-ref="destination"
p:sessionTransacted="true"
p:receiveTimeout="5000"/>
Now, when I run the test by starting ActiveMQ and then running my application, I see that the messages are published to the "destination" queue from the logs. However, I am not able to retrieve any of these from the queue. Even in JConsole, it shows the queueSize and counts as 0.
How do I make it work so that I can send to and receives messages from this queue? Please help.
Why do you specify the transport connector to: tcp://localhost:0 and the ConnectionFactory URL to tcp://localhost:61616?
I would use the same URL for both.
A spring quartz process runs every 15 minutes in my project i.e 96 times a day. This fetch certain records from database and POST it on a REST service (running on JBoss 7). These records are in general 50 to 100 in count.
On REST service there is jms event publisher that publishes this message on a topic. There are two consumers on this topic.
That process message and sends push notification messages on mobile
Talk to third party (generally takes 4 to 5 second to complete the call)
Since it is topic both consumers receive all messages but they filter them out based on some property, so few messages are processed by one and rest by another consumer.
My problem is; which is being observed recently since a week time; that consumer #1 receives response from APNS as invalid token multiple times; token is used to send push notification to mobile; after some time this consumer stops and do not respond at all while second one keeps running.
Below are configurations:
<amq:broker id="broker" useJmx="false" persistent="false">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://localhost:0"/>
</amq:transportConnectors>
</amq:broker>
<!-- ActiveMQ Destination -->
<amq:topic id="topicName" physicalName="topicPhysicalName"/>
<!-- JMS ConnectionFactory to use, configuring the embedded broker using XML -->
<amq:connectionFactory id="jmsFactory" brokerURL="vm://localhost"/>
<!-- JMS Producer Configuration -->
<bean id="jmsProducerConnectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory"
depends-on="broker"
p:targetConnectionFactory-ref="jmsFactory"/>
<!-- JMS Templates-->
<bean id="jmsTemplate"
class="org.springframework.jms.core.JmsTemplate"
p:connectionFactory-ref="jmsProducerConnectionFactory"/>
<!-- Publisher-->
<bean name="jmsEventPublisher"
class="com.jhi.mhm.services.event.jms.publisher.JMSEventPublisher">
<property name="jmsTemplate" ref="jmsTemplate"/>
<property name="topic">
<map>
<entry key="keyname" value-ref="topicName"/>
</map>
</property>
</bean>
<!-- JMS Consumer Configuration -->
<bean name="consumer2" class="Consumer2"/>
<bean name="consumer1" class="Consumer1"/>
<bean id="jmsConsumerConnectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory"
depends-on="broker"
p:targetConnectionFactory-ref="jmsFactory"/>
<jms:listener-container container-type="default"
connection-factory="jmsConsumerConnectionFactory"
acknowledge="auto"
destination-type="topic">
<jms:listener destination="topicPhysicalName" ref="consumer1"/>
<jms:listener destination="topicPhysicalName" ref="consumer2"/>
</jms:listener-container>
I search another posted questions but could not find anything related.
Your thoughts would be really helpful.
shailu - I went through similar problem. What we did is upgrade the version of MQ. Although this did not solved the problem completely as MQ shows random behavior and at the end we simply merged our endpoint and call destination as per biz logic
I have a production application that seems to be losing messages intermittently. One example that I have identified is the sending system logs 3 messages that are sent milliseconds apart. The receiving system logs that it got message 1 and message 3...but not message 2. The queue's depth is 0. I see no errors in the logs of the application that reads the messages to indicate something bad happened.
I have verified that there are no other "rogue" clients creating a race condition for reading messages.
Here is my configuration; we have a primary and secondary/failover queue...
<bean id="primaryProficiencyInboundQueue" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueManagerName" value="${lsm.primary.outbound.manager}"/>
<property name="baseQueueName" value="${lsm.proficiency.inbound.queue}"/>
</bean>
<bean id="secondaryProficiencyInboundQueue" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueManagerName" value="${lsm.secondary.outbound.manager}"/>
<property name="baseQueueName" value="${lsm.proficiency.inbound.queue}"/>
</bean>
<int:channel id="inboundProficiencyChannel">
<int:interceptors>
<int:wire-tap channel="logger" />
</int:interceptors>
</int:channel>
<!-- adapter that connects the inbound proficiency queues to the inboundProficiencyChannel -->
<jms:message-driven-channel-adapter id="primaryProficiencyInboundAdapter"
connection-factory="primaryConnectionFactory"
destination="primaryProficiencyInboundQueue"
channel="inboundProficiencyChannel" />
<jms:message-driven-channel-adapter id="secondaryProficiencyInboundAdapter"
connection-factory="secondaryConnectionFactory"
destination="secondaryProficiencyInboundQueue"
channel="inboundProficiencyChannel" />
<int:service-activator id="proficiencyInboundActivator"
input-channel="inboundProficiencyChannel">
<bean class="com.myactivator.LSMInboundProficiencyActivator" />
</int:service-activator>
Is it possible that something is failing silently?
One other noteworthy thing is that there are two production servers that are both configured as above....meaning that there are two servers that have JMS adapters configured to read the messages. I mentioned above that I see message 1 and message 3 being read. In that case, I see server one process message one and server two process mesage two...so it seems that having two servers configured like above has no negative impact. Any other thoughts on what could be happening or how I can debug the case of the missing/dropped messages?
Version information:
Spring - 3.0.5.RELEASE
Spring Integration - 2.0.3.RELEASE
Nothing springs to mind that would cause lost messages; I don't recall that ever being reported; but 2.0.3 is quite ancient (3rd birthday coming up soon). We don't support 2.0.x any more but you should at least update to 2.0.6 which is the last release of 2.0.x. But preferably move up to the latest 3.0.0 release (or at least 2.2.6).
I have activemq used in my system and what i see is the following message:
TopicSubscription: consumer=...: Pending message cursor [org.apache.activemq.broker.region.cursors.VMPendingMessageCursor#1684f89c] is full, temp usage (0%) or memory usage (100%) limit reached, blocking message add() pending the release of resources.
This is because if i understand correct my consumer is slow while my producer is fast. The result out of it is that eventually my producer is blocked untill consumer reads the message and frees some memory. What i whant is that my producer is not blocked and also when memory is full old messages are being discurded.
Given my understanding of what i have read the following configuration should do the trick (messageEvictionStrategy, pendingMessageLimitStrategy) but it is not working for me and i cannot figure out why.
I have specified low memoryusage limit low (35Mb) to make issue apear faster for testing reasons, but the case is that i need it eventually when the problem apears for activemq to just drop old messages.
I have found one non satisfactory solution of setting in ActiveMQConnectionFactory useAsyncSend=true and specifying sendTimeout. This makes producer not blocked but in this way the newest message is dropped and not the olderst one.
Finally, i am talking for non durable topics.
Any help guys would pe perfect. Below i have activemq configuration
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false" memoryLimit="35 Mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
<messageEvictionStrategy>
<oldestMessageEvictionStrategy/>
</messageEvictionStrategy>
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="10"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="35 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="5000 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
activemq version 5.7.0
i use spring template to send messages:
<bean class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="pooledJmsConnectionFactory"/>
<property name="timeToLive" value="100"/>
</bean>
I transmit javax.jms.ObjectMessage, relativelly small in size.
I found the problem in customer premisses I have many toppics in my application but managed to reproduce it loccally sending from 1 thread, non-stop messages continiusly always to the same topic. The message send was just a small string.
I have only one producer and problem seems to appear when i have 1 (or more) slow consumer(s) -but one slow consumer is enough-. if no slow consumer exists, problem does not appear.
I do not think it makes any difference but i use
<transportConnectors>
<transportConnector name="openwire" uri="nio://0.0.0.0:33029?wireFormat.maxInactivityDuration=60000&wireFormat.maxInactivityDurationInitalDelay=60000"/>
</transportConnectors>
How can I recreate this? How many producers/consumers are attached to this topic? Is it only one topic?
Your settings look okay, but you don't need to set memoryLimit=35mb on the policy. It kinda doesn't make sense to set it the same as the overall system usage. The idea is that memory limit for all topics combined would equal the system memory limit. So for example, if you have two topics, each would use 35MB (2 * 35 == 70MB) and that would exceed the overall system memory settings. I don't think this is the specific cause of what you're seeing, but something to keep in mind.
Also what version of ActiveMQ is this? If you have some tests already written that can produce this, let me know.
It turns out that when using JmsTemplate in order to have asynchronous sent and then messages that cannot be delivered we need to enable explicitQosEnabled and set deliveryMode=1 (non persistent).
Also on client side, the consumer needs to have a smaller prefetch limit
server
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="pooledJmsConnectionFactory"/>
<property name="explicitQosEnabled" value="true"/>
<property name="deliveryMode" value="1"/>
</bean>
client
<jms:listener-container .. prefetch="1000">
...
</jms:listener-container>
dont ask me why... but this seems to have solved my problem.
Basically non 100% needed but if someone can explain this to me would be perfect