Spring JMS / Integration thread routing - spring

I have a Spring JMS-to-database microservice using Spring Integration to route incoming messages through a sequence of filter, transform, and service activator endpoints (the last persisting the filtered and transformed messages to an Oracle database). Messages have a primary key used in the database. We have a 5 concurrent consumer set up, based on MQ constraints in prod. All of this works.
Recently we found that the upstream producer can send two or more messages with the same primary key back-to-back. Since two separate threads are given the two messages, even MERGE SQL fails with a SQLIntegrityConstraintViolationException (both threads attempt the ON NOT MATCHED INSERT... portion of the MERGE with the database server).
So we thought to change the listener to guarantee that a specific thread would get a message with any given primary key, using a Hash of the key moduloed by the concurrent consumer count. We need to ACK the message once it is saved to the database, not before, so I want to preserve transaction semantics. But I have been unable to get this working with either a Router, or a LoadBalancer on the channel coming from our DefaultMessageListenerContainer. I've also looked at sub-classing DefaultMessageListenerContainer, but don't see the appropriate point to select a given thread.
Any recommendations?
Bean/channel XML setup, our classes renamed for brevity:
...
<bean id="listenerContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:connectionFactory-ref="consumerConnectionFactory"
p:concurrentConsumers="5"
p:maxConcurrentConsumers="5"
p:destinationName="IN_QUEUE"
p:idleConsumerLimit="5"
p:sessionTransacted="true"
p:errorHandler-ref="errorHandler"
p:autoStartup="true"
p:recoveryInterval="60000"
p:receiveTimeout="5000"/>
...
`<int-jms:message-driven-channel-adapter id="xxx-message-in"
container="listenerContainer"
error-channel="errorChannel"/>
<int:transformer id="unmarshaller"
method="unmarshalPayload"
input-channel="message-in" output-channel="filterIt">
<bean class="com.db.xyz.endpoint.UnmarshallerEndpoint"/>
</int:transformer>
<int:filter id="identifyIt" input-channel="filterIt"
output-channel="transformItC" method="isItOurs">
<bean class="com.db.xyz.service.endpoint.ItFilterEndpoint"/>
</int:filter>
<int:transformer id="transformIt"
input-channel="transformItC"
method="transform" output-channel="persistItC">
<bean class="com.db.xzy.service.endpoint.ItTransformEndpoint"/>
</int:transformer>
<int:service-activator id="persistIt" method="publish"
input-channel="persistItC">
<bean class="com.db.xyz.ilr.service.endpoint.PersistEndpoint"/>
</int:service-activator>
Along with error channel definitions, etc. The consumerConnectionFactory is just a class that selects a Solace JMS, MQ Series, or Active MQ factory based on properties.

Related

Consume from channel only if the number of messages reaches a count or the message is in the channel since a while

I have a custom sink module and I would like to consume the messages from the input only if the the number of messages reaches a count or if they are in the channel since some time. In nutshell, I want to do a bulk push.
I have tried aggregating the number of messages after consuming and storing them in an aggregated channel backed by SimpleMessageStore and have MessageGroupStoreReaper checking for messages in the channel.
I am not satisfied with this approach as I am consuming the messages and storing them in an in-memory store, I am aware of the JDBC store as well but I don't want to follow this approach as the message channels in spring XD are backed by redis/mq I would like to consume from the input channel based on my conditions.
My current bean configuration is as shown below:
<int:aggregator id="messageAggregator" ref="messageAggregatorBean"
method="aggregate" input-channel="input" output-channel="aggregatorOutputChannel"
release-strategy="messageReleaseStrategyBean" release-strategy-method="canRelease"
send-partial-result-on-expiry="true" message-store="resultMessageStore">
</int:aggregator>
<int:service-activator id="contributionIndexerService"
ref="contributionIndexerBean" method="bulkIndex" input-channel="aggregatorOutChannel" />
<bean id="resultMessageStore"
class="org.springframework.integration.store.SimpleMessageStore" />
<bean id="resultMessageStoreReaper"
class="org.springframework.integration.store.MessageGroupStoreReaper">
<property name="messageGroupStore" ref="resultMessageStore" />
<property name="timeout" value="60000" />
</bean>
<task:scheduled-tasks>
<task:scheduled ref="resultMessageStoreReaper" method="run"
fixed-rate="10000" />
</task:scheduled-tasks>
Any thoughts or comments?
Thanks in advance.
I'm not sure that you will be able to determine the count of messages in the Broker's queue (Redis/RabbitMQ or even just normal JMS), more over how much they have been there.
You definitely should consume them to do such a logic.
And yes, I think Aggregator can help you. But right: that must be a Persistent Message Store.
for the case
if they are in the channel since some time
The Aggregator suggest an option like group-timeout to release those groups which haven't reached the releaseStrategy condition, but you would like to emit them anyway over some time: http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to

Why PollSkipStrategy.skipPoll method is getting called on every message polled from queue?

I'm using inbound poller to process failed requests from backout queue. For scheduling, I'm using corn expression '0 0/2 * * * *' i.e. execute poller every two minutes. The scheduling is working fine as per corn, but PollSkipStrategy.skipPoll method is getting called for every message polled. I was under impression is, poll skip strategy will be execute once for each poll and not for each record polled. I have implementation for PollSkipStrategy.skipPoll, which returns true or false based on peoperty. I'm missing something here? Here is my configuration
<bean id="RegistrationEventPoller"
class="com.poller.RegistrationEventPoller">
<property name="RegistrationEventRetryCount" value="$env{RegistrationEventRetryCount}"/>
</bean>
<bean id="PollSkipAdvice" class="org.springframework.integration.scheduling.PollSkipAdvice">
<constructor-arg ref="PollSkipStrategy"/>
</bean>
<bean id="PollSkipStrategy"
class="com..poller.PollSkipStrategy">
<property name="RegistrationPollerOnOff" value="$env{RegistrationPollerOnOff}"/>
</bean>
The advice is an around advice on the whole flow (MessageSource.receive() and sending the message). When the poller fires it calls the flow for up to maxMessagesPerPoll so, yes, the advice is actually called for each message found within the poll, not just on the first poll. It simply provides a mechanism to stop calling the message source if some condition prevents you from handling messages.
A more sophisticated Smart Polling feature was added in 4.2 which gives you much more flexibility.

Spring RabbitTemplate is not creating dead letter queue with TTL

I am using spring-rabbit1.1 and RabbitMQ 3.3.1 ,
My spring configuration will create any queue with the help of RabbitTemplate on Rabbit MQ but if the queue has been configured with x-dead-letter-exchange and x-message-ttl , it just creates the queue with out the TTL and dead letter exchange.
For Eg : the below queue will create the queue but TTL and dead letter exhange is not getting created .
<rabbit:queue name="hello.queue.dead">
<rabbit:queue-arguments>
<entry key="x-dead-letter-exchange" value="hello.activity-task.topic"/>
<entry key="x-message-ttl" value="10000"/>
</rabbit:queue-arguments>
</rabbit:queue>
So i had to go and delete the queue from Rabbit MQ and create with all the required values manually to make it work .
Can anyone help me if there is any option to solve this issue ???
You have to explicitly declare the queue and exchange...
<rabbit:queue name="q.with.dlx">
<rabbit:queue-arguments>
<entry key="x-dead-letter-exchange" value="dlx"/>
<entry key="x-message-ttl" value="10000" value-type="java.lang.Long"/>
</rabbit:queue-arguments>
</rabbit:queue>
<rabbit:queue name="dlq"/>
<rabbit:direct-exchange name="dlx">
<rabbit:bindings>
<rabbit:binding key="q.with.dlx" queue="dlq"/>
</rabbit:bindings>
</rabbit:direct-exchange>
This assumes you routed the original message using the default direct exchange (routing by queue name). Hence the dead letter routing uses the same routing key (queue name). If you route using an explicit routing key, you would use that.
By the way, the RabbitTemplate does not declare these elements, it's the RabbitAdmin instance.

How to route messages in RabbitMQ receiver?

I would like to process incoming messages differently basing on the message header. How to implement it efficiently in RabbitMQ?
My listener is listenining to messages coming from presence.queue channels. The message header KEY has different keys: key1, key2. Messages with key1 should be processed by messageService.method1, messages with key2 should be processed with messageService.method2.
Here is the code I am playing with but obviously it is wrong. I probably should route messages from presence.queue to first.queue and second.queue ?
<!-- RECEIVER -->
<rabbit:queue id="presence.queue" name="presence" durable="true"/>
<rabbit:queue id="first.queue" name="first"/>
<rabbit:queue id="second.queue" name="second"/>
<rabbit:direct-exchange name="presence.direct" durable="true" >
<rabbit:bindings>
<rabbit:binding queue="presence.queue" key="key1"/>
<rabbit:binding queue="presence.queue" key="key2"/>
</rabbit:bindings>
</rabbit:direct-exchange>
<!-- RabbitMQ Asynchronous Receiver from PRESENCE_ENGINE queue -->
<rabbit:listener-container connection-factory="connectionFactory" acknowledge="auto" >
<rabbit:listener queues="first.queue" ref="messageService" method="method1" />
<rabbit:listener queues="second.queue" ref="messageService" method="method2" />
</rabbit:listener-container>
Why do you need presence.queue? Just bind your consumer queues with the appropriate routing key. If you want to use message headers for routing, use a Headers Exchange.
See Getting Started and Headers Exchange.
Correction. Producers send messages to the exchange, not the queue. Come back, please, to the AMQP theory.
They may send Messages with the same routing key as well, but as Gary suggests, you can use Headers Exchange and build binding to your queues by some value from MessageProperties.
Especially you already have that value.
There is nohing to change on producer side. But from other side: you are the server (consumer) and everything in your hadns: you can configure AMQP routing as it would be comfortably for you.
That's why AMQP is better than JMS!

Spring Integration - creating an anonymous queue

Problem:
I am using Spring Integration backed by RabbitMQ. I have a Fanout Exchange to which messages are published. I wish to launch multiple instances of the same service which creates a queue bound to this exchange. When a message is published to this exchange, I want all the launched services to receive the message. This does not happen as although there are multiple services, they all use the same queue
Solution:
Create a non-durable, exclusive, anonymous queue for each consumer.
Problem:
How do I make the queue anonymous?
Here is my code so far, the problem is the binding requires a queue attribute, and the queue requires a name - and therefore is named, and not anonymous.
<rabbit:fanout-exchange id="structureUpdateExchange" name="publish.registry.update" >
<rabbit:bindings>
<rabbit:binding queue="publish.registry.update.queue" />
</rabbit:bindings>
</rabbit:fanout-exchange>
<rabbit:queue durable="false" auto-delete="true" exclusive="true" name="publish.registry.update.queue" />
<si:channel id="publishInformationChannel" />
<int-amqp:inbound-gateway request-channel="publishInformationChannel"
queue-names="publish.registry.update.queue" />
Any help would be very much appreciated. Thanks.
So yes, as has said by Phillipp define the queue with id but no name.
<rabbit:queue id="queueTemp"/>
You need to use SpEL to reference the queue name. See the queue-names attribute below:
<int-amqp:inbound-channel-adapter
channel="fromRabbit"
queue-names="#{#queueTemp.name}"
connection-factory="connectionFactory" />
You might already be doing this but note that on the consumer side you also define the fanout exchange with the binding to your temporary queue. So something like:
<rabbit:fanout-exchange id="myFanoutExchange" name="test-fanex" >
<rabbit:bindings>
<rabbit:binding queue="queueTemp" />
</rabbit:bindings>
</rabbit:fanout-exchange>
You can directly define a channel that is backed by an AMQP queue:
<int-amqp:publish-subscribe-channel id="publishInformationChannel"
exchange="publish.registry.update" />
This will create an exchange publish.registry.update, an anonymous queue and an incoming queue adapter for the channel publishInformationChannel.

Resources