Problem:
I am using Spring Integration backed by RabbitMQ. I have a Fanout Exchange to which messages are published. I wish to launch multiple instances of the same service which creates a queue bound to this exchange. When a message is published to this exchange, I want all the launched services to receive the message. This does not happen as although there are multiple services, they all use the same queue
Solution:
Create a non-durable, exclusive, anonymous queue for each consumer.
Problem:
How do I make the queue anonymous?
Here is my code so far, the problem is the binding requires a queue attribute, and the queue requires a name - and therefore is named, and not anonymous.
<rabbit:fanout-exchange id="structureUpdateExchange" name="publish.registry.update" >
<rabbit:bindings>
<rabbit:binding queue="publish.registry.update.queue" />
</rabbit:bindings>
</rabbit:fanout-exchange>
<rabbit:queue durable="false" auto-delete="true" exclusive="true" name="publish.registry.update.queue" />
<si:channel id="publishInformationChannel" />
<int-amqp:inbound-gateway request-channel="publishInformationChannel"
queue-names="publish.registry.update.queue" />
Any help would be very much appreciated. Thanks.
So yes, as has said by Phillipp define the queue with id but no name.
<rabbit:queue id="queueTemp"/>
You need to use SpEL to reference the queue name. See the queue-names attribute below:
<int-amqp:inbound-channel-adapter
channel="fromRabbit"
queue-names="#{#queueTemp.name}"
connection-factory="connectionFactory" />
You might already be doing this but note that on the consumer side you also define the fanout exchange with the binding to your temporary queue. So something like:
<rabbit:fanout-exchange id="myFanoutExchange" name="test-fanex" >
<rabbit:bindings>
<rabbit:binding queue="queueTemp" />
</rabbit:bindings>
</rabbit:fanout-exchange>
You can directly define a channel that is backed by an AMQP queue:
<int-amqp:publish-subscribe-channel id="publishInformationChannel"
exchange="publish.registry.update" />
This will create an exchange publish.registry.update, an anonymous queue and an incoming queue adapter for the channel publishInformationChannel.
Related
I have a Spring JMS-to-database microservice using Spring Integration to route incoming messages through a sequence of filter, transform, and service activator endpoints (the last persisting the filtered and transformed messages to an Oracle database). Messages have a primary key used in the database. We have a 5 concurrent consumer set up, based on MQ constraints in prod. All of this works.
Recently we found that the upstream producer can send two or more messages with the same primary key back-to-back. Since two separate threads are given the two messages, even MERGE SQL fails with a SQLIntegrityConstraintViolationException (both threads attempt the ON NOT MATCHED INSERT... portion of the MERGE with the database server).
So we thought to change the listener to guarantee that a specific thread would get a message with any given primary key, using a Hash of the key moduloed by the concurrent consumer count. We need to ACK the message once it is saved to the database, not before, so I want to preserve transaction semantics. But I have been unable to get this working with either a Router, or a LoadBalancer on the channel coming from our DefaultMessageListenerContainer. I've also looked at sub-classing DefaultMessageListenerContainer, but don't see the appropriate point to select a given thread.
Any recommendations?
Bean/channel XML setup, our classes renamed for brevity:
...
<bean id="listenerContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:connectionFactory-ref="consumerConnectionFactory"
p:concurrentConsumers="5"
p:maxConcurrentConsumers="5"
p:destinationName="IN_QUEUE"
p:idleConsumerLimit="5"
p:sessionTransacted="true"
p:errorHandler-ref="errorHandler"
p:autoStartup="true"
p:recoveryInterval="60000"
p:receiveTimeout="5000"/>
...
`<int-jms:message-driven-channel-adapter id="xxx-message-in"
container="listenerContainer"
error-channel="errorChannel"/>
<int:transformer id="unmarshaller"
method="unmarshalPayload"
input-channel="message-in" output-channel="filterIt">
<bean class="com.db.xyz.endpoint.UnmarshallerEndpoint"/>
</int:transformer>
<int:filter id="identifyIt" input-channel="filterIt"
output-channel="transformItC" method="isItOurs">
<bean class="com.db.xyz.service.endpoint.ItFilterEndpoint"/>
</int:filter>
<int:transformer id="transformIt"
input-channel="transformItC"
method="transform" output-channel="persistItC">
<bean class="com.db.xzy.service.endpoint.ItTransformEndpoint"/>
</int:transformer>
<int:service-activator id="persistIt" method="publish"
input-channel="persistItC">
<bean class="com.db.xyz.ilr.service.endpoint.PersistEndpoint"/>
</int:service-activator>
Along with error channel definitions, etc. The consumerConnectionFactory is just a class that selects a Solace JMS, MQ Series, or Active MQ factory based on properties.
I have a custom sink module and I would like to consume the messages from the input only if the the number of messages reaches a count or if they are in the channel since some time. In nutshell, I want to do a bulk push.
I have tried aggregating the number of messages after consuming and storing them in an aggregated channel backed by SimpleMessageStore and have MessageGroupStoreReaper checking for messages in the channel.
I am not satisfied with this approach as I am consuming the messages and storing them in an in-memory store, I am aware of the JDBC store as well but I don't want to follow this approach as the message channels in spring XD are backed by redis/mq I would like to consume from the input channel based on my conditions.
My current bean configuration is as shown below:
<int:aggregator id="messageAggregator" ref="messageAggregatorBean"
method="aggregate" input-channel="input" output-channel="aggregatorOutputChannel"
release-strategy="messageReleaseStrategyBean" release-strategy-method="canRelease"
send-partial-result-on-expiry="true" message-store="resultMessageStore">
</int:aggregator>
<int:service-activator id="contributionIndexerService"
ref="contributionIndexerBean" method="bulkIndex" input-channel="aggregatorOutChannel" />
<bean id="resultMessageStore"
class="org.springframework.integration.store.SimpleMessageStore" />
<bean id="resultMessageStoreReaper"
class="org.springframework.integration.store.MessageGroupStoreReaper">
<property name="messageGroupStore" ref="resultMessageStore" />
<property name="timeout" value="60000" />
</bean>
<task:scheduled-tasks>
<task:scheduled ref="resultMessageStoreReaper" method="run"
fixed-rate="10000" />
</task:scheduled-tasks>
Any thoughts or comments?
Thanks in advance.
I'm not sure that you will be able to determine the count of messages in the Broker's queue (Redis/RabbitMQ or even just normal JMS), more over how much they have been there.
You definitely should consume them to do such a logic.
And yes, I think Aggregator can help you. But right: that must be a Persistent Message Store.
for the case
if they are in the channel since some time
The Aggregator suggest an option like group-timeout to release those groups which haven't reached the releaseStrategy condition, but you would like to emit them anyway over some time: http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to
Hey so i'm using spring integration's jms:outbound-channel-adapterand need to set a priority on a message before i push it through to the messaging system.
Now in plain JMS i had two ways of doing it.
Either set the priority on the MessageProducer:
this.producer.setPriority(i);
Or on the send method itself:
channel.send(message, DeliveryMode.PERSISTENT, 5, 1000);
Neither of these options are available for me anymore since the channel adapter abstracts me away from these details.
Setting the priority on the message itself works only in the in memory channels of spring integration and loses effect soon as i put it into an actual queue. And turns out setting the priority on the message isn't an option at all: JMS message priority not working on Message
There's an attribute on the channel adapter where i can set the priority, but this is static.
<jms:outbound-channel-adapter id="101Out"
channel="101MessageChannel"
connection-factory="101Factory"
destination="QUEUE_NAME"
priority="1" />
The max i can do i read it from a property file. (Or so i think. I'm not sure). I can use the destination-expression attribute to inspect the incoming message and dynamically route it to different destinations, but there's no priority-expression counter part for me to do the same with the priority.
I have a work around of sorts, but it's not a very good one:
<jms:outbound-channel-adapter id="101HighPriorityOut"
channel="101HighPriorityChannel"
connection-factory="101Factory"
destination-expression="headers.QUEUE_NAME"
priority="1"
explicit-qos-enabled="true" />
<jms:outbound-channel-adapter id="101LowPriorityOut"
channel="101LowPriorityChannel"
connection-factory="101Factory"
destination-expression="headers.QUEUE_NAME"
priority="0"
explicit-qos-enabled="true" />
I just route the messages to the appropriate outbound adapter once i determine what the priority needs to be. But if the number of priorities increases i'll be in trouble. Even if it doesn't, having two outbound adapters instead of one just coz i couldn't dynamically assign a priority is kinda clumsy i thought.
Appreciate the help :-)
Oh and i'm using Websphere MQ as my message broker. I don't know if this has anything to do with the message broker though.
Simply set the priority header in the message...
<int:header-enricher ...>
<int:priority value="2" />
</int:heaer-enricher>
The priority in the adapter configuration is a default which is used when there's no priority header (you can use a property placeholder to set it from a properties file).
Or, use an expression...
<int:header-enricher ...>
<int:priority expression="payload.foo == 'bar' ? 1 : 2" />
</int:heaer-enricher>
<int:header-enricher ...>
<int:priority expression="payload.priority" />
</int:heaer-enricher>
<int:header-enricher ...>
<int:priority expression="#someBean.calculatePriority(payload)" />
</int:heaer-enricher>
I am using spring-rabbit1.1 and RabbitMQ 3.3.1 ,
My spring configuration will create any queue with the help of RabbitTemplate on Rabbit MQ but if the queue has been configured with x-dead-letter-exchange and x-message-ttl , it just creates the queue with out the TTL and dead letter exchange.
For Eg : the below queue will create the queue but TTL and dead letter exhange is not getting created .
<rabbit:queue name="hello.queue.dead">
<rabbit:queue-arguments>
<entry key="x-dead-letter-exchange" value="hello.activity-task.topic"/>
<entry key="x-message-ttl" value="10000"/>
</rabbit:queue-arguments>
</rabbit:queue>
So i had to go and delete the queue from Rabbit MQ and create with all the required values manually to make it work .
Can anyone help me if there is any option to solve this issue ???
You have to explicitly declare the queue and exchange...
<rabbit:queue name="q.with.dlx">
<rabbit:queue-arguments>
<entry key="x-dead-letter-exchange" value="dlx"/>
<entry key="x-message-ttl" value="10000" value-type="java.lang.Long"/>
</rabbit:queue-arguments>
</rabbit:queue>
<rabbit:queue name="dlq"/>
<rabbit:direct-exchange name="dlx">
<rabbit:bindings>
<rabbit:binding key="q.with.dlx" queue="dlq"/>
</rabbit:bindings>
</rabbit:direct-exchange>
This assumes you routed the original message using the default direct exchange (routing by queue name). Hence the dead letter routing uses the same routing key (queue name). If you route using an explicit routing key, you would use that.
By the way, the RabbitTemplate does not declare these elements, it's the RabbitAdmin instance.
I would like to process incoming messages differently basing on the message header. How to implement it efficiently in RabbitMQ?
My listener is listenining to messages coming from presence.queue channels. The message header KEY has different keys: key1, key2. Messages with key1 should be processed by messageService.method1, messages with key2 should be processed with messageService.method2.
Here is the code I am playing with but obviously it is wrong. I probably should route messages from presence.queue to first.queue and second.queue ?
<!-- RECEIVER -->
<rabbit:queue id="presence.queue" name="presence" durable="true"/>
<rabbit:queue id="first.queue" name="first"/>
<rabbit:queue id="second.queue" name="second"/>
<rabbit:direct-exchange name="presence.direct" durable="true" >
<rabbit:bindings>
<rabbit:binding queue="presence.queue" key="key1"/>
<rabbit:binding queue="presence.queue" key="key2"/>
</rabbit:bindings>
</rabbit:direct-exchange>
<!-- RabbitMQ Asynchronous Receiver from PRESENCE_ENGINE queue -->
<rabbit:listener-container connection-factory="connectionFactory" acknowledge="auto" >
<rabbit:listener queues="first.queue" ref="messageService" method="method1" />
<rabbit:listener queues="second.queue" ref="messageService" method="method2" />
</rabbit:listener-container>
Why do you need presence.queue? Just bind your consumer queues with the appropriate routing key. If you want to use message headers for routing, use a Headers Exchange.
See Getting Started and Headers Exchange.
Correction. Producers send messages to the exchange, not the queue. Come back, please, to the AMQP theory.
They may send Messages with the same routing key as well, but as Gary suggests, you can use Headers Exchange and build binding to your queues by some value from MessageProperties.
Especially you already have that value.
There is nohing to change on producer side. But from other side: you are the server (consumer) and everything in your hadns: you can configure AMQP routing as it would be comfortably for you.
That's why AMQP is better than JMS!