My current setup (partly) is like this:
<channel id="chainInboundChannel">
<queue message-store="mongoDbMessageStore"/>
</channel>
<chain id="myChain" input-channel="chainInboundChannel">
<service-activator ref="service1" ... />
<service-activator ref="service2" ... />
<service-activator ref="service3" ... />
<service-activator ref="service4" ... />
</chain>
Presumably my messages are now persisted. This seems to be true for the time the messages are in the queue. But as a message is received by the chain, it is removed from the message store. So if the application shuts down while the message is being processed by e.g. 'service3', the message is simply lost.
How can I keep the messages persisted while they are being processed inside the chain?
OPTIONAL:
What if any of the services returns null at some specific scenario? In this case the message should be removed since it has come to the end of its lifecycle.
Related
I have a custom sink module and I would like to consume the messages from the input only if the the number of messages reaches a count or if they are in the channel since some time. In nutshell, I want to do a bulk push.
I have tried aggregating the number of messages after consuming and storing them in an aggregated channel backed by SimpleMessageStore and have MessageGroupStoreReaper checking for messages in the channel.
I am not satisfied with this approach as I am consuming the messages and storing them in an in-memory store, I am aware of the JDBC store as well but I don't want to follow this approach as the message channels in spring XD are backed by redis/mq I would like to consume from the input channel based on my conditions.
My current bean configuration is as shown below:
<int:aggregator id="messageAggregator" ref="messageAggregatorBean"
method="aggregate" input-channel="input" output-channel="aggregatorOutputChannel"
release-strategy="messageReleaseStrategyBean" release-strategy-method="canRelease"
send-partial-result-on-expiry="true" message-store="resultMessageStore">
</int:aggregator>
<int:service-activator id="contributionIndexerService"
ref="contributionIndexerBean" method="bulkIndex" input-channel="aggregatorOutChannel" />
<bean id="resultMessageStore"
class="org.springframework.integration.store.SimpleMessageStore" />
<bean id="resultMessageStoreReaper"
class="org.springframework.integration.store.MessageGroupStoreReaper">
<property name="messageGroupStore" ref="resultMessageStore" />
<property name="timeout" value="60000" />
</bean>
<task:scheduled-tasks>
<task:scheduled ref="resultMessageStoreReaper" method="run"
fixed-rate="10000" />
</task:scheduled-tasks>
Any thoughts or comments?
Thanks in advance.
I'm not sure that you will be able to determine the count of messages in the Broker's queue (Redis/RabbitMQ or even just normal JMS), more over how much they have been there.
You definitely should consume them to do such a logic.
And yes, I think Aggregator can help you. But right: that must be a Persistent Message Store.
for the case
if they are in the channel since some time
The Aggregator suggest an option like group-timeout to release those groups which haven't reached the releaseStrategy condition, but you would like to emit them anyway over some time: http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to
I got this error.
org.springframework.messaging.MessageDeliveryException: failed to send message to channel 'org.springframework.integration.handler.MessageHandlerChain$ReplyForwardingMessageChannel#261b0001' within timeout: -1
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:112)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:44)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:94)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:260)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:241)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:205)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:199)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:177)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.handler.MessageHandlerChain.handleMessageInternal(MessageHandlerChain.java:131)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
And my spring integration xml is like this.
<int-ws:inbound-gateway id="asmTestWSGateway" request-channel="webserviceInboundChannel"
reply-channel="outboundStartChannel" marshaller="asmMarshaller"
unmarshaller="asmMarshaller" />
<bean id="asmMarshaller" class="org.springframework.oxm.castor.CastorMarshaller">
<property name="targetClasses" value="com.sds.redca.test.vo.DependencyVO"></property>
<property name="mappingLocation" value="classpath:mapping-asm.xml"></property>
</bean>
<int:chain input-channel="webserviceInboundChannel" output-channel="checkAuthentificationChannel">
<int:service-activator>
<bean class="com.sds.redca.core.endpoint.activator.BuildMsgActivator" />
</int:service-activator>
<int:filter ref="sampleFilter" />
<int:header-enricher>
<int:header name="protocolType" value="#{T(com.sds.redca.cmm.RedCAConstants).WS_PROTOCOL}"/>
</int:header-enricher>
</int:chain>
<int:chain input-channel="checkAuthentificationChannel" output-channel="outboundStartChannel">
<int:service-activator>
<bean class="com.sds.redca.core.endpoint.activator.AuthentificationServiceActivator" />
</int:service-activator>
</int:chain>
Webservice gateway received soap message with marshalled message translated to vo object.
and normalized and checked whether or not it is valid then go to back to the reply channel for returning the client.
I don't know how to solve this. Even if I make logging level debug, i can't find the reason.
Any help would be greatly appreciated!
Show, please, your outboundStartChannel config.
Tipically failed to send message to channel ... within timeout: -1 is happened, when we use QueueChannel and can't overcome its current capacity, becuase there is not enough resources to poll messages from that queue to process.
Actoually I don't see reason for your use-case to use reply-channel on <int-ws:inbound-gateway> at all. Just remove it and output-channel="outboundStartChannel" from the last <chain> as well. And rely on the TemporaryReplyChannel from the MessageHeaders in that case to achieve a short circuit for the entire flow.
I would like to process incoming messages differently basing on the message header. How to implement it efficiently in RabbitMQ?
My listener is listenining to messages coming from presence.queue channels. The message header KEY has different keys: key1, key2. Messages with key1 should be processed by messageService.method1, messages with key2 should be processed with messageService.method2.
Here is the code I am playing with but obviously it is wrong. I probably should route messages from presence.queue to first.queue and second.queue ?
<!-- RECEIVER -->
<rabbit:queue id="presence.queue" name="presence" durable="true"/>
<rabbit:queue id="first.queue" name="first"/>
<rabbit:queue id="second.queue" name="second"/>
<rabbit:direct-exchange name="presence.direct" durable="true" >
<rabbit:bindings>
<rabbit:binding queue="presence.queue" key="key1"/>
<rabbit:binding queue="presence.queue" key="key2"/>
</rabbit:bindings>
</rabbit:direct-exchange>
<!-- RabbitMQ Asynchronous Receiver from PRESENCE_ENGINE queue -->
<rabbit:listener-container connection-factory="connectionFactory" acknowledge="auto" >
<rabbit:listener queues="first.queue" ref="messageService" method="method1" />
<rabbit:listener queues="second.queue" ref="messageService" method="method2" />
</rabbit:listener-container>
Why do you need presence.queue? Just bind your consumer queues with the appropriate routing key. If you want to use message headers for routing, use a Headers Exchange.
See Getting Started and Headers Exchange.
Correction. Producers send messages to the exchange, not the queue. Come back, please, to the AMQP theory.
They may send Messages with the same routing key as well, but as Gary suggests, you can use Headers Exchange and build binding to your queues by some value from MessageProperties.
Especially you already have that value.
There is nohing to change on producer side. But from other side: you are the server (consumer) and everything in your hadns: you can configure AMQP routing as it would be comfortably for you.
That's why AMQP is better than JMS!
I have a situation I don't know how to manage it.
The flow I neeed is the following one:
The first service makes its job and creates a message that needs to be retrieved in the chain's end.
When the first service finishes, I need to invoke a push notification server via a new service with a particular message but with some info related to the one created in step 1.
Finally, I the push notification has been sent successfully, I have to retrieve the message created in step 1.
The question is, how can I keep message created in step 1 when the outbound-gateway calling was produced and retrieved me the message from notification push server?
<int:chain input-channel="v1.inputChannel.input" output-channel="v1.inputChannel.output" send-timeout="50000">
<int:header-enricher>
<int:error-channel ref="v1.inputChannel.error" />
</int:header-enricher>
<int:service-activator ref="v1.input.service" method="methodName"/>
<int:service-activator ref="v1.notificationPusher.service" method="pushNotification"/>
<int-http:outbound-gateway url="http://example.com/api/elements/:element_id/objects" http-method="POST">
<int-http:uri-variable name="element_id" expression="#pathVariables.elementId"/>
</int-http:outbound-gateway>
<!-- here the transformer needs to get the messsage from v1.input.service -->
<int:object-to-json-transformer/>
</int:chain>
After some researching in 'Spring Integration in Action' I think the best option for resolving this situation is to use the wire tap pattern, having the push notification service as secondary flow.
Here you can see the book's example where, If I have understood it in a correct way, the auditChannel acts as a secondary flow separated from the main one.
<channel id="debitChannel">
<interceptors>
<wire-tap channel="auditChannel"/>
</interceptors>
</channel>
<service-activator input-channel="debitChannel" method="process">
<beans:bean class="siia.monitoring.wiretap.DebitService"/>
</service-activator>
<filter input-channel="auditChannel" expression="payload.amount > 10000" output-channel="logger"/>
<logging-channel-adapter id="logger" expression="'auditing debit: ' + payload"/>
I think you could achieve this with something like that:
Dupĺicate your message before send it to the outbound
Send one to the outbound and the another one to some channel
Use an Aggregator with the same timeout of your outbound to "join" them, but you could, in fact, only pass the message from the first step through (I haven't tested it)
Send it to the json transformer
To use this approach, I believe, you have to put the outbound and the json-transformer outside the chain as well as your logic to duplicate the message.
I can think of one way, which is to squirrel away the payload into a header after step 1, then retrieve and repopulate the payload right before you invoke step 3:
<int:service-activator ref="v1.input.service" method="methodName"/>
<int:header-enricher>
<int:header name="originalpayload" expression="payload"/>
</int:header-enricher>
<int:service-activator ref="v1.notificationPusher.service" method="pushNotification"/>
<int:enricher expression="headers.originalpayload"/>
<int-http:outbound-gateway url="http://xxx.com/api/elements/:element_id/objects" http-method="POST">
<int-http:uri-variable name="element_id" expression="#pathVariables.elementId"/>
</int-http:outbound-gateway>
Problem:
I am using Spring Integration backed by RabbitMQ. I have a Fanout Exchange to which messages are published. I wish to launch multiple instances of the same service which creates a queue bound to this exchange. When a message is published to this exchange, I want all the launched services to receive the message. This does not happen as although there are multiple services, they all use the same queue
Solution:
Create a non-durable, exclusive, anonymous queue for each consumer.
Problem:
How do I make the queue anonymous?
Here is my code so far, the problem is the binding requires a queue attribute, and the queue requires a name - and therefore is named, and not anonymous.
<rabbit:fanout-exchange id="structureUpdateExchange" name="publish.registry.update" >
<rabbit:bindings>
<rabbit:binding queue="publish.registry.update.queue" />
</rabbit:bindings>
</rabbit:fanout-exchange>
<rabbit:queue durable="false" auto-delete="true" exclusive="true" name="publish.registry.update.queue" />
<si:channel id="publishInformationChannel" />
<int-amqp:inbound-gateway request-channel="publishInformationChannel"
queue-names="publish.registry.update.queue" />
Any help would be very much appreciated. Thanks.
So yes, as has said by Phillipp define the queue with id but no name.
<rabbit:queue id="queueTemp"/>
You need to use SpEL to reference the queue name. See the queue-names attribute below:
<int-amqp:inbound-channel-adapter
channel="fromRabbit"
queue-names="#{#queueTemp.name}"
connection-factory="connectionFactory" />
You might already be doing this but note that on the consumer side you also define the fanout exchange with the binding to your temporary queue. So something like:
<rabbit:fanout-exchange id="myFanoutExchange" name="test-fanex" >
<rabbit:bindings>
<rabbit:binding queue="queueTemp" />
</rabbit:bindings>
</rabbit:fanout-exchange>
You can directly define a channel that is backed by an AMQP queue:
<int-amqp:publish-subscribe-channel id="publishInformationChannel"
exchange="publish.registry.update" />
This will create an exchange publish.registry.update, an anonymous queue and an incoming queue adapter for the channel publishInformationChannel.