Right way to split, enrich items then send each item to another channel? - spring

Is this the right way to split a list of items, enrich each item and then send each of those enriched items to another channel?
It seems like even though each item is being enriched only the last one is sent to the output channel...
Here is the snipper from my test where I see from the flow for only page2 being invoked.
this.sitePackage = new Package();
this.sitePackage.add(page1);
this.sitePackage.add(page2);
this.sitePackage.add(page3);
//Publish using gateway
this.publishingService.publish(sitePackage);
If I do this however...
this.sitePackage.add(page1);
this.sitePackage.add(page1);
this.sitePackage.add(page2);
this.sitePackage.add(page2);
this.sitePackage.add(page3);
this.sitePackage.add(page3);
I see all the pages being published but the last one is page2 not page3 (even though from debugging I can see the instance has page 3 properties).
It seems like every other item is being seen by the flows...
My flows go like this...
Starting with the PublishPackage flow. This is the main entry flow and intended to split the items out of the package and send each of them, after enriching the payload, to flows who are attached to the publishPackageItem channel...
#Bean
IntegrationFlow flowPublishPackage()
{
return flow -> flow
.channel(this.publishPackageChannel())
.<Package>handle((p, h) -> this.savePackage(p))
.split(Package.class, this::splitPackage)
.channel(this.publishPackageItemChannel());
}
#Bean
#PublishPackageChannel
MessageChannel publishPackageChannel()
{
return MessageChannels.direct().get();
}
#Bean
#PublishPackageItemChannel
MessageChannel publishPackageItemChannel()
{
return MessageChannels.direct().get();
}
#Splitter
List<PackageEntry> splitPackage(final Package bundle)
{
final List<PackageEntry> enrichedEntries = new ArrayList<>();
for (final PackageEntry entry : bundle.getItems())
{
enrichedEntries.add(entry);
}
return enrichedEntries;
}
#Bean
GatewayProxyFactoryBean publishingGateway()
{
final GatewayProxyFactoryBean proxy = new GatewayProxyFactoryBean(PublishingService.class);
proxy.setBeanFactory(this.beanFactory);
proxy.setDefaultRequestChannel(this.publishPackageChannel());
proxy.setDefaultReplyChannel(this.publishPackageChannel());
proxy.afterPropertiesSet();
return proxy;
}
Next, the CMS publish flows are attached to the publishPackageItem channel and based on the type after splitting, routed to a specific element channel for handling. After splitting the page only specific element types may have a subscribing flow.
#Inject
public CmsPublishFlow(#PublishPackageItemChannel final MessageChannel channelPublishPackageItem)
{
this.channelPublishPackageItem = channelPublishPackageItem;
}
#Bean
#PublishPageChannel
MessageChannel channelPublishPage()
{
return MessageChannels.direct().get();
}
#Bean
IntegrationFlow flowPublishContent()
{
return flow -> flow
.channel(this.channelPublishPackageItem)
.filter(PackageEntry.class, p -> p.getEntry() instanceof Page)
.transform(PackageEntry.class, PackageEntry::getEntry)
.split(Page.class, this::traversePageElements)
.<Content, String>route(Content::getType, mapping -> mapping
.resolutionRequired(false)
.subFlowMapping(PAGE, sf -> sf.channel(channelPublishPage()))
.subFlowMapping(IMAGE, sf -> sf.channel(channelPublishAsset()))
.defaultOutputToParentFlow());
//.channel(IntegrationContextUtils.NULL_CHANNEL_BEAN_NAME);
}
Finally, my goal is to subscribe to the channel and handle each element accordingly. I subscribe this flow to the channelPublishPage. Each subscriber may handle the element differently.
#Inject
#PublishPageChannel
private MessageChannel channelPublishPage;
#Bean
IntegrationFlow flowPublishPage()
{
return flow -> flow
.channel(this.channelPublishPage)
.publishSubscribeChannel(c -> c
.subscribe(s -> s
.<Page>handle((p, h) -> this
.generatePage(p))));
}

I somehow feel that the problem is here:
proxy.setDefaultRequestChannel(this.publishPackageChannel());
proxy.setDefaultReplyChannel(this.publishPackageChannel());
Consider do not use the same channel for requests and for waiting replies. This way you bring some loop and really unexpected behavior.

Related

Implementing Polling in Middle of spring Integration flow DSL

I am writing a spring integration DSL flow.
Which will look like below diagram.
As you can see in flow I need to read 1 mil entities from database. I want o avoid reading those in single go.
I want to implement polling which will read N entities in fixed interval and send it for processing.
In the examples I read for polling, The polling is used as the first step of the Flow. in my case I want to implement in in middle of the flow.
Please let me know how do I implement this.
Any help is appreciated.
Thanks in Advance.
If you want to trigger the start of some polled flow using some external stimulus, use a control bus:
#SpringBootApplication
public class So63337649Application {
public static void main(String[] args) {
SpringApplication.run(So63337649Application.class, args);
}
#Bean
IntegrationFlow trigger(ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Amqp.inboundAdapter(connectionFactory, "foo"))
.transform(msg -> "#poller.start()")
.channel("control.input")
.get();
}
#Bean
IntegrationFlow control() {
return f -> f.controlBus();
}
#Bean
IntegrationFlow mainFlow() {
return IntegrationFlows.from(() -> "foo", e -> e
.id("poller")
.autoStartup(false)
.poller(Pollers.fixedDelay(5000)))
.handle(System.out::println)
.get();
}
}

how to send partial source data attributes to different targets

I am building an integration between a source and two targets, here source data object has 10 attributes , of which one target needs around 6 attributes and another target needs 4 attributes only, appreciate any help here on how i can achieve with spring
You can configure the source to send the Message to a PublishSubscribeChannel .
Then configure two Transformers to subscribe to this pub-sub channel. One of the transformer will transform the message the 6 attributes version while other to the 4 attributes version. Both transformers will then send the transformed messages to a separate channel .The two target system will look for the messages sent to these separated channels and process them.
In term of annotation configuration , it looks like the following: (Assuming the message the source sent out is Foo)
#Bean
public MessageChannel pubSubChannel() {
return new PublishSubscribeChannel();
}
#Bean
public MessageChannel outputChannelWith4Attributes() {
return new DirectChannel();
}
#Bean
public MessageChannel outputChannelWith6Attributes() {
return new DirectChannel();
}
#Component
public class MyTransformer {
#Transformer(inputChannel = "pubSubChannel", outputChannel = "outputChannelWith4Attributes")
public Foo transformTo4Attribute(Foo foo) {
//do the transformation logic here
return result;
}
#Transformer(inputChannel = "pubSubChannel", outputChannel = "outputChannelWith6Attributes")
public Foo transformTo6Attribute(Foo foo) {
//do the transformation logic here
return result;
}
}
And configure the source to send the message with payload Foo to pubSubChannel .Also configure the targets to process message from outputChannelWith4Attributes and outputChannelWith6Attributes.

Spring Integration HTTP to Scatter Gather

I'm new to Spring Integration and trying to make use of the enterprise pattern of scatter-gather, but I'm struggling with implementation details and struggling with available examples I can find online.
In short my scenario is:
An HTTP request is sent from a user to system A.
Before responding (aka synchronous), system A sends N number of messages to N number of system Xs, asynchronously.
System A sits and waits for responses.
Once there is a response from each of the request systems, system A collates the responses into one larger response.
System A finally responds to the user with the larger response.
Basically, as far as the original consumer is concerned, a single is request that responds with an answer, without having to 'come back later'. However, that request was actually to a facade that masks the complexity that lies behind it (potentially hitting hundreds of systems, making synchronous requests at back-end non-performant and infeasible).
So far I have this implementation (scrubbed details so may not be 1:1 example of what I'm playing with, for example the correlationStrategy I've since worked out isn't doing what I'd expect):
#Bean
public IntegrationFlow overallRequest(final AmqpTemplate amqpTemplate) {
return IntegrationFlows.from( // HTTP endpoint to user makes requests on
Http.inboundChannelAdapter("/request-overall-document")
.requestMapping(m -> m.methods(HttpMethod.POST))
.requestPayloadType(String.class))
.log()
// Arbitrary header to simplify example, realistically would generate a UUID
// and attach to some correlating header that works for systems involved
.enrichHeaders(p -> p.header("someHeader", "someValue"))
.log()
.scatterGather(
recipientListRouterSpec ->
recipientListRouterSpec
.applySequence(true)
.recipientFlow(
flow ->
flow.handle( // Straight pass through of msg received to see in response
Amqp.outboundAdapter(amqpTemplate)
.exchangeName( // RabbitMQ fanout exchange to N queues to N systems
"request-overall-document-exchange"))),
aggregatorSpec ->
aggregatorSpec
// Again for example, arbitrary once two correlated responses
.correlationStrategy(msg -> msg.getHeaders().get("someHeader"))
.releaseStrategy(gm -> gm.size() == 2)
// Simple string concatenation for overall response
.outputProcessor(
msgrp ->
msgrp.getMessages().stream()
.map(msg -> msg.getPayload().toString())
.reduce("Overall response: ", (nexus, txt) -> nexus + "|" + txt))
// Reset group on each response
.expireGroupsUponCompletion(true),
scatterGatherSpec ->
scatterGatherSpec.gatherChannel(
responseChannel())) // The channel to listen for responses to request on
.log()
.get();
}
With this as the response channel configuration:
#Bean
public MessageChannel responseChannel() {
return new QueueChannel();
}
#Bean
public AmqpInboundChannelAdapter responseChannelAdapter(
SimpleMessageListenerContainer listenerContainer,
#Qualifier("responseChannel") MessageChannel channel) {
AmqpInboundChannelAdapter adapter = new AmqpInboundChannelAdapter(listenerContainer);
adapter.setOutputChannel(channel);
return adapter;
}
#Bean
public SimpleMessageListenerContainer responseContainer(ConnectionFactory connectionFactory) {
SimpleMessageListenerContainer container =
new SimpleMessageListenerContainer(connectionFactory);
container.setQueueNames("request-overall-document-responses");
return container;
}
With all responses being sent to a separate Spring application that just pipes the request payloads back again (aka for testing without having to integrate with actual systems):
#Bean
public IntegrationFlow systemOneReception(final ConnectionFactory connectionFactory, final AmqpTemplate amqpTemplate) {
return IntegrationFlows.from(Amqp.inboundAdapter(connectionFactory, "request-overall-document-system-1"))
.log()
.handle(Amqp.outboundAdapter(amqpTemplate).routingKey("request-overall-document-responses"))
.get();
}
#Bean
public IntegrationFlow systemTwoReception(final ConnectionFactory connectionFactory, final AmqpTemplate amqpTemplate) {
return IntegrationFlows.from(Amqp.inboundAdapter(connectionFactory, "request-overall-document-system-2"))
.log()
.handle(Amqp.outboundAdapter(amqpTemplate).routingKey("request-overall-document-responses"))
.get();
}
And I get the following error in system A upon successful release as per the aggregation / release strategy in the scatter-gather implementation:
2020-02-29 20:06:39.255 ERROR 152 --- [ask-scheduler-1] o.s.integration.handler.LoggingHandler : org.springframework.messaging.MessageDeliveryException: The 'gatherResultChannel' header is required to deliver the gather result., failedMessage=GenericMessage [payload=Overall response: |somerequesttobesent|somerequesttobesent, headers={amqp_receivedDeliveryMode=PERSISTENT, content-length=19, amqp_deliveryTag=2, sequenceSize=1, amqp_redelivered=false, amqp_contentEncoding=UTF-8, host=localhost:18081, someHeader=someValue, connection=keep-alive, correlationId=182ee203-85ab-9ef6-7b19-3a8e2da8f5a7, id=994a0cf5-ad2b-02c3-dc93-74fae2f5092b, cache-control=no-cache, contentType=text/plain, timestamp=1583006799252, http_requestMethod=POST, sequenceNumber=1, amqp_consumerQueue=request-overall-document-responses, accept=*/*, amqp_receivedRoutingKey=request-overall-document-responses, amqp_timestamp=Sat Feb 29 20:06:39 GMT 2020, amqp_messageId=3341deae-7ed0-a042-0bb7-d2d2be871165, http_requestUrl=http://localhost:18081/request-overall-document, amqp_consumerTag=amq.ctag-ULxwuAjp8ZzcopBZYvcbZQ, accept-encoding=gzip, deflate, br, user-agent=PostmanRuntime/7.22.0}]
at org.springframework.integration.scattergather.ScatterGatherHandler.lambda$doInit$2(ScatterGatherHandler.java:160)
at org.springframework.integration.channel.FixedSubscriberChannel.send(FixedSubscriberChannel.java:77)
at org.springframework.integration.channel.FixedSubscriberChannel.send(FixedSubscriberChannel.java:71)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:431)
at org.springframework.integration.handler.AbstractMessageProducingHandler.doProduceOutput(AbstractMessageProducingHandler.java:284)
at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:265)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:223)
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.completeGroup(AbstractCorrelatingMessageHandler.java:823)
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.handleMessageInternal(AbstractCorrelatingMessageHandler.java:475)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:169)
at org.springframework.integration.endpoint.PollingConsumer.handleMessage(PollingConsumer.java:143)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:390)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:329)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$null$1(AbstractPollingEndpoint.java:277)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$2(AbstractPollingEndpoint.java:274)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Now I understand I have a few gaps, but I'm struggling to work out how to move forward:
The given error: that there isn't some 'gatherResultChannel' output. I'd have thought this would have been subsequent 'handles' / 'logs' / w.e on the result of the scatterGather(...) call, but not so on playing around.
There needs to be some form of mapping from the result of the scatter-gather aggregation back to the original Http.XXX request.
EDIT: from further digging, the issue given seems to be because when going out via AMQP (in my case, RabbitMQ), the header in question is deliberately dropped as it's a MessageChannel (see lines 230 to 257). Unsure if the implication here is that splitting/aggregation isn't intended to cross between multiple independent applications (my assumption is that it's dropped because it's an instance of a Java object, which would be problematic to pass around)...
FURTHER EDIT: with fresh eyes noticed something I hadn't before, the exception I pasted in quotes the failed message, and it seems to be a clear result of the output processing (while fiddling, flicked between DirectChannel and QueueChannel, only DirectChannel does not print the payload so wasn't looking for it). To be sure it wasn't doing some cloning or something weird, updated the stub service to transform and append unique postfixes (as below), and yes it was actually aggregating.
.transform(msg -> MessageFormat.format("{0}_system1response", msg))
.transform(msg -> MessageFormat.format("{0}_system2response", msg))
The 'gatherResultChannel' header is required to deliver the gather result., failedMessage=GenericMessage [payload=Overall response: |sometext_system2response|sometext_system1response, hea...
So it seems like scattering, gathering and aggregation is all working, the only thing that's not is that the given processing doesn't know where to push the messages after that?
ONCE MORE: As per Gary's response, replaced all adapters with gateways, however in doing so can no longer fanout? So removed scatterGatherSpec argument from the scatterGather call, and replaced / added in two recipient as follows:
.recipientFlow(flow -> flow.handle(Amqp.asyncOutboundGateway(asyncTemplate).routingKeyFunction(m -> "request-overall-document-system-1"), e -> e.id("sytemOneOutboundGateway")))
.recipientFlow(flow -> flow.handle(Amqp.asyncOutboundGateway(asyncTemplate).routingKeyFunction(m -> "request-overall-document-system-2"), e -> e.id("sytemTwoOutboundGateway")))
which is the closest I can get to a working example, however, while this does sort-of work, it results in reprocessing of message multiple times on/off queues, where my expected output for a POST with 'msgtosend' would have been:
Overall message: |msgtosend_system1response|msgtosend_system2response
Instead I get sporadic outputs like:
Overall message: |msgtosend|msgtosend_system1response
Overall message: |msgtosend_system2response|msgtosend_system1response_system1response
Overall message: |msgtosend|msgtosend_system1response_system1response
Overall message: |msgtosend_system2response|msgtosend_system1response_system1response
I assume there's some config / bean overlap but try as I might I can't isolate what it is, i.e. connection factory, listener container, async template, etc. etc.
Use an AMQP outbound gateway instead of outbound and inbound channel adapters; that way the channel header will be retained. There is an AsyncAmqpOutboundGateway which is probably best for your purposes.
If you must use channel adapters for some reason, use a header enricher together with a Header Channel Registry to convert the channel to a String representation so it can be retained.
EDIT
Here is a simple example:
#SpringBootApplication
public class So60469260Application {
public static void main(String[] args) {
SpringApplication.run(So60469260Application.class, args);
}
#Bean
public IntegrationFlow flow(AsyncRabbitTemplate aTemp) {
return IntegrationFlows.from(Gate.class)
.enrichHeaders(he -> he.headerExpression("corr", "payload"))
.scatterGather(rlr -> rlr
.applySequence(true)
.recipientFlow(f1 -> f1.handle(Amqp.asyncOutboundGateway(aTemp)
.routingKey("foo")))
.recipientFlow(f2 -> f2.handle(Amqp.asyncOutboundGateway(aTemp)
.routingKey("bar"))),
agg -> agg.correlationStrategy(msg -> msg.getHeaders().get("corr")))
.get();
}
#Bean
public AsyncRabbitTemplate aTemp(RabbitTemplate template) {
return new AsyncRabbitTemplate(template);
}
#Bean
#DependsOn("flow")
public ApplicationRunner runner(Gate gate) {
return args -> System.out.println(gate.doIt("foo"));
}
#RabbitListener(queues = "foo")
public String foo(String in) {
return in.toUpperCase();
}
#RabbitListener(queues = "bar")
public String bar(String in) {
return in + in;
}
}
interface Gate {
List<String> doIt(String in);
}
[foofoo, FOO]

How to dead letter a RabbitMQ messages when an exceptions happens in a service after an aggregator's forceRelease

I am trying to figure out the best way to handle errors that might have occurred in a service that is called after a aggregate's group timeout occurred that mimics the same flow as if the releaseExpression was met.
Here is my setup:
I have a AmqpInboundChannelAdapter that takes in messages and send them to my aggregator.
When the releaseExpression has been met and before the groupTimeout has expired, if an exception gets thrown in my ServiceActivator, the messages get sent to my dead letter queue for all the messages in that MessageGroup. (10 messages in my example below, which is only used for illustrative purposes) This is what I would expect.
If my releaseExpression hasn't been met but the groupTimeout has been met and the group times out, if an exception gets throw in my ServiceActivator, then the messages do not get sent to my dead letter queue and are acked.
After reading another blog post,
link1
it mentions that this happens because the processing happens in another thread by the MessageGroupStoreReaper and not the one that the SimpleMessageListenerContainer was on. Once processing moves away from the SimpleMessageListener's thread, the messages will be auto ack.
I added the configuration mentioned in the link above and see the error messages getting sent to my error handler. My main question, is what is considered the best way to handle this scenario to minimize message getting lost.
Here are the options I was exploring:
Use a BatchRabbitTemplate in my custom error handler to publish the failed messaged to the same dead letter queue that they would have gone to if the releaseExpression was met. (This is the approach I outlined below but I am worried about messages getting lost, if an error happens during publishing)
Investigate if there is away I could let the SimpleMessageListener know about the error that occurred and have it send the batch of messages that failed to a dead letter queue? I doubt this is possible since it seems the messages are already acked.
Don't set the SimpleMessageListenerContainer to AcknowledgeMode.AUTO and manually ack the messages when they get processed via the Service when the releaseExpression being met or the groupTimeOut happening. (This seems kinda of messy, since there can be 1..N message in the MessageGroup but wanted to see what others have done)
Ideally, I want to have a flow that will that will mimic the same flow when the releaseExpression has been met, so that the messages don't get lost.
Does anyone have recommendation on the best way to handle this scenario they have used in the past?
Thanks for any help and/or advice!
Here is my current configuration using Spring Integration DSL
#Bean
public SimpleMessageListenerContainer workListenerContainer() {
SimpleMessageListenerContainer container =
new SimpleMessageListenerContainer(rabbitConnectionFactory);
container.setQueues(worksQueue());
container.setConcurrentConsumers(4);
container.setDefaultRequeueRejected(false);
container.setTransactionManager(transactionManager);
container.setChannelTransacted(true);
container.setTxSize(10);
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
return container;
}
#Bean
public AmqpInboundChannelAdapter inboundRabbitMessages() {
AmqpInboundChannelAdapter adapter = new AmqpInboundChannelAdapter(workListenerContainer());
return adapter;
}
I have defined a error channel and defined my own taskScheduler to use for the MessageStoreRepear
#Bean
public ThreadPoolTaskScheduler taskScheduler(){
ThreadPoolTaskScheduler ts = new ThreadPoolTaskScheduler();
MessagePublishingErrorHandler mpe = new MessagePublishingErrorHandler();
mpe.setDefaultErrorChannel(myErrorChannel());
ts.setErrorHandler(mpe);
return ts;
}
#Bean
public PollableChannel myErrorChannel() {
return new QueueChannel();
}
public IntegrationFlow aggregationFlow() {
return IntegrationFlows.from(inboundRabbitMessages())
.transform(Transformers.fromJson(SomeObject.class))
.aggregate(a->{
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.releaseExpression("size() == 10");
a.transactional(true);
}
)
.handle("someService", "processMessages")
.get();
}
Here is my custom error flow
#Bean
public IntegrationFlow errorResponse() {
return IntegrationFlows.from("myErrorChannel")
.<MessagingException, Message<?>>transform(MessagingException::getFailedMessage,
e -> e.poller(p -> p.fixedDelay(100)))
.channel("myErrorChannelHandler")
.handle("myErrorHandler","handleFailedMessage")
.log()
.get();
}
Here is the custom error handler
#Component
public class MyErrorHandler {
#Autowired
BatchingRabbitTemplate batchingRabbitTemplate;
#ServiceActivator(inputChannel = "myErrorChannelHandler")
public void handleFailedMessage(Message<?> message) {
ArrayList<SomeObject> payload = (ArrayList<SomeObject>)message.getPayload();
payload.forEach(m->batchingRabbitTemplate.convertAndSend("some.dlq","#", m));
}
}
Here is the BatchingRabbitTemplate bean
#Bean
public BatchingRabbitTemplate batchingRabbitTemplate() {
ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
scheduler.setPoolSize(5);
scheduler.initialize();
BatchingStrategy batchingStrategy = new SimpleBatchingStrategy(10, Integer.MAX_VALUE, 30000);
BatchingRabbitTemplate batchingRabbitTemplate = new BatchingRabbitTemplate(batchingStrategy, scheduler);
batchingRabbitTemplate.setConnectionFactory(rabbitConnectionFactory);
return batchingRabbitTemplate;
}
Update 1) to show custom MessageGroupProcessor:
public class CustomAggregtingMessageGroupProcessor extends AbstractAggregatingMessageGroupProcessor {
#Override
protected final Object aggregatePayloads(MessageGroup group, Map<String, Object> headers) {
return group;
}
}
Example Service:
#Slf4j
public class SomeService {
#ServiceActivator
public void processMessages(MessageGroup messageGroup) throws IOException {
Collection<Message<?>> messages = messageGroup.getMessages();
//Do business logic
//ack messages in the group
for (Message<?> m : messages) {
com.rabbitmq.client.Channel channel = (com.rabbitmq.client.Channel)
m.getHeaders().get("amqp_channel");
long deliveryTag = (long) m.getHeaders().get("amqp_deliveryTag");
log.debug(" deliveryTag = {}",deliveryTag);
log.debug("Channel = {}",channel);
channel.basicAck(deliveryTag, false);
}
}
}
Updated integrationFlow
public IntegrationFlow aggregationFlowWithCustomMessageProcessor() {
return IntegrationFlows.from(inboundRabbitMessages()).transform(Transformers.fromJson(SomeObject.class))
.aggregate(a -> {
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.releaseExpression("size() == 10");
a.transactional(true);
a.outputProcessor(new CustomAggregtingMessageGroupProcessor());
}).handle("someService", "processMessages").get();
}
New ErrorHandler to do nack
public class MyErrorHandler {
#ServiceActivator(inputChannel = "myErrorChannelHandler")
public void handleFailedMessage(MessageGroup messageGroup) throws IOException {
if(messageGroup!=null) {
log.debug("Nack messages size = {}", messageGroup.getMessages().size());
Collection<Message<?>> messages = messageGroup.getMessages();
for (Message<?> m : messages) {
com.rabbitmq.client.Channel channel = (com.rabbitmq.client.Channel)
m.getHeaders().get("amqp_channel");
long deliveryTag = (long) m.getHeaders().get("amqp_deliveryTag");
log.debug("deliveryTag = {}",deliveryTag);
log.debug("channel = {}",channel);
channel.basicNack(deliveryTag, false, false);
}
}
}
}
Update 2 Added custom ReleaseStratgedy and change to aggegator
public class CustomMeasureGroupReleaseStratgedy implements ReleaseStrategy {
private static final int MAX_MESSAGE_COUNT = 10;
public boolean canRelease(MessageGroup messageGroup) {
return messageGroup.getMessages().size() >= MAX_MESSAGE_COUNT;
}
}
public IntegrationFlow aggregationFlowWithCustomMessageProcessorAndReleaseStratgedy() {
return IntegrationFlows.from(inboundRabbitMessages()).transform(Transformers.fromJson(SomeObject.class))
.aggregate(a -> {
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.transactional(true);
a.releaseStrategy(new CustomMeasureGroupReleaseStratgedy());
a.outputProcessor(new CustomAggregtingMessageGroupProcessor());
}).handle("someService", "processMessages").get();
}
There are some flaws in your understanding.If you use AUTO, only the last message will be dead-lettered when an exception occurs. Messages successfully deposited in the group, before the release, will be ack'd immediately.
The only way to achieve what you want is to use MANUAL acks.
There is no way to "tell the listener container to send messages to the DLQ". The container never sends messages to the DLQ, it rejects a message and the broker sends it to the DLX/DLQ.

Correlate messages between 2 JMS queues using Spring integration components

I have 2 JMS queues and my application subscribes to both of them with Jms.messageDrivenChannelAdapter(...) component.
First queue receives messages of type Paid. Second queue receives messages of type Reversal.
Business scenario defines correlation between messages of type Paid and type Reversal.
Reversal should wait for Paid in order to be processed.
How can I achieve such "wait" pattern with Spring Integration?
Is it possible to correlate messages between 2 JMS queues?
See the documentation about the Aggregator.
The aggregator correlates messages using some correlation strategy and releases the group based on some release strategy.
The Aggregator combines a group of related messages, by correlating and storing them, until the group is deemed to be complete. At that point, the aggregator creates a single message by processing the whole group and sends the aggregated message as output.
The output payload is a list of the grouped message payloads by default, but you can provide a custom output processor.
EDIT
#SpringBootApplication
public class So55299268Application {
public static void main(String[] args) {
SpringApplication.run(So55299268Application.class, args);
}
#Bean
public IntegrationFlow in1(ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Jms.messageDrivenChannelAdapter(connectionFactory)
.destination("queue1"))
.channel("aggregator.input")
.get();
}
#Bean
public IntegrationFlow in2(ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Jms.messageDrivenChannelAdapter(connectionFactory)
.destination("queue2"))
.channel("aggregator.input")
.get();
}
#Bean
public IntegrationFlow aggregator() {
return f -> f
.aggregate(a -> a
.correlationExpression("headers.jms_correlationId")
.releaseExpression("size() == 2")
.expireGroupsUponCompletion(true)
.expireGroupsUponTimeout(true)
.groupTimeout(5_000L)
.discardChannel("discards.input"))
.handle(System.out::println);
}
#Bean
public IntegrationFlow discards() {
return f -> f.handle((p, h) -> {
System.out.println("Aggregation timed out for " + p);
return null;
});
}
#Bean
public ApplicationRunner runner(JmsTemplate template) {
return args -> {
send(template, "one", "two");
send(template, "three", null);
};
}
private void send(JmsTemplate template, String one, String two) {
template.convertAndSend("queue1", one, m -> {
m.setJMSCorrelationID(one);
return m;
});
if (two != null) {
template.convertAndSend("queue2", two, m -> {
m.setJMSCorrelationID(one);
return m;
});
}
}
}
and
GenericMessage [payload=[two, one], headers={jms_redelivered=false, jms_destination=queue://queue1, jms_correlationId=one, id=784535fe-8861-1b22-2cfa-cc2e67763674, priority=4, jms_timestamp=1553290921442, jms_messageId=ID:Gollum2.local-55540-1553290921241-4:1:3:1:1, timestamp=1553290921457}]
2019-03-22 17:42:06.460 INFO 55396 --- [ask-scheduler-1] o.s.i.a.AggregatingMessageHandler : Expiring MessageGroup with correlationKey[three]
Aggregation timed out for three

Resources