I am using the splitter to split the messages and to pass it to the respective other channels for further processing.
But I wanted to send the splitter output to one channel which will write it into another file. Also wanted to send the splitter output to another channel which will perform some task.
I am able to do the same by using the below, but it doesn't seems to be working if failed to process any of the split record in channel2. Its stopping the process and not writing the remaining records in channel1.
<int:splitter input-channel="inputchannel" ref="splitter" method="doSplit" output-channel="routingChannel"/>
<int-recipient-list-router id="customRouter" input-channel="routingChannel"
<int:recipient channel="channel1"/> <!--Write to file-->
<int:recipient channel="channel2"/> <!-- logic to process -->
</int:reciepient-list-router>
Is there any other way I can pass it to the separate channels independently.
The recipient-list-router has an option like:
/**
* Specify whether send failures for one or more of the recipients should be ignored. By default this is
* <code>false</code> meaning that an Exception will be thrown whenever a send fails. To override this and suppress
* Exceptions, set the value to <code>true</code>.
* #param ignoreSendFailures true to ignore send failures.
*/
public void setIgnoreSendFailures(boolean ignoreSendFailures) {
Or if you wish for XML configuration:
<xsd:attribute name="ignore-send-failures">
<xsd:annotation>
<xsd:documentation><![CDATA[
If set to "true", failures to send to a message channel will
be ignored. If set to "false", a MessageDeliveryException will be
thrown instead, and if the router resolves more than one channel,
any subsequent channels will not receive the message.
Please be aware that when using direct channels (single threaded),
send-failures can be caused by exceptions thrown by components
much further down-stream.
This attribute defaults to false.
]]></xsd:documentation>
</xsd:annotation>
<xsd:simpleType>
<xsd:union memberTypes="xsd:boolean xsd:string" />
</xsd:simpleType>
</xsd:attribute>
Related
I've a spring-integration implementation with following:
Multiple publishing channels publishing on one common channel.
All the channels returns same response object.
Aggregator trying to aggregate response from all the above channels
Issue: Aggregator not able to combine all the responses together and provided method gets invoked on the first reponse from the channels
Here are the details. What is that I've to do to aggregate the responses?
<int:publish-subscribe-channel id="aggregate-channel" apply-sequence="true"/>
<int:publish-subscribe-channel id="input-channel" apply-sequence="true"/>
<int:service-activator input-channel="input-channel" output-channel="aggregate-channel" ref="...A" method="...A">
<int:service-activator input-channel="input-channel" output-channel="aggregate-channel" ref="...B" method="...B">
<int:service-activator input-channel="input-channel" output-channel="aggregate-channel" ref="...C" method="...C">
<int:service-activator input-channel="input-channel" output-channel="aggregate-channel" ref="...D" method="...D">
<!--This is the aggregator.
**Expecting a list of size 4 but then it gets list of size 1 for each response channel
-->
<int:aggregator input-channel="aggregate-channel" output-channel="gateway-response-channel" ref="Service" method="responseListProcessor"/>
Solution
Replace
<int:publish-subscribe-channel id="aggregate-channel" apply-sequence="true"/>
with
<int:publish-subscribe-channel id="aggregate-channel"/>
Issue
Initially it starts with sequence size 4 because input-channel has 4 subscribers
But when you add the attribute apply-sequence="true" to aggregate-channel, it reset the sequence size to 1 because aggregate-channel has only one subscriber which is the aggregator.
Reference
https://docs.spring.io/spring-integration/docs/5.1.7.RELEASE/reference/html/#channel-configuration-pubsubchannel
If you provide a aggregator downstream from a PublishSubscribeChannel, you can set the 'apply-sequence' property on the channel to true.
Doing so indicates that the channel should set the sequence-size and sequence-number message headers as well as the correlation ID prior to passing along the messages.
For example, if there are five subscribers, the sequence-size would be set to 5, and the messages would have sequence-number header values ranging from 1 to 5.
I would say your requirement is fully covered by specific for this kinda of tasks EIP - Scatter-Gather: https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#scatter-gather
I'm following the sample below to implement spring-integration gateway.
Issue: Not able to get response on the Gateway which is sent from a service-activator.
Gateway reply-channel is "bestQuoteResponseChannel".
Service-activator writes to the same channel.
All the custom methods are executed and response is composed but never gets sent out. Any idea what's missing here?
My configuration also has a scatter-gather in the flow.
<!--Gateway:-->
<channel id="bestQuoteResponseChannel"/>
<gateway id="loanBrokerGateway"
default-request-channel="loanRequestsChannel"
service-interface="org.springframework.integration.samples.loanbroker.LoanBrokerGateway">
<method name="getBestLoanQuote" reply-channel="bestQuoteResponseChannel">
<header name="RESPONSE_TYPE" value="BEST"/>
</method>
</gateway>
<chain input-channel="loanRequestsChannel">
<header-enricher>
<header name="creditScore" expression="#creditBureau.getCreditReport(payload).score"/>
</header-enricher>
<recipient-list-router apply-sequence="true">
<recipient selector-expression="headers.creditScore > 800" channel="exclusiveBankChannel"/>
<recipient selector-expression="headers.creditScore > 750" channel="premiereBankChannel"/>
<recipient selector-expression="headers.creditScore > 700" channel="qualityBankChannel"/>
<recipient selector-expression="headers.creditScore > 650" channel="friendlyBankChannel"/>
<recipient channel="easyBankChannel"/>
</recipient-list-router>
</chain>
<!-- Messages are sent to the banks via bank channels and will be received and processed by an aggregator -->
<aggregator input-channel="loanQuotesChannel" output-channel="input" method="aggregateQuotes">
<beans:bean class="org.springframework.integration.samples.loanbroker.LoanQuoteAggregator"/>
</aggregator>
<service-activator ref="findBestQuoteService" method="findBestQuote" input-channel="input" output-channel="bestQuoteResponseChannel"/>
First of all for such a scenario you don't need a reply-channel on a gateway and therefore you don't need that output-channel on the service activator. All the reply logic should land on the replyChannel header which is always populated by the framework when gateway method is called.
It would be great to see what your findBestQuoteService.findBestQuote does since the LoanQuoteAggregator does everything for us already:
/**
* Aggregates LoanQuote Messages to return a single reply Message.
*
* #param quotes list of loan quotes received from upstream lenders
* #param responseType header that indicates the response type
* #return the best {#link LoanQuote} if the 'RESPONSE_TYPE' header value is 'BEST' else all quotes
*/
public Object aggregateQuotes(List<LoanQuote> quotes,
#Header(value="RESPONSE_TYPE", required=false) String responseType) {
Collections.sort(quotes);
return ("BEST".equals(responseType)) ? quotes.get(0) : quotes;
}
So, what is the point of your custom service thereafter?
Also consider to turn on DEBUG level for the org.springframework.integration category to see how your messages are traveling. Possibly we will see some issue in logs, too.
I don't see how your logic could lose a replyChannel header, but we need more info to investigate.
I had scatter-gather in the spring-integration flow before it sends reply to the Gateway. That was somehow was preventing it to send the response to Gateway.
Changes made:
I replaced scatter-gather component with the publish-subscribe channel and aggregator combination.
It worked fine after this change.
https://docs.spring.io/spring-integration/docs/5.3.0.M1/reference/html/scatter-gather.html#scatter-gather-namespace
I am getting the string messages in a Queue. I am consuming it and doing further processing using the below channel processMessage.
Now I have a requirement where I need to send this messages to another queue based on the selector expression mentioned below.
At the same time I need to make sure that the I am not missing any records in processMessage channel. Assuming that the below configuration is good to go?
or is there any better way to handle this situtaion:
<int:recipient-list-router id="router" input-channel="incomingMsg">
<int:recipient channel="publishToQueue" selector-expression="payload.contains('test trade') "/>
<int:recipient channel="processMessage"/>
</int:recipient-list-router>
<task:executor id="executor" pool-size="10" />
<int:publish-subscribe-channel id="publishToQueue" task-executor="executor"/>
The logical explanation is :
IF payload.contains(test trade)
THEN
PublishToQueue
END IF
ProcessMessage
Your configuration is correct and it is really a classical sample for the Recipient List Router. You send message to the processMessage unconditionally and the same message is sent to the publishToQueue if it fits to the selector expression.
Another way to reach the same is possible via Publish-Subscribe Channel and a Filter in the beginning of the second flow. But I would say your way with the Recipient List Router is really good.
Otherwise it isn't clear what is your question and why you are in doubt to go ahead.
I am using redis as a queue (using the spring queue-in/outbound-channel-adapter) to distribute tasks (a message into the queue, etc)
As the throughput is quite high we observed that, although the messages were sent to the redis queue, a lot of them were lost and no messages arrived to the component after the inbound (a header router)
The channel config is attached below; the point is that we though that the problem was in this header router after the inbound addapter, that was unable to manage the rate of messages read from the queue, so they were lost.
We have use an intermediate element between the inbound adapter and this component (that is a header-router) and add a queue to fix this.
This works fine, but actually we don't fully understand the solution and if this is the proper one.
An expert view and opinion about this configuration will be wellcome!
Thanks
<!-- a Queue Inbound Channel Adapter is available to 'right pop' messages
from a Redis List. -->
<redis:queue-inbound-channel-adapter
id="fromRedis" channel="in" queue="${name}"
receive-timeout="1000" recovery-interval="3000" expect-message="true"
auto-startup="true"/>
<!-- a queue to avoid lost messages before the header router -->
<int:channel id="in">
<int:queue capacity="1000"/>
</int:channel>
<!-- a bridge to connect channels and have a poller -->
<int:bridge input-channel="in" output-channel="out">
<int:poller fixed-delay="500" />
</int:bridge>
<int:header-value-router id="router" timeout="15000"
input-channel="out" header-name="decision"
resolution-required="false" default-output-channel="defaultChannel" />
---added on 26/02
To insert messages into redis we have a web service, but actually is as you said, simply write messages into redis (
for... channel.send(msg)
Nothing more
About your answer I am now thinking in remove the in channel and its queue and use directly the header-value-router; but I have more questions:
I think the right solution is a low value for timeout in header-value-router, so I'll have the error notification faster if we don't have a consumer available. If I don't use a value as timeout, it will block indefinitely and this is a bad idea, isn't it?
I don't know how to manage the MesssageDeliveryException because the router don't have an error-channel configuration, ???
I think that if I can manage this error and get the message back I can re-send it to redis again. There are other servers that get the messages from redis and they luckily could attend it.
I add my proposed solution, but is not complete and we are not sure about the error management as I explained above
<!-- a Queue Inbound Channel Adapter is available to 'right pop' messages
from a Redis List. -->
<redis:queue-inbound-channel-adapter
id="fromRedis" channel="in" queue="${name}"
receive-timeout="1000" recovery-interval="3000" expect-message="true"
auto-startup="true"/>
<!-- a header-value-router with quite low timeout -->
<int:header-value-router id="router" timeout="150"
input-channel="in" header-name="decision"
resolution-required="false" default-output-channel="defaultChannel" />
<!-- ¿if MessageDeliveryException???? what to do??? -->
<int:channel id="someConsumerHeaderValue">
<int:dispatcher task-executor="ConsumerExecutor" />
</int:channel>
<!-- If 5 threads are busy we queue messages up to 5; if queue is full we can increase to 5 more working threads; if no more threads we have a... ¿¿MessageDeliveryException?? -->
<task:executor id="ConsumerExecutor" pool-size="5-5"
queue-capacity="5" />
Well, that's great to see such an observation. That might improve the Framework somehow.
So, I'd like to see:
Some test-case to reproduce from the Framework perspective.
Although I guess there is just enough to send a lot of messages to the Redis and use your config to consume. (Correct me if there is need anything else)
The downstream flow after the <int:header-value-router>. Look, you use there timeout="15000" which is synonym to the send-timeout :
Specify the maximum amount of time in milliseconds to wait
when sending Messages to the target MessageChannels if blocking
is possible (e.g. a bounded queue channel that is currently full).
By default the send will block indefinitely.
Synonym for 'timeout' - only one can be supplied.
From here I can say that if your downstream consumer if enough slow on some QueueChannel there you end up with the:
/**
* Inserts the specified element at the tail of this queue, waiting if
* necessary up to the specified wait time for space to become available.
*
* #return {#code true} if successful, or {#code false} if
* the specified waiting time elapses before space is available
* #throws InterruptedException {#inheritDoc}
* #throws NullPointerException {#inheritDoc}
*/
public boolean offer(E e, long timeout, TimeUnit unit)
....
while (count.get() == capacity) {
if (nanos <= 0)
return false;
nanos = notFull.awaitNanos(nanos);
}
Pay attention to that return false; indicating exactly the message lost.
That is also know like back-pressure drop strategy.
Let me know if you have different picture there.
You may consider to remove that timeout="15000" to meet the same in queue channel behavior.
UPDATE
Well, the error handling works a bit different way. The "guilty" component just throws Exception, like it is with raw Java and it is OK that this component isn't responsible for Exception catching that is up to the caller.
And caller in our case an upstream component - <redis:queue-inbound-channel-adapter>.
Any inbound channel adapter has an error-channel option. Through the <poller> if it is MessageSource or directly when it is MessageProducer.
I'm sure you will be able to handle:
if (!sent) {
throw new MessageDeliveryException(message,
"failed to send message to channel '" + channel + "' within timeout: " + timeout);
}
in that error-channel sub-flow and achieve your requirements for recovery.
I am using a publish-subscribe channel after the inbound gateway receives a String message to parallely send it to the logger to log the message and to a transformer to transform the message. I want both these acivities to happen in parallel.
My question is very simple - Does the publish subscribe channel in spring integration sends messages to it's subscibers parallely?
Below is the code snippet from the source of spring-integration-context.xml.
<int:gateway id="gateway" service-interface="com.test.Gateway">
</int:gateway>
<int:publish-subscribe-channel id="publishsubscribechannel" />
<int:service-activator input-channel="publishsubscribechannel"
method="transformEvent" ref="transformer" output-channel="transformerreplychannel">
</int:service-activator>
<int:service-activator input-channel="publishsubscribechannel"
method="logMessage" ref="logger">
</int:service-activator>
Here the transformer and the logger are 2 subscribers to the publishsubscribechannel. In this setup will the message flow to logger and transformer from the gateway happen asynchronously by default??...OR I need to do some other configuration to achieve the same.
... or using JavaConfig
#Bean
public MessageChannel publishsubscribechannel() {
return new PublishSubscribeChannel(executor());
}
#Bean
public ThreadPoolTaskExecutor executor() {
ThreadPoolTaskExecutor pool = new ThreadPoolTaskExecutor();
pool.setCorePoolSize(10);
pool.setMaxPoolSize(10);
pool.setWaitForTasksToCompleteOnShutdown(true);
return pool;
}
By default, it runs in sequence. So in your case, it will be transformer, then logger. If you want to run it in parallel, you need to specify task-executor
<int:publish-subscribe-channel id="publishsubscribechannel" task-executor="executor" />
...
<task:executor id="executor" pool-size="10" />
And by using task-executor, the message handling is performed asynchronously.
As is from the Spring Integration documentation
PublishSubscribeChannel
The PublishSubscribeChannel implementation broadcasts any Message sent to it to all of its subscribed handlers. This is most often used for sending Event Messages whose primary role is notification as opposed to Document Messages which are generally intended to be processed by a single handler. Note that the PublishSubscribeChannel is intended for sending only. Since it broadcasts to its subscribers directly when its send(Message) method is invoked, consumers cannot poll for Messages (it does not implement PollableChannel and therefore has no receive() method). Instead, any subscriber must be a MessageHandler itself, and the subscriber's handleMessage(Message) method will be invoked in turn.
Prior to version 3.0, invoking the send method on a PublishSubscribeChannel that had no subscribers returned false. When used in conjunction with a MessagingTemplate, a MessageDeliveryException was thrown. Starting with version 3.0, the behavior has changed such that a send is always considered successful if at least the minimum subscribers are present (and successfully handle the message). This behavior can be modified by setting the minSubscribers property, which defaults to 0.
[Note] Note
If a TaskExecutor is used, only the presence of the correct number of subscribers is used for this determination, because the actual handling of the message is performed asynchronously.
Please note the Note.
It does mention if the TaskExecutor is used, the message handling would be asynchronous.
So, yes you have to add a TaskExecutor for this to be asynchronous.