I'm following the sample below to implement spring-integration gateway.
Issue: Not able to get response on the Gateway which is sent from a service-activator.
Gateway reply-channel is "bestQuoteResponseChannel".
Service-activator writes to the same channel.
All the custom methods are executed and response is composed but never gets sent out. Any idea what's missing here?
My configuration also has a scatter-gather in the flow.
<!--Gateway:-->
<channel id="bestQuoteResponseChannel"/>
<gateway id="loanBrokerGateway"
default-request-channel="loanRequestsChannel"
service-interface="org.springframework.integration.samples.loanbroker.LoanBrokerGateway">
<method name="getBestLoanQuote" reply-channel="bestQuoteResponseChannel">
<header name="RESPONSE_TYPE" value="BEST"/>
</method>
</gateway>
<chain input-channel="loanRequestsChannel">
<header-enricher>
<header name="creditScore" expression="#creditBureau.getCreditReport(payload).score"/>
</header-enricher>
<recipient-list-router apply-sequence="true">
<recipient selector-expression="headers.creditScore > 800" channel="exclusiveBankChannel"/>
<recipient selector-expression="headers.creditScore > 750" channel="premiereBankChannel"/>
<recipient selector-expression="headers.creditScore > 700" channel="qualityBankChannel"/>
<recipient selector-expression="headers.creditScore > 650" channel="friendlyBankChannel"/>
<recipient channel="easyBankChannel"/>
</recipient-list-router>
</chain>
<!-- Messages are sent to the banks via bank channels and will be received and processed by an aggregator -->
<aggregator input-channel="loanQuotesChannel" output-channel="input" method="aggregateQuotes">
<beans:bean class="org.springframework.integration.samples.loanbroker.LoanQuoteAggregator"/>
</aggregator>
<service-activator ref="findBestQuoteService" method="findBestQuote" input-channel="input" output-channel="bestQuoteResponseChannel"/>
First of all for such a scenario you don't need a reply-channel on a gateway and therefore you don't need that output-channel on the service activator. All the reply logic should land on the replyChannel header which is always populated by the framework when gateway method is called.
It would be great to see what your findBestQuoteService.findBestQuote does since the LoanQuoteAggregator does everything for us already:
/**
* Aggregates LoanQuote Messages to return a single reply Message.
*
* #param quotes list of loan quotes received from upstream lenders
* #param responseType header that indicates the response type
* #return the best {#link LoanQuote} if the 'RESPONSE_TYPE' header value is 'BEST' else all quotes
*/
public Object aggregateQuotes(List<LoanQuote> quotes,
#Header(value="RESPONSE_TYPE", required=false) String responseType) {
Collections.sort(quotes);
return ("BEST".equals(responseType)) ? quotes.get(0) : quotes;
}
So, what is the point of your custom service thereafter?
Also consider to turn on DEBUG level for the org.springframework.integration category to see how your messages are traveling. Possibly we will see some issue in logs, too.
I don't see how your logic could lose a replyChannel header, but we need more info to investigate.
I had scatter-gather in the spring-integration flow before it sends reply to the Gateway. That was somehow was preventing it to send the response to Gateway.
Changes made:
I replaced scatter-gather component with the publish-subscribe channel and aggregator combination.
It worked fine after this change.
https://docs.spring.io/spring-integration/docs/5.3.0.M1/reference/html/scatter-gather.html#scatter-gather-namespace
Related
I currently have a Spring Integration application which is utilizing a number of TCP inbound and outbound adapter combinations for message handling. All of these adapter combinations utilize the same single MessageEndpoint for request processing and the same single MessagingGateway for response sending.
The MessageEndpoint’s final output channel is a DirectChannel that is also the DefaultRequestChannel of the MessageGateway. This DirectChannel utilizes the default RoundRobinLoadBalancingStrategy which is doing a Round Robin search for the correct Outbound Adapter to send the given response through. Of course, this round robin search does not always find the appropriate Outbound Adapter on first search and when it doesn’t it logs accordingly. Not only is this producing a large amount of unwanted logging but it also raises some performance concerns as I anticipate several hundred inbound/outbound adapter combinations existing at any given time.
I am wondering if there is a way in which I can more closely correlate the inbound and outbound adapters in a way that there is no need for the round robin processing and each response can be sent directly to the corresponding outbound adapter? Ideally, I would like this to be implemented in a way that the use of a single MessageEndpoint and single MessageGateway can be maintained.
Note: Please limit solutions to those which use the Inbound/Outbound Adapter combinations. The use of TcpInbound/TcpOutboundGateways is not possible for my implementation as I need to send multiple responses to a single request and, to my knowledge, this can only be done with the use of inbound/outbound adapters.
To add some clarity, below is a condensed version of the current implementation described. I have tried to clear out any unrelated code just to make things easier to read...
// Inbound/Outbound Adapter creation (part of a service that is used to dynamically create varying number of inbound/outbound adapter combinations)
public void configureAdapterCombination(int port) {
TcpNioServerConnectionFactory connectionFactory = new TcpNioServerConnectionFactory(port);
// Connection Factory registered with Application Context bean factory (removed for readability)...
TcpReceivingChannelAdapter inboundAdapter = new TcpReceivingChannelAdapter();
inboundAdapter.setConnectionFactory(connectionFactory);
inboundAdapter.setOutputChannel(context.getBean("sendFirstResponse", DirectChannel.class));
// Inbound Adapter registered with Application Context bean factory (removed for readability)...
TcpSendingMessageHandler outboundAdapter = new TcpSendingMessageHandler();
outboundAdapter.setConnectionFactory(connectionFactory);
// Outbound Adapter registered with Application Context bean factory (removed for readability)...
context.getBean("outboundResponse", DirectChannel.class).subscribe(outboundAdapter);
}
// Message Endpoint for processing requests
#MessageEndpoint
public class RequestProcessor {
#Autowired
private OutboundResponseGateway outboundResponseGateway;
// Direct Channel which is using Round Robin lookup
#Bean
public DirectChannel outboundResponse() {
return new DirectChannel();
}
// Removed additional, unrelated, endpoints for readability...
#ServiceActivator(inputChannel="sendFirstResponse", outputChannel="sendSecondResponse")
public Message<String> sendFirstResponse(Message<String> message) {
// Unrelated message processing/response generation excluded...
outboundResponseGateway.sendOutboundResponse("First Response", message.getHeaders().get(IpHeaders.CONNECTION_ID, String.class));
return message;
}
// Service Activator that puts second response on the request channel of the Message Gateway
#ServiceActivator(inputChannel = "sendSecondResponse", outputChannel="outboundResponse")
public Message<String> processQuery(Message<String> message) {
// Unrelated message processing/response generation excluded...
return MessageBuilder.withPayload("Second Response").copyHeaders(message.getHeaders()).build();
}
}
// Messaging Gateway for sending responses
#MessagingGateway(defaultRequestChannel="outboundResponse")
public interface OutboundResponseGateway {
public void sendOutboundResponse(#Payload String payload, #Header(IpHeaders.CONNECTION_ID) String connectionId);
}
SOLUTION:
#Artem's suggestions in the comments/answers below seem to do the trick. Just wanted to make a quick note about how I was able to add a replyChannel to each Outbound Adapter on creation.
What I did was create two maps that are being maintained by the application. The first map is populated whenever a new Inbound/Outbound adapter combination is created and it is a mapping of ConnectionFactory name to replyChannel name. The second map is a map of ConnectionId to replyChannel name and this is populated on any new TcpConnectionOpenEvent via an EventListener.
Note that every TcpConnectionOpenEvent will have a ConnectionFactoryName and ConnectionId property defined based on where/how the connection is established.
From there, whenever a new request is received I use theses maps and the 'ip_connectionId' header on the Message to add a replyChannel header to the Message. The first response is sent by manually grabbing the corresponding replyChannel (based on the value of the replyChannel header) from the application's context and sending the response on that channel. The second response is sent via Spring Integration using the replyChannel header on the message as Artem describes in his responses.
This solution was implemented as a quick proof of concept and is just something that worked for my current implementation. Including this to hopefully jumpstart other viewer's own implementations/solutions.
Well, I see now your point about round-robin. You create many similar TCP channel adapters against the same channels. In this case it is indeed hard to distinguish one flow from another because you have a little control over those channels and their subscribers.
On of the solution would be grate with Spring Integration Java DSL and its dynamic flows: https://docs.spring.io/spring-integration/reference/html/dsl.html#java-dsl-runtime-flows
So, you would concentrate only on the flows and won't worry about runtime registration. But since you are not there and you deal just with plain Java & Annotations configuration, it is much harder for you to achieve a goal. But still...
You may be know that there is something like replyChannel header. It is taken into an account when we don't have a outputChannel configured. This way you would be able to have an isolated channel for each flow and the configuration would be really the same for all the flows.
So,
I would create a new channel for each configureAdapterCombination() call.
Propagate this one into that method for replyChannel.subscribe(outboundAdapter);
Use this channel in the beginning of your particular flow to populate it into a replyChannel header.
This way your processQuery() service-activator should go without an outputChannel. It is going to be selected from the replyChannel header for a proper outbound channel adapter correlation.
You don't need a #MessagingGateway for such a scenario since we don't have a fixed defaultRequestChannel any more. In the sendFirstResponse() service method you just take a replyChannel header and send a newly created message manually. Technically it is exactly the same what you try to do with a mentioned #MessagingGateway.
For Java DSL variant I would go with a filter on the PublishSubscribeChannel to discard those messages which don't belong to the current flow. Anyway it is a different story.
Try to figure out how you can have a reply channel per flow when you configure particular configureAdapterCombination().
I am getting the string messages in a Queue. I am consuming it and doing further processing using the below channel processMessage.
Now I have a requirement where I need to send this messages to another queue based on the selector expression mentioned below.
At the same time I need to make sure that the I am not missing any records in processMessage channel. Assuming that the below configuration is good to go?
or is there any better way to handle this situtaion:
<int:recipient-list-router id="router" input-channel="incomingMsg">
<int:recipient channel="publishToQueue" selector-expression="payload.contains('test trade') "/>
<int:recipient channel="processMessage"/>
</int:recipient-list-router>
<task:executor id="executor" pool-size="10" />
<int:publish-subscribe-channel id="publishToQueue" task-executor="executor"/>
The logical explanation is :
IF payload.contains(test trade)
THEN
PublishToQueue
END IF
ProcessMessage
Your configuration is correct and it is really a classical sample for the Recipient List Router. You send message to the processMessage unconditionally and the same message is sent to the publishToQueue if it fits to the selector expression.
Another way to reach the same is possible via Publish-Subscribe Channel and a Filter in the beginning of the second flow. But I would say your way with the Recipient List Router is really good.
Otherwise it isn't clear what is your question and why you are in doubt to go ahead.
I am using the splitter to split the messages and to pass it to the respective other channels for further processing.
But I wanted to send the splitter output to one channel which will write it into another file. Also wanted to send the splitter output to another channel which will perform some task.
I am able to do the same by using the below, but it doesn't seems to be working if failed to process any of the split record in channel2. Its stopping the process and not writing the remaining records in channel1.
<int:splitter input-channel="inputchannel" ref="splitter" method="doSplit" output-channel="routingChannel"/>
<int-recipient-list-router id="customRouter" input-channel="routingChannel"
<int:recipient channel="channel1"/> <!--Write to file-->
<int:recipient channel="channel2"/> <!-- logic to process -->
</int:reciepient-list-router>
Is there any other way I can pass it to the separate channels independently.
The recipient-list-router has an option like:
/**
* Specify whether send failures for one or more of the recipients should be ignored. By default this is
* <code>false</code> meaning that an Exception will be thrown whenever a send fails. To override this and suppress
* Exceptions, set the value to <code>true</code>.
* #param ignoreSendFailures true to ignore send failures.
*/
public void setIgnoreSendFailures(boolean ignoreSendFailures) {
Or if you wish for XML configuration:
<xsd:attribute name="ignore-send-failures">
<xsd:annotation>
<xsd:documentation><![CDATA[
If set to "true", failures to send to a message channel will
be ignored. If set to "false", a MessageDeliveryException will be
thrown instead, and if the router resolves more than one channel,
any subsequent channels will not receive the message.
Please be aware that when using direct channels (single threaded),
send-failures can be caused by exceptions thrown by components
much further down-stream.
This attribute defaults to false.
]]></xsd:documentation>
</xsd:annotation>
<xsd:simpleType>
<xsd:union memberTypes="xsd:boolean xsd:string" />
</xsd:simpleType>
</xsd:attribute>
I am using redis as a queue (using the spring queue-in/outbound-channel-adapter) to distribute tasks (a message into the queue, etc)
As the throughput is quite high we observed that, although the messages were sent to the redis queue, a lot of them were lost and no messages arrived to the component after the inbound (a header router)
The channel config is attached below; the point is that we though that the problem was in this header router after the inbound addapter, that was unable to manage the rate of messages read from the queue, so they were lost.
We have use an intermediate element between the inbound adapter and this component (that is a header-router) and add a queue to fix this.
This works fine, but actually we don't fully understand the solution and if this is the proper one.
An expert view and opinion about this configuration will be wellcome!
Thanks
<!-- a Queue Inbound Channel Adapter is available to 'right pop' messages
from a Redis List. -->
<redis:queue-inbound-channel-adapter
id="fromRedis" channel="in" queue="${name}"
receive-timeout="1000" recovery-interval="3000" expect-message="true"
auto-startup="true"/>
<!-- a queue to avoid lost messages before the header router -->
<int:channel id="in">
<int:queue capacity="1000"/>
</int:channel>
<!-- a bridge to connect channels and have a poller -->
<int:bridge input-channel="in" output-channel="out">
<int:poller fixed-delay="500" />
</int:bridge>
<int:header-value-router id="router" timeout="15000"
input-channel="out" header-name="decision"
resolution-required="false" default-output-channel="defaultChannel" />
---added on 26/02
To insert messages into redis we have a web service, but actually is as you said, simply write messages into redis (
for... channel.send(msg)
Nothing more
About your answer I am now thinking in remove the in channel and its queue and use directly the header-value-router; but I have more questions:
I think the right solution is a low value for timeout in header-value-router, so I'll have the error notification faster if we don't have a consumer available. If I don't use a value as timeout, it will block indefinitely and this is a bad idea, isn't it?
I don't know how to manage the MesssageDeliveryException because the router don't have an error-channel configuration, ???
I think that if I can manage this error and get the message back I can re-send it to redis again. There are other servers that get the messages from redis and they luckily could attend it.
I add my proposed solution, but is not complete and we are not sure about the error management as I explained above
<!-- a Queue Inbound Channel Adapter is available to 'right pop' messages
from a Redis List. -->
<redis:queue-inbound-channel-adapter
id="fromRedis" channel="in" queue="${name}"
receive-timeout="1000" recovery-interval="3000" expect-message="true"
auto-startup="true"/>
<!-- a header-value-router with quite low timeout -->
<int:header-value-router id="router" timeout="150"
input-channel="in" header-name="decision"
resolution-required="false" default-output-channel="defaultChannel" />
<!-- ¿if MessageDeliveryException???? what to do??? -->
<int:channel id="someConsumerHeaderValue">
<int:dispatcher task-executor="ConsumerExecutor" />
</int:channel>
<!-- If 5 threads are busy we queue messages up to 5; if queue is full we can increase to 5 more working threads; if no more threads we have a... ¿¿MessageDeliveryException?? -->
<task:executor id="ConsumerExecutor" pool-size="5-5"
queue-capacity="5" />
Well, that's great to see such an observation. That might improve the Framework somehow.
So, I'd like to see:
Some test-case to reproduce from the Framework perspective.
Although I guess there is just enough to send a lot of messages to the Redis and use your config to consume. (Correct me if there is need anything else)
The downstream flow after the <int:header-value-router>. Look, you use there timeout="15000" which is synonym to the send-timeout :
Specify the maximum amount of time in milliseconds to wait
when sending Messages to the target MessageChannels if blocking
is possible (e.g. a bounded queue channel that is currently full).
By default the send will block indefinitely.
Synonym for 'timeout' - only one can be supplied.
From here I can say that if your downstream consumer if enough slow on some QueueChannel there you end up with the:
/**
* Inserts the specified element at the tail of this queue, waiting if
* necessary up to the specified wait time for space to become available.
*
* #return {#code true} if successful, or {#code false} if
* the specified waiting time elapses before space is available
* #throws InterruptedException {#inheritDoc}
* #throws NullPointerException {#inheritDoc}
*/
public boolean offer(E e, long timeout, TimeUnit unit)
....
while (count.get() == capacity) {
if (nanos <= 0)
return false;
nanos = notFull.awaitNanos(nanos);
}
Pay attention to that return false; indicating exactly the message lost.
That is also know like back-pressure drop strategy.
Let me know if you have different picture there.
You may consider to remove that timeout="15000" to meet the same in queue channel behavior.
UPDATE
Well, the error handling works a bit different way. The "guilty" component just throws Exception, like it is with raw Java and it is OK that this component isn't responsible for Exception catching that is up to the caller.
And caller in our case an upstream component - <redis:queue-inbound-channel-adapter>.
Any inbound channel adapter has an error-channel option. Through the <poller> if it is MessageSource or directly when it is MessageProducer.
I'm sure you will be able to handle:
if (!sent) {
throw new MessageDeliveryException(message,
"failed to send message to channel '" + channel + "' within timeout: " + timeout);
}
in that error-channel sub-flow and achieve your requirements for recovery.
I am using a publish-subscribe channel after the inbound gateway receives a String message to parallely send it to the logger to log the message and to a transformer to transform the message. I want both these acivities to happen in parallel.
My question is very simple - Does the publish subscribe channel in spring integration sends messages to it's subscibers parallely?
Below is the code snippet from the source of spring-integration-context.xml.
<int:gateway id="gateway" service-interface="com.test.Gateway">
</int:gateway>
<int:publish-subscribe-channel id="publishsubscribechannel" />
<int:service-activator input-channel="publishsubscribechannel"
method="transformEvent" ref="transformer" output-channel="transformerreplychannel">
</int:service-activator>
<int:service-activator input-channel="publishsubscribechannel"
method="logMessage" ref="logger">
</int:service-activator>
Here the transformer and the logger are 2 subscribers to the publishsubscribechannel. In this setup will the message flow to logger and transformer from the gateway happen asynchronously by default??...OR I need to do some other configuration to achieve the same.
... or using JavaConfig
#Bean
public MessageChannel publishsubscribechannel() {
return new PublishSubscribeChannel(executor());
}
#Bean
public ThreadPoolTaskExecutor executor() {
ThreadPoolTaskExecutor pool = new ThreadPoolTaskExecutor();
pool.setCorePoolSize(10);
pool.setMaxPoolSize(10);
pool.setWaitForTasksToCompleteOnShutdown(true);
return pool;
}
By default, it runs in sequence. So in your case, it will be transformer, then logger. If you want to run it in parallel, you need to specify task-executor
<int:publish-subscribe-channel id="publishsubscribechannel" task-executor="executor" />
...
<task:executor id="executor" pool-size="10" />
And by using task-executor, the message handling is performed asynchronously.
As is from the Spring Integration documentation
PublishSubscribeChannel
The PublishSubscribeChannel implementation broadcasts any Message sent to it to all of its subscribed handlers. This is most often used for sending Event Messages whose primary role is notification as opposed to Document Messages which are generally intended to be processed by a single handler. Note that the PublishSubscribeChannel is intended for sending only. Since it broadcasts to its subscribers directly when its send(Message) method is invoked, consumers cannot poll for Messages (it does not implement PollableChannel and therefore has no receive() method). Instead, any subscriber must be a MessageHandler itself, and the subscriber's handleMessage(Message) method will be invoked in turn.
Prior to version 3.0, invoking the send method on a PublishSubscribeChannel that had no subscribers returned false. When used in conjunction with a MessagingTemplate, a MessageDeliveryException was thrown. Starting with version 3.0, the behavior has changed such that a send is always considered successful if at least the minimum subscribers are present (and successfully handle the message). This behavior can be modified by setting the minSubscribers property, which defaults to 0.
[Note] Note
If a TaskExecutor is used, only the presence of the correct number of subscribers is used for this determination, because the actual handling of the message is performed asynchronously.
Please note the Note.
It does mention if the TaskExecutor is used, the message handling would be asynchronous.
So, yes you have to add a TaskExecutor for this to be asynchronous.