Using spring integration, for every min i need to read list of orders from database whose status is in-progress and make a 3rd party rest call for each order in sequence or parallel. Below is my code
<int:inbound-channel-adapter ref="orderReader" method="readOrderRecords"
channel="orderListChannel">
<int:poller cron="0/60 * * * * *"/>
</int:inbound-channel-adapter>
<int:splitter input-channel="orderListChannel" method="split" ref="orderSplitter"
output-channel="processOrders">
</int:splitter>
<int:publish-subscribe-channel id="processOrders"/>
<int:chain id="orderProcess_Chain"
input-channel="processOrders">
...contain the REST call config
</int.chain>
The above code is not working as expected, if there are n records in the database with in-progress, its processing only the first order (orderProcess_Chain is called only for 1st order)
What is wrong in the code
First of all we need to see your DB reading logic, that your orderReader.readOrderRecords().
From the other hand you should be sure that finish your REST call properly.
It is actually request/reply protocol, but I see only one-way interaction.
Right, that is correct for your purpose, but you should void the REST response somehow.
And one more clue. You always can switch on DEBUG for the org.springframework.integration category and analyze how your messages travel there. For example you may have some Exception, but there is no any exception handling in your config.
Related
I've been using Spring boot and getting rid of all XML files in my project.
Unfortunately it also uses Spring integration which from my experience is very heavily XML based.
I have a scenario which requires me to have an aggregator, and have that aggregator polled every x amount of seconds.
This can be done using XML like so (example taken from a previous SO question):
<!--
the poller will process 100 messages every minute
if the size of the group is 100 (the poll reached the max messages) or 60 seconds time out (poll has less than 100 messages) then the payload with the list of messages is passed to defined output channel
-->
<int:aggregator input-channel="logEntryChannel" output-channel="logEntryAggrChannel"
send-partial-result-on-expiry="true"
group-timeout="60000"
correlation-strategy-expression="T(Thread).currentThread().id"
release-strategy-expression="size() == 100">
<int:poller max-messages-per-poll="100" fixed-rate="60000"/>
</int:aggregator>
I've managed to find a class that kinda sorta does the trick and it's bean definition is:
#Bean(name = "aggregatingMessageHandler")
public AggregatingMessageHandler aggregatingMessageHandler() {
AggregatingMessageHandler aggregatingMessageHandler =
new AggregatingMessageHandler(messageGroupProcessorBean(),
new SimpleMessageStore(10));
aggregatingMessageHandler.setCorrelationStrategy(customCorrelationStrategyBean());
aggregatingMessageHandler.setReleaseStrategy(
new TimeoutCountSequenceSizeReleaseStrategy(3,
TimeoutCountSequenceSizeReleaseStrategy.DEFAULT_TIMEOUT));
aggregatingMessageHandler.setExpireGroupsUponCompletion(true);
aggregatingMessageHandler.setOutputChannel(outputAggregatedChannelBean());
return aggregatingMessageHandler;
}
However this triggers the canRelease() method of the ReleaseStrategy only when a new message is received in the inboundChannel associated with this handler, and not at a fixed time interval which is not the desired result.
I want all groups older than one minute to be redirected to the output channel.
My question is - is there a way to programmatically attach a Poller such as the one in the XML definition?
For Java & Annotation configuration take a look here and here.
The Aggregator component has AggregatorFactoryBean for easier Java Configuration.
Anyway you have to pay attention that there is a #ServiceActivator annotation together with a #Bean on that handler definition. And exactly #ServiceActivator has poller attribute.
Also pay attention that there is a Java DSL for Spring Integration.
Another part of your question is a bit confusion. The poller fully isn't related to the release strategy. Its responsibility in this case to receive messages from the PollableChannel which is that logEntryChannel. And only after that already polled messages are placed to the aggregator for it correlation and release logic.
What is done in that sample is fully different story and we can discuss it in the separate SO thread.
At unit test time, I try to bridge the Spring Integration default channel to a queued channel, since I want to check the correctness of the amount of message flow into this channel.
<int:filter input-channel="prevChannel" output-channel="myChannel">
<int:bridge input-channel="myChannel" output-channel="aggregateChannel">
// the reason I have above bridge is I want to check the amount of message after filter.
// I cannot check prevChannel since it is before filtering, and I cannot check aggregateChannel
// because it has other processing branch
// in test xml I import above normal workflow xml and added below configuration to
// redirect message from myChannel to checkMyChannel to checking.
<int:bridge input-channel="myChannel"
output-channel="checkMyChannel"/>
<int:channel id="checkMyChannel">
<int:queue/>
</int:channel>
I autowired checkMyChannel in my unit test
but checkMyChannel.getqueuesize() always return 0.
Is there sth I did wrong?
You missed to share test-case with us. We don't have the entire picture. And looks like you have a race condition there. Someone polls all your messages from the checkMyChannel before you start assert that getqueuesize().
In our tests we don't use <poller> for such a cases. We use PollableChannel.receive(timeout) manually.
got this fixed, I have to declare myChannel to be a publish-subscribe-channel
How to test Spring Integration
This one helps, for my case there is a race condition since.
"A regular channel with multiple subscribers will round-robin ..."
I have an spring integration implemetation where I have two Client subscribers listening to the same JMS Topic. I am using JDBC message store( Different REGIONS) in both the implementation to save the incoming messages. While processing the data I get the exception:
org.springframework.dao.EmptyResultDataAccessException: Incorrect result size: expected 1, actual 0
I know this is Jira issue : https://jira.spring.io/browse/INT-2912
As now I cant upgrade the Spring version. I am unable to understand the workaround "The work-around is either to always use a different groupKey or to use separate tables for each Message Store. We will need to add a REGION column to the INT_GROUP_TO_MESSAGE as well."
How can i create a different groupKey?
My implementation is as below:
<bean
id="jdbcMessageStore"
class="org.springframework.integration.jdbc.JdbcMessageStore"
p:dataSource-ref="datasource"
p:region="REPORTS"/>
<si:aggregator
send-partial-result-on-expiry="false"
message-store="jdbcMessageStore"
discard-channel="discardedLogger"/>
The "groupKey" mentioned there is the correlation strategy result; by default it just uses the correlationId header.
You can use correlation-strategy-expression="'foo' + headers['correlationId']" and correlation-strategy-expression="'bar' + headers['correlationId']" to use a different group key for each app.
I am using redis as a queue (using the spring queue-in/outbound-channel-adapter) to distribute tasks (a message into the queue, etc)
As the throughput is quite high we observed that, although the messages were sent to the redis queue, a lot of them were lost and no messages arrived to the component after the inbound (a header router)
The channel config is attached below; the point is that we though that the problem was in this header router after the inbound addapter, that was unable to manage the rate of messages read from the queue, so they were lost.
We have use an intermediate element between the inbound adapter and this component (that is a header-router) and add a queue to fix this.
This works fine, but actually we don't fully understand the solution and if this is the proper one.
An expert view and opinion about this configuration will be wellcome!
Thanks
<!-- a Queue Inbound Channel Adapter is available to 'right pop' messages
from a Redis List. -->
<redis:queue-inbound-channel-adapter
id="fromRedis" channel="in" queue="${name}"
receive-timeout="1000" recovery-interval="3000" expect-message="true"
auto-startup="true"/>
<!-- a queue to avoid lost messages before the header router -->
<int:channel id="in">
<int:queue capacity="1000"/>
</int:channel>
<!-- a bridge to connect channels and have a poller -->
<int:bridge input-channel="in" output-channel="out">
<int:poller fixed-delay="500" />
</int:bridge>
<int:header-value-router id="router" timeout="15000"
input-channel="out" header-name="decision"
resolution-required="false" default-output-channel="defaultChannel" />
---added on 26/02
To insert messages into redis we have a web service, but actually is as you said, simply write messages into redis (
for... channel.send(msg)
Nothing more
About your answer I am now thinking in remove the in channel and its queue and use directly the header-value-router; but I have more questions:
I think the right solution is a low value for timeout in header-value-router, so I'll have the error notification faster if we don't have a consumer available. If I don't use a value as timeout, it will block indefinitely and this is a bad idea, isn't it?
I don't know how to manage the MesssageDeliveryException because the router don't have an error-channel configuration, ???
I think that if I can manage this error and get the message back I can re-send it to redis again. There are other servers that get the messages from redis and they luckily could attend it.
I add my proposed solution, but is not complete and we are not sure about the error management as I explained above
<!-- a Queue Inbound Channel Adapter is available to 'right pop' messages
from a Redis List. -->
<redis:queue-inbound-channel-adapter
id="fromRedis" channel="in" queue="${name}"
receive-timeout="1000" recovery-interval="3000" expect-message="true"
auto-startup="true"/>
<!-- a header-value-router with quite low timeout -->
<int:header-value-router id="router" timeout="150"
input-channel="in" header-name="decision"
resolution-required="false" default-output-channel="defaultChannel" />
<!-- ¿if MessageDeliveryException???? what to do??? -->
<int:channel id="someConsumerHeaderValue">
<int:dispatcher task-executor="ConsumerExecutor" />
</int:channel>
<!-- If 5 threads are busy we queue messages up to 5; if queue is full we can increase to 5 more working threads; if no more threads we have a... ¿¿MessageDeliveryException?? -->
<task:executor id="ConsumerExecutor" pool-size="5-5"
queue-capacity="5" />
Well, that's great to see such an observation. That might improve the Framework somehow.
So, I'd like to see:
Some test-case to reproduce from the Framework perspective.
Although I guess there is just enough to send a lot of messages to the Redis and use your config to consume. (Correct me if there is need anything else)
The downstream flow after the <int:header-value-router>. Look, you use there timeout="15000" which is synonym to the send-timeout :
Specify the maximum amount of time in milliseconds to wait
when sending Messages to the target MessageChannels if blocking
is possible (e.g. a bounded queue channel that is currently full).
By default the send will block indefinitely.
Synonym for 'timeout' - only one can be supplied.
From here I can say that if your downstream consumer if enough slow on some QueueChannel there you end up with the:
/**
* Inserts the specified element at the tail of this queue, waiting if
* necessary up to the specified wait time for space to become available.
*
* #return {#code true} if successful, or {#code false} if
* the specified waiting time elapses before space is available
* #throws InterruptedException {#inheritDoc}
* #throws NullPointerException {#inheritDoc}
*/
public boolean offer(E e, long timeout, TimeUnit unit)
....
while (count.get() == capacity) {
if (nanos <= 0)
return false;
nanos = notFull.awaitNanos(nanos);
}
Pay attention to that return false; indicating exactly the message lost.
That is also know like back-pressure drop strategy.
Let me know if you have different picture there.
You may consider to remove that timeout="15000" to meet the same in queue channel behavior.
UPDATE
Well, the error handling works a bit different way. The "guilty" component just throws Exception, like it is with raw Java and it is OK that this component isn't responsible for Exception catching that is up to the caller.
And caller in our case an upstream component - <redis:queue-inbound-channel-adapter>.
Any inbound channel adapter has an error-channel option. Through the <poller> if it is MessageSource or directly when it is MessageProducer.
I'm sure you will be able to handle:
if (!sent) {
throw new MessageDeliveryException(message,
"failed to send message to channel '" + channel + "' within timeout: " + timeout);
}
in that error-channel sub-flow and achieve your requirements for recovery.
In a spring integration project I have a outbound gateway to send and receive messages synchronously. I would like to log the time taken for such an operation.
<jms:outbound-gateway id="jmsOutGateway"
request-destination="outQueue"
request-channel="outboundJmsRequests"
reply-channel="jmsReplies"/>
I tried searching but could only find examples of using AOP and PerformanceMonitorInterceptor to trace methods.
You can do it like this:
<int-jms:outbound-gateway id="jmsOutGateway"
request-destination="outQueue"
request-channel="outboundJmsRequests"
reply-channel="jmsReplies">
<int-jms:request-handler-advice-chain>
<bean class="org.springframework.aop.interceptor.PerformanceMonitorInterceptor"/>
</int-jms:request-handler-advice-chain>
</int-jms:outbound-gateway>
The <request-handler-advice-chain> is applied for the AbstractReplyProducingMessageHandler.handleRequestMessage method and <int-jms:outbound-gateway> populates JmsOutboundGateway - an implementation of AbstractReplyProducingMessageHandler.