At unit test time, I try to bridge the Spring Integration default channel to a queued channel, since I want to check the correctness of the amount of message flow into this channel.
<int:filter input-channel="prevChannel" output-channel="myChannel">
<int:bridge input-channel="myChannel" output-channel="aggregateChannel">
// the reason I have above bridge is I want to check the amount of message after filter.
// I cannot check prevChannel since it is before filtering, and I cannot check aggregateChannel
// because it has other processing branch
// in test xml I import above normal workflow xml and added below configuration to
// redirect message from myChannel to checkMyChannel to checking.
<int:bridge input-channel="myChannel"
output-channel="checkMyChannel"/>
<int:channel id="checkMyChannel">
<int:queue/>
</int:channel>
I autowired checkMyChannel in my unit test
but checkMyChannel.getqueuesize() always return 0.
Is there sth I did wrong?
You missed to share test-case with us. We don't have the entire picture. And looks like you have a race condition there. Someone polls all your messages from the checkMyChannel before you start assert that getqueuesize().
In our tests we don't use <poller> for such a cases. We use PollableChannel.receive(timeout) manually.
got this fixed, I have to declare myChannel to be a publish-subscribe-channel
How to test Spring Integration
This one helps, for my case there is a race condition since.
"A regular channel with multiple subscribers will round-robin ..."
Related
I want to write to a channel adapter only if the previous channel adapter write has been written successfully. I’m trying to do this by:
#Bean
public IntegrationFlow buildFlow() {
return IntegrationFlows.from(someChannelAdapter)
.handle(outboundChannelAdapter1)
.handle(outboundChannelAdapter2)
.get();
}
But I’m getting the following exception: The ‘currentComponent’ (…ReactiveMessageHandlerAdapter) is a one-way 'MessageHandler’ and it isn’t appropriate to configure ‘outputChannel’. This is the end of the integration flow.
How can I perform this?
If your handler implementation is one-way, fire-n-forget, then indeed there is no justification to continue the flow. It can go ahead with the configuration if the current handler is reply-producing and there will be something we can build a message to send to the next channel.
In your case .handle(outboundChannelAdapter1) is just void, so the next .handle(outboundChannelAdapter2) is not going to have anything to continue the flow. So, the framework gives you a hint that such a configuration is wrong. It is called flow for a reason: the result of the current endpoint is going to be an input for the next one. If no result, no continuation. How else it could work in your opinion?
The point is that there need to be something to write to your channel adapter. One of the solution is a PublishSubscribeChannel which distributes the same input message to all its subscribers. If that is what would fit to your expectations, then take a look into its support in Java DSL: https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-subflows.
Another way is a RecipientListRouter pattern: https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#router-implementations-recipientlistrouter.
You may achieve the same with WireTap as well, but it depends on a business logic of your solution: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-wiretap.
But anyway: you need to understand that the second handler can be called only if there is an input message for its channel. In all those cases I showed you it is exactly the same message you send to a first handler. If your expectations are different, please elaborate what kind of message you'd like to have for a second handler if the first does not return anything.
I've been using Spring boot and getting rid of all XML files in my project.
Unfortunately it also uses Spring integration which from my experience is very heavily XML based.
I have a scenario which requires me to have an aggregator, and have that aggregator polled every x amount of seconds.
This can be done using XML like so (example taken from a previous SO question):
<!--
the poller will process 100 messages every minute
if the size of the group is 100 (the poll reached the max messages) or 60 seconds time out (poll has less than 100 messages) then the payload with the list of messages is passed to defined output channel
-->
<int:aggregator input-channel="logEntryChannel" output-channel="logEntryAggrChannel"
send-partial-result-on-expiry="true"
group-timeout="60000"
correlation-strategy-expression="T(Thread).currentThread().id"
release-strategy-expression="size() == 100">
<int:poller max-messages-per-poll="100" fixed-rate="60000"/>
</int:aggregator>
I've managed to find a class that kinda sorta does the trick and it's bean definition is:
#Bean(name = "aggregatingMessageHandler")
public AggregatingMessageHandler aggregatingMessageHandler() {
AggregatingMessageHandler aggregatingMessageHandler =
new AggregatingMessageHandler(messageGroupProcessorBean(),
new SimpleMessageStore(10));
aggregatingMessageHandler.setCorrelationStrategy(customCorrelationStrategyBean());
aggregatingMessageHandler.setReleaseStrategy(
new TimeoutCountSequenceSizeReleaseStrategy(3,
TimeoutCountSequenceSizeReleaseStrategy.DEFAULT_TIMEOUT));
aggregatingMessageHandler.setExpireGroupsUponCompletion(true);
aggregatingMessageHandler.setOutputChannel(outputAggregatedChannelBean());
return aggregatingMessageHandler;
}
However this triggers the canRelease() method of the ReleaseStrategy only when a new message is received in the inboundChannel associated with this handler, and not at a fixed time interval which is not the desired result.
I want all groups older than one minute to be redirected to the output channel.
My question is - is there a way to programmatically attach a Poller such as the one in the XML definition?
For Java & Annotation configuration take a look here and here.
The Aggregator component has AggregatorFactoryBean for easier Java Configuration.
Anyway you have to pay attention that there is a #ServiceActivator annotation together with a #Bean on that handler definition. And exactly #ServiceActivator has poller attribute.
Also pay attention that there is a Java DSL for Spring Integration.
Another part of your question is a bit confusion. The poller fully isn't related to the release strategy. Its responsibility in this case to receive messages from the PollableChannel which is that logEntryChannel. And only after that already polled messages are placed to the aggregator for it correlation and release logic.
What is done in that sample is fully different story and we can discuss it in the separate SO thread.
Using spring integration, for every min i need to read list of orders from database whose status is in-progress and make a 3rd party rest call for each order in sequence or parallel. Below is my code
<int:inbound-channel-adapter ref="orderReader" method="readOrderRecords"
channel="orderListChannel">
<int:poller cron="0/60 * * * * *"/>
</int:inbound-channel-adapter>
<int:splitter input-channel="orderListChannel" method="split" ref="orderSplitter"
output-channel="processOrders">
</int:splitter>
<int:publish-subscribe-channel id="processOrders"/>
<int:chain id="orderProcess_Chain"
input-channel="processOrders">
...contain the REST call config
</int.chain>
The above code is not working as expected, if there are n records in the database with in-progress, its processing only the first order (orderProcess_Chain is called only for 1st order)
What is wrong in the code
First of all we need to see your DB reading logic, that your orderReader.readOrderRecords().
From the other hand you should be sure that finish your REST call properly.
It is actually request/reply protocol, but I see only one-way interaction.
Right, that is correct for your purpose, but you should void the REST response somehow.
And one more clue. You always can switch on DEBUG for the org.springframework.integration category and analyze how your messages travel there. For example you may have some Exception, but there is no any exception handling in your config.
I am using redis as a queue (using the spring queue-in/outbound-channel-adapter) to distribute tasks (a message into the queue, etc)
As the throughput is quite high we observed that, although the messages were sent to the redis queue, a lot of them were lost and no messages arrived to the component after the inbound (a header router)
The channel config is attached below; the point is that we though that the problem was in this header router after the inbound addapter, that was unable to manage the rate of messages read from the queue, so they were lost.
We have use an intermediate element between the inbound adapter and this component (that is a header-router) and add a queue to fix this.
This works fine, but actually we don't fully understand the solution and if this is the proper one.
An expert view and opinion about this configuration will be wellcome!
Thanks
<!-- a Queue Inbound Channel Adapter is available to 'right pop' messages
from a Redis List. -->
<redis:queue-inbound-channel-adapter
id="fromRedis" channel="in" queue="${name}"
receive-timeout="1000" recovery-interval="3000" expect-message="true"
auto-startup="true"/>
<!-- a queue to avoid lost messages before the header router -->
<int:channel id="in">
<int:queue capacity="1000"/>
</int:channel>
<!-- a bridge to connect channels and have a poller -->
<int:bridge input-channel="in" output-channel="out">
<int:poller fixed-delay="500" />
</int:bridge>
<int:header-value-router id="router" timeout="15000"
input-channel="out" header-name="decision"
resolution-required="false" default-output-channel="defaultChannel" />
---added on 26/02
To insert messages into redis we have a web service, but actually is as you said, simply write messages into redis (
for... channel.send(msg)
Nothing more
About your answer I am now thinking in remove the in channel and its queue and use directly the header-value-router; but I have more questions:
I think the right solution is a low value for timeout in header-value-router, so I'll have the error notification faster if we don't have a consumer available. If I don't use a value as timeout, it will block indefinitely and this is a bad idea, isn't it?
I don't know how to manage the MesssageDeliveryException because the router don't have an error-channel configuration, ???
I think that if I can manage this error and get the message back I can re-send it to redis again. There are other servers that get the messages from redis and they luckily could attend it.
I add my proposed solution, but is not complete and we are not sure about the error management as I explained above
<!-- a Queue Inbound Channel Adapter is available to 'right pop' messages
from a Redis List. -->
<redis:queue-inbound-channel-adapter
id="fromRedis" channel="in" queue="${name}"
receive-timeout="1000" recovery-interval="3000" expect-message="true"
auto-startup="true"/>
<!-- a header-value-router with quite low timeout -->
<int:header-value-router id="router" timeout="150"
input-channel="in" header-name="decision"
resolution-required="false" default-output-channel="defaultChannel" />
<!-- ¿if MessageDeliveryException???? what to do??? -->
<int:channel id="someConsumerHeaderValue">
<int:dispatcher task-executor="ConsumerExecutor" />
</int:channel>
<!-- If 5 threads are busy we queue messages up to 5; if queue is full we can increase to 5 more working threads; if no more threads we have a... ¿¿MessageDeliveryException?? -->
<task:executor id="ConsumerExecutor" pool-size="5-5"
queue-capacity="5" />
Well, that's great to see such an observation. That might improve the Framework somehow.
So, I'd like to see:
Some test-case to reproduce from the Framework perspective.
Although I guess there is just enough to send a lot of messages to the Redis and use your config to consume. (Correct me if there is need anything else)
The downstream flow after the <int:header-value-router>. Look, you use there timeout="15000" which is synonym to the send-timeout :
Specify the maximum amount of time in milliseconds to wait
when sending Messages to the target MessageChannels if blocking
is possible (e.g. a bounded queue channel that is currently full).
By default the send will block indefinitely.
Synonym for 'timeout' - only one can be supplied.
From here I can say that if your downstream consumer if enough slow on some QueueChannel there you end up with the:
/**
* Inserts the specified element at the tail of this queue, waiting if
* necessary up to the specified wait time for space to become available.
*
* #return {#code true} if successful, or {#code false} if
* the specified waiting time elapses before space is available
* #throws InterruptedException {#inheritDoc}
* #throws NullPointerException {#inheritDoc}
*/
public boolean offer(E e, long timeout, TimeUnit unit)
....
while (count.get() == capacity) {
if (nanos <= 0)
return false;
nanos = notFull.awaitNanos(nanos);
}
Pay attention to that return false; indicating exactly the message lost.
That is also know like back-pressure drop strategy.
Let me know if you have different picture there.
You may consider to remove that timeout="15000" to meet the same in queue channel behavior.
UPDATE
Well, the error handling works a bit different way. The "guilty" component just throws Exception, like it is with raw Java and it is OK that this component isn't responsible for Exception catching that is up to the caller.
And caller in our case an upstream component - <redis:queue-inbound-channel-adapter>.
Any inbound channel adapter has an error-channel option. Through the <poller> if it is MessageSource or directly when it is MessageProducer.
I'm sure you will be able to handle:
if (!sent) {
throw new MessageDeliveryException(message,
"failed to send message to channel '" + channel + "' within timeout: " + timeout);
}
in that error-channel sub-flow and achieve your requirements for recovery.
I am using a publish-subscribe channel after the inbound gateway receives a String message to parallely send it to the logger to log the message and to a transformer to transform the message. I want both these acivities to happen in parallel.
My question is very simple - Does the publish subscribe channel in spring integration sends messages to it's subscibers parallely?
Below is the code snippet from the source of spring-integration-context.xml.
<int:gateway id="gateway" service-interface="com.test.Gateway">
</int:gateway>
<int:publish-subscribe-channel id="publishsubscribechannel" />
<int:service-activator input-channel="publishsubscribechannel"
method="transformEvent" ref="transformer" output-channel="transformerreplychannel">
</int:service-activator>
<int:service-activator input-channel="publishsubscribechannel"
method="logMessage" ref="logger">
</int:service-activator>
Here the transformer and the logger are 2 subscribers to the publishsubscribechannel. In this setup will the message flow to logger and transformer from the gateway happen asynchronously by default??...OR I need to do some other configuration to achieve the same.
... or using JavaConfig
#Bean
public MessageChannel publishsubscribechannel() {
return new PublishSubscribeChannel(executor());
}
#Bean
public ThreadPoolTaskExecutor executor() {
ThreadPoolTaskExecutor pool = new ThreadPoolTaskExecutor();
pool.setCorePoolSize(10);
pool.setMaxPoolSize(10);
pool.setWaitForTasksToCompleteOnShutdown(true);
return pool;
}
By default, it runs in sequence. So in your case, it will be transformer, then logger. If you want to run it in parallel, you need to specify task-executor
<int:publish-subscribe-channel id="publishsubscribechannel" task-executor="executor" />
...
<task:executor id="executor" pool-size="10" />
And by using task-executor, the message handling is performed asynchronously.
As is from the Spring Integration documentation
PublishSubscribeChannel
The PublishSubscribeChannel implementation broadcasts any Message sent to it to all of its subscribed handlers. This is most often used for sending Event Messages whose primary role is notification as opposed to Document Messages which are generally intended to be processed by a single handler. Note that the PublishSubscribeChannel is intended for sending only. Since it broadcasts to its subscribers directly when its send(Message) method is invoked, consumers cannot poll for Messages (it does not implement PollableChannel and therefore has no receive() method). Instead, any subscriber must be a MessageHandler itself, and the subscriber's handleMessage(Message) method will be invoked in turn.
Prior to version 3.0, invoking the send method on a PublishSubscribeChannel that had no subscribers returned false. When used in conjunction with a MessagingTemplate, a MessageDeliveryException was thrown. Starting with version 3.0, the behavior has changed such that a send is always considered successful if at least the minimum subscribers are present (and successfully handle the message). This behavior can be modified by setting the minSubscribers property, which defaults to 0.
[Note] Note
If a TaskExecutor is used, only the presence of the correct number of subscribers is used for this determination, because the actual handling of the message is performed asynchronously.
Please note the Note.
It does mention if the TaskExecutor is used, the message handling would be asynchronous.
So, yes you have to add a TaskExecutor for this to be asynchronous.