I have an integration flow where some of the steps are async and some of sync. I want to use the barrier to block the Main thread until all async tasks have completed. Based on the documentation, there are two ways to use the barrier.
Send a second trigger message to the input channel of the barrier.
Invoke the trigger method manually of the barrier
In my use case a message comes in the flow and then goes through several components until it reaches the completed channel. I want the main thread to be blocked until the original messages reaches the completed channel. So it seems appropriate to use the option #2 and invoke the barrier trigger method after reaching the completed state. This doesnt seem to work. Here is a simplified version of my flow.
<int:gateway
service-interface="...BarrierGateway"
id="barrierGateway" default-request-channel="input">
</int:gateway>
<int:channel id="input">
<int:dispatcher task-executor="executor" />
</int:channel>
<int:service-activator input-channel="input" output-channel="completed">
<bean class="...BarrierSA" />
</int:service-activator>
<int:channel id="completed" />
<int:service-activator input-channel="completed"
ref="barrier1.handler" method="trigger" />
<int:barrier id="barrier1" input-channel="input" timeout="10000" />
I am sending a message to the gateway which passes it to the input channel which is using a dispatcher so a new thread is started to pass the message forward. At this point, I want to block the main thread while the Executor-1 thread goes through the flow. The rest of the flow is simple. My service-activator sleeps for 3 seconds before returning the message to simulate a delay. Once the message is received in the completed channel, the service-activator should invoke the barrier trigger method and only at this point, the main thread should be released. Instead, the main thread is released right after the dispatcher starts a new thread. I have tried specifying a constant correlation id ('abc') but that didnt help.
I see you're caught in a trap.
The <int:barrier> suspends the Thread just on the message message, but only that Thread which brings that message to him. Looking to your config it is the same input channel with Executor. The purpose of the ExecutorChannel to shift message to a different thread, but not suspend the caller's thread.
From other side you have one more mistake around that input. You declare two subscribers for him, where only one of them will be called by the round-robin balancing strategy.
To fix your task we should have one more top-level channel as <publish-subscribe-channel>. And right, already now you can have two subscribers.
One of them should be a <bridge> to your input ExecutorChannel. And another the desired <barrier>. And only now it can suspend (block in your terms) the main thread from the <gateway>.
From other side the more simpler solution would be don't use the <barrier> at all. The <gateway> has an ability to to block the caller's thread and wait for reply. Of course, that works when the gateway methods isn't void.
And one more point to your config: if you don't wait for reply in the gateway, the <barrier> will fail with the
throw new DestinationResolutionException("no output-channel or replyChannel header available");
So, consider to use something as an output-channel there a well.
Related
I am implementing an kafka based application where I would like to manually acknowledge incoming messages. Architecture forces me to do it in a separate thread.
The question is: is it possible and safe to execute Acknowledgement.acknowledge() in a different thread than consumer?
Yes it is, as long as you use MANUAL and not MANUAL_IMMEDIATE, but I don't think you'll get what you expect.
Kafka doesn't track each message, just offsets within the partition.
Let's say message 1 arrives and you hand off to another thread. Then message 2 arrives and it is handed off to yet another thread.
When the offset for message 2 is ack'd, you are effectively acking both messages.
I have a huge xml that come as an input payload to my Spring integration flow. So I am using claim check in transformer instead of header enricher to retain my payload. I am using an in-memory message store.
Later on in my SI flow, I have a splitter that splits the payload into multiple threads and each thread will invoke different channel based on one of the attribute payload. I am using a router for achieve this. Each flow or each thread uses a claim check out transformer to retrieve the initial payload then us it for building the required response. Each thread will produce a response and I don't have to aggregate them. So I will have multiple responses coming out from my flow which will then be dropped into a queue.
I cannot remove the message during the check out as other thread will also try to check out the same message. What is the best way to remove the message from the message store?
Sample configuration
`<int:chain input-channel="myInputChannel"
output-channel="myOutputchannel">
<int:claim-check-in />
<int:header-enricher>
<int:header name="myClaimCheckID" expression="payload"/>
</int:header-enricher>
</int:chain>`
all the other components in the flow are invoked before the splitter
<int:splitter input-channel="mySplitterChannel" output-channel="myRouterChannel" expression="mySplitExpression">
</int:splitter>
`<int:router input-channel="myRouterChannel" expression="routerExpression"
resolution-required="true">
<int:mapping value="A" channel="aChannel" />
<int:mapping value="B" channel="bChannel" />
<int:mapping value="C" channel="cChannel" />
</int:router>`
Each channel has a claim check out transformer for the initial payload. So how do I make sure the message is removed after all the threads have been processed?
When you know you are done with the message you can simply invoke the message store's remove() method. You could use a service activator with
... expression="#store.remove(headers['myClaimCheckID'])" ...
However, if you are using an in-memory message store there is really no point in using the claim check pattern.
If you simply promote the payload to a header, it will use no more memory than putting it in a store.
Even if it ends up in multiple messages on multiple threads, it makes no difference since they'll all be pointing to the same object on the heap.
I have a sync process that waits, by polling a database, for an async process to respond. Basically the async process inserts a new record into a database, and the sync process checks periodically for an update to it, continuing its execution after an update is found. The async process is not invoked by the sync one.
I'm believing that this isn't the only solution available, concerning a sync process waiting for an async one. My question here is, is there any better way to do it? Concerning performance specially.
If there's already an answer to this question, i'm sorry, but i haven't found it.
A better way would be to send a message from the async process to the sync one. The latter could just block with a receive activity until the message arrives. This is better from a performance viewpoint, because the sync process does not have to wake up regularily and check the database. Instead, it could just sleep until the engine detects the arrival of the message and wakes up the sync process.
Please note that this implies that you have to use correlationSets [1] on the side of the sync process, which have to be initiated some point in time before waiting on the actual message (for instance during your initial receive). This is needed so that the engine can decide to which process instance to forward the message to. You also might want to add an additional partnerLink with myRole to the sync process so that there is an operation to invoke by the async process (which then also needs a corresponding partnerLink). The structure of the sync process which I mean looks somehow like this:
<sequence>
<receive name="initialSyncReceive" createInstance="yes" partnerLink="originalPartnerLink" ... />
<!-- some logic before waiting on async process -->
<receive name="notificationFromAsync" partnerLink="newPartnerLinkForAsyncOnly" ... />
<!-- some logic after notification from async process -->
<reply name="replyToSyncRequest" partnerLink="originalPartnerLink" ... />
<sequence>
Disclaimer: This is just an outline and you cannot just copy/paste the code. Getting this scenario to work requires a number of non-trivial changes to both processes, e.g. adding correlationSets, additional WSDLs, propertyAliases, and partnerlinks.
I have a queue channel, and a service activator with a poller which reads from that queue. I'd like to have configuration to say "I want 50 threads to poll that queue, and each time you poll and get a message back, on this thread, invoke the service the service-activator points to."
The service has no #Async annotations, but is stateless and safe to run in a concurrent fashion.
Will the below do that? Are there other preferred ways of achieving this?
<int:channel id="titles">
<int:queue/>
</int:channel>
<int:service-activator output-channel="resolvedIds" ref="searchService" method="searchOnTitle" input-channel="titles">
<int:poller fixed-delay="100" time-unit="MILLISECONDS" task-executor="taskExecutor"></int:poller>
</int:service-activator>
<task:executor id="taskExecutor" pool-size="50" keep-alive="120" />
Yes I think it does what you want. Once you introduce a QueueChannel the interaction becomes async - you don't need #Async. If you don't explicitly set up a poller it will use the default poller.
What you have outlined is the best way to achieve it. You might also consider putting a limit on the queue size - so that in case there is a lag in keeping up with the producer it doesn't lead to out of memory issue. If a size is specified then the send calls on the channel will block - acting as a throttle.
The configuration you have will work as you expect. The only issue is that once you start creating executors and pollers for each end point it becomes difficult to figure out the optimal configuration for the entire application. It is ok to do this kind of optimization for a few specific steps - but not for all the endpoints (nothing in you questions suggests that you are doing it, just thought that I will raise it anyway.
I'm using Oracle ODP.Net for enqueue and dequeue.
Process A : Enqueue
Process B : Dequeue with MessageAvailable event
If Process A and B are running, there is no problem. On the "Process B", the event is always fired.
But, if "Process B" is off and "Process A" is on, when "Process B" restarts, the queues inserted during the off time are lost.
Is there an option for to fire the event for all queues inserted in the past ?
Many Thanks
There seem to be two approaches to address this issue:
Call the Listen() method of the OracleAQQueue class (after registering for message notification) to pick up "orphaned" messages sitting in the queue. Note that Listen() blocks until a message is received or a timeout occurs. So you'd want to specify a (short) timeout to return back to the processing thread in the event no messages are on the queue.
Call the Dequeue() method and trap Oracle error 25228 (no message available to dequeue). See the following thread from the Oracle support forums: https://forums.oracle.com/forums/thread.jspa?threadID=2186496.
I've been scratching my head on this topic. If you still have to "manually" test for new messages, what is the benefit of using the MessageAvaiable event callback in the first place? One route I've pondered is to wrap the Listen() method in an async call so that the caller isn't blocking on the thread (until a message is received or a timeout occurs). I wrapped Listen() and Dequeue() in a custom Receive() method and created my own MessageReceived event handler to pass the message details to the calling thread. Seems somewhat redundant, since ODP.NET provides the out-of-box callback, but I don't have to deal with the issue you describe (or write code to "manually" test for "orphaned" messages.
Any comments/thoughts on approach are welcomed.
I've been looking at this one too and have ended up doing something similar to Greg. I've not used the Listen() method though as I don't think it offers me anything over and above a simple Dequeue() - Listen() seems to be beneficial when you want to listen on behalf of multiple consumers, which in my instance is not relevant (see Oracle Docs).
So, in my 'Process B' I first register for notifications before initiating a polling process to check for any existing messages. It doesn't Listen(), it just calls Dequeue() within a controlled loop with a Wait period of a couple of seconds set. If the polling process encounters an Oracle timeout the wait period has expired and polling stops. I may need to consider dealing with timeouts if the wait period hasn't expired (though not 100% sure this if this is likely to happen).
I've noticed that any messages which are enqueued whilst polling will call the message notification method but by the time this connects and tries to retrieve the message the polling process always seems to have taken it. So inside the message notification method I capture and ignore any OracleExceptions with number 25263 (no message in queue <...> with message ID <...>).