I've a spring-integration implementation with following:
Multiple publishing channels publishing on one common channel.
All the channels returns same response object.
Aggregator trying to aggregate response from all the above channels
Issue: Aggregator not able to combine all the responses together and provided method gets invoked on the first reponse from the channels
Here are the details. What is that I've to do to aggregate the responses?
<int:publish-subscribe-channel id="aggregate-channel" apply-sequence="true"/>
<int:publish-subscribe-channel id="input-channel" apply-sequence="true"/>
<int:service-activator input-channel="input-channel" output-channel="aggregate-channel" ref="...A" method="...A">
<int:service-activator input-channel="input-channel" output-channel="aggregate-channel" ref="...B" method="...B">
<int:service-activator input-channel="input-channel" output-channel="aggregate-channel" ref="...C" method="...C">
<int:service-activator input-channel="input-channel" output-channel="aggregate-channel" ref="...D" method="...D">
<!--This is the aggregator.
**Expecting a list of size 4 but then it gets list of size 1 for each response channel
-->
<int:aggregator input-channel="aggregate-channel" output-channel="gateway-response-channel" ref="Service" method="responseListProcessor"/>
Solution
Replace
<int:publish-subscribe-channel id="aggregate-channel" apply-sequence="true"/>
with
<int:publish-subscribe-channel id="aggregate-channel"/>
Issue
Initially it starts with sequence size 4 because input-channel has 4 subscribers
But when you add the attribute apply-sequence="true" to aggregate-channel, it reset the sequence size to 1 because aggregate-channel has only one subscriber which is the aggregator.
Reference
https://docs.spring.io/spring-integration/docs/5.1.7.RELEASE/reference/html/#channel-configuration-pubsubchannel
If you provide a aggregator downstream from a PublishSubscribeChannel, you can set the 'apply-sequence' property on the channel to true.
Doing so indicates that the channel should set the sequence-size and sequence-number message headers as well as the correlation ID prior to passing along the messages.
For example, if there are five subscribers, the sequence-size would be set to 5, and the messages would have sequence-number header values ranging from 1 to 5.
I would say your requirement is fully covered by specific for this kinda of tasks EIP - Scatter-Gather: https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#scatter-gather
Related
I have a Spring JMS-to-database microservice using Spring Integration to route incoming messages through a sequence of filter, transform, and service activator endpoints (the last persisting the filtered and transformed messages to an Oracle database). Messages have a primary key used in the database. We have a 5 concurrent consumer set up, based on MQ constraints in prod. All of this works.
Recently we found that the upstream producer can send two or more messages with the same primary key back-to-back. Since two separate threads are given the two messages, even MERGE SQL fails with a SQLIntegrityConstraintViolationException (both threads attempt the ON NOT MATCHED INSERT... portion of the MERGE with the database server).
So we thought to change the listener to guarantee that a specific thread would get a message with any given primary key, using a Hash of the key moduloed by the concurrent consumer count. We need to ACK the message once it is saved to the database, not before, so I want to preserve transaction semantics. But I have been unable to get this working with either a Router, or a LoadBalancer on the channel coming from our DefaultMessageListenerContainer. I've also looked at sub-classing DefaultMessageListenerContainer, but don't see the appropriate point to select a given thread.
Any recommendations?
Bean/channel XML setup, our classes renamed for brevity:
...
<bean id="listenerContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:connectionFactory-ref="consumerConnectionFactory"
p:concurrentConsumers="5"
p:maxConcurrentConsumers="5"
p:destinationName="IN_QUEUE"
p:idleConsumerLimit="5"
p:sessionTransacted="true"
p:errorHandler-ref="errorHandler"
p:autoStartup="true"
p:recoveryInterval="60000"
p:receiveTimeout="5000"/>
...
`<int-jms:message-driven-channel-adapter id="xxx-message-in"
container="listenerContainer"
error-channel="errorChannel"/>
<int:transformer id="unmarshaller"
method="unmarshalPayload"
input-channel="message-in" output-channel="filterIt">
<bean class="com.db.xyz.endpoint.UnmarshallerEndpoint"/>
</int:transformer>
<int:filter id="identifyIt" input-channel="filterIt"
output-channel="transformItC" method="isItOurs">
<bean class="com.db.xyz.service.endpoint.ItFilterEndpoint"/>
</int:filter>
<int:transformer id="transformIt"
input-channel="transformItC"
method="transform" output-channel="persistItC">
<bean class="com.db.xzy.service.endpoint.ItTransformEndpoint"/>
</int:transformer>
<int:service-activator id="persistIt" method="publish"
input-channel="persistItC">
<bean class="com.db.xyz.ilr.service.endpoint.PersistEndpoint"/>
</int:service-activator>
Along with error channel definitions, etc. The consumerConnectionFactory is just a class that selects a Solace JMS, MQ Series, or Active MQ factory based on properties.
I am getting the string messages in a Queue. I am consuming it and doing further processing using the below channel processMessage.
Now I have a requirement where I need to send this messages to another queue based on the selector expression mentioned below.
At the same time I need to make sure that the I am not missing any records in processMessage channel. Assuming that the below configuration is good to go?
or is there any better way to handle this situtaion:
<int:recipient-list-router id="router" input-channel="incomingMsg">
<int:recipient channel="publishToQueue" selector-expression="payload.contains('test trade') "/>
<int:recipient channel="processMessage"/>
</int:recipient-list-router>
<task:executor id="executor" pool-size="10" />
<int:publish-subscribe-channel id="publishToQueue" task-executor="executor"/>
The logical explanation is :
IF payload.contains(test trade)
THEN
PublishToQueue
END IF
ProcessMessage
Your configuration is correct and it is really a classical sample for the Recipient List Router. You send message to the processMessage unconditionally and the same message is sent to the publishToQueue if it fits to the selector expression.
Another way to reach the same is possible via Publish-Subscribe Channel and a Filter in the beginning of the second flow. But I would say your way with the Recipient List Router is really good.
Otherwise it isn't clear what is your question and why you are in doubt to go ahead.
I am using the splitter to split the messages and to pass it to the respective other channels for further processing.
But I wanted to send the splitter output to one channel which will write it into another file. Also wanted to send the splitter output to another channel which will perform some task.
I am able to do the same by using the below, but it doesn't seems to be working if failed to process any of the split record in channel2. Its stopping the process and not writing the remaining records in channel1.
<int:splitter input-channel="inputchannel" ref="splitter" method="doSplit" output-channel="routingChannel"/>
<int-recipient-list-router id="customRouter" input-channel="routingChannel"
<int:recipient channel="channel1"/> <!--Write to file-->
<int:recipient channel="channel2"/> <!-- logic to process -->
</int:reciepient-list-router>
Is there any other way I can pass it to the separate channels independently.
The recipient-list-router has an option like:
/**
* Specify whether send failures for one or more of the recipients should be ignored. By default this is
* <code>false</code> meaning that an Exception will be thrown whenever a send fails. To override this and suppress
* Exceptions, set the value to <code>true</code>.
* #param ignoreSendFailures true to ignore send failures.
*/
public void setIgnoreSendFailures(boolean ignoreSendFailures) {
Or if you wish for XML configuration:
<xsd:attribute name="ignore-send-failures">
<xsd:annotation>
<xsd:documentation><![CDATA[
If set to "true", failures to send to a message channel will
be ignored. If set to "false", a MessageDeliveryException will be
thrown instead, and if the router resolves more than one channel,
any subsequent channels will not receive the message.
Please be aware that when using direct channels (single threaded),
send-failures can be caused by exceptions thrown by components
much further down-stream.
This attribute defaults to false.
]]></xsd:documentation>
</xsd:annotation>
<xsd:simpleType>
<xsd:union memberTypes="xsd:boolean xsd:string" />
</xsd:simpleType>
</xsd:attribute>
Hey so i'm using spring integration's jms:outbound-channel-adapterand need to set a priority on a message before i push it through to the messaging system.
Now in plain JMS i had two ways of doing it.
Either set the priority on the MessageProducer:
this.producer.setPriority(i);
Or on the send method itself:
channel.send(message, DeliveryMode.PERSISTENT, 5, 1000);
Neither of these options are available for me anymore since the channel adapter abstracts me away from these details.
Setting the priority on the message itself works only in the in memory channels of spring integration and loses effect soon as i put it into an actual queue. And turns out setting the priority on the message isn't an option at all: JMS message priority not working on Message
There's an attribute on the channel adapter where i can set the priority, but this is static.
<jms:outbound-channel-adapter id="101Out"
channel="101MessageChannel"
connection-factory="101Factory"
destination="QUEUE_NAME"
priority="1" />
The max i can do i read it from a property file. (Or so i think. I'm not sure). I can use the destination-expression attribute to inspect the incoming message and dynamically route it to different destinations, but there's no priority-expression counter part for me to do the same with the priority.
I have a work around of sorts, but it's not a very good one:
<jms:outbound-channel-adapter id="101HighPriorityOut"
channel="101HighPriorityChannel"
connection-factory="101Factory"
destination-expression="headers.QUEUE_NAME"
priority="1"
explicit-qos-enabled="true" />
<jms:outbound-channel-adapter id="101LowPriorityOut"
channel="101LowPriorityChannel"
connection-factory="101Factory"
destination-expression="headers.QUEUE_NAME"
priority="0"
explicit-qos-enabled="true" />
I just route the messages to the appropriate outbound adapter once i determine what the priority needs to be. But if the number of priorities increases i'll be in trouble. Even if it doesn't, having two outbound adapters instead of one just coz i couldn't dynamically assign a priority is kinda clumsy i thought.
Appreciate the help :-)
Oh and i'm using Websphere MQ as my message broker. I don't know if this has anything to do with the message broker though.
Simply set the priority header in the message...
<int:header-enricher ...>
<int:priority value="2" />
</int:heaer-enricher>
The priority in the adapter configuration is a default which is used when there's no priority header (you can use a property placeholder to set it from a properties file).
Or, use an expression...
<int:header-enricher ...>
<int:priority expression="payload.foo == 'bar' ? 1 : 2" />
</int:heaer-enricher>
<int:header-enricher ...>
<int:priority expression="payload.priority" />
</int:heaer-enricher>
<int:header-enricher ...>
<int:priority expression="#someBean.calculatePriority(payload)" />
</int:heaer-enricher>
I have a situation I don't know how to manage it.
The flow I neeed is the following one:
The first service makes its job and creates a message that needs to be retrieved in the chain's end.
When the first service finishes, I need to invoke a push notification server via a new service with a particular message but with some info related to the one created in step 1.
Finally, I the push notification has been sent successfully, I have to retrieve the message created in step 1.
The question is, how can I keep message created in step 1 when the outbound-gateway calling was produced and retrieved me the message from notification push server?
<int:chain input-channel="v1.inputChannel.input" output-channel="v1.inputChannel.output" send-timeout="50000">
<int:header-enricher>
<int:error-channel ref="v1.inputChannel.error" />
</int:header-enricher>
<int:service-activator ref="v1.input.service" method="methodName"/>
<int:service-activator ref="v1.notificationPusher.service" method="pushNotification"/>
<int-http:outbound-gateway url="http://example.com/api/elements/:element_id/objects" http-method="POST">
<int-http:uri-variable name="element_id" expression="#pathVariables.elementId"/>
</int-http:outbound-gateway>
<!-- here the transformer needs to get the messsage from v1.input.service -->
<int:object-to-json-transformer/>
</int:chain>
After some researching in 'Spring Integration in Action' I think the best option for resolving this situation is to use the wire tap pattern, having the push notification service as secondary flow.
Here you can see the book's example where, If I have understood it in a correct way, the auditChannel acts as a secondary flow separated from the main one.
<channel id="debitChannel">
<interceptors>
<wire-tap channel="auditChannel"/>
</interceptors>
</channel>
<service-activator input-channel="debitChannel" method="process">
<beans:bean class="siia.monitoring.wiretap.DebitService"/>
</service-activator>
<filter input-channel="auditChannel" expression="payload.amount > 10000" output-channel="logger"/>
<logging-channel-adapter id="logger" expression="'auditing debit: ' + payload"/>
I think you could achieve this with something like that:
Dupĺicate your message before send it to the outbound
Send one to the outbound and the another one to some channel
Use an Aggregator with the same timeout of your outbound to "join" them, but you could, in fact, only pass the message from the first step through (I haven't tested it)
Send it to the json transformer
To use this approach, I believe, you have to put the outbound and the json-transformer outside the chain as well as your logic to duplicate the message.
I can think of one way, which is to squirrel away the payload into a header after step 1, then retrieve and repopulate the payload right before you invoke step 3:
<int:service-activator ref="v1.input.service" method="methodName"/>
<int:header-enricher>
<int:header name="originalpayload" expression="payload"/>
</int:header-enricher>
<int:service-activator ref="v1.notificationPusher.service" method="pushNotification"/>
<int:enricher expression="headers.originalpayload"/>
<int-http:outbound-gateway url="http://xxx.com/api/elements/:element_id/objects" http-method="POST">
<int-http:uri-variable name="element_id" expression="#pathVariables.elementId"/>
</int-http:outbound-gateway>