First want to say that Spring Integration is some great stuff. Hats off to the team for such a solid framework.
Here is my current challenge. My goal is to handle the crosscutting concern of auditing information within a message flow. For example, store the current SI Message in flight, its Message ID, all Payloads contained within the Message and context specific "attributes" that belong to the Message such as "orderId", "customerId", "partId", etc.
You can imagine a synchronous flow that may be running for 6 months where reports might need to be run to determine availability of that particular service (e.g. failed transactions versus successful, transactions with particular "attribute" values, transactions in a certain status, failover occurrences, etc).
If I have the following flow:
gateway->channel1->object-to-json-transformer->channel2->outbound-gateway
gateway has a single method which takes an argument of MyRequest and returns a type of MyResponse. When the flow starts, I can wire-tap channel1 and route all data on that channel to an audit channel, auditChannel, for example.
<int:channel id="auditChannel"/>
<int-jdbc:outbound-channel-adapter data-source="auditDataSource" channel="auditChannel"
query="insert into MESSAGE (PAYLOAD,CREATED_DATE) values (:payload, :createdDate)"
sql-parameter-source-factory="messageSpelSource"/>
<bean id="messageSpelSource"
class="org.springframework.integration.jdbc.ExpressionEvaluatingSqlParameterSourceFactory">
<property name="parameterExpressions">
<map>
<entry key="payload" value="payload.toString()"/>
<entry key="createdDate" value="new java.util.Date()"/>
</map>
</property>
</bean>
The above subflow (from channel1 through auditChannel) does not result in a Message object for the payload Map entry. Instead the type is MyRequest. This makes sense since I would not want to marshall a Message instance outbound, but it still leaves me in a dilemma of not having access to the Message envelope for auditing purposes.
If my intention is to provide a generic auditing facility that persists on demand to a common integration database schema (e.g. to a MESSAGE (message_id, correlation_id, payload, timestamp) table and MESSAGE_ATTRIBUTE (attribute_id, message_id, name, value) table), how can I ensure that I always have access to the core Message instance whenever I wire-tap a channel within the flow?
This use case is something that I have had to deal with many years ago with a custom integration framework so I know it is a valid concern.
I hope my request is not too far fetched. Perhaps there is a simple way to handle this and I am just not seeing it.
It's not entirely clear what you consider to be the problem; you can add more parameters, such as...
<entry key="timestamp" value="headers['timestamp']"/>
...what am I missing in your question?
The whole message is available using "#this".
Related
I have a workflow whose message payload (MasterObj) is being enriched several times. During the 2nd enrichment () an UnknownHostException was thrown by an outbound gateway. My error channel on the enricher is called but the message the error-channel receives is an exception, and the failed msg in that exception is no longer my MasterObj (original payload) but it is now the object gotten from request-payload-expression on the enricher.
The enricher calls an outbound-gateway and business-wise this is optional. I just want to continue my workflow with the payload that I've been enriching. The docs say that the error-channel on the enricher can be used to provide an alternate object (to what the enricher's request-channel would return) but even when I return an object from the enricher's error-channel, it still takes me to the workflow's overall error channel.
How do I trap errors from enricher's + outbound-gateways, and continue processing my workflow with the same payload I've been working on?
Is trying to maintain a single payload object for the entire workflow the right strategy? I need to be able to access it whenever I need.
I was thinking of using a bean scoped to the session where I store the payload but that seems to defeat the purpose of SI, no?
Thanks.
Well, if you worry about your MasterObj in the error-channel flow, don't use that request-payload-expression and let the original payload go to the enricher's sub-flow.
You always can use in that flow a simple <transformer expression="">.
On the other hand, you're right: it isn't good strategy to support single object through the flow. You carry messages via channel and it isn't good to be tied on each step. The Spring Integration purpose is to be able to switch from different MessageChannel types at any time with small effort for their producers and consumers. Also you can switch to the distributed mode when consumers and producers are on different machines.
If you still need to enrich the same object several times, consider to write some custom Java code. You can use a #MessagingGateway on the matter to still have a Spring Integration gain.
And right, scope is not good for integration flow, because you can simply switch there to a different channel type and lose a ThreadLocal context.
I am developing a Clustered Web Application with different WARS deployed, so I need session sharing (and not only this). I've started using Ignite as a good platform for Clustered(Replicated) cache server.
The issue I reached is this:
My cache Key is String and Value is a HashMap
CacheConfiguration<Integer, Map<String,String>> cfg = new CacheConfiguration<>("my_cache");
I am using this cache as a WEBSESSION. The issue is where one servlet gets the Map, Put some session specific values, and put it back to Ignite. After the first servlet gets the cache, second one enters and because it finishes after the first one, the second put will kill first one changes.
So my exact question is, what's the pattern to have concurrent map access issue solved is a high efficient way (without whole object locking).
Regards
It sounds a bit weird to me, because this scenario should be only possible when two there are two concurrent requests working with the same session. How is this possible?
But in any case, you can use TRANSACTIONAL cache for web session data. This will guarantee that these two requests will be processed within a lock and the data will be updated atomically.
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="web-sessions-cache"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
</bean>
I have a huge xml that come as an input payload to my Spring integration flow. So I am using claim check in transformer instead of header enricher to retain my payload. I am using an in-memory message store.
Later on in my SI flow, I have a splitter that splits the payload into multiple threads and each thread will invoke different channel based on one of the attribute payload. I am using a router for achieve this. Each flow or each thread uses a claim check out transformer to retrieve the initial payload then us it for building the required response. Each thread will produce a response and I don't have to aggregate them. So I will have multiple responses coming out from my flow which will then be dropped into a queue.
I cannot remove the message during the check out as other thread will also try to check out the same message. What is the best way to remove the message from the message store?
Sample configuration
`<int:chain input-channel="myInputChannel"
output-channel="myOutputchannel">
<int:claim-check-in />
<int:header-enricher>
<int:header name="myClaimCheckID" expression="payload"/>
</int:header-enricher>
</int:chain>`
all the other components in the flow are invoked before the splitter
<int:splitter input-channel="mySplitterChannel" output-channel="myRouterChannel" expression="mySplitExpression">
</int:splitter>
`<int:router input-channel="myRouterChannel" expression="routerExpression"
resolution-required="true">
<int:mapping value="A" channel="aChannel" />
<int:mapping value="B" channel="bChannel" />
<int:mapping value="C" channel="cChannel" />
</int:router>`
Each channel has a claim check out transformer for the initial payload. So how do I make sure the message is removed after all the threads have been processed?
When you know you are done with the message you can simply invoke the message store's remove() method. You could use a service activator with
... expression="#store.remove(headers['myClaimCheckID'])" ...
However, if you are using an in-memory message store there is really no point in using the claim check pattern.
If you simply promote the payload to a header, it will use no more memory than putting it in a store.
Even if it ends up in multiple messages on multiple threads, it makes no difference since they'll all be pointing to the same object on the heap.
Problem
When my web application updates an item in the database, it sends a message containing the item ID via Camel onto an ActiveMQ queue, the consumer of which will get an external service (Solr) updated. The external service reads from the database independently.
What I want is that if the web application sends another message with the same item ID while the old one is still on queue, that the new message be dropped to avoid running the Solr update twice.
After the update request has been processed and the message with that item ID is off the queue, new request with the same ID should again be accepted.
Is there a way to make this work out of the box? I'm really tempted to drop ActiveMQ and simply implement the update request queue as a database table with a unique constraint, ordered by timestamp or a running insert id.
What I tried so far
I've read this and this page on Stackoverflow. These are the solutions mentioned there:
Idempotent consumers in Camel: Here I can specify an expression that defines what constitutes a duplicate, but that would also prevent all future attempts to send the same message, i.e. update the same item. I only want new update requests to be dropped while they are still on queue.
"ActiveMQ already does duplicate checks, look at auditDepth!": Well, this looks like a good start and definitely closest to what I want, but this determines equality based on the Message ID which I cannot set. So either I find a way to make ActiveMQ generate the Message ID for this queue in a certain way or I find a way to make the audit stuff look at my item ID field instead of the Message ID. (One comment in my second link even suggests using "a well defined property you set on the header", but fails to explain how.)
Write a custom plugin that redirects incoming messages to the deadletter queue if they match one that's already on the queue. This seems to be the most complete solution offered so far, but it feels so overkill for what I perceive as a fairly mundane and every-day task.
PS: I found another SO page that asks the same thing without an answer.
What you want is not message broker functionality, repeat after me, "A message broker is not a database, A message broker is not a database", repeat as necessary.
The broker's job is get messages reliably from point A to point B. The client offers some filtering capabilities via message selectors but this is minimal and mainly useful in keeping only specific messages that a single client is interested in from flowing there and not others which some other client might be in charge of processing.
Your use case calls for a more stateful database centric solution as you've described. Creating a broker plugin to walk the Queue to check for a message is reinventing the wheel and prone to error if the Queue depth is large as ActiveMQ might not even page in all the messages for you based on memory constraints.
I'm looking for an easy way on some of my flows to be able to log when some "event" occurs.
In my simple case an "event" might be whenever any message flows down a channel, or whenever a certain # of messages flow down a channel, I'd like to print out some info to a log file.
I know there currently is a logging-channel-adapter but in the case just described I'd need to be able to tailor my own log message and I'd also need to have some sort of counter or metrics keeping track of things (so the expression on the adapter wouldn't suffice since that grants access to the payload but not info about the channel or flow).
I'm aware that Spring Integration already exposes a lot of metrics to JMX via ManagedResources and MetricType and ManagedMetrics.
I've also watched Russell's "Managing and Monitoring Spring Integration Applications" YouTube video several times: https://www.youtube.com/watch?v=TetfR7ULnA8
and I realize that Spring Integration component metrics can be polled via jmx-attribute-polling-channel-adapter
There are certainly many ways to get what I'm after.
A few examples:
ServiceAdapter that has a counter in it that also has a reference to a logger
Hook into the advice-chain of a poller
Poll JMX via jmx-attribute-polling-channel-adapter
It might be useful however to offer a few components that users could put into the middle of a flow that could provide some basic functionality to easily satisfy the use-case I described.
Sample flow might look like:
inbound-channel-adapter -> metric-logging-channel-interceptor -> componentY -> outbound-channel-adapter
Very high-level such a component might look like a hybrid of the logging-channel-adapter and a ChannelInterceptor with a few additional fields:
<int:metric-logging-channel-interceptor>
id=""
order=""
phase=""
auto-startup=""
ref=""
method=""
channel=""
outchannel=""
log-trigger-expression="(SendCount % 10) = 0"
level=""
logger-name=""
log-full-message=""
expression=""
/>
Internally the class implementing that would need to keep a few basic stats, I think the ones exposed on messageChannel would be a good (i.e. SendCount, MaxSendDuration, etc).
The log-trigger-expression and expression attributes would need access to the internal counters as well.
Please let me know if there is something that already does what I'm describing or if I'm overcomplicating this. If it does not currently exist though I think that being able to quickly drop a component into a flow without having to write a custom ServiceActivator just for logging purposes provides benefit.
Interesting question. You can already do something similar with a selective wire-tap...
<si:publish-subscribe-channel id="seconds">
<si:interceptors>
<si:wire-tap channel="thresholdLogger" selector="selector" />
</si:interceptors>
</si:publish-subscribe-channel>
<bean id="selector" class="org.springframework.integration.filter.ExpressionEvaluatingSelector">
<constructor-arg
value="#mbeanServer.getAttribute('org.springframework.integration:type=MessageChannel,name=seconds', 'SendCount') > 5" />
</bean>
<si:logging-channel-adapter id="thresholdLogger" />
There are a couple of things going on here...
The stats are actually held in the MBean for the channel, not the channel itself so the expression has to get the value via the MBean server.
Right now, the wire-tap doesn't support selector-expression, just selector so I had to use a reference to an expression evaluating selector. It would be a useful improvement to support selector-expression directly.
Even though the selector in this example acts on the stats for the tapped channel, it can actually reference any MBean.
I can see some potential improvements here.
Support selector-expression.
Maintain the stats in the channel itself instead of the MBean so we can just use #channelName.sendCount > 5.
Feel free to open JIRA 'improvement' issue(s).
Hope that helps.