Spring integration channel message count - spring

I have 2 questions. Here they are:
Is there any way to determine the number of messages waiting to be
processed in a spring integration channel while the number of
customers keep increasing along the time?
In the app context, I want to be able to define x instances of
bean y which x and y, both consumes from channel p, programmatically
increase or decrease the consumers based on the load.
There is an example showed in spring 2gx but it used rabbitmq to determine load.

For 1), Spring Integration channels are just beans like any other bean. Assuming that you're using a standard pollable channel, you can autowire it in by name and then get the number of messages waiting:
http://docs.spring.io/spring-integration/api/org/springframework/integration/channel/QueueChannel.html#getQueueSize()
You may also be able to introspect this through JMX if you want.
For 2)... To do that, you'll want to use an AOP proxy on top of an object pool, then wire the proxy into (what I assume is a) Service Activator. Those properties can be externalized with a PropertyPlaceholderConfigurer. So, as an example:
<bean id="propertyPlaceholder"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="systemPropertiesMode" value="2" />
<property name="locations" value="classpath:my_config_params.properties" />
</bean>
<bean id="myBusinessObjectImpl" class="com.mycompany.whatever.impl.MyServiceImpl"
scope="prototype" autowire-candidate="false" />
<bean id="myBusinessObjPool" class="org.springframework.aop.target.CommonsPoolTargetSource">
<property name="targetBeanName" value="myBusinessObjectImpl" />
<!-- THIS IS THE KEY. myconfig.params.poolsize is the name of the property in the my_config_params.properties above. -->
<property name="maxSize" value="${myconfig.params.poolsize}" />
</bean>
<bean id="myBusinessObject" class="org.springframework.aop.framework.ProxyFactoryBean">
<property name="targetSource" ref="myBusinessObjPool" />
</bean>
<int:channel id="myInputChannel">
<int:queue size="500" />
</int:channel>
<int:service-activator inputChannel="myInputChannel" ref="myBusinessObject" method="processMessages">
<int:poller max-messages-per-poll="10" fixed-rate="5000"/>
</int:service>
This also allows you to make the Service Activator stateful.
Obligatory link to object pooling functionality:
http://docs.spring.io/spring/docs/3.0.x/spring-framework-reference/html/aop-api.html#aop-ts-pool

Related

Queue channel after integrator channel not working

I am using Spring Integration for processing file. Here's my configuration
<int:channel id="startChannel">
<int:splitter input-channel="startChannel" output-channel="parseChannel" ref="splitter" method="split" />
<int:channel id="parseChannel">
<int:queue size="50"/>
</int:channel>
<int:service-activator input-channel="parseChannel" output-channel="aggregatChannel" ref="process" method="process" />
<int:channel id="aggregateChannel">
<int:queue size="50"/>
</int:channel>
<int:aggregate input-channel="aggregateChannel" output-channel="postProcessingChannel" ...(other attributes) />
<int:channel id="postProcessingChannel">
<int:queue size="50"/>
</int:channel>
<int:channel id="outboundChannel">
<int:queue size="50"/>
</int:channel>
<int:service-activator input-channel="postProcessingChannel" output-channel="outbound-channel" ... />
<int:outbound-adapter channel="outputChannel" ... />
..global poller
..global taskexecutor of size 40
My configuration works fine till the aggregator. Aggregator is able to put messages at the postProcessingChannel but nobody is reading from the postProcessing Channel.
On a closer look, i observe -
1) post processing channel is filled up with messages to its capacity.
2) threads are blocked in aggregator implementation, because they are not able to put messages in the postPostProcessing Channel.
My question is why no thread is reading from PostProcessingchannel ? From JProfiler I can see, many threads are idle/free not doing anything.
Can someone please help me understand this behavior.
You have a lot of queue channels; it is generally not necessary to make every channel a queue channel.
If you have more configuration, you are probably experiencing thread starvation in the default taskScheduler bean - it only has 10 threads by default.
If you have close to 10 queue channels in your application you will run out of threads because, by default, each queue channel will block a thread for 1 second (receiveTimeout).
Consider reducing the number of queue channels (as I said, it is not generally needed to have so many; you don't need to hand off to another thread between every component).
Or, see Configuring the Task Scheduler to increase the available threads for scheduled tasks.

Consume from channel only if the number of messages reaches a count or the message is in the channel since a while

I have a custom sink module and I would like to consume the messages from the input only if the the number of messages reaches a count or if they are in the channel since some time. In nutshell, I want to do a bulk push.
I have tried aggregating the number of messages after consuming and storing them in an aggregated channel backed by SimpleMessageStore and have MessageGroupStoreReaper checking for messages in the channel.
I am not satisfied with this approach as I am consuming the messages and storing them in an in-memory store, I am aware of the JDBC store as well but I don't want to follow this approach as the message channels in spring XD are backed by redis/mq I would like to consume from the input channel based on my conditions.
My current bean configuration is as shown below:
<int:aggregator id="messageAggregator" ref="messageAggregatorBean"
method="aggregate" input-channel="input" output-channel="aggregatorOutputChannel"
release-strategy="messageReleaseStrategyBean" release-strategy-method="canRelease"
send-partial-result-on-expiry="true" message-store="resultMessageStore">
</int:aggregator>
<int:service-activator id="contributionIndexerService"
ref="contributionIndexerBean" method="bulkIndex" input-channel="aggregatorOutChannel" />
<bean id="resultMessageStore"
class="org.springframework.integration.store.SimpleMessageStore" />
<bean id="resultMessageStoreReaper"
class="org.springframework.integration.store.MessageGroupStoreReaper">
<property name="messageGroupStore" ref="resultMessageStore" />
<property name="timeout" value="60000" />
</bean>
<task:scheduled-tasks>
<task:scheduled ref="resultMessageStoreReaper" method="run"
fixed-rate="10000" />
</task:scheduled-tasks>
Any thoughts or comments?
Thanks in advance.
I'm not sure that you will be able to determine the count of messages in the Broker's queue (Redis/RabbitMQ or even just normal JMS), more over how much they have been there.
You definitely should consume them to do such a logic.
And yes, I think Aggregator can help you. But right: that must be a Persistent Message Store.
for the case
if they are in the channel since some time
The Aggregator suggest an option like group-timeout to release those groups which haven't reached the releaseStrategy condition, but you would like to emit them anyway over some time: http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to

wsgateway failed to send message to channel '' within timeout: -1 in spring integration

I got this error.
org.springframework.messaging.MessageDeliveryException: failed to send message to channel 'org.springframework.integration.handler.MessageHandlerChain$ReplyForwardingMessageChannel#261b0001' within timeout: -1
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:112)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:44)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:94)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:260)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:241)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:205)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:199)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:177)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.handler.MessageHandlerChain.handleMessageInternal(MessageHandlerChain.java:131)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
And my spring integration xml is like this.
<int-ws:inbound-gateway id="asmTestWSGateway" request-channel="webserviceInboundChannel"
reply-channel="outboundStartChannel" marshaller="asmMarshaller"
unmarshaller="asmMarshaller" />
<bean id="asmMarshaller" class="org.springframework.oxm.castor.CastorMarshaller">
<property name="targetClasses" value="com.sds.redca.test.vo.DependencyVO"></property>
<property name="mappingLocation" value="classpath:mapping-asm.xml"></property>
</bean>
<int:chain input-channel="webserviceInboundChannel" output-channel="checkAuthentificationChannel">
<int:service-activator>
<bean class="com.sds.redca.core.endpoint.activator.BuildMsgActivator" />
</int:service-activator>
<int:filter ref="sampleFilter" />
<int:header-enricher>
<int:header name="protocolType" value="#{T(com.sds.redca.cmm.RedCAConstants).WS_PROTOCOL}"/>
</int:header-enricher>
</int:chain>
<int:chain input-channel="checkAuthentificationChannel" output-channel="outboundStartChannel">
<int:service-activator>
<bean class="com.sds.redca.core.endpoint.activator.AuthentificationServiceActivator" />
</int:service-activator>
</int:chain>
Webservice gateway received soap message with marshalled message translated to vo object.
and normalized and checked whether or not it is valid then go to back to the reply channel for returning the client.
I don't know how to solve this. Even if I make logging level debug, i can't find the reason.
Any help would be greatly appreciated!
Show, please, your outboundStartChannel config.
Tipically failed to send message to channel ... within timeout: -1 is happened, when we use QueueChannel and can't overcome its current capacity, becuase there is not enough resources to poll messages from that queue to process.
Actoually I don't see reason for your use-case to use reply-channel on <int-ws:inbound-gateway> at all. Just remove it and output-channel="outboundStartChannel" from the last <chain> as well. And rely on the TemporaryReplyChannel from the MessageHeaders in that case to achieve a short circuit for the entire flow.

Change isolation level with Spring 4.0 & Hibernate 4.2.8

we are developing an app with Spring 4.0.0, Hibernate 4.2.8 and Ms SQL Server 8, which uses a custom sequence backed with a DB table and mapped with a Hibernate VO (CustomSequence)
This sequence is accesed within a service call:
Main service starts its own transaction
Execute code, do some things, queries...
Calls sequence service for a sequence value (SequenceService)
SequenceService starts its own transaction (REQUIRES_NEW)
SequenceService finds object, returns value and saves next value
Main service gets value, sets in a business object and saves (at this point the sequence value is already commited by the inner new transaction)
Exits
Snippet of the service which manages custom sequence:
#Transactional(propagation = Propagation.REQUIRES_NEW, isolation = Isolation.SERIALIZABLE)
#Service("sequenceService")
public class SequenceService implements ISequenceService {
#Autowired
private ISequenceDao sequenceDao;
private Integer getSequence() {
CustomSequence sequenceOut = sequenceDao.find();
final Integer nextVal = sequenceOut.getNextVal();
sequenceOut.setNextVal(nextVal + 1);
sequenceDao.save(sequenceOut);
return nextVal;
}
}
Our problem is serializable attribute is completely ignored so 2 concurrent threads access getSequence method and obtain the same value.
If we check isolation with TransactionSynchronizationManager the value seems correct for serializable (value=8):
...
Integer isolation = TransactionSynchronizationManager.getCurrentTransactionIsolationLevel();
...
Our spring xml file is this one:
<context:annotation-config />
<context:component-scan base-package="dev.app"/>
<tx:annotation-driven />
<bean name="transactionManager" class="org.springframework.orm.hibernate4.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<bean id="dataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:comp/env/jdbc/appDatasource"/>
</bean>
<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean" lazy-init="false" >
<property name="dataSource"> <ref bean="dataSource" /></property>
<property name="packagesToScan" value="dev.app.model"/>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.SQLServerDialect</prop>
<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.format_sql">true</prop>
<!-- Disable LOB creation as connection -->
<prop key="hibernate.temp.use_jdbc_metadata_defaults">false</prop>
</props>
</property>
</bean>
I've checked database serializable capabilities with MS SQL Management Studio with those commands and then execute app code, and it worked (blocked code until studio did commit):
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRAN
UPDATE CUSTOM_SEQUENCE set NEXTVAL = 1000;
WAITFOR DELAY '00:1:00'
COMMIT
¿any clues for what's going on? I've read lot of info on internet but to no avail
lot of thanks in advance!
According to the code of HibernateTransactionManager, this could be because something is setting the prepareConnection flag of the transaction manager to false:
/**
* Set whether to prepare the underlying JDBC Connection of a transactional
* Hibernate Session, that is, whether to apply a transaction-specific
* isolation level and/or the transaction's read-only flag to the underlying
* JDBC Connection.
* <p>Default is "true". If you turn this flag off, the transaction manager
* will not support per-transaction isolation levels anymore. ...
*/
public void setPrepareConnection(boolean prepareConnection) {
this.prepareConnection = prepareConnection;
}
Try and put a breakpoint to see if it's the case. Also this flag is always used together with isSameConnectionForEntireSession(session):
if (this.prepareConnection && isSameConnectionForEntireSession(session)) {
....
}
isSameConnectionForEntireSession says:
/**
* Return whether the given Hibernate Session will always hold the same
* JDBC Connection. This is used to check whether the transaction manager
* can safely prepare and clean up the JDBC Connection used for a transaction.
* <p>The default implementation checks the Session's connection release mode
* to be "on_close".
*/
protected boolean isSameConnectionForEntireSession(Session session) ...
This means a custom isolation level can only be applied if: the flag to do so is enabled at the transaction manager AND there is the guarantee that the same database connection will always be used for the same hibernate session.
If it's not the case, then the transaction manager does not change the isolation settings, because if a session could use multiple sessions for different queries, the transaction manager would not know when the sessions are being sent back to the pool.
This basically means the transaction manager only changes the isolation settings of a database session if it has the guarantee that it can cleanup those same settings before the session is sent to the pool.

Is it possible to create a synchronous Gateway

I'm currently using a spring integration channel through a gateway and see a very strange (and annoying) behaviour. messages are processed extremely late (more than a minute) after they were send.
Since the channel is handled asynchronously by its own thread pool i guess these threads simply aren't active. since sending of these messages is pretty important i would like to force the handling of the message synchronously in the current thread.
is this possible?
My current config:
<int:annotation-config />
<task:executor id="integrationPool" pool-size="0-100" />
<int:poller default="true" task-executor="integrationPool" fixed-rate="50" />
<int:channel id="loggableevents">
<int:queue />
</int:channel>
<int:channel id="outgoingevents">
<int:queue />
<int:interceptors>
<int:wire-tap channel="loggableevents" />
</int:interceptors>
</int:channel>
<int:outbound-channel-adapter channel="outgoingevents" method="handleMessage" ref="momadapter" />
<bean id="momadapter" class="my.project.mom.EventSendingMessageHandler" />
<!-- Logging -->
<int:outbound-channel-adapter channel="loggableevents" ref="loggingadapter" method="handleMessage" />
<bean id="loggingadapter" class="my.project.logging.EventLoggingHandler" />
<int:gateway id="eventgateway" service-interface="my.project.mom.EventGateway" default-request-channel="outgoingevents" />
I use the gateway for convenience, would it be better to access the channel directly?
Just remove the <queue/> element from the outgoingEvents channel and it will all run on the calling thread.
http://static.springsource.org/spring-integration/docs/2.1.2.RELEASE/reference/html/messaging-channels-section.html
If you get this answer and are using the interface based gateway. Using the message channel is mandatory. The solution in that case is, use the direct channel. The following code sample shows how to do this using the java DSL.
#Bean(name = "spa.import.zipcodes")
public MessageChannel queueRecordsChannel() {
return MessageChannels.direct().get();
}
Check this page for more information on the DirectChannel:
http://docs.spring.io/spring-integration/reference/html/messaging-channels-section.html

Resources