We are consuming messages from a HiveMQ MQTT Broker and process the data with spring integration. As the last processing step it is common for us to execute an update/insert statement on a relational database using an int-jpa:outbound-channel-adapter.
Let's consider a situation where the database connection is lost but mqtt messages are still consumed by the inbound mqtt adataper. Currently we lose these messages as we are not handling the failed database operations.
How should we handle the messages if the database connection is currently not available?
Should we implement a Message Store backed by another highly available database which is persisting messages from spring integration?
Should we implement a retry advice and periodically retry database operations?
As our starting point we tried to implement the retry advice with the following spring-context.xml. As a result the message was retried once and then an exception occured as it is not able to perform the database operation. No further retry was executed after the exception occurred.
<?xml version="1.0" encoding="UTF-8"?>
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int-jpa="http://www.springframework.org/schema/integration/jpa"
xmlns:task="http://www.springframework.org/schema/task"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/jpa http://www.springframework.org/schema/integration/jpa/spring-integration-jpa.xsd
http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd">
<int:channel id="mqttInputChannel" />
<!-- MQTT INBOUND ADAPTER IS CONFIGURED IN JAVA -->
<int-jpa:outbound-channel-adapter
channel="mqttInputChannel"
flush-size=""
entity-class="com.iot.db.DefaultBean"
persist-mode="PERSIST"
entity-manager-factory="entityManagerFactory">
<int-jpa:transactional transaction-manager="transactionManager" />
<int-jpa:request-handler-advice-chain>
<ref bean="retryAdvice" />
</int-jpa:request-handler-advice-chain>
</int-jpa:outbound-channel-adapter>
<int:handler-retry-advice id="retryAdvice" />
</beans>
Another approach was to back up the messages via a message store but we weren't able to bind the message store to the jpa adapters transaction. Here is the corresponding spring-context.xml file with the message store:
<?xml version="1.0" encoding="UTF-8"?>
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int-jpa="http://www.springframework.org/schema/integration/jpa"
xmlns:task="http://www.springframework.org/schema/task"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/jpa
http://www.springframework.org/schema/integration/jpa/spring-integration-jpa.xsd
http://www.springframework.org/schema/task
http://www.springframework.org/schema/task/spring-task-3.0.xsd">
<int:channel id="mqttInputChannel">
<int:queue message-store="messageStore" />
</int:channel>
<bean
id="messageStore"
class="org.springframework.integration.jdbc.store.JdbcChannelMessageStore">
<property
name="dataSource"
ref="dataSource" />
<property
name="channelMessageStoreQueryProvider"
ref="queryProvider" />
</bean>
<bean
id="queryProvider"
class="org.springframework.integration.jdbc.store.channel.H2ChannelMessageStoreQueryProvider" />
<!-- MQTT INBOUND ADAPTER IS CONFIGURED IN JAVA -->
<int-jpa:outbound-channel-adapter
channel="mqttInputChannel"
flush-size=""
entity-class="com.iot.db.DefaultBean"
persist-mode="PERSIST"
entity-manager-factory="entityManagerFactory">
<int-jpa:transactional transaction-manager="transactionManager" />
<int-jpa:request-handler-advice-chain>
<ref bean="retryAdvice" />
</int-jpa:request-handler-advice-chain>
</int-jpa:outbound-channel-adapter>
<int:handler-retry-advice id="retryAdvice" />
</beans>
Are there any best practices or well known pattern how to handle this situation?
thanks in advance
Related
I have a simple tcp client using spring integration that takes in a message, and writes it to file.
I want the client to connect by default to thing.input.primary.host, but in the event of a connection failure, I want it to reconnect to thing.input.secondary.host, and then send the message on like normal; is there support in spring integration for this behavior?
Edit: for clarity. What I'd like to do, is to have it switch from thingClient to thingClientSecondary, if thingClient fails.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-ip="http://www.springframework.org/schema/integration/ip"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xmlns="http://www.springframework.org/schema/beans"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/ip http://www.springframework.org/schema/integration/ip/spring-integration-ip.xsd
http://www.springframework.org/schema/integration/file http://www.springframework.org/schema/integration/file/spring-integration-file.xsd">
<int-ip:tcp-inbound-channel-adapter
id="thingInGateway"
connection-factory="thingClientPrimary"
channel="fileOutputChain"
client-mode="true"
auto-startup="true"
retry-interval="1000"/>
<int-ip:tcp-connection-factory
id="thingClientPrimary"
type="client"
host="${thing.input.primary.host}"
port="${thing.input.primary.port}"
deserializer="thingSerializer"
serializer="thingSerializer"
socket-support="thingSocketSupport"
so-keep-alive="true"
using-nio="true"/>
<int-ip:tcp-connection-factory
id="thingClientSecondary"
type="client"
host="${thing.input.secondary.host}"
port="${thing.input.secondary.port}"
deserializer="thingSerializer"
serializer="thingSerializer"
socket-support="thingSocketSupport"
so-keep-alive="true"
using-nio="true"/>
<int:chain input-channel="fileOutputChain" >
<int-file:outbound-channel-adapter directory="${thing.output.directory}"/>
</int:chain>
</beans>
No; simply define a second channel adapter that uses the other factory and the same channel; then any data coming in on either port will go to the same place.
"Failover" is an unusual term on the server side. Normally, clients fail over when a server connection is lost.
On the server side, we are passive and the term "failover" makes no sense here.
I have found several really awkward problems which took me 3hr+ to resolve. I was wondering if anyone could explain me why this is the case. I believe both of them belong to the very much same context so I have two questions. I hope reader will have patience as for me this is both intimidating and interesting behavior of sf.
I only know the error and resolution, but not satisfied until I understand:
Guidance I follow: Have one configuration file - use package scan inside of your root customConfig only(declared via mapping in web.xml), make sure servlet-context.xml only scans controllers' package. All other context files import via import directive at the very beginning of your customConfig.
1.1 Error if you do it other way: Dependency injection(of various components) will dramatically fail with multiple overlapping config package scanning.
1.2 Error if you do it other way: Transaction during service request of transactionManager in context with entityManagerFactory will fail if servlet-context.xml scans the same package. (i.e. same service package as your customConfig scans)
2: LocaleChangeInterceptor can only be declared in servlet-context - won't work in custom root configuration, reason unknown(doesn't work even if adding package scan for controllers package inside customConfig however now funny bit - SessionLocaleResolver on the other hand will work ok if defined in custom config! )
Q1: So it is me to blame who as human was mistakenly adding overlapping context-component package scan or it would be logical for Spring to resolve these collisions? Or they should be resolved but it doesn't work for me for some reason?
I observed fellow developer and smiled when he told it is best not to touch spring configuration nor try to improve it nor try to update it. I smiled, now I clearly don't(I now find myself intimated by this sf config violence), after all of this do you think it is ok to place everything inside just a single configuration file like servlet-context.xml ?
Q2: What is the magic behind LocaleChangeInterceptor, I've spent around 5 hours fixing it until just moved in "try-and-fail" mood it into servlet-context and it worked.
Second is a pure mystery to solve. Nothing fancy inside customConfig
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:p="http://www.springframework.org/schema/p"
xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:mvc="http://www.springframework.org/schema/mvc"
xsi:schemaLocation="http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-3.1.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-3.2.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd">
<!--
<import resource="securityContext.xml"/> -->
<import resource="jpaContext.xml"/>
<context:annotation-config />
<context:component-scan base-package="com.org.app" />
<bean id="messageSource" class="org.springframework.context.support.ReloadableResourceBundleMessageSource">
<property name="basename" value="/WEB-INF/messages" />
<property name="defaultEncoding" value="UTF-8"/>
</bean>
<bean id="localeResolver" class="org.springframework.web.servlet.i18n.SessionLocaleResolver">
<property name="defaultLocale" value="en_GB" />
</bean>
<mvc:interceptors>
<bean
class="org.springframework.web.servlet.i18n.LocaleChangeInterceptor"
p:paramName="lang" />
</mvc:interceptors> ...
After firing ?lang=locale_LOCALE request nothing will happen - no error, no indication , app will load successfully and page will just be reloaded under the same locale.
However placing this interceptor code inside servlet-context.xml below will resolve on locale request successfully.
<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/mvc"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:beans="http://www.springframework.org/schema/beans"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:p="http://www.springframework.org/schema/p"
xsi:schemaLocation="http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">
<!-- DispatcherServlet Context: defines this servlet's request-processing
infrastructure -->
<!-- Enables the Spring MVC #Controller programming model -->
<annotation-driven />
<context:component-scan base-package="com.org.app.controller" />
<!-- Handles HTTP GET requests for /resources/** by efficiently serving
up static resources in the ${webappRoot}/resources directory -->
<resources mapping="/resources/**" location="/resources/" />
<!-- Resolves views selected for rendering by #Controllers to .jsp resources
in the /WEB-INF/views directory -->
<beans:bean
class="org.springframework.web.servlet.view.InternalResourceViewResolver">
<beans:property name="prefix" value="/WEB-INF/views/" />
<beans:property name="suffix" value=".jsp" />
</beans:bean>
<interceptors>
<beans:bean
class="org.springframework.web.servlet.i18n.LocaleChangeInterceptor"
p:paramName="lang" />
</interceptors>
</beans:beans>
Your LocalChangeInterceptor and LocaleResolver must be defined in the servlet-context.xml. <context:annotation-driven /> is already implied by the use of <context:component-scan />
In your root context you are also scanning for #Controller you need to exclude them.
<context:component-scan base-package="com.org.app">
<context:exclude-filter type="annotation" value="org.springframework.stereotype.Controller" />
</context:compoment-scan>
Basically all the web related things (and the things which are used by the DispatcherServlet) must be loaded by the DispatcherServlet. Due to its nature it will only look into its own local application context for the beans it needs instead of in its parents.
The reason for this is that you can have multiple DispatcherServlets each with its own configuration, this would break if it would load the configuration from the root application context.
I am trying to create a persistent event queue using Spring Integration. In a first version I want to use JPA (with a MySQL database), and in the future it is very possible to migrate to a JMS version (I want the transition to be so easy as possible).
I did an example version (without persisting with JPA), that do something similar to what I need:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xsi:schemaLocation="http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/beans">
<int:channel id="customerEventChannel">
<int:queue/>
</int:channel>
<bean id="customerEventService" class="springintegration.CustomerEventService"/>
<int:outbound-channel-adapter ref="customerEventService" method="handleCustomerEvent" channel="customerEventChannel">
<int:poller fixed-delay="3000" max-messages-per-poll="1" />
</int:outbound-channel-adapter>
</beans>
But when I try to modify the example to use JPA inbound and outbound channel adapters, I cannot make it work.
My idea is to read the database every 5 seconds (configurable), and then process the elements in the "queue" recovered from database. On the other hand, I want to insert new elements in the database calling a method in a service class. For doing this, I tried:
<int:channel id="customerEventInputChannel" />
<int:channel id="customerEventOutputChannel" />
<int-jpa:inbound-channel-adapter channel="customerEventInputChannel"
entity-class="springintegration.CustomerEvent"
entity-manager-factory="entityManagerFactory"
auto-startup="true"
expect-single-result="true"
delete-after-poll="true">
<int:poller fixed-rate="5000">
<int:transactional propagation="REQUIRED" transaction-manager="transactionManager"/>
</int:poller>
</int-jpa:inbound-channel-adapter>
<int-jpa:outbound-channel-adapter channel="customerEventOutputChannel"
entity-class="springintegration.CustomerEvent"
entity-manager-factory="entityManagerFactory">
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:outbound-channel-adapter>
But I am missing something because I cannot read or write to database. I tried also with service-activator, bridge, gateway, etc., with same results.
Any help would be appreciated.
I recently found out that queueChannels can become persistent with a Jdbc.
See http://docs.spring.io/spring-integration/reference/html/jdbc.html chapter Backing Message Channels
I have defined DataSource in Context.xml(JNDI) and i would like to use with JPA transaction manager in Spring application Context.xml. I don't want to use JTA Transaction since i am using tomcat server.
Could anyone help me how can i achieve this with an example? I am using #transactional annotations in DAO's and services.
Regards
Vijay
you can Define DataSource: Use JndiObjectFactoryBean class. To define the data source by providing JNDI binding name.
<!– Data Source JNDI –>
<bean id=”dataSource” class=”org.springframework.jndi.JndiObjectFactoryBean”>
<property name=”jndiName”>
<value>jdbc/SampleDS</value>
</property>
</bean>
Here is an example:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:jee="http://www.springframework.org/schema/jee"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/jee
http://www.springframework.org/schema/jee/spring-jee-3.0.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx-3.0.xsd">
<!-- Provides access to the JNDI datasource -->
<jee:jndi-lookup id="dataSource" jndi-name="jdbc/jpetstore"/>
<!-- Defines transaction manager -->
<bean id="txManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource"/>
</bean>
<!-- Enables transactional behavior -->
<tx:annotation-driven />
<!-- other <bean/> definitions here -->
</beans>
Use <jee:jndi-lookup/> to get access to the JNDI data source. Then, pass the obtained datasource reference to an appropriate PlatformTransactionManager such as DataSourceTransactionManager.
Also, <tx:annotation-driven /> should be provided for the #Transactional annotations to be activated.
To obtain a good understanding of Spring transaction management, I would recommend reading chapter 10 of Spring reference available here.
8.2. Fail fast mode
I'm looking for where I might configure this option in Spring, but I'm not sure this is a part of JSR 303 so much as it is a Hibernate configuration for their own validator implementation. This is important because if I have multiple constraint violations I only want the first one "thrown".
Assuming you're using Spring's LocalValidatorFactoryBean to set up your validator, you can specify provider-specific configuration properties within the validationPropertyMap attribute.
The property name of Hibernate Validator's fail-fast attribute is "hibernate.validator.fail_fast", so you would set up your validator like this:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.1.xsd">
<bean name="validator" class="org.springframework.validation.beanvalidation.LocalValidatorFactoryBean">
<property name="validationPropertyMap">
<util:map>
<entry key="hibernate.validator.fail_fast" value="true"/>
</util:map>
</property>
</bean>
</beans>