We have an application where we need to poll a folder and process the files.
We are using clustered environment and on each server files are getting processed using multiple threads. I am using FileInBoundAdapter, poller and Task-executor. But I am seeing the same files are getting processed in different threads. After reading Spring integration documentation it seems ConcurrentMetadataStore/RedisMetadataStore will help to avoid this issue.
I am trying to find out a sample code for RedisMetadataStore API.
Can someone help me sample code or suggest different solution?
Thanks,
Mohan
The sample is pretty simple. You just need to supply the RedisConnectionFactory to the same Redis server for all your cluster nodes and inject FileSystemPersistentAcceptOnceFileListFilter to your <int-file:inbound-channel-adapter>s:
<bean id="redisMetadataStore" class="org.springframework.integration.redis.metadata.RedisMetadataStore">
<constructor-arg ref="redisConnectionFactory"/>
</bean>
<bean id="acceptOnceFilter" class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter">
<constructor-arg ref="redisMetadataStore"/>
<constructor-arg value="the_key"/>
</bean>
<int-file:inbound-channel-adapter filter="acceptOnceFilter"/>
Related
We have a batch job running in Spring XD which reads from MongoDB using the standard MongoItemReader which converts mongo records to our domain model. Up to Spring XD version 1.1.3 this worked fine, however in versions 1.2.0 and 1.2.1 the job is failing with the following error (package name shortened)
java.lang.NoClassDefFoundError: c/s/r/b/b/domain/IndexId
at c.s.r.b.b.domain.IndexId_Instantiator_hxmj4p.newInstance(Unknown Source) ~[na:na]
at
org.springframework.data.convert.BytecodeGeneratingEntityInstantiator$EntityInstantiatorAdapter.createInstance(BytecodeGeneratingEntityInstantiator.java:193) ~
[spring-data-commons-1.10.0.RELEASE.jar:na]
at org.springframework.data.convert.BytecodeGeneratingEntityInstantiator.createInstance
(BytecodeGeneratingEntityInstantiator.java:76) ~[spring-data-commons-1.10.0.RELEASE.jar:na]
at
org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:250) ~[spring-data-mongodb-1.7.0.RELEASE.jar:na]
at
org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:231) ~[spring-data-mongodb-1.7.0.RELEASE.jar:na]
Looking into this I found the threads NoClassDefFoundError when making a query in spring-data-solr within a play framework application, and NoClassDefFoundError after upgrading to 1.7.0.RELEASE which suggest this is due to a change in spring-data-mongo 1.7.0 and the underlying spring-data-commons to change the default entity instantiation technique to improve performance.
Based on the suggested fix in those threads I've modified the mongo template in my job module XML definition as follows and this fixes the problem:
<bean id="mappingConverter" class="org.springframework.data.mongodb.core.convert.MappingMongoConverter">
<constructor-arg ref="dbRefResolver"/>
<constructor-arg ref="mongoMappingContext"/>
<property name="instantiators" ref="entityInstantiators" />
</bean>
<bean id="dbRefResolver" class="org.springframework.data.mongodb.core.convert.DefaultDbRefResolver">
<constructor-arg ref="mongoDbFactory"/>
</bean>
<bean id="mongoMappingContext" class="org.springframework.data.mongodb.core.mapping.MongoMappingContext"/>
<bean id="entityInstantiators" class="org.springframework.data.convert.EntityInstantiators">
<constructor-arg value="#{T(org.springframework.data.convert.ReflectionEntityInstantiator).INSTANCE}"/>
</bean>
<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate">
<constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
<constructor-arg name="mongoConverter" ref="mappingConverter" />
</bean>
However this is verbose and obviously isn't an ideal fix. The problem doesn't show up in our job module integration test so I have a hunch its caused by a combination of the default entity instantiation change and the fact that when a module executes in Spring XD the domain classes will be in the module's class loader and not visible to the spring data mongo classes in XD's main class loader.
So should this be regarded as a bug in Spring XD or Spring Data Mongo? One fix might be an improvement to the Spring Data Mongo mongo:mapping-converter XML configuration to allow forcing the use of the ReflectionEntityInstantiator which would at least reduce the amount of XML needed above. Alternatively maybe Spring XD should handle this scenario automatically?
I don't think there is anything we can do from the XD side since this is a custom job. We have to rely on the spring-data-mongodb functionality.
It looks like you're running into DATACMNS-710, which is fixed in Fowler SR1 (equivalent to Spring Data MongoDB 1.7.1). You might wanna try the just released Gosling release, too.
I have a spring message listener configured with
<bean id="processListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="concurrentConsumers" value="1"/>
<property name="clientId" value="process-execute"/>
<property name="connectionFactory" ref="topicConnectionFactory"/>
<property name="destination" ref="processExecuteQueue"/>
<property name="messageListener" ref="processExecuteListener"/>
</bean>
This is running on a cluster with 2 nodes. I see that it's creating 1 consumer per node rather than 1 per cluster. They're both configured with the above xml so they have the same clientId. Yet, when 2 notifications are posted to the queue, both of the listeners are running, each gets a notification, and both execute in parallel. This is a problem because the notifications need to be handled sequentially.
I can't seem to find out how to make it have only one message listener per cluster rather than per node.
I solved the problem by having the jms queue block the next consumer until the previous returned. This is a feature in the weblogic server I'm using called Unit of Order. The documentation says you just need to enable it on the queue (I used hash). However, I found that I needed to enable it on the connection factory as well and set a default name. Now I see an MDP per node but 1 waits for 2 to complete before processing and vice versa. Not the solution I intended but it's working nontheless. While oracle specific, it's actually slightly better than a single MDP solution.
Note: I did not set the unit of order name in the spring jmstemplate producer as I do not know if that's possible. I have weblogic setting a default name when none is provided by the producer.
I have developed a small webapp using and SpringMVC(3.1.3.RELEASE) and Hibernate 4.2.0.Final.
I'm trying to convert it to be a multi-tenant application.
Similar topics have been covered in other threads, but I couldn't find a definitive solution to my problem.
What I am trying to achieve is to design a web app which is able to:
Read a datasource configuration at startup (an XML file containing multiple datasource definitions, which is placed outside the WAR file and it's not the application-context or hibernate configuration file)
Create a session factory for each one of them (considering that each datasource is a database with a different schema).
How can i set my session factory scope as session? ( OR Can i reuse the same session factory ?) .
Example:
Url for client a - URL: http://project.com/a/login.html
Url for client b - URL: http://project.com/b/login.html
If client "a" make request,read the datasource configuration file and Create a session factory using that XML file for the client "a".
This same process will be repeating if the client "b" will send a request.
What I am looking, how to implement datasource creation upon customer subscription without editing the Spring configuration file. It needs to be automated.
Here is my code ,that i have done so far.
Please anyone tell me,What modifications i need to be made?
Please give an answer with some example code..I am quite new in spring and hibernate world.
Spring.xml
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close" p:driverClassName="${jdbc.driverClassName}"
p:url="${jdbc.databaseurl}"
p:username="${jdbc.username}" p:password="${jdbc.password}" />
<bean id="sessionFactory"
class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="configLocation">
<value>classpath:hibernate.cfg.xml</value>
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">${jdbc.dialect}</prop>
<prop key="hibernate.show_sql">true</prop>
</props>
</property>
</bean>
<bean id="transactionManager"
class="org.springframework.orm.hibernate4.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory" />
JDBC.properties File
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.dialect=org.hibernate.dialect.MySQLDialect
jdbc.databaseurl=jdbc:mysql://localhost:3306/Logistics
jdbc.username=root
jdbc.password=rot#pspl#12
hibernate.cfg.xml File
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<mapping class="pepper.logis.organizations.model.Organizaions" />
<mapping class="pepper.logis.assets.model.Assets" />
</session-factory>
</hibernate-configuration>
Thanks,
First create a table for Tenant with tenant_id and associate it with all users.Now, you can fetch this details while the user logs in and set it in session.
We are using AbstractRoutingDataSource to switch DataSource for every request on Spring Boot. I think it is Hot Swapable targets/datasource mentioned by #bhantol above.
It solves our problems but I don't think it is sound solution. I guess JNDI could be a better one than AbstractRoutingDataSource.
Wondering what you ended up with.
Here are some ideas for you.
Option 1) Single Application Instance.
It is somewhat ambitious to to this using what you are actually trying to achieve.
The gist is to simply deploy the same exact application with different context root on the same JVM. You can still tune the JVM as a whole like you would have if you had a truely multi-tenant application. But this comes at the expense of duplication of classes, contexts, local caching, start up times etc.
But as of today the Spring Framework 4.0 does not provide much of an multi-tenancy support (other than Hot Swapable targets/datasource) etc. I am looking for a good framework but it may be a wash to move away from Spring at this time for me.
Option 2) Multiple deployments of same application (more practical as of today)
Just have your same exact application deploy to the same application server JVM instance or even different.
If you use the same instance you may now need to bootstrap your app to pickup a DataSource based on what the instance should serve e.g. client=a property would be enough to pickup a **a**DataSource" or **b**DataSource I myself ended up going this approach.
If you have a different application server instance you could just configure a different JNDI path and treat things generically. No need for client="a" property because you have liberty to define your datasource differently with the same name.
I'm currently trying to configure hibernate search via spring across 3 machines for the purpose of using a JMS distributed index. Due to the way we deploy our software I have to use the same configuration across all three machines but I need a way to set one of them to be the JMS Master.
Currently hibernate is being configured via Spring using the following bean declaration:
<bean class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean"
id="productSessionFactory">
<property name="dataSource">
<ref local="productDataSource"/>
</property>
<property name="configLocation">
<value>classpath:hibernate.cfg.xml</value>
</property>
<property name="entityInterceptor" ref="builderInterceptor"/>
<property name="eventListeners">
<map key-type="java.lang.String" value-type="java.lang.Object">
<entry key="save" value-ref="saveEventListener"/>
</map>
</property>
</bean>
On one of the three machines I need to set the property hibernate.search.default.directory_provider to filesystem-master and on the other two I need to set it to filesystem-slave.
I have the ability to set flags on the individual machines to identify which machine should be the master but due to all the configuration being XML I dont have any ability to add logic to set the parameters correctly.
Is there an way to set this parameter programmatically while leaving the rest of the configuration alone?
Thanks!
A programmatic way is generally possible, but I am not sure exactly how you do that in Spring. Instead of putting your properties into a config file you would have to build the properties dynamically (or at least partly dynamically) and pass it to AnnotationSessionFactoryBean. If I am not mistaken there are hooks in the Spring SPI which should allow you to do that, eg BeanDefinitionRegistryPostProcessor.
The other approach would be to write your own DirectoryProvider. Have a look at org.hibernate.search.store.impl.FSMasterDirectoryProvider and org.hibernate.search.store.impl.FSSlaveDirectoryProvider and write a provider which can act as slave or master depending on the flag you can read on the machine.
One application I'm working on has several URLs and other information that is instance specific. The first pass uses a typical Spring PropertyPlaceholderConfigurer with a properties file:
<bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="classpath:application.properties"/>
</bean>
The main issue with this is of course the property file is an artifact that must be checked in, and for starting a new instance would require updating that artifact. For a streamline deployment, I would like to have the ApplicationContext bootstrap itself based on database table(s). I have seen solutions like this forum post, does anyone here know of better tools or is this defacto approach to this problem? I would also like to be able to update/reload the settings at runtime using JMX or other facilities, but having to restart the app after changes to the database would still be a better solution to the current one.
The way we did it was to put some configuration information in the environment and then pull the relevant info from there.
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="searchSystemEnvironment" value="true" />
</bean>
If configuration changes then the app will need to be restarted. Can also put all the different configurations into the environment and nest the variables like the following:
<bean id="db" class="org.DataSource"
p:databaseServer="${${MODE}_DBSERVER}"
p:databaseName="${${MODE}_DBNAME}" />
where $MODE = dev, qa, etc.