Currently I am using JdbcCursorItemReader and FlatFileItemWriter in a job step.
Due to performance issue we have to use stored procedure.
Is there a way to make a call to stored proc in Spring Batch 2.0.. RELEASE?
<bean id="jdbcCursorItemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource"/>
<property name="sql"
value="SELECT X,Y,Z
FROM V_VIEW "/>
<property name="mapper">
<bean class="com.mapping.SomeMapper"/>
</property>
</bean>
<bean class="org.springframework.batch.item.file.FlatFil eItemWriter" id="flatFileItemWriter">
<property name="resource" ref="resource"/>
<property name="fieldSetCreator">
<bean class="org.springframework.batch.item.file.mapping .PassThroughFieldSetMapper"/>
</property>
</bean>
... Other config
how to write a custom database reader wih callable statement...
Sample code is appreciated.. Thank You,.
There is a StoredProceedureItemReader that is built just for this use case. You can read more about it in the documentation here: https://docs.spring.io/spring-batch/apidocs/org/springframework/batch/item/database/StoredProcedureItemReader.html
I could find my answer:
<bean id="jdbcCursorItemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource"/>
<property name="sql" value="Call schema.StoredProcName"/>
<property name="mapper">
<bean class="com.mapping.SomeMapper"/>
</property>
</bean>
Related
I need to create a batch job that reads from the database millions of rows and I am thinking which is the best approach to do it the fastest that can be done; I think that using sql native to read will be the best option but I am also want to know more infos about the spring batch configuration and I don't seem to finding what I am searching for.
I know there are 2 ways of reading data in a Spring batch job using a Reader.
inside a bean eg:
<batch:step id="fetchData" >
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk reader="dataReader" processor="dataProcessor"
writer="dataWriter" commit-interval="100"/>
</batch:tasklet>
<bean id="dataReader"
class="org.springframework.batch.item.database.JdbcPagingItemReader"
scope="step">
<property name="dataSource" ref="dataSourceNative"/>
<property name="pageSize" value="100"/>
<property name="rowMapper" ref="RowsMapper"/>
<property name="queryProvider">
<bean class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
<property name="dataSource" ref="dataSourceNative"/>
<property name="selectClause"
value="SELECT a,b "/>
<property name="fromClause"
value="from table"/>
<property name="whereClause"
value="WHERE a="blabla"/>
</bean>
</property>
</bean>
using a Java Class
<property name="targetObject" ref="myClassDao" />
<property name="targetMethod" value="list" />
<property name="arguments">
<list>
<!-- add arguments list -->
</list>
</property>
</bean>
<bean id="myClassDao" class="path.to.MyClassDAO"
Is it the same thing to do any of this approaches or the 2nd one gives you more liberty?
If the answer is no, which option is better/fastest?
I am thinking if I want to read from the database but using hibernate to do it. Can be implemented with any of these approaches?
I am using a data source defined in tomcat in my spring configuration as shown in the below xml.
It can happen sometimes that this data source may not be defined in the context.xml of tomcat.
In such cases , the context initialization fails since myDS is not found.
Is it possible to configure the datasource as optional so that application initialisation is not impacted ?
There can be a run time error when this data source is accessed , which is acceptable
<bean id="myDataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:comp/env/jdbc/myDS"/>
</bean>
<bean id="myEntityManagerFactory"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="myDataSource" />
<property name="persistenceXmlLocation" value="classpath:META-INF/persistence.xml" />
<property name="packagesToScan" value="com..XX.XX" />
<property name="persistenceUnitName" value="myPU" />
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="showSql" value="#{systemProperties['showSql'] == null ? 'true' : systemProperties['showSql'] }" />
</bean>
</property>
<property name="persistenceUnitPostProcessors">
<list>
<ref bean="wrkflw-punitpostprocessor" />
</list>
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.dialect">#{systemProperties['dbDialect']}</prop>
</props>
</property>
</bean>
Thanks
Muhad
You may check the DelegatingDataSource, you could encapsulate the logic to load the datasource from JNDI within its instantiation. For your application there will be always a DataSource there, but in some cases (whenever its not able to load the DataSource from JNDI) there is no delegation.
I am using Spring Batch 2. version.I have generated the csv file and able to save in csv format on local.
Now I want to generate the same file but it will be stored on SFTP server.
I had gone through some tutorial which generates file on sftp server but they are using spring integration with Spring Batch.
Is it possible to generate the file on SFTP server using Spring Batch only?
Below is itemReader bean defined::
<bean id="itemReader"
class="org.springframework.batch.item.database.JdbcCursorItemReader"
scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql"
value="select u.ID, u.USER_LOGIN, u.PASSWORD, u.AGE from USERS u" />
</property>
<property name="rowMapper">
<bean class="com.example.UserRowMapper" />
</property>
</bean>
ItemWriter Bean::
<bean id="flatFileItemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="file:csv/user.csv" />
<property name="appendAllowed" value="true" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value="," />
<property name="fieldExtractor">
<bean
class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="name,age,id,password"/>
</bean>
</property>
</bean>
You have to customize your writer class
I doing unit testing on spring-hibernate DAOs... configured using
#ContextConfiguration(
locations = {
"classpath:test-applicationContext.xml"
})
but it looks like its transacting with the actual database.
How do I use a temporary datastore with out working on actual database
Define your data store on a separate file and include that xml file with your mail application xml. When testing, include a separate xml file to hold your data store pointing to another database such as hsql. Then this will be the data source referred to by your main applicationContext.xml.
Thanks Guys I used H2 and got it working:
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="org.h2.Driver" />
<property name="url"
value="jdbc:h2:mem:processdb;INIT=RUNSCRIPT FROM 'classpath:create.sql'" />
</bean>
<bean id="sessionFactory"
class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="configLocation">
<value>classpath:hibernate.cfg.xml</value>
</property>
<property name="configurationClass">
<value>org.hibernate.cfg.AnnotationConfiguration</value>
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.H2Dialect</prop>
<prop key="hibernate.show_sql">true</prop>
</props>
</property>
</bean>
<tx:annotation-driven />
<bean id="transactionManager"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<bean id="stateDAO" class="com.tutorial.jquery.dao.impl.StateDAOImpl"></bean>
<bean id="stateService" class="com.tutorial.jquery.service.impl.StateServiceImpl"></bean>
I found this topic How do i exclude everything but text/html from a heritrix crawl?
I have changed bean to this
<property name="shouldProcessRule">
<bean class="org.archive.modules.deciderules.ContentTypeMatchesRegexDecideRule">
<property name="decision" value="ACCEPT" />
<property name="regex" value="^application/pdf.*"/>
</bean>
</property>
</bean>
But heritrix still saves every file to mirror dir.
I believe you are missing a reject rule above your accept rule. I have the following that works:
<property name="shouldProcessRule">
<bean class="org.archive.modules.deciderules.DecideRuleSequence">
<property name="rules">
<list>
<bean class="org.archive.modules.deciderules.RejectDecideRule">
</bean>
<bean class="org.archive.modules.deciderules.ContentTypeMatchesRegexDecideRule">
<property name="decision" value="ACCEPT" />
<property name="regex" value="^application/pdf.*"/>
</bean>
</list>
</property>
</bean>
</property>
This rejects everything, then accepts everything listed in the following rules.