Spring batch ItemProcessor order of processing the items - spring

Here is my spring configuration file.
<batch:job id="empTxnJob">
<batch:step id="stepOne">
<batch:partition partitioner="partitioner" step="worker" handler="partitionHandler" />
</batch:step>
</batch:job>
<bean id="asyncTaskExecutor" class="org.springframework.core.task.SimpleAsyncTaskExecutor" />
<bean id="partitionHandler" class="org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler" scope="step">
<property name="taskExecutor" ref="asyncTaskExecutor" />
<property name="step" ref="worker" />
<property name="gridSize" value="${batch.gridsize}" />
</bean>
<bean id="partitioner" class="com.spring.mybatch.EmpTxnRangePartitioner">
<property name="empTxnDAO" ref="empTxnDAO" />
</bean>
<batch:step id="worker">
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk reader="databaseReader" writer="databaseWriter" commit-interval="25" processor="itemProcessor">
</batch:chunk>
</batch:tasklet>
</batch:step>
<bean name="databaseReader" class="org.springframework.batch.item.database.JdbcCursorItemReader" scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql">
<value>
<![CDATA[
select *
from
emp_txn
where
emp_txn_id >= #{stepExecutionContext['minValue']}
and
emp_txn_id <= #{stepExecutionContext['maxValue']}
]]>
</value>
</property>
<property name="rowMapper">
<bean class="com.spring.mybatch.EmpTxnRowMapper" />
</property>
<property name="verifyCursorPosition" value="false" />
</bean>
<bean id="databaseWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
<property name="dataSource" ref="dataSource" />
<property name="sql">
<value><![CDATA[update emp_txn set txn_status=:txnStatus where emp_txn_id=:empTxnId]]></value>
</property>
<property name="itemSqlParameterSourceProvider">
<bean class="org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider" />
</property>
</bean>
<bean id="itemProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor" scope="step">
<property name="delegates">
<list>
<ref bean="processor1" />
<ref bean="processor2" />
</list>
</property>
</bean>
My custom range partitioner will split it based on primary key of emp_txn records.
Assume that an emp(primary key - emp_id) can have multiple emp_txn(primary key - emp_txn_id) to be processed. With my current setup, Its possible in ItemProcessor(either processor1 or processor 2) that 2 threads can process the emp_txn for same employee(i.e., for same emp_id).
Unfortunately the back end logic that process(in processor2) the emp_txn is not capable of handling transactions for same emp in parallel. Is there a way in spring batch to control the order of such processing?

With the use case you are describing, I think you're partitioning by the wrong thing. I'd partition by emp instead of emp-txn. That would group the emp-txns and you could order them there. It would also prevent the risk of emp-txns from being processed out of order based on which thread gets to it first.
To answer your direct question, no. There is no way to order items going through processors in separate threads. Once you break the step up into partitioning, each partition works independently.

Related

Spring Batch taking too long

i am new spring batch. i recently tried a batch which will read records from file and insert into MariaDB. But for inserting 10k records its taking 2min 30sec.
I know its too much time. Table have only 3 columns without any Keys.
Here is my job-XML
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:batch="http://www.springframework.org/schema/batch" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/batch
http://www.springframework.org/schema/batch/spring-batch-2.2.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
">
<import resource="../../context.xml" />
<import resource="../../database.xml" />
<bean id="itemProcessor" class="com.my.sbatch.processors.CustomItemProcessor" />
<batch:job id="file_to_db">
<batch:step id="step1">
<batch:tasklet transaction-manager="transactionManager" start-limit="100">
<batch:chunk reader="cvsFileItemReader"
writer="databaseItemWriter" commit-interval="10">
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="multiResourceReader"
class=" org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources"
value="file:batch/csv/processing/*.csv" />
<property name="delegate" ref="cvsFileItemReader" />
</bean>
<bean id="mappingBean" class="com.my.sbatch.bean.Batch1Bean"
scope="prototype" />
<bean name="customFieldSetMapper" class="com.my.sbatch.core.CustomFieldSetMapper">
<property name="classObj" ref="mappingBean"/>
</bean>
<bean id="cvsFileItemReader" class="com.my.sbatch.customReader.CustomItemReader" scope="step">
<property name="resource" value="file:#{jobParameters['inputFile']}" />
<property name="lineMapper">
<bean class="com.my.sbatch.core.CustomLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="#{jobParameters['delimiter']}" />
</bean>
</property>
<property name="fieldSetMapper" ref="customFieldSetMapper" />
</bean>
</property>
</bean>
<bean id="databaseItemWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter" scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql">
<value>
<![CDATA[
#{jobParameters['insert_JobQuery']}
]]>
</value>
</property>
<property name="ItemPreparedStatementSetter">
<bean class="com.my.sbatch.core.CustomPreparedStatement" />
</property>
Here is my context.xml
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.2.xsd">
<!-- stored job-meta in memory -->
<!--
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
<property name="transactionManager" ref="transactionManager" />
</bean>
-->
<!-- stored job-meta in database -->
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="transactionManager" ref="transactionManager" />
<property name="databaseType" value="mysql" />
</bean>
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
In com.my.sbatch.core.CustomFieldSetMapper, com.my.sbatch.core.CustomPreparedStatement classes i am using reflections for mapping fields from File -> bean and Bean -> DB(Prepared statement).
Can you please advice me any solution for why it is taking too much time
In this example, your batch process is executed by a single thread. This may be the reason it takes so long. I recommend you use multhreading with TaskExecutor bean.
For example:
<batch:job id="file_to_db">
<batch:step id="step1">
<batch:tasklet task-executor="taskExecutor" transaction-manager="transactionManager" start-limit="100">
<batch:chunk reader="cvsFileItemReader"
writer="databaseItemWriter" commit-interval="10">
</batch:chunk>
</batch:tasklet>
</batch:step>
This is the simplest solution for multithreading but you may have problems in concurrent environments with access to information.
I recommend that you read this information to see the different scalability strategies.
More information about scalability here

How to use ORALCE SQL SEQUENCE in spring batch writer.?

can someone please let me know how to use oracle sequence in spring batch writer?.
I tried using custseq.nextval in the insert statement, but its failing.
<bean id="testSSRReader"
class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="bconnectedDataSource" />
<property name="sql"
value="select CUST_USA_ID , CUST_FIRST_NAME , CUST_LAST_NAME from BL_CUSTOMER fetch first 100 rows only" />
<property name="rowMapper">
<bean class="com.macys.batch.rowmapper.TestSSRRowMapper" />
</property>
</bean>
<bean id="testSSRProcessor" class="com.macys.batch.processor.TestSSRProcessor" />
<bean id="testSSRWriter"
class="org.springframework.batch.item.database.JdbcBatchItemWriter">
<property name="dataSource" ref="ocDataSource" />
<property name="sql">
<value>
<![CDATA[
insert into TESTTABLESSR(CUSTOMER_ID,CUSTOMER_NAME,CITY)
VALUES (custseq.nextval,:firstName,:lastName)
]]>
</value>
</property>
<property name="itemSqlParameterSourceProvider" ref="itemSqlParameterSourceProvider" />
</bean>

JDBCBatchItemWriter not receiving the List for batch update

I'm a new bee to spring batch. My requirement is to fetch records from a DB table, process them(each record can be processed independently so i'm partitioning and using a task executor) and then update the status column in the same table based on processing status.
Simplified version of my code is below.
Item Reader (My custom column partitioner will decide the min & max value below):
<bean name="databaseReader" class="org.springframework.batch.item.database.JdbcCursorItemReader" scope="step">
<property name="dataSource" ref="dataSource"/>
<property name="sql">
<value>
<![CDATA[
select id,user_login,user_pass,age from users where id >= #{stepExecutionContext['minValue']} and id <= #{stepExecutionContext['maxValue']}
]]>
</value>
</property>
<property name="rowMapper">
<bean class="com.springapp.batch.UserRowMapper" />
</property>
<property name="verifyCursorPosition" value="false"/>
</bean>
Item Processor:
<bean id="itemProcessor" class="com.springapp.batch.UserItemProcessor" scope="step"/>
....
public class UserItemProcessor implements ItemProcessor<Users, Users>
{
#Override
public Users process(Users users) throws Exception {
// do some processing here..
//update users status
//users.setStatus(users.getId() + ": Processed by :" + Thread.currentThread().getName() + ": Processed at :" + new GregorianCalendar().getTime().toString());
//System.out.println("Processing user :" + users + " :" +Thread.currentThread().getName());
return users;
}
}
Item Writer:
<bean id="databaseWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
<property name="dataSource" ref="dataSource" />
<property name="sql">
<value>
<![CDATA[
update users set status = :status where id= :id
]]>
</value>
</property>
<property name="itemSqlParameterSourceProvider">
<bean class="org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider" />
</property>
</bean>
Step configuration:
<batch:job id="usersJob">
<batch:step id="stepOne">
<batch:partition step="worker" partitioner="myColumnRangepartitioner" handler="partitionHandler" />
</batch:step>
</batch:job>
<batch:step id="worker" >
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk reader="databaseReader" writer="databaseWriter" commit-interval="5" processor="itemProcessor" />
</batch:tasklet>
</batch:step>
<bean id="asyncTaskExecutor" class="org.springframework.core.task.SimpleAsyncTaskExecutor" />
<bean id="partitionHandler" class="org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler" scope="step">
<property name="taskExecutor" ref="asyncTaskExecutor"/>
<property name="step" ref="worker" />
<property name="gridSize" value="3" />
</bean>
Since i have specified the commit interval as 5 my understanding is that when 5 items are processed by a partition, it will call JDBCItemWriter with a List of 5 Users object to perform a batch JDBC update. However with the current setup, i'm receiving 1 User object at a time during batch update.
Is my understanding above correct or am i missing any step/configuration ?
Note: I'm using HSQL file based database for testing.
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="org.hsqldb.jdbc.JDBCDriver"/>
<property name="url" value="jdbc:hsqldb:file:C://users.txt"/>
<property name="username" value="sa"/>
<property name="password" value=""/>
</bean>

Spring MDP not Consuming Message

I am implementing Spring MDP + JMSTemplate to send and receive the message. The message send mechanism is working fine, however the MDP is not getting invoked. I tried testing the via plain receiver, and was able to retrieve the message, but not via MDP. What could be the problem? I can see the messages getting accumulated in the request queue, but somehow the MDP is not getting trigger. Am I missing anything here in configurations or something else needs to be taken care of?
Here's the Spring Config. The Java class to send and received are pretty much standard ones.
<bean id="cookieRequestListener" class="com.text.jms.mq.mdp.RequestQueueMDP">
<property name="logger" ref="mqLogger" />
<property name="scoringEngine" ref="scoringEngine" />
<property name="mqSender" ref="jmsMQSender" />
</bean>
<bean id="CookieRequestContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="cachedConnectionFactory" />
<property name="destination" ref="jmsRequestQueue" />
<property name="messageListener" ref="cookieRequestListener" />
</bean>
<bean id="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate">
<ref bean="jndiTemplate" />
</property>
<property name="jndiName">
<value>java:/jms/queueCF</value>
</property>
</bean>
<!-- A cached connection to wrap the Queue connection -->
<bean id="cachedConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="jmsConnectionFactory"/>
<property name="sessionCacheSize" value="10" />
</bean>
<!-- jms Request Queue Configuration -->
<bean id="jmsRequestQueue" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate">
<ref bean="jndiTemplate" />
</property>
<property name="jndiName">
<value>java:/jms/cookieReqQ</value>
</property>
</bean>
<!-- jms Response Queue Configuration -->
<bean id="jmsResponseQueue" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate">
<ref bean="jndiTemplate" />
</property>
<property name="jndiName">
<value>java:/jms/cookieResQ</value>
</property>
</bean>
<bean id="jmsJMSTemplate" class="org.springframework.jms.core.JmsTemplate" >
<property name="connectionFactory" ref="cachedConnectionFactory" />
</bean>
<!-- jms MQ Utility -->
<bean id="jmsMQSender" class="com.text.jms.util.MQSender">
<property name="jmsTemplate">
<ref bean="jmsJMSTemplate"></ref>
</property>
<property name="defaultDestination">
<ref bean="jmsRequestQueue" />
</property>
<property name="logger" ref="mqLogger" />
</bean>

Spring + MongoDB + Quartz = OptimisticLockingFailureException

on my Spring app I have a job with following setup:
<!-- Spring Quartz Job -->
<bean id="runMeJob" class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean">
<property name="targetObject" ref="com.pixolut.mrb.ob.ss.SsGateway" />
<property name="targetMethod" value="scheduler" />
</bean>
<bean id="simpleTrigger" class="org.springframework.scheduling.quartz.SimpleTriggerBean">
<property name="jobDetail" ref="runMeJob" />
<property name="repeatInterval" value="5000" />
<property name="startDelay" value="1000" />
</bean>
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="jobDetails">
<list>
<ref bean="runMeJob" />
</list>
</property>
<property name="triggers">
<list>
<ref bean="simpleTrigger" />
</list>
</property>
</bean>
The problem is that when I try to save an object by using the MongoTemplate save function I get OptimisticLockingFailureException
Is it because Quarts doen't support Mongo?
This issue was caused from a null property on my model.

Resources