Spring Batch partition doesnt work composite itemprocessor - spring

I have a Spring Batch partition job. I’m using CompositeProcessor, read data from DB and save these items into an CopyOnWriteArrayList. Because the environment is concurrent but my CopyOnWriteArrayList is being utilized for other threads and mix information, I don’t know why and what I am doing bad, and the output writing them into files for each thread.
public class CustomerItemProcessor implements ItemProcessor<beangenerico,CopyOnWriteArrayList<beanCustomer>> {
private CustomerDAO customerDAO;
private CopyOnWriteArrayList<beanCustomer> listbean;
public CopyOnWriteArrayList<beanCustomer> process(beangenerico rangos) throws Exception {
listbean = customerDAO.getAccAgentes(rangos);
if(listbean != null) {
//customer.setId(currentCustomer.getId());
return listbean;
}else{
return null;
}
}
The configuration of my batch im XML:
<batch:job id="partitionJob" xmlns="http://www.springframework.org/schema/batch">
<batch:step id="masterStep">
<batch:partition step="slave" partitioner="rangePartitioner">
<batch:handler grid-size="10" task-executor="taskExecutor"/>
</batch:partition>
</batch:step>
</batch:job>
<!-- each thread will run this job, with different stepExecutionContext values. -->
<batch:step id="slave" xmlns="http://www.springframework.org/schema/batch">
<batch:tasklet task-executor="taskExecutor" throttle-limit="1">
<batch:chunk reader="beaniniendreader" writer="tempRecordsWriter" processor="completeItemProcessor" commit-interval="1" />
</batch:tasklet>
</batch:step>
<bean id="taskExecutor" class="org.springframework.core.task.SimpleAsyncTaskExecutor" />
<bean id="rangePartitioner" class="my.package.springbatch.RangePartitioner" />
<bean id="beaniniendreader" class="my.package.springbatch.FormiikReader" scope="step"></bean>
<bean id="beanprocessor" class="my.package.springbatch.FormiikProcessor" scope="step">
<property name="accountExecutiveDao" ref="accountExecutiveDao"/>
</bean>
<bean id="beanprocessor2" class="my.package.springbatch.CustomerItemProcessor" scope="step">
<property name="customerDAO" ref="customerAccDao"/>
</bean>
<bean id="completeItemProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
<property name="delegates">
<list>
<ref bean="beanprocessor2"/>
<ref bean="accItemprocessor"/>
<ref bean="beanaccDataItem"/>
</list>
</property>
</bean>
<bean id="tempRecordsWriter" class="my.package.springbatch.ListDelegateWriter" scope="step">
<property name="delegate" ref="flatFileItemWriterPartition"/>
</bean>
<!-- csv file writer -->
<bean id="flatFileItemWriterPartition" class="org.springframework.batch.item.file.FlatFileItemWriter"
scope="step" >
<property name="resource"
value="file:csv/outputs/users.processed#{stepExecutionContext[fromId]}-#{stepExecutionContext[toId]}.csv" />
<property name="appendAllowed" value="false" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value="," />
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="cuenta, name, purchasedPackage" />
</bean>
</property>
</bean>
</property>
</bean>

I going back to take the subject of my code, it is advised to me using Threadlocal for storing thread-specific data whereby it´s works . Here I put my code again. Thanks for your replies.
public class CustomerItemProcessor implements ItemProcessor<beangenerico,ThreadLocal<CopyOnWriteArrayList<beanCustomer>>> {
private CustomerDAO customerDAO;
private ThreadLocal<CopyOnWriteArrayList<beanCustomer>> listbean = new ThreadLocal<CopyOnWriteArrayList<beanCustomer>>();
public ThreadLocal<CopyOnWriteArrayList<beanCustomer>> process(beangenerico rangos) throws Exception {
listbean.set(new CopyOnWriteArrayList<beanCustomer>());
listbean = customerDAO.getAccAgentes(rangos);
if(listbean != null) {
return listbean;
} else {
return null;
}
}
public void setCustomerDAO(CustomerDAO customerDAO) {
this.customerDAO = customerDAO;
}
}

Related

Spring batch jobs are not getting triggered after spring boot upgrade to 2.2

I have the below xml based batch configuration but the jobs are not getting executed. I do not see any error while bringing up the application, I suspect there is a misconfiguration which Im not able to figure out. The below is the configuration:
Application.java look like the below:
#SpringBootApplication(exclude= {ValidationAutoConfiguration.class, WebMvcAutoConfiguration.class })
#ImportResource("classpath:/application.xml")
#ComponentScan(value = "com.abc.xyz.app.configuration") //retrives dataSource
#EnableScheduling
#EnableTransactionManagement
public class Application extends SpringBootServletInitializer{
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
application.xml
<import resource="classpath:/app-batch-context.xml"/>
<import resource="classpath:/job_1.xml"/>
<import resource="classpath:/schdeuler_1.xml"/>
app-batch-context.xml has the below contents.
<bean id="jobRegistry" class="org.springframework.batch.core.configuration.support.MapJobRegistry"/>
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="appJobRepository"/>
</bean>
<bean id="jobExplorer" class="org.springframework.batch.core.explore.support.JobExplorerFactoryBean">
<property name="dataSource" ref="dataSource"/>
</bean>
<bean id="jobOperator" class="org.springframework.batch.core.launch.support.SimpleJobOperator">
<property name="jobExplorer" ref="jobExplorer"/>
<property name="jobLauncher" ref="jobLauncher"/>
<property name="jobRepository" ref="appJobRepository"/>
<property name="jobRegistry" ref="jobRegistry"/>
</bean>
<bean id="appJobRepository" class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="lobHandler" ref="lobHandler"/>
<property name="serializer" ref="xStreamExecutionContextStringSerializer"/>
<property name="transactionManager" ref="transactionManager"/>
</bean>
<bean id="xStreamExecutionContextStringSerializer" class="org.springframework.batch.core.repository.dao.XStreamExecutionContextStringSerializer"/>
<bean id="lobHandler" class="org.springframework.jdbc.support.lob.DefaultLobHandler"/>
<!-- Wrapper bean to hold the applicationcontext, and make it accessible to all the objects-->
<bean id="appContext" class="com.app.core.AppContextInitializer"></bean>
<bean id="stepListner" class="com.app.core.step.APPStepExceutionListner"></bean>
<bean id="jobListner" class="com.app.core.job.APPJobExecutionListner"></bean>
<!-- abstract base Job, all jobs will extend this Job -->
<batch:job id="APPBaseJob" abstract="true" job-repository="appJobRepository">
<batch:listeners>
<batch:listener ref="jobListner"/>
</batch:listeners>
</batch:job>
<!-- abstract base Job, all the steps will extend this step-->
<batch:step id="abstractStep" abstract = "true">
<batch:listeners>
<batch:listener ref ="stepListner"/>
</batch:listeners>
</batch:step>
job_1.xml
<batch:job id="myTestJob" parent="APPBaseJob">
<batch:step id="myTestPreProcessorStep" next="myTestStopProcessingDecider">
<batch:tasklet ref="myTestPreProcessorTasklet"/>
</batch:step>
<batch:decision id="myTestStopProcessingDecider" decider="stopProcessingDecider">
<batch:next on="CONTINUE" to="myTestFileNameValidatorStep" />
<batch:end on="COMPLETED"/>
</batch:decision>
.
.
.
<batch:step id="myTestCustomInputValidatorStep" next="myTestTransformStep">
<batch:tasklet ref="myTestCustomInputValidatorTasklet"/>
</batch:step>
<batch:step id="myTestTransformStep" parent="abstractStep" next="myTestFileTransferStep">
<batch:tasklet>
<batch:chunk reader="myTestFileItemReader" processor="myTestXmlProcessor" writer="myTestItemWriter"
commit-interval="#{stepExecutionContext.get('APPBATCHCONTEXT').appBatch.toBeProcessedSize}"/>
</batch:tasklet>
</batch:step>
<batch:step id="myTestFileTransferStep" next="myTestPostProcessorStep">
<batch:tasklet ref="myTestFileTransferTasklet"/>
</batch:step>
</batch:job>
<!-- File Reader -->
<bean class="com.app.core.reader.v1.APPJaxbFileEntityReader" id="myFileItemReader" scope="step">
<property name="batchContext" value="#{stepExecutionContext.get('APPBATCHCONTEXT')}"></property>
<property name="packageName" value="com.abc.jaxb.xyz.extract"></property>
</bean>
<!-- File Content Writer-->
<bean class="com.app.core.writer.v1.APPXmlJaxbItemWriter" id="myItemWriter" scope="step">
<property name="batchContext" value="#{stepExecutionContext.get('APPBATCHCONTEXT')}"></property>
<property name="packageName" value="com.abc.jaxb.xyz.extract"></property>
</bean>
<bean id="myXmlProcessor" class="com.abc.app.xyz.customprocessor.XMLDocumentProcessor" scope="step">
<property name="batchContext" value="#{stepExecutionContext.get('APPBATCHCONTEXT')}"></property>
<property name="somePropertyDataService" ref="somePropertyDataService"/>
</bean>
<bean id="myFileTransferTasklet" class="com.abc.xyz.customsender.mySenderTasklet">
<property name="stepSkipDeciders" ref="skipStepDeciders"></property>
<property name="router" ref="myRouter"></property>
</bean>
<bean class="com.abc.xyz.app.customsender.ABCRouter"
id="myRouter"></bean>
<bean id="transactionManager" class="org.springframework.transaction.jta.WebSphereUowTransactionManager" />
Scheduler has the below info : Using a custom jobLauncher but in turn uses a org.springframework.batch.core.launch.JobLauncher to run the jobs.
<task:scheduler id="myScheduler" pool-size="1"/>
<bean id="myLauncher" class="com.abc.xyz.job.APPJobLauncher">
<property name="jobCode" value="abc"></property>
</bean>
<task:scheduled-tasks scheduler="abcScheduler">
<task:scheduled ref="myLauncher" method="startJob" cron="${abcFreq}"/>
</task:scheduled-tasks>
Job frequency will be in properties file with a cron expression * */5 * * * *
Please do let me know if any pointer is there that I should check.
Spring boot upgrade from 1.5.x to 2.2.x
Thanks in advance.
try simply with #EnableBatchProcessing in your Application.java
#SpringBootApplication(exclude= {ValidationAutoConfiguration.class, WebMvcAutoConfiguration.class })
#ImportResource("classpath:/application.xml")
#ComponentScan(value = "com.abc.xyz.app.configuration") //retrives dataSource
#EnableScheduling
#EnableBatchProcessing
#EnableTransactionManagement
public class Application extends SpringBootServletInitializer{
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}

Spring Batch not invoking ItemReader after Partitioner execution

I would like configure SpringBatch using partitioner. Parttioner will partition a list of data into multiple chunks and ItemReader will process those data. But in my case after successfully invoking Parttioner it is not invoking ItemReader's read() method. Below are the code snippet and code. Could you please let me know whats wrong?
<batch:job id="cycleJob">
<batch:step id="step1">
<batch:partition partitioner="cyclePartitioner">
<batch:step>
<batch:tasklet task-executor="taskExecutor" throttle-limit="1">
<batch:chunk processor="itemProcessor" reader="itemReader" writer="itemWriter" commit-interval="10">
</batch:chunk>
</batch:tasklet>
</batch:step>
<batch:handler task-executor="taskExecutor" grid-size="${maxThreads}" />
</batch:partition>
</batch:step>
</batch:job>
<bean id="itemProcessor" class="com.navisys.besystem.batch.CycleItemProcessor">
<property name="transactionTemplate" ref="txTemplate"/>
<property name="processorType" value="Batch.executeCycle"/>
</bean>
<bean id="itemWriter" class="com.navisys.besystem.batch.CycleItemWriter" />
<bean id="taskExecutor" class="org.springframework.core.task.SimpleAsyncTaskExecutor">
<constructor-arg type="java.lang.String" value="cycle-" />
<property name="concurrencyLimit" value="${maxThreads}" />
</bean>
<bean id="itemReader" scope="step" class="com.navisys.besystem.batch.CycleItemReader">
<property name="dao" ref="cycledao" />
<property name="cycleDate" value="${cycleDate}" />
<property name="batchIds" value="${batchIds}" />
<property name="switches" value="${switches}" />
<property name="workItemsPerMessage" value="${workItemsPerMessage}" />
<property name="policyMask" value="${policyMask}"></property>
<property name="mainFile" value="${mainFile}" />
<property name="datafileLocation" value="${datafileLocation}"></property>
<property name="data" value="#{stepExecutionContext['data']}" />
</bean>
<bean id="cyclePartitioner" class="com.navisys.besystem.batch.CyclePartitioner">
<property name="dao" ref="cycledao" />
<property name="cycleDate" value="${cycleDate}" />
<property name="batchIds" value="${batchIds}" />
<property name="currentSwitch" value="R"></property>
</bean>
public class CyclePartitioner implements Partitioner {
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
final Map<String, ExecutionContext> contextMap = new HashMap<>();
List<BatchContractIdData> list = initialize();
int partionCount = 0;
int itemsPerList = (null == list || list.isEmpty())?1:(int)(Math.ceil(list.size()/gridSize));
for(List<BatchContractIdData> data:Lists.partition(list, itemsPerList)){
ExecutionContext context = new ExecutionContext();
context.put("data", new ArrayList<BatchContractIdData>(data));
contextMap.put(getPartitionName(++partionCount), context);
}
return contextMap;
}
}

How to create a SocketConfig class in Spring using factory-method?

I am using apache commons httpclient 4.3.x along with spring3. I am trying to wire up a connectionpool with it's associated socketconfig instance.
http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.html
http://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/config/SocketConfig.html?is-external=true
My code looks like this:
<bean id="socketConfig" class="org.apache.http.config.SocketConfig" factory-method="custom" init-method="build">
<property name="soTimeout" value="60000"/>
<property name="soLinger" value="5" />
</bean>
<bean name="poolingHttpConnectionManager" class="org.apache.http.impl.conn.PoolingHttpClientConnectionManager" depends-on="socketConfig">
<property name="maxTotal" value="20" />
<property name="defaultMaxPerRoute" value="20" />
<property name="defaultSocketConfig" ref="socketConfig" />
</bean>
However, this is not working. The instance type that is used to setDefaultSocketConfig() on PoolingHttpClientConnectionManager is of type SocketConfig.Builder instead of SocketConfig.
What I want to have happen is as follows:
SocketConfig config = SocketConfig.custom()
.setSoTimeout(60000)
.setSoLinger(5)
.build()
So, I expect that the socketConfig bean type should be a SocketConfig instance, not a SocketConfig.Builder instance.
As per spring docs, I thought this should work.
http://docs.spring.io/spring/docs/3.0.x/spring-framework-reference/htmlsingle/spring-framework-reference.html#beans-factory-class-static-factory-method
is there anything I am doing wrong? Or is this just not supported in spring?
It turns out that the socketconfig builder instance is not designed to work with spring very well.
I had to use a spring beanfactory implementation to create the instance.
The bean class:
import org.apache.http.config.SocketConfig;
import org.springframework.beans.factory.FactoryBean;
public class SocketConfigFactoryBean implements FactoryBean<SocketConfig> {
int soLinger;
int soTimeout;
public SocketConfig getObject() throws Exception {
return SocketConfig.custom()
.setSoLinger(soLinger)
.setSoTimeout(soTimeout)
.build();
}
public Class<?> getObjectType() {
return SocketConfig.class;
}
public boolean isSingleton() {
return true;
}
public int getSoLinger() {
return soLinger;
}
public void setSoLinger(int soLinger) {
this.soLinger = soLinger;
}
public int getSoTimeout() {
return soTimeout;
}
public void setSoTimeout(int soTimeout) {
this.soTimeout = soTimeout;
}
}
The bean definition
<bean name="poolingHttpConnectionManager" class="org.apache.http.impl.conn.PoolingHttpClientConnectionManager">
<property name="maxTotal" value="20" />
<property name="defaultMaxPerRoute" value="20" />
<property name="defaultSocketConfig">
<bean class="org.apache.http.config.SocketConfig" factory-method="custom" init-method="build">
<bean class="com.ex.spring.beans.factory.SocketConfigFactoryBean">
<property name="soTimeout" value="60000"/>
<property name="soLinger" value="5" />
</bean>
</property>
</bean>
I was able to achieve it by doing the next configuration in spring:
<bean id="socketConfig" class="org.apache.http.config.SocketConfig" factory-method="custom">
<property name="soTimeout" value="1000" />
<property name="soLinger" value="5" />
</bean>
<bean name="poolingHttpConnectionManager" class="org.apache.http.impl.conn.PoolingHttpClientConnectionManager">
<property name="maxTotal" value="20" />
<property name="defaultMaxPerRoute" value="20" />
<property name="defaultSocketConfig">
<bean factory-bean="socketConfig" factory-method="build" />
</property>
</bean>

write method in spring batch not called

I have a problem whereby the write method is not called.
Spring Context
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:batch="http://www.springframework.org/schema/batch" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/batch
http://www.springframework.org/schema/batch/spring-batch-2.2.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
">
<!-- spring batch context -->
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
<property name="transactionManager" ref="transactionManager" />
</bean>
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- Must set this -->
<bean
class="org.springframework.batch.core.configuration.support.JobRegistryBeanPostProcessor">
<property name="jobRegistry" ref="jobRegistry" />
</bean>
<bean id="jobRegistry"
class="org.springframework.batch.core.configuration.support.MapJobRegistry" />
<!-- spring batch context -->
<bean id="report" class="core.Report" scope="prototype" />
<bean id="customWriter" class="spring.jobs.CustomWriter" />
<batch:job id="reportJob">
<batch:step id="step1">
<batch:tasklet>
<batch:chunk reader="cvsFileItemReader" writer="customWriter"
commit-interval="10">
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="cvsFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
<!-- Read a csv file -->
<property name="resource" value="file:input/report.csv" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="names" value="id,impressions" />
</bean>
</property>
<property name="fieldSetMapper">
<bean
class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="report" />
</bean>
</property>
</bean>
</property>
</bean>
<!-- run every 5 seconds -->
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="triggers">
<bean id="cronTrigger"
class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">
<property name="jobDetail" ref="jobDetail" />
<property name="cronExpression" value="*/5 * * * * ?" />
</bean>
</property>
</bean>
<bean id="jobDetail"
class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
<property name="jobClass" value="quartz.JobLauncherDetails" />
<property name="group" value="quartz-batch" />
<property name="durability" value="true" />
<property name="jobDataAsMap">
<map>
<entry key="jobName" value="reportJob" />
<entry key="jobLocator" value-ref="jobRegistry" />
<entry key="jobLauncher" value-ref="jobLauncher" />
<entry key="param1" value="j1" />
<entry key="param2" value="j2" />
</map>
</property>
</bean>
</beans>
CustomWriter.java
import java.util.List;
import org.springframework.batch.item.ItemWriter;
import core.Report;
public class CustomWriter implements ItemWriter<Report> {
#Override
public void write(List<? extends Report> items) throws Exception{
System.out.println("writer..." + items.size());
for(Report item : items){
System.out.println(item);
}
}
}
quartz.JobLauncherDetails class
package quartz;
import java.util.Date;
import java.util.Map;
import java.util.Map.Entry;
import org.quartz.JobExecutionContext;
import org.springframework.batch.core.JobExecutionException;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.JobParametersBuilder;
import org.springframework.batch.core.configuration.JobLocator;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.scheduling.quartz.QuartzJobBean;
public class JobLauncherDetails extends QuartzJobBean {
static final String JOB_NAME = "jobName";
private JobLocator jobLocator;
private JobLauncher jobLauncher;
public JobLauncherDetails() {
System.out.println("JobLauncherDetails initialize...");
}
public void setJobLocator(JobLocator jobLocator) {
this.jobLocator = jobLocator;
}
public void setJobLauncher(JobLauncher jobLauncher) {
this.jobLauncher = jobLauncher;
}
#SuppressWarnings("unchecked")
protected void executeInternal(JobExecutionContext context) {
System.out.println("executeInternal...");
Map<String, Object> jobDataMap = context.getMergedJobDataMap();
String jobName = (String) jobDataMap.get(JOB_NAME);
JobParameters jobParameters = getJobParametersFromJobMap(jobDataMap);
try {
jobLauncher.run(jobLocator.getJob(jobName), jobParameters);
} catch (JobExecutionException e) {
e.printStackTrace();
}
}
// get params from jobDataAsMap property, job-quartz.xml
private JobParameters getJobParametersFromJobMap(
Map<String, Object> jobDataMap) {
JobParametersBuilder builder = new JobParametersBuilder();
for (Entry<String, Object> entry : jobDataMap.entrySet()) {
String key = entry.getKey();
Object value = entry.getValue();
if (value instanceof String && !key.equals(JOB_NAME)) {
builder.addString(key, (String) value);
} else if (value instanceof Float || value instanceof Double) {
builder.addDouble(key, ((Number) value).doubleValue());
} else if (value instanceof Integer || value instanceof Long) {
builder.addLong(key, ((Number) value).longValue());
} else if (value instanceof Date) {
builder.addDate(key, (Date) value);
} else {
// JobDataMap contains values which are not job parameters
// (ignoring)
}
}
// need unique job parameter to rerun the same job
builder.addDate("run date", new Date());
return builder.toJobParameters();
}
}
In this case, the writer.... in write method is not printed.. Couldn't find the problem up till now..
I tried your config with latest version of batch (2.2.1.RELEASE), spring (3.2.3.RELEASE) and quartz (2.2.0) and all works fine substituting file:input/report.csv with file:${user.dir}/src/main/java/input/report.csv.
Are you sure file exists and doesn't throw a
IllegalStateException: Input resource must exist (reader is in
'strict' mode): URL [file:input/report.csv]
You can try use classpath:input/report.csv or build the real path of file (or check if your file is not empty...)

in Spring Batch, can multiple JdbcBatchItemWriters be configured to write in parallel?

In my spring batch job, my item processor splits the object, which the item reader reads, into seven lists of variable lengths. These lists have to be written to seven tables in the DB and any errors (like db rejecting records for any reason) must cause the transaction to rollback on all seven tables.
Currently, I create a wrapped object with these seven lists which are passed to the custom item writer. The writer takes all these items, creates its own seven lists so that it only has seven batched writes (using DAOs based on JdbcTemplate) for a batch of the wrapped objects returned by the item processor.
My writer calls the insert function for each of these tables sequentially which I would like to speed up. I was wondering if I could write the lists, to their respective tables, in parallel so that the overall execution time is the time of the longest write. One requirement I cannot compromise is that this has to be in a single transaction which needs to be rolled back should any of the writers have any exceptions.
here's a simple solution utilizing a TaskExecutor and extending on the org.springframework.batch.item.support.CompositeItemWriter.
package de.incompleteco.spring.batch.item.support;
import java.util.List;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.support.CompositeItemWriter;
import org.springframework.core.task.TaskExecutor;
import org.springframework.util.Assert;
import de.incompleteco.spring.domain.SimpleEntity;
public class ParallelCompositeItemWriter extends CompositeItemWriter<SimpleEntity> {
private List<ItemWriter<? super SimpleEntity>> delegates;
private TaskExecutor taskExecutor;
#Override
public void write(final List<? extends SimpleEntity> item) throws Exception {
for (final ItemWriter<? super SimpleEntity> writer : delegates) {
taskExecutor.execute(new Runnable() {
#Override
public void run() {
try {
writer.write(item);
} catch (Throwable t) {
rethrow(t);
}
}
private void rethrow(Throwable t) {
if (t instanceof RuntimeException) {
throw (RuntimeException) t;
}
else if (t instanceof Error) {
throw (Error) t;
}
throw new IllegalStateException(t);
}
});
}//end for
}
public void setTaskExecutor(TaskExecutor taskExecutor) {
this.taskExecutor = taskExecutor;
}
#Override
public void setDelegates(List<ItemWriter<? super SimpleEntity>> delegates) {
this.delegates = delegates;
super.setDelegates(delegates);
}
#Override
public void afterPropertiesSet() throws Exception {
super.afterPropertiesSet();
Assert.notNull(taskExecutor,"Task executor needs to be set");
}
}
an example configuration would look something like this;
<batch:job id="simpleJob">
<batch:step id="simpleJob.step1">
<batch:tasklet>
<batch:chunk reader="reader" writer="writer" commit-interval="10"/>
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="reader" class="org.springframework.batch.item.support.IteratorItemReader">
<constructor-arg ref="itemList"/>
</bean>
<bean id="writer" class="de.incompleteco.spring.batch.item.support.ParallelCompositeItemWriter">
<property name="delegates" ref="writerDelegates"/>
<property name="taskExecutor" ref="writerTaskExecutor"/>
</bean>
<util:list id="writerDelegates">
<bean class="org.springframework.batch.item.database.JdbcBatchItemWriter">
<property name="dataSource" ref="dataSource1"/>
<property name="sql" value="insert into test_table (idcol,stuff) values (:idCol,:stuff)"/>
<property name="itemSqlParameterSourceProvider">
<bean class="org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider"/>
</property>
</bean>
<bean class="org.springframework.batch.item.database.JdbcBatchItemWriter">
<property name="dataSource" ref="dataSource2"/>
<property name="sql" value="insert into test_table (idcol,stuff) values (:idCol,:stuff)"/>
<property name="itemSqlParameterSourceProvider">
<bean class="org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider"/>
</property>
</bean>
</util:list>
<util:list id="itemList">
<bean class="de.incompleteco.spring.domain.SimpleEntity">
<constructor-arg value="stuff1"/>
</bean>
<bean class="de.incompleteco.spring.domain.SimpleEntity">
<constructor-arg value="stuff2"/>
</bean>
<bean class="de.incompleteco.spring.domain.SimpleEntity">
<constructor-arg value="stuff3"/>
</bean>
</util:list>
<task:executor id="writerTaskExecutor" pool-size="3"/>
<bean id="dataSource1" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init" destroy-method="close">
<property name="className" value="org.h2.jdbcx.JdbcDataSource" />
<property name="uniqueName" value="#{T(System).currentTimeMillis()}" />
<property name="allowLocalTransactions" value="true"/>
<property name="maxPoolSize" value="2" />
<property name="driverProperties">
<props>
<prop key="URL">jdbc:h2:mem:a;DB_CLOSE_DELAY=-1</prop>
</props>
</property>
</bean>
<bean id="dataSource2" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init" destroy-method="close">
<property name="className" value="org.h2.jdbcx.JdbcDataSource" />
<property name="uniqueName" value="#{T(System).currentTimeMillis()}" />
<property name="allowLocalTransactions" value="true"/>
<property name="maxPoolSize" value="2" />
<property name="driverProperties">
<props>
<prop key="URL">jdbc:h2:mem:b;DB_CLOSE_DELAY=-1</prop>
</props>
</property>
</bean>
<jdbc:initialize-database data-source="dataSource1">
<jdbc:script location="classpath:/META-INF/sql/schema-h2.sql"/>
</jdbc:initialize-database>
<jdbc:initialize-database data-source="dataSource2">
<jdbc:script location="classpath:/META-INF/sql/schema-h2.sql"/>
</jdbc:initialize-database>
<!-- XA transaction -->
<bean id="btmConfig" factory-method="getConfiguration" class="bitronix.tm.TransactionManagerServices"/>
<bean id="BitronixTransactionManager" factory-method="getTransactionManager"
class="bitronix.tm.TransactionManagerServices" depends-on="btmConfig" destroy-method="shutdown" />
<bean id="transactionManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="transactionManager" ref="BitronixTransactionManager" />
<property name="userTransaction" ref="BitronixTransactionManager" />
</bean>
this example uses the following;
Bitronix JTA to support transactions across multiple databases
a very simple model of a simple entity into a simple jdbc record
(the stuff in the database is very crude and just an example)

Resources