Spring Batch - reading multiple PDF files and passing them to ItemProcessor - spring

I would like to read a multiple pdf files and process them one by one.
I use MultiResourceItemReader and a custom delegate:
public class MyItemReader implements ResourceAwareItemReaderItemStream<MyItem> {
private Resource resource;
#Override
public MyItem read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
return null; //create MyItem
}
#Override
public void setResource(Resource resource) {
this.resource = resource;
}
#Override
public void open(ExecutionContext executionContext) throws ItemStreamException {
}
#Override
public void update(ExecutionContext executionContext) throws ItemStreamException {
}
#Override
public void close() throws ItemStreamException {
}
}
The problem I have is that the read method is ivoked infinitly and my ItemProcessor is not invoked.
The resources property is correctly set - files are set.
Could anyone explain me this? Thanks in advance.
I finally decided to use ResourcesItemReader instead of MultiResourceItemReader with custom delegate. This solution is simpler.
<!--suppress SpringBatchModel -->
<batch:job id="my-import">
<batch:step id="myFileStep">
<batch:tasklet>
<batch:chunk reader="resourcesItemReader"
processor="sddeImportProcessor"
writer="sddeImportJpaItemWriter"
commit-interval="${commit.interval:500}"/>
</batch:tasklet>
</batch:step>
<batch:listeners>
<batch:listener ref="sftpImportExecutionListener"/>
<batch:listener ref="longRunningJobExecutionNotificator"/>
<batch:listener ref="exitStatusJobExecutionListener"/>
<batch:listener ref="afterJobExecutionMailSender"/>
</batch:listeners>
</batch:job>
<bean id="sftpImportExecutionListener"
class="my.batches.shared.listener.SftpImportJobListener">
<constructor-arg name="ftsReadService" ref="ftsReadService"/>
<constructor-arg name="ftsWriterService" ref="ftsWriterService"/>
<constructor-arg name="localDir" value="${voe.batch.sdde.unterschriftenblatt.import.local.folder}"/>
<constructor-arg name="remoteDir" value="${voe.batch.sdde.unterschriftenblatt.import.remote.folder}"/>
<constructor-arg name="multipleFilesImport" value="true" />
</bean>
<bean id="resourcesItemReader" class="org.springframework.batch.item.file.ResourcesItemReader" scope="step">
<property name="resources" value="#{jobExecutionContext['import.input.file.path']}"/>
</bean>
<bean id="myImportProcessor" class="my.MyProcessor">
<property name="myUpdateService" ref="defaultUpdateService" />
</bean>
<bean id="myImportJpaItemWriter" class="org.springframework.batch.item.database.JpaItemWriter">
<property name="entityManagerFactory" ref="entityManagerFactory"/>
</bean>

Related

StatefulRetryOperationsInterceptor not working when included with TransactionInterceptor in interceptor chain

I have the below configuration in application context xml file
<bean id="methodMapWithDefaultTxAttributeSource" class="org.springframework.transaction.interceptor.MatchAlwaysTransactionAttributeSource">
<property name="transactionAttribute" value="PROPAGATION_REQUIRES_NEW,timeout_60"/>
</bean>
<bean id="methodMapTxInterceptor"
class="org.springframework.transaction.interceptor.TransactionInterceptor">
<property name="transactionManager" ref="txManager"/>
<property name="transactionAttributeSource" ref="methodMapWithDefaultTxAttributeSource"/>
</bean>
<bean class="org.springframework.aop.framework.autoproxy.BeanNameAutoProxyCreator">
<property name="interceptorNames">
<list>
<idref bean="retryAdvice"/>
<idref bean="methodMapTxInterceptor"/>
</list>
</property>
<property name="beanNames">
<value>service</value>
</property>
</bean>
<bean id="txProxyTemplate"
class="org.springframework.transaction.interceptor.TransactionProxyFactoryBean"
abstract="true">
<property name="transactionManager" ref="txManager" />
<property name="transactionAttributes">
<props>
<prop key="*">PROPAGATION_REQUIRES_NEW,timeout_60</prop>
</props>
</property>
</bean>
<bean id="manager1" class="package2.Manager1">
<constructor-arg ref="dataSource"/>
</bean>
<bean id="manager2" class="package2.Manager2">
<constructor-arg ref="dataSource"/>
</bean>
<bean id="manager1TxProxy" parent="txProxyTemplate">
<property name="proxyTargetClass" value="true" />
<property name="target" ref="manager1" />
</bean>
<bean id="manager2TxProxy" parent="txProxyTemplate">
<property name="proxyTargetClass" value="true" />
<property name="target" ref="manager2"/>
</bean>
<bean id="retryPolicy" class="org.springframework.retry.policy.SimpleRetryPolicy">
<constructor-arg name="maxAttempts" value="3"/>
</bean>
<bean id="retryTemplate" class="org.springframework.retry.support.RetryTemplate">
<property name="retryPolicy" ref="retryPolicy"/>
</bean>
<bean id="rollbackClassifier" class="org.springframework.classify.BinaryExceptionClassifier">
<constructor-arg name="typeMap">
<util:map map-class="java.util.HashMap" key-type="java.lang.Class" value-type="java.lang.Boolean">
<entry key="java.lang.NullPointerException" value="false"/>
</util:map>
</constructor-arg>
<constructor-arg name="defaultValue" value="true"/>
<constructor-arg name="traverseCauses" value="true"/>
</bean>
<bean id="retryAdvice" class="org.springframework.retry.interceptor.StatefulRetryOperationsInterceptor">
<property name="retryOperations" ref="retryTemplate"/>
<property name="rollbackClassifier" ref="rollbackClassifier"/>
<property name="label" value="label"/>
</bean>
<bean id="service" class="package2.Service">
<property name="manager1" ref="manager1"/>
<property name="manager2" ref="manager2TxProxy"/>
</bean>
As you can see i have wrapped a interceptor chain around Service class method. The goal is add retry and transaction facility to all Service class method. I have modified the Service class below method to throw exception whenever it is called
public void executeWithException() {
manager1.execute();
throw new NullPointerException();
//manager2.execute();
}
Now in the first try, the interceptor chain has StatefulRetryOperationsInterceptor and TransactionInterceptor and before calling the Service class method transaction is created. The Service class method throws exception and it will retry.
Now in the second retry, the interceptor chain will have only StatefulRetryOperationsInterceptor and not TransactionInterceptor. I feel this is wrong. Even for second retry a new transaction has to be created. The javadoc says that. But is not happening here. The TransactionInterceptor is skipped.
Am i missing some configuration here.
Please help me out.
Screenshot of call stacktrace on first retry
Screenshot of call stacktrace on second retry
Hi Gary, I tried your example. I created my own transaction manager as shown below
public class MyTransactionManager extends AbstractPlatformTransactionManager {
private int i = 0;
#Override
protected Object doGetTransaction() throws TransactionException {
return new Object();
}
#Override
protected void doBegin(Object transaction, TransactionDefinition definition) throws TransactionException {
System.out.println("Transaction" + i);
i = i + 1;
}
#Override
protected void doCommit(DefaultTransactionStatus status) throws TransactionException {
}
#Override
protected void doRollback(DefaultTransactionStatus status) throws TransactionException {
}
}
Used it in the xml file
<bean id="txManager" class="package2.MyTransactionManager"/>
Below is the console output
Transaction0
Manager1 Execute
Manager1 Execute
Manager1 Execute
Exception in thread "main"
As you see transaction doBegin method is called once printing "Transaction0". This shows new transactions are not created for every retry.
Below is the main method
public class Example2 {
public static void main(String[] args) {
ApplicationContext context = new ClassPathXmlApplicationContext("example2.xml");
Service service1 = (Service)context.getBean("service");
service1.executeWithException();
}
}
When I debugged the code, TransactionInterceptor is in the chain but it is skipped on subsequent retry.
It makes no sense that the interceptor would change between calls; you must be mistaken.
It works fine for me with similar configuration as yours:
#SpringBootApplication
#ImportResource("so70609332-context.xml")
public class So70609332Application {
public static void main(String[] args) {
SpringApplication.run(So70609332Application.class, args);
}
#Bean
TransactionInterceptor txInterceptor(TransactionManager tm) {
return new TransactionInterceptor(tm, new MatchAlwaysTransactionAttributeSource());
}
#Bean
ApplicationRunner runner(Service service, MyTransactionManager tm) {
return args -> {
while (true) {
try {
callIt(service);
}
catch (IllegalStateException e) {
}
catch (Exception e) {
System.out.println(tm.begins);
break;
}
}
};
}
private void callIt(Service nt) {
try {
nt.foo();
}
catch (IllegalStateException e) {
throw e;
}
}
}
class Service {
void foo() {
System.out.println("called: " + TransactionSynchronizationManager.isActualTransactionActive());
throw new IllegalStateException();
}
}
#Component
class MyTransactionManager extends AbstractPlatformTransactionManager {
int begins;
#Override
protected Object doGetTransaction() throws TransactionException {
return new Object();
}
#Override
protected void doBegin(Object transaction, TransactionDefinition definition) throws TransactionException {
this.begins++;
}
#Override
protected void doCommit(DefaultTransactionStatus status) throws TransactionException {
}
#Override
protected void doRollback(DefaultTransactionStatus status) throws TransactionException {
}
}
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans https://www.springframework.org/schema/beans/spring-beans.xsd">
<bean
class="org.springframework.aop.framework.autoproxy.BeanNameAutoProxyCreator">
<property name="interceptorNames">
<list>
<idref bean="retryAdvice" />
<idref bean="txInterceptor" />
</list>
</property>
<property name="beanNames">
<value>service</value>
</property>
</bean>
<bean id="retryTemplate"
class="org.springframework.retry.support.RetryTemplate">
</bean>
<bean id="retryAdvice"
class="org.springframework.retry.interceptor.StatefulRetryOperationsInterceptor">
<property name="retryOperations" ref="retryTemplate" />
<property name="label" value="label" />
</bean>
<bean id="service" class="com.example.demo.Service" />
</beans>
called: true
called: true
called: true
3

Spring Batch-How to process multiple records at the same time in the processor?

I have a file to parse and process records from. It is working fine as line-by-line (parsing one record at a time). My requirement is I've to parse thru multiple line and fetch the required information from each records and then after combining the fetched info from all the records, I call a service to perform business logic. I have to perform this logic inside my Processor class. The data looks like as below example:
001 123456 987654321551580 Wayne DR 1
001 123456 987654321552APT 786 1
001 123456 987654321553LOS ANGELES 1
001 123456 987654321554CA 1
001 123456 98765432155590001 1
The data element available at columns 30-32 is what I am interested to fetch from each record. In the above example, the values 551, 552, 553, 554, 555 in each line respectively. They all come in together in the file. So basically when the current item in my processor parses the first line and finds out that its '551' (means Address Line1 in business code), then I want to fetch the rest of the address that follows this line and save them in one complete address. At the end I want to pass this address to the service class from the processor and then move on to parse the next record available in the file. My problem is that the processor works on line by line for each record so this way I am not able to keep track/association between all these related lines.
Sorry if I am not able to explain my problem in an easier way..I am new to Spring Batch and still learning.
If you know the associated data records will be next to one another in the file (as opposed to spread out randomly), you can leverage the SingleItemPeekableItemReader to associate multiple lines to create one complete object. This older answer has a bit more info.
Example Context File:
<bean id="peekingReader" class="com.package.whatever.YourPeekingReader">
<property name="delegate" ref="flatFileItemReader"/>
</bean>
<bean id="flatFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" value="file://temp/file.txt" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer" ref="yourTokenizer"/>
<property name="fieldSetMapper" ref="yourMapper"/>
</bean>
</property>
</bean>
Example Peeking Reader:
public class YourPeekingReader extends SingleItemPeekableItemReader<YourObject> {
#Override
public YourObject read() {
YourObject item = super.read();
if (item == null) {
return null;
}
while (true) {
YourObject possibleRelatedObject = peek();
if (possibleRelatedObject == null) {
return item;
}
//logic to determine if next line in file relates to same object
boolean matches = false;
if (matches) {
item.addRelatedInfo(super.read());
} else {
return item;
}
}
}
}
#Dean..thanks again. To be more precise with my code, here it is
Customer-record-reader.xml
<batch:job id="myFileReaderJob">
<batch:step id="stepA" next="stepSuccess">
<batch:tasklet>
<batch:chunk reader="myInputReader" processor="myProcessor" writer="myWriter" commit-interval="1"/>
</batch:tasklet>
</batch:step>
<batch:step id="stepSuccess">
<batch:tasklet ref="successTasklet" />
</batch:step>
</batch:job>
<bean id="myInputReader" scope="step" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="lineMapper" ref="myLineMapper" />
</bean>
<bean id="myLineMapper" class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean id="fixedLengthLineTokenizer" class="org.springframework.batch.item.file.transform.FixedLengthTokenizer">
<property name="names" value="custRecord,tranId,partyId,uniquePartyId,deNum,deVal" />
<property name="columns" value="1-75,1-3,6-11,21-29,30-32,33-62" />
<property name="strict" value="false" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="myInputData" />
</bean>
</property>
</bean>
As you can see, I am not using a custom implementation of ItemReader to wrap the FlatFileItemReader. Can you elaborate more in detail, on how to make changes in this code above to implement the SingleItemPeekableItemReader.
Thanks
#Dean
I tried implementing as per your suggestion
Config1.xml
<import resource="classpath*:/META-INF/java-batchlauncher/mainConfig.xml" />
<batch:job id="prT813FileReaderJob">
<batch:step id="stepA" next="stepB">
<batch:tasklet ref="aTasklet" />
</batch:step>
<batch:step id="stepB" next="stepSuccess">
<batch:tasklet>
<batch:chunk reader="prT813MultiReader" processor="participantRecordT813Processor" writer="prT813ItemWriter" commit-interval="1"/>
<batch:listeners>
<batch:listener ref="enabledFeaturesStepListener"/>
</batch:listeners>
<batch:transaction-attributes propagation="NEVER"/>
</batch:tasklet>
</batch:step>
<batch:step id="stepSuccess">
<batch:tasklet ref="successTasklet" />
</batch:step>
</batch:job>
My mainConfig.xml file changes:
<bean id="prT813MultiReader" scope="step" class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="#{jobParameters[INPUT_FILES]}" />
<property name="delegate" ref="prT813InputReader" />
</bean>
<bean id="prT813MultiThreadedReader" scope="step" class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="#{stepExecutionContext[fileName]}" />
<property name="delegate" ref="prT813InputReader" />
</bean>
<bean id="prT813InputReader" scope="step" class="com.fileprocessing.ParticipantRecordT813ItemReader">
<property name="delegate" ref="prT813CustomPeekableItemReader" />
</bean>
<bean id="prT813CustomPeekableItemReader" scope="step" class="org.springframework.batch.item.support.SingleItemPeekableItemReader">
<property name="delegate" ref="participantRecordT813ItemReader" />
</bean>
<bean id="participantRecordT813ItemReader" scope="step" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="lineMapper" ref="prT813LineMapper" />
</bean>
Created a new Reader class:
public class ParticipantRecordT813ItemReader extends SingleItemPeekableItemReader<ParticipantRecordT813InputData> {
private static final String CLASS = "ParticipantRecordT813ItemReader";
#Override
public ParticipantRecordT813InputData read() throws UnexpectedInputException, ParseException, Exception {
ParticipantRecordT813InputData item = super.read();
Log.report(CLASS, "I am in the reader ::::");
if (item != null) {
while (item.getDeNum()=="551") {
Log.report(CLASS, "I am in the reader at DE551::::" + item.getDeNum());
ParticipantRecordT813InputData possibleRelatedObject = peek();
if (possibleRelatedObject == null) {
return item;
}
//logic to determine if next line in file relates to same object
boolean matches = possibleRelatedObject.getDeNum()=="552";
if (matches) {
Log.report(CLASS, "I am in the reader at DE552::::" + possibleRelatedObject.getDeNum());
} else {
return item;
}
}
}
return item;
}
}
I am getting the below exception:
ERROR [main] (AbstractStep.java:225)- Encountered an error executing step stepB in job prT813FileReaderJob
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.prT813MultiReader' defined in URL []: Initialization of bean failed; nested exception is org.springframework.beans.ConversionNotSupportedException: Failed to convert property value of type 'com.sun.proxy.$Proxy10 implementing org.springframework.batch.item.ItemStreamReader,org.springframework.batch.item.PeekableItemReader,java.io.Serializable,org.springframework.aop.scope.ScopedObject,org.springframework.aop.framework.AopInfrastructureBean,org.springframework.aop.SpringProxy,org.springframework.aop.framework.Advised' to required type 'org.springframework.batch.item.file.ResourceAwareItemReaderItemStream' for property 'delegate'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [com.sun.proxy.$Proxy10 implementing org.springframework.batch.item.ItemStreamReader,org.springframework.batch.item.PeekableItemReader,java.io.Serializable,org.springframework.aop.scope.ScopedObject,org.springframework.aop.framework.AopInfrastructureBean,org.springframework.aop.SpringProxy,org.springframework.aop.framework.Advised] to required type [org.springframework.batch.item.file.ResourceAwareItemReaderItemStream] for property 'delegate': no matching editors or conversion strategy found
As you can see that prT813MultiReader and prT813MultiThreadedReader of type MultiResourceItemReader and I delegate them to prT813InputReader of type SingleItemPeekableItemReader.
I tried implementing ResourceAwareItemReaderItemStream in my reader class which get rid of the above exception but then it complaints on ParticipantRecordT813InputData item = super.read(); for nullPointerException.
public class ParticipantRecordT813ItemReader extends SingleItemPeekableItemReader<ParticipantRecordT813InputData> implements ResourceAwareItemReaderItemStream<ParticipantRecordT813InputData> {
private static final String CLASS = "ParticipantRecordT813ItemReader";
SingleItemPeekableItemReader<ParticipantRecordT813InputData> delegate = new SingleItemPeekableItemReader<ParticipantRecordT813InputData>();
#Override
public ParticipantRecordT813InputData read() throws UnexpectedInputException, ParseException, Exception {
ParticipantRecordT813InputData item = super.read();
Log.report(CLASS, "I am in the reader ::::");
if (item != null) {
while (item.getDeNum()=="551") {
Log.report(CLASS, "I am in the reader at DE551::::" + item.getDeNum());
ParticipantRecordT813InputData possibleRelatedObject = peek();
if (possibleRelatedObject == null) {
return item;
}
//logic to determine if next line in file relates to same object
boolean matches = possibleRelatedObject.getDeNum()=="552";
if (matches) {
Log.report(CLASS, "I am in the reader at DE552::::" + possibleRelatedObject.getDeNum());
} else {
return item;
}
}
}
return item;
}
#Override
public void close() throws ItemStreamException {
// TODO Auto-generated method stub
super.close();
}
#Override
public void open(ExecutionContext arg0) throws ItemStreamException {
// TODO Auto-generated method stub
super.open(arg0);
}
#Override
public void update(ExecutionContext arg0) throws ItemStreamException {
// TODO Auto-generated method stub
super.update(arg0);
}
#Override
public void setResource(Resource arg0) {
// TODO Auto-generated method stub
super.setDelegate(delegate);
}
}
Any idea where I am wrong????

Spring Batch- Xml based configuration performance over Java based

I am trying to convert the spring batch configuration from xml based to annotation based.
Below is my xml based configuration.
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- Step will need a transaction manager -->
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="dbMapper" class="org.test.DBValueMapper">
</bean>
<bean id="dbMapperFlatfile" class="org.test.FlatFileRowMapper">
</bean>
<bean id="paramSetter" class="org.test.DBParamSetter">
</bean>
<bean id="dbReader" class="org.test.DBValueReader"
scope="step">
<property name="paramSetter" ref="paramSetter"/>
<property name="verifyCursorPosition" value="false" />
<property name="dataSource" ref="dataSource" />
<property name="sql" value="#{jobParameters['SQL_QUERY']}" />
<property name="rowMapper" ref="dbMapper" />
<property name="fetchSize" value="5000" />
</bean>
<bean id="dbWriterIO" class="org.test.TemplateWritterIO"
scope="step">
<property name="velocityEngine" ref="velocityEngine" />
<!-- <property name="rptConfig" value="#{jobParameters['RPT_CONFIGVAL']}" /> -->
<property name="headerCallback" ref="dbWriterIO" />
<property name="footerCallback" ref="dbWriterIO" />
</bean>
<batch:job id="fileGenJobNio">
<batch:step id="fileGenJobStempNio">
<batch:tasklet>
<batch:chunk reader="dbReader" writer="dbWriterNIO"
commit-interval="5000">
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
Below is the equivalent Java based configuration:
#EnableBatchProcessing
#Import({ServiceConfiguration.class})
public class SRBatchGenerator extends DefaultBatchConfigurer{
#Autowired
private JobBuilderFactory jobBuilders;
#Autowired
private StepBuilderFactory stepBuilders;
#Autowired
private VelocityEngine velocityEngine;
#Autowired
private DBValueMapper mapper;
#Autowired
private DbHelper dbhelper;
#Autowired
private DataSource datasource;
#Bean
public Step step(){
return stepBuilders.get("step")
.chunk(5000)
.reader(reader())
//.processor(processor())
.writer(writer())
//.listener(logProcessListener())
.faultTolerant()
//.skipLimit(10)
//.skip(UnknownGenderException.class)
//.listener(logSkipListener())
.build();
}
#Bean
public Job fileGeneratorJob(){
return jobBuilders.get("fileGeneratorJob")
//.listener(protocolListener())
.start(step())
.build();
}
#Bean
public DBValueMapper mapper(){
return new DBValueMapper();
}
#Bean
#StepScope
public DBValueReader3 reader(){
String query="Select Test1,Test2,test3,test4 from RPt_TEST";
DBValueReader3 dbread = new DBValueReader3();
dbread.setSql(query);
dbread.setRowMapper(mapper);
dbread.setDataSource(datasource);
return dbread;
}
#Bean
#StepScope
public TemplateWritterIO writer(){
TemplateWritterIO writer=new TemplateWritterIO();
writer.setVelocityEngine(velocityEngine);
return writer;
}
#Override
protected JobRepository createJobRepository() throws Exception {
MapJobRepositoryFactoryBean factory =
new MapJobRepositoryFactoryBean();
factory.afterPropertiesSet();
return (JobRepository) factory.getObject();
}
}
When I execute my Job using xml based it took 27sec to write 1 Million record into Flat file.
But to write same 1 million record, Java based job took about 2 hours to write.
I am not sure what I am missing here. Can anyone help me or guide me why it is slow in Java based configuration.

spring batch - how to execute purge tasklet

I have written a simple spring batch tasklet which calls a dao method which in turn does some deletes. But I am not sure what I should be doing to call the job.
public class RemoveSpringBatchHistoryTasklet implements Tasklet {
#Autowired
private SpringBatchDao springBatchDao;
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext)
throws Exception {
contribution.incrementWriteCount(springBatchDao.purge());
return RepeatStatus.FINISHED;
}
}
So far to execute my spring batch jobs I am using quartz triggers with a setup like so. Each job has it's own xml file which has a read and a writer.
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="jobDetails">
<list>
<ref bean="dailyTranCountJobDetail" />
</list>
</property>
<property name="triggers">
<list>
<ref bean="dailyTranCountCronTrigger" />
</list>
</property>
</bean>
<bean id="dailyTranCountCronTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean">
<property name="jobDetail" ref="dailyTranCountJobDetail" />
<property name="cronExpression" value="#{batchProps['cron.dailyTranCounts']}" />
</bean>
<bean id="dailyTranCountJobDetail" class="org.springframework.scheduling.quartz.JobDetailBean">
<property name="jobClass" value="com.myer.reporting.batch.JobLauncherDetails" />
<property name="group" value="quartz-batch" />
<property name="jobDataAsMap">
<map>
<entry key="jobName" value="job-daily-tran-counts" />
<entry key="jobLocator" value-ref="jobRegistry" />
<entry key="jobLauncher" value-ref="jobLauncher" />
</map>
</property>
</bean>
And then here is an example of the job file itself with a reader and a writer.
<job id="job-daily-tran-counts" xmlns="http://www.springframework.org/schema/batch">
<step id="job-daily-tran-counts-step1">
<tasklet transaction-manager="custDbTransactionManager">
<chunk
reader="dailyTranCountJdbcCursorItemReader"
writer="dailyTranCountItemWriter"
commit-interval="1000" />
</tasklet>
</step>
</job>
<bean id="dailyTranCountJdbcCursorItemReader"
class="com.myer.reporting.dao.itemreader.DailyTranCountJdbcCursorItemReader"
scope="step"
parent="abstractEposJdbcDao">
<property name="rowMapper">
<bean class="com.myer.reporting.dao.mapper.DailyTranCountMapper" />
</property>
</bean>
<bean id="dailyTranCountItemWriter"
class="com.myer.reporting.dao.itemwriter.DailyTranCountItemWriter"
parent="abstractCustDbJdbcDao"/>
Obviously for this new job there is no reader or writer. So what it he best/correct way for me to execute my new tasklet?
thanks
I prefer java configuration instead of xml. You can configure your tasklet with the following code:
#Configuration
#EnableBatchProcessing
public class BatchCleanUpJobsConfiguration {
#Bean
public Job batchCleanUpJob(final JobBuilderFactory jobBuilderFactory,
final StepBuilderFactory stepBuilderFactory,
final RemoveSpringBatchHistoryTasklet removeSpringBatchHistoryTasklet) {
return jobBuilderFactory.get("batchCleanUpJob")
.start(stepBuilderFactory.get("batchCleanUpStep")
.tasklet(removeSpringBatchHistoryTasklet)
.build())
.build();
}
#Bean
public RemoveSpringBatchHistoryTasklet batchCleanUpTasklet(final JdbcTemplate jdbcTemplate) {
final var tasklet = new RemoveSpringBatchHistoryTasklet();
tasklet.setJdbcTemplate(jdbcTemplate);
return tasklet;
}
}
To schedule your new job use the following code:
#Component
#RequiredArgsConstructor
public class BatchCleanUpJobsScheduler {
private final Job batchCleanUpJob;
private final JobLauncher launcher;
#Scheduled(cron = "0 0 0 * * MON-FRI")
public void launchBatchCleanupJob()
throws JobParametersInvalidException, JobExecutionAlreadyRunningException,
JobRestartException, JobInstanceAlreadyCompleteException {
launcher.run(
batchCleanUpJob,
new JobParametersBuilder()
.addLong("launchTime", System.currentTimeMillis())
.toJobParameters());
}
}

Spring Batch - Validate Header Lines in input csv file and skip the file if it invalidates

I have a simple job as below:
<batch:step id="step">
<batch:tasklet>
<batch:chunk reader="itemReader" processor="itemProcessor" writer="itemWriter" commit- interval="5000" />
</batch:tasklet>
</batch:step>
itemReader is as below:
<bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader"
scope="step">
<property name="linesToSkip" value="1"></property>
<property name="skippedLinesCallback" ref="skippedLinesCallback" ></property>
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer" ref="lineTokenizer">
<property name="delimiter" value="," />
</bean>
</property>
<property name="fieldSetMapper">
<bean
class="org.springframework.batch.item.file.mapping.PassThroughFieldSetMapper" />
</property>
</bean>
</property>
<property name="resource" value="#{stepExecutionContext['inputKeyName']}" />
</bean>
<bean id"lineTokenizer" class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<bean id="skippedLinesCallback" class="com.test.IteMReaderHeader" >
<property name="lineTokenizer" ref="lineTokenizer">
</bean>
I am setting the "names" of the input fields in "com.test.IteMReaderHeader" class by injecting "lineTokenizer" in it.
I need to validate the header lines which is the 1st line in the input csv file with a fixed header value and if the header line invalidates then in that case I need to fail the step and skip the entire file so that the next file can be used for reading.
Please suggest a suitable way of achieving it.
I would really appreciate your time and valuable inputs.
Thanks !!
Looking code of FlatFileItemReader file stop condition is managed;
with private field boolean noInput
with private function readLine() used in protected doRead()
IMHO the best solution is to throw a runtime exception from your skippedLineCallback and manage error as reader exhaustion condition.
Foe example writing your delegate in this way
class SkippableItemReader<T> implements ItemStreamReader<T> {
private ItemStreamReader<T> flatFileItemReader;
private boolean headerError = false;
void open(ExecutionContext executionContext) throws ItemStreamException {
try {
flatFileItemReader.open(executionContext);
} catch(MyCustomExceptionHeaderErrorException e) {
headerError = true;
}
}
public T read() {
if(headerError)
return null;
return flatFileItemReader.read();
}
// Other functions delegation
}
(you have to register delegate as stream manually,of course)
or extending FlatFileItemReader as
class SkippableItemReader<T> extends FlatFileItemReader<T> {
private boolean headerError = false;
protected void doOpen() throws Exception {
try {
super.doOpen();
} catch(MyCustomExceptionHeaderErrorException e) {
headerError = true;
}
}
protected T doRead() throws Exception {
if(headerError)
return null;
return super.doRead();
}
}
The code has been written directly without test so there can be errors, but I hope you can understand my point.
Hope can solve your problem

Resources