Spring Batch-How to process multiple records at the same time in the processor? - spring

Question

I have a file to parse and process records from. It is working fine as line-by-line (parsing one record at a time). My requirement is I've to parse thru multiple line and fetch the required information from each records and then after combining the fetched info from all the records, I call a service to perform business logic. I have to perform this logic inside my Processor class. The data looks like as below example:
001 123456 987654321551580 Wayne DR 1
001 123456 987654321552APT 786 1
001 123456 987654321553LOS ANGELES 1
001 123456 987654321554CA 1
001 123456 98765432155590001 1
The data element available at columns 30-32 is what I am interested to fetch from each record. In the above example, the values 551, 552, 553, 554, 555 in each line respectively. They all come in together in the file. So basically when the current item in my processor parses the first line and finds out that its '551' (means Address Line1 in business code), then I want to fetch the rest of the address that follows this line and save them in one complete address. At the end I want to pass this address to the service class from the processor and then move on to parse the next record available in the file. My problem is that the processor works on line by line for each record so this way I am not able to keep track/association between all these related lines.
Sorry if I am not able to explain my problem in an easier way..I am new to Spring Batch and still learning.

Answer 1

If you know the associated data records will be next to one another in the file (as opposed to spread out randomly), you can leverage the SingleItemPeekableItemReader to associate multiple lines to create one complete object. This older answer has a bit more info.
Example Context File:
<bean id="peekingReader" class="com.package.whatever.YourPeekingReader">
<property name="delegate" ref="flatFileItemReader"/>
</bean>
<bean id="flatFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" value="file://temp/file.txt" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer" ref="yourTokenizer"/>
<property name="fieldSetMapper" ref="yourMapper"/>
</bean>
</property>
</bean>
Example Peeking Reader:
public class YourPeekingReader extends SingleItemPeekableItemReader<YourObject> {
#Override
public YourObject read() {
YourObject item = super.read();
if (item == null) {
return null;
}
while (true) {
YourObject possibleRelatedObject = peek();
if (possibleRelatedObject == null) {
return item;
}
//logic to determine if next line in file relates to same object
boolean matches = false;
if (matches) {
item.addRelatedInfo(super.read());
} else {
return item;
}
}
}
}

Answer 2

#Dean..thanks again. To be more precise with my code, here it is
Customer-record-reader.xml
<batch:job id="myFileReaderJob">
<batch:step id="stepA" next="stepSuccess">
<batch:tasklet>
<batch:chunk reader="myInputReader" processor="myProcessor" writer="myWriter" commit-interval="1"/>
</batch:tasklet>
</batch:step>
<batch:step id="stepSuccess">
<batch:tasklet ref="successTasklet" />
</batch:step>
</batch:job>
<bean id="myInputReader" scope="step" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="lineMapper" ref="myLineMapper" />
</bean>
<bean id="myLineMapper" class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean id="fixedLengthLineTokenizer" class="org.springframework.batch.item.file.transform.FixedLengthTokenizer">
<property name="names" value="custRecord,tranId,partyId,uniquePartyId,deNum,deVal" />
<property name="columns" value="1-75,1-3,6-11,21-29,30-32,33-62" />
<property name="strict" value="false" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="myInputData" />
</bean>
</property>
</bean>
As you can see, I am not using a custom implementation of ItemReader to wrap the FlatFileItemReader. Can you elaborate more in detail, on how to make changes in this code above to implement the SingleItemPeekableItemReader.
Thanks

Answer 3

#Dean
I tried implementing as per your suggestion
Config1.xml
<import resource="classpath*:/META-INF/java-batchlauncher/mainConfig.xml" />
<batch:job id="prT813FileReaderJob">
<batch:step id="stepA" next="stepB">
<batch:tasklet ref="aTasklet" />
</batch:step>
<batch:step id="stepB" next="stepSuccess">
<batch:tasklet>
<batch:chunk reader="prT813MultiReader" processor="participantRecordT813Processor" writer="prT813ItemWriter" commit-interval="1"/>
<batch:listeners>
<batch:listener ref="enabledFeaturesStepListener"/>
</batch:listeners>
<batch:transaction-attributes propagation="NEVER"/>
</batch:tasklet>
</batch:step>
<batch:step id="stepSuccess">
<batch:tasklet ref="successTasklet" />
</batch:step>
</batch:job>
My mainConfig.xml file changes:
<bean id="prT813MultiReader" scope="step" class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="#{jobParameters[INPUT_FILES]}" />
<property name="delegate" ref="prT813InputReader" />
</bean>
<bean id="prT813MultiThreadedReader" scope="step" class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="#{stepExecutionContext[fileName]}" />
<property name="delegate" ref="prT813InputReader" />
</bean>
<bean id="prT813InputReader" scope="step" class="com.fileprocessing.ParticipantRecordT813ItemReader">
<property name="delegate" ref="prT813CustomPeekableItemReader" />
</bean>
<bean id="prT813CustomPeekableItemReader" scope="step" class="org.springframework.batch.item.support.SingleItemPeekableItemReader">
<property name="delegate" ref="participantRecordT813ItemReader" />
</bean>
<bean id="participantRecordT813ItemReader" scope="step" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="lineMapper" ref="prT813LineMapper" />
</bean>
Created a new Reader class:
public class ParticipantRecordT813ItemReader extends SingleItemPeekableItemReader<ParticipantRecordT813InputData> {
private static final String CLASS = "ParticipantRecordT813ItemReader";
#Override
public ParticipantRecordT813InputData read() throws UnexpectedInputException, ParseException, Exception {
ParticipantRecordT813InputData item = super.read();
Log.report(CLASS, "I am in the reader ::::");
if (item != null) {
while (item.getDeNum()=="551") {
Log.report(CLASS, "I am in the reader at DE551::::" + item.getDeNum());
ParticipantRecordT813InputData possibleRelatedObject = peek();
if (possibleRelatedObject == null) {
return item;
}
//logic to determine if next line in file relates to same object
boolean matches = possibleRelatedObject.getDeNum()=="552";
if (matches) {
Log.report(CLASS, "I am in the reader at DE552::::" + possibleRelatedObject.getDeNum());
} else {
return item;
}
}
}
return item;
}
}
I am getting the below exception:
ERROR [main] (AbstractStep.java:225)- Encountered an error executing step stepB in job prT813FileReaderJob
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.prT813MultiReader' defined in URL []: Initialization of bean failed; nested exception is org.springframework.beans.ConversionNotSupportedException: Failed to convert property value of type 'com.sun.proxy.$Proxy10 implementing org.springframework.batch.item.ItemStreamReader,org.springframework.batch.item.PeekableItemReader,java.io.Serializable,org.springframework.aop.scope.ScopedObject,org.springframework.aop.framework.AopInfrastructureBean,org.springframework.aop.SpringProxy,org.springframework.aop.framework.Advised' to required type 'org.springframework.batch.item.file.ResourceAwareItemReaderItemStream' for property 'delegate'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [com.sun.proxy.$Proxy10 implementing org.springframework.batch.item.ItemStreamReader,org.springframework.batch.item.PeekableItemReader,java.io.Serializable,org.springframework.aop.scope.ScopedObject,org.springframework.aop.framework.AopInfrastructureBean,org.springframework.aop.SpringProxy,org.springframework.aop.framework.Advised] to required type [org.springframework.batch.item.file.ResourceAwareItemReaderItemStream] for property 'delegate': no matching editors or conversion strategy found
As you can see that prT813MultiReader and prT813MultiThreadedReader of type MultiResourceItemReader and I delegate them to prT813InputReader of type SingleItemPeekableItemReader.
I tried implementing ResourceAwareItemReaderItemStream in my reader class which get rid of the above exception but then it complaints on ParticipantRecordT813InputData item = super.read(); for nullPointerException.
public class ParticipantRecordT813ItemReader extends SingleItemPeekableItemReader<ParticipantRecordT813InputData> implements ResourceAwareItemReaderItemStream<ParticipantRecordT813InputData> {
private static final String CLASS = "ParticipantRecordT813ItemReader";
SingleItemPeekableItemReader<ParticipantRecordT813InputData> delegate = new SingleItemPeekableItemReader<ParticipantRecordT813InputData>();
#Override
public ParticipantRecordT813InputData read() throws UnexpectedInputException, ParseException, Exception {
ParticipantRecordT813InputData item = super.read();
Log.report(CLASS, "I am in the reader ::::");
if (item != null) {
while (item.getDeNum()=="551") {
Log.report(CLASS, "I am in the reader at DE551::::" + item.getDeNum());
ParticipantRecordT813InputData possibleRelatedObject = peek();
if (possibleRelatedObject == null) {
return item;
}
//logic to determine if next line in file relates to same object
boolean matches = possibleRelatedObject.getDeNum()=="552";
if (matches) {
Log.report(CLASS, "I am in the reader at DE552::::" + possibleRelatedObject.getDeNum());
} else {
return item;
}
}
}
return item;
}
#Override
public void close() throws ItemStreamException {
// TODO Auto-generated method stub
super.close();
}
#Override
public void open(ExecutionContext arg0) throws ItemStreamException {
// TODO Auto-generated method stub
super.open(arg0);
}
#Override
public void update(ExecutionContext arg0) throws ItemStreamException {
// TODO Auto-generated method stub
super.update(arg0);
}
#Override
public void setResource(Resource arg0) {
// TODO Auto-generated method stub
super.setDelegate(delegate);
}
}
Any idea where I am wrong????