Reading multiple files resides in a file system which matches the job parameters using MultiResourceItemReader - spring

Question

Use Case :
I would like to launch a job which takes employee id as job parameters, which will be multiple employee ids.
In a file system, files will be residing which contains employee ids as part of the file name (It is a remote file system, not local)
i need to process those files where file name contains the employee-id and passing it to the reader.
I am thinking of using MultiResourceItemReader but i am confused how to match the file name with Employee Id (Job Parameter) which is there in a file system.
Please suggest.

Answer 1

The class MultiResourceItemReader has a method setResources(Resources[] resources) which lets you specify resources to read either with an explicit list or with a wildcard expression (or both).
Example (explicit list) :
<bean class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources">
<list>
<value>file:C:/myFiles/employee-1.csv</value>
<value>file:C:/myFiles/employee-2.csv</value>
</list>
</property>
</bean>
Example (wildcard) :
<bean class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="file:C:/myFiles/employee-*.csv" />
</bean>
As you may know, you can use job parameters in configuration by using #{jobParameters['key']} :
<bean class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="file:C:/myFiles/employee-#{jobParameters['id']}.csv" />
</bean>
Unfortunately, wildcard expressions can't manage an OR expression over a list of value with a separator (id1, id2, id3...). And I'm guessing you don't know how many distinct values you'll have to declare an explicit list with a predefined number of variables.
However a working solution would be to use the Loop mechanism of Spring Batch with a classic FlatFileItemReader. The principle is basically to set the next="" on the last step to the first step until you have exhausted every item to read. I will provide code samples if needed.
EDIT
Let's say you have a single chunk to read one file at a time. First of all, you'd need to put the current id from the job parameter in the context to pass it to the reader.
public class ParametersManagerTasklet implements Tasklet, StepExecutionListener {
private Integer count = 0;
private Boolean repeat = true;
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
// Get job parameter and split
String[] ids = chunkContext.getStepContext().getJobParameters().getString(PARAMETER_KEY).split(DELIMITER);
// Check for end of list
if (count >= ids.length) {
// Stop loop
repeat = false;
} else {
// Save current id and increment counter
chunkContext.getStepContext().getJobExecutionContext().put(CURRENT_ID_KEY, ids[count++];
}
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
if (!repeat) {
return new ExitStatus("FINISHED");
} else {
return new ExitStatus("CONTINUE");
}
}
}
Now you declare this step in your XML and create a loop :
<batch:step id="ParametersStep">
<batch:tasklet>
<bean class="xx.xx.xx.ParametersManagerTasklet" />
</batch:tasklet>
<batch:next on="CONTINUE" to="ReadStep" />
<batch:end on="FINISHED" />
</batch:step>
<batch:step id="ReadStep">
<batch:tasklet>
<batch:chunk commit-interval="10">
<batch:reader>
<bean class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="file:C:/myFiles/employee-#{jobExecutionContext[CURRENT_ID_KEY]}.csv" />
</bean>
</batch:reader>
<batch:writer>
</batch:writer>
</batch:chunk>
</batch:tasklet>
<batch:next on="*" to="ParametersStep" />
</batch:step>

Answer 2

You can write your own FactoryBean to perform a custom resources search.
public class ResourcesFactoryBean extends AbstractFactoryBean<Resource[]> {
String[] ids;
String path;
public void setIds(String[] ids) {
this.ids = ids;
}
public void setPath(String path) {
this.path = path;
}
#Override
protected Resource[] createInstance() throws Exception {
final List<Resource> l = new ArrayList<Resource>();
final PathMatchingResourcePatternResolver x = new PathMatchingResourcePatternResolver();
for(final String id : ids)
{
final String p = String.format(path, id);
l.addAll(Arrays.asList(x.getResources(p)));
}
return l.toArray(new Resource[l.size()]);
}
#Override
public Class<?> getObjectType() {
return Resource[].class;
}
}
---
<bean id="reader" class="org.springframework.batch.item.file.MultiResourceItemReader" scope="step">
<property name="delegate" ref="itemReader" />
<property name="resources">
<bean class="ResourcesFactoryBean">
<property name="path"><value>file:C:/myFiles/employee-%s.cvs</value> </property>
<property name="ids">
<value>#{jobParameters['id']}</value>
</property>
</bean>
</property>
</bean>
jobParameter 'id' is a comma separated list of your ID.