Ability to read multiple files then write content per source file - spring

Question

Reading file is working good but writing file is not.
I would like to read multiple files then use MultiResourceItemWriter to write it separately, like:
Read Files:
source/abc.csv
source/cbd.csv
source/efg.csv
Should write files separate like:
target/abc.csv
target/cbd.csv
target/efg.csv
But currently it's putting all data in one file.
#Bean
public MultiResourceItemWriter<FooCsv> multipleCsvWriter(#Value("${directory.destination}") Resource folder) throws Exception {
MultiResourceItemWriter<FooCsv> writer = new MultiResourceItemWriter<>();
writer.setResource(folder);
writer.setDelegate(csvWriter(file));
return writer;
}
Note this is like copy and paste from source folder to target folder.

Answer 1

There are two ways to this-
Write custom reader by extending multiresourceitemreader, then in read() method you can get your current source file name. Then set this file to your csvFoo and pass this source file name dynamically to you are writer.
Write one extra step and taskley which will list all your available source files then add them to jobparameter and pass both source and dest file name dynamically to next step and call this step in cycle until all list files are not proceed.

Answer 2

You can use MultiResourcePartitioner to achieve the same. Here is sample Batch Config
#Configuration
#EnableBatchProcessing
public class BatchConfig {
#Autowired
private ResourcePatternResolver resourcePatternResolver;
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
public DataSource dataSource;
#Autowired
ApplicationContext context;
#Bean
#JobScope
public MultiResourcePartitioner paritioner(#Value("#{jobParameters[srcDir]}") String src) throws IOException {
Resource[] resources = resourcePatternResolver.getResources(src);
MultiResourcePartitioner partitioner = new MultiResourcePartitioner();
partitioner.partition(1);
partitioner.setResources(resources);
return partitioner;
}
#Bean
#StepScope
public FlatFileItemReader<String> reader(#Value("#{stepExecutionContext[fileName]}") Resource file) {
FlatFileItemReader<String> reader = new FlatFileItemReader<String>();
reader.setResource(file);
reader.setLineMapper(new PassThroughLineMapper());
return reader;
}
#Bean
#StepScope
public FlatFileItemWriter<String> writer(#Value("#{jobParameters[destDir]}") String dest,
#Value("#{stepExecutionContext[fileName]}") Resource file) {
String destFile = dest + file.getFilename();
System.out.println(destFile);
FlatFileItemWriter<String> writer = new FlatFileItemWriter<String>();
writer.setLineAggregator(new PassThroughLineAggregator<>());
writer.setResource(resourcePatternResolver.getResource(destFile));
return writer;
}
#Bean
public Job kpJob() {
return jobBuilderFactory.get("kpJob").incrementer(new RunIdIncrementer()).flow(step1()).end().build();
}
#Bean
public Step step1() {
Partitioner partitioner = context.getBean(MultiResourcePartitioner.class);
return stepBuilderFactory.get("step1").partitioner(slaveStep()).partitioner("step1.slave", partitioner).build();
}
#Bean
public Step slaveStep() {
ItemReader<String> reader = context.getBean(FlatFileItemReader.class);
ItemWriter<String> writer = context.getBean(FlatFileItemWriter.class);
return stepBuilderFactory.get("step1.slave").<String, String>chunk(10).reader(reader).writer(writer).build();
}
}
And pass srcDir and dstDir as job parameter.

Answer 3

simple solution to your problem - why not just create Steps for each of the source file. Given that and based in your scenario, file source and target is one to one. MultiResourceItemWriter is just a way to create new output file once it reaches the limit you set setItemCountLimitPerResource(int) (that i dont find in your example). You can even run the steps in parallel if you have any issue with performance, if there are no dependencies or sequence.