MultiResourcePartitioner - Multiple Partitions Loading Same Resource - spring

Question

I'm using local partitioning in spring batch to write xml files to the database. I have already split the original file to smaller files and i have used MultiResourcePartitioner to process each one of them as each file will be processed by one thread. I'm getting a violation of primary Key constraint error i don't know how to deal with this issue
List of files
The partitionner
#Bean
public Partitioner partitioner1(){
MultiResourcePartitioner partitioner = new MultiResourcePartitioner();
Resource[] resources;
try {
resources = resourcePatternResolver.getResources("file:src/main/resources/data/*.xml");
} catch (IOException e) {
throw new RuntimeException("I/O problems when resolving the input file pattern.",e);
}
partitioner.setResources(resources);
return partitioner;
}
The StaxEventItemReader using XML file as an input for the reader
#Bean
#StepScope
public StaxEventItemReader<Customer> CustomerItemReader() {
XStreamMarshaller unmarshaller = new XStreamMarshaller();
Map<String, Class> aliases = new HashMap<>();
aliases.put("customer", Customer.class);
unmarshaller.setAliases(aliases);
StaxEventItemReader<Customer> reader = new StaxEventItemReader<>();
reader.setResource(new ClassPathResource("data/customerOutput1-25000.xml"));
reader.setFragmentRootElementName("customer");
reader.setUnmarshaller(unmarshaller);
return reader;
}
The JdbcBatchItemWriter (writing to the database)
#Bean
#StepScope
public JdbcBatchItemWriter<Customer> customerItemWriter() {
JdbcBatchItemWriter<Customer> itemWriter = new JdbcBatchItemWriter<>();
itemWriter.setDataSource(this.dataSource);
itemWriter.setSql("INSERT INTO NEW_CUSTOMER VALUES (:id, :firstName, :lastName, :birthdate)");
itemWriter.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider());
itemWriter.afterPropertiesSet();
return itemWriter;
}
Thanks for any help

Answer 1

Your reader has this line, which causes all the partitions to load the same file:
reader.setResource(new ClassPathResource("data/customerOutput1-25000.xml"));
It should instead take the resource from the Step Execution Context. You can access the execution context either in the open() method using the ItemStream interface or the beforeStep() method of the StepExectionListener interface. A bit of personal preference here, but I generally thing using ItemStream is the "better" solution.