Spring Batch AsyncItemProcessor not processing in parallel - spring

Question

I have a AsyncItemProcessor which I want to run in parallel with the following config. But, the processing is not happening in parallel.
#Configuration
#EnableBatchProcessing
#EnableAsync
public class JobConfig {
#Autowired
private JobBuilderFactory jobBuilder;
#Autowired
private StepBuilderFactory stepBuilder;
#Autowired
#Qualifier("writer")
private ItemWriter writer;
#Bean
#JobScope
public ItemProcessor itemProcessor() {
ItemProcessor itemProcessor = new ItemProcessor();
return itemProcessor;
}
#Bean
#JobScope
public AsyncItemProcessor asyncItemProcessor() throws IOException {
AsyncItemProcessor asyncItemProcessor = new AsyncItemProcessor();
asyncItemProcessor.setDelegate(itemProcessor());
asyncItemProcessor.setTaskExecutor(getAsyncExecutor());
asyncItemProcessor.afterPropertiesSet();
return asyncItemProcessor;
}
#Bean(name = "asyncExecutor")
public TaskExecutor getAsyncExecutor() {
SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor() {
#Override
protected void doExecute(Runnable task) {
final JobExecution jobExecution = JobSynchronizationManager.getContext().getJobExecution();
super.doExecute(() -> {
JobSynchronizationManager.register(jobExecution);
try {
task.run();
} finally {
JobSynchronizationManager.close();
}
});
}
};
simpleAsyncTaskExecutor.setThreadNamePrefix("processing 1-");
simpleAsyncTaskExecutor.setConcurrencyLimit(100);
return simpleAsyncTaskExecutor;
}
#Bean
#JobScope
public AsyncItemWriter asyncItemWriter() throws IOException {
AsyncItemWriter asyncItemWriter = new AsyncItemWriter<>();
asyncItemWriter.setDelegate(writer);
asyncItemWriter.afterPropertiesSet();
return asyncItemWriter;
}
#Bean
#JobScope
public FlatFileItemReader<Result> requestFileReader() {
DefaultLineMapper lineMapper = new DefaultLineMapper();
......
FlatFileItemReader<Result> itemReader = new FlatFileItemReader<>();
itemReader.setLineMapper(lineMapper);
return itemReader;
}
#Bean
public Step simpleFileStep() throws IOException {
return stepBuilder.get("simpleFileStep").chunk(100).reader(fileReader).processor(asyncItemProcessor())
.writer(asyncItemWriter()).build();
}
#Bean(name = "customExecutionContext")
#JobScope
public ExecutionContext customExecutionContext() {
ExecutionContext executionContext = new ExecutionContext();
return executionContext;
}
}
The Processor Class :
#JobScope
public class RequestProcessor implements ItemProcessor<Result, List<Item>> {
#Value("#{jobExecution}")
private JobExecution jobExecution;
#Autowired
#Qualifier("customExecutionContext")
private ExecutionContext storedContext;
#Override
public List<Item> process(Result result) throws Exception {
Date start = new Date();
// Processing logic
Date end = new Date();
long diff = end.getTime() - start.getTime();
log.info("Time taken to process the
items:"+TimeUnit.SECONDS.convert(diff, TimeUnit.MILLISECONDS));
return items;
}
}
I want to process File with 1000 records in parallel in this scenario, but only 100 items are getting processed at a time and writing at a time.
Please let me know if there is some issue with config.
Also after processing each chunk of 100 items, I am having a delay of 2 minutes before processing next chunk. In that time, I can see only following logs:
[GC concurrent-string-deduplication, 16.2K->0.0B(16.2K), avg 88.7%, 0.0000820 secs]
[GC pause (G1 Evacuation Pause) (young) 687M->302M(768M), 0.0138859 secs]

Answer 1

Concurrency is different than parallelism. The AsyncItemProcessor is designed to work hand in hand with the AsyncItemWriter to process items concurrently. In your case, a single chunk of 100 items will be processed concurrently, but chunks will not be processed in parallel. It is still a serial execution of chunks, but every chunk is processed concurrently by different threads from the task executor.
There is no way to process chunks in parallel in Spring Batch. What Spring Batch provides though is partitioning, where partitions can be processed in parallel (either with local threads, or with remote JVMs).
So what you can do is partition your input (for example with 1000 items in each partition) and configure a partitioned step to process partitions in parallel. Note that each partition can further process chunks concurrently as well. You can find more details and a code example in the reference documentation here: Scaling and Parallel Processing.