Unable to limit the parallelism using ThreadPoolExecutor with Spring Batch - spring

Question

Here's my configuration:
#StepScope
#Bean(name = "mySlaveStep")
public Step mySlaveStep(
#Qualifier(value = "myReader") ItemReader reader,
#Qualifier(value = "myWriter") ItemWriter writer,
StepBuilderFactory stepBuilderFactory) {
return stepBuilderFactory.get("MySlaveStep")
.<SomeObject, SomeObject>chunk(1000)
.reader(reader)
.writer(writer)
.build();
}
#Bean(name = "myStep")
public Step myStep(
#Qualifier(value = "myPartitioner") Partitioner partitioner, // with #StepScope
#Qualifier(value = "myExecutor") TaskExecutor executor, // With/without #StepScope
#Qualifier(value = "myStep") Step step, // With #StepScope
StepBuilderFactory stepBuilderFactory) {
return stepBuilderFactory
.get("MyStep")
.partitioner("MyPartition", partitioner)
.taskExecutor(executor)
.step(step)
.build();
}
#StepScope // With or without
#Bean(name = "taskExecutor")
public TaskExecutor taskExecutor() {
final ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setQueueCapacity(Integer.MAX_VALUE);
executor.setCorePoolSize(2);
executor.setMaxPoolSize(2);
return executor;
}
The exception I'm getting is:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.mySlaveStep': Scope 'step' is not active for the current thread; consider defining a scoped proxy for this bean if you intend to refer to it from a singleton; nested exception is java.lang.IllegalStateException: No context holder available for step scope
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:368) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
at org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35) ~[spring-aop-5.2.0.RELEASE.jar:5.2.0.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:192) ~[spring-aop-5.2.0.RELEASE.jar:5.2.0.RELEASE]
at com.sun.proxy.$Proxy144.execute(Unknown Source) ~[na:na]
at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:138) ~[spring-batch-core-4.2.0.RELEASE.jar:4.2.0.RELEASE]
at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:135) ~[spring-batch-core-4.2.0.RELEASE.jar:4.2.0.RELEASE]
at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: java.lang.IllegalStateException: No context holder available for step scope
at org.springframework.batch.core.scope.StepScope.getContext(StepScope.java:167) ~[spring-batch-core-4.2.0.RELEASE.jar:4.2.0.RELEASE]
at org.springframework.batch.core.scope.StepScope.get(StepScope.java:99) ~[spring-batch-core-4.2.0.RELEASE.jar:4.2.0.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:356) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
So I assume this is related to this. Removing/adding #StepScope from TaskExecutor bean does not change the outcome, however, removing TaskExecutor altogether resolves the issue. I'm only trying to limit the number of parallel partitions being handled as per here. How do I go about it?

Answer 1

First, I will address the scope of the task executor. The task executor should not be "SpringBatch-scoped" (job-scoped or step-scoped) or even scoped at all (IMO the default singleton scope is the correct scope of such a compnent for most use cases). Spring Batch does not create or manage threads, it delegates that to task executors in different parts of the framework. Therefore, such a component should not be impacted by any scope of Spring Batch and should not impact the behaviour of a Spring Batch job by any mean. If this is the case, that would be a bug in Spring Batch.
Now let me address the scope of a step. A step in Spring Batch cannot be step-scoped. That does not make sense. Marking the step as step-scoped means do not create that step bean until the job enclosing it is running (ie at runtime). But, at that time, the step was not configured yet. A batch artefact of a step (reader, writer, listener, tasklet, partitioner, etc) can be step-scoped though, but not the step itself. There is a note about that in the reference documentation here: Late Binding of Job and Step Attributes. Removing the step scope on mySlaveStep should fix your issue.
While I see valid use cases for step components to be scoped (to use late-binding for instance), I do not see any valid use case to scope the step itself.