I have some questions about using Spring's #Scheduled and #Async functionality together.
Lets say my requirement is to process 5 rows from a DB every 1 second, so in 1 pass of the scheduler thread would create 5 asynchronous threads
to process each row from a Database.
My questions are as follows:
1) Is there an alternative way to creating 5 ascynchonis threads instead of using a while loop within the scheduled method?
One problem I see with this approach is the thread pools active count may not equal the max pool size and therefore the loop will not break before 1 second has passed.
2) In some cases the log statement in the AsyncService i.e. Executing dbItem on the following asyncExecutor : task-scheduler-1 displays task-scheduler-1 as the thread name and not async_thread_ as i would always expect?
3) If my scheduler thread takes longer than 1 second to run, what happens the subsequent pass of the scheduler?
The asyncExecutor:
#Override
#Bean(name="asyncExecutor")
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(5);
threadPoolTaskExecutor.setQueueCapacity(5);
threadPoolTaskExecutor.setMaxPoolSize(10);
threadPoolTaskExecutor.setThreadNamePrefix("async_thread_");
threadPoolTaskExecutor.setWaitForTasksToCompleteOnShutdown(true);
return threadPoolTaskExecutor;
}
which is injected into a class with a scheduled method:
#Autowired
#Qualifier("asyncExecutor")
private ThreadPoolTaskExecutor threadPoolTaskExecutor;
#Autowired
AsyncService asyncService;
#Autowired
private DaoService daoService;
#Scheduled(fixedRateString = "1000")
public void schedulerMetod() {
try {
while (threadPoolTaskExecutor.getActiveCount() < threadPoolTaskExecutor.getMaxPoolSize()) {
DbItem dbItem = daoService.retrieveNewItemFromDB();
if (dbItem != null){
asyncService.processNewItem(dbItem);
}
}
} catch (ObjectOptimisticLockingFailureException ole){
log.info(ole.getMessage());
} catch (Exception ex){
log.error(ex.getMessage());
}
}
#Service
public class AsyncServiceImpl implements AsyncService {
#Autowired
private TaskService taskService;
#Override
#Transactional
#Async("asyncExecutor")
public void processNewItem(DbItem dbItem) {
log.debug("Executing dbItem on the following asyncExecutor : " + Thread.currentThread().getName());
taskService.processNewItem(dbItem);
}
}
Using ThreadPoolTaskScheduler bean you will have twofold purpose as it implements both TaskExecutor (supports #Async) and TaskScheduler (supports #Scheduled) interfaces. You can configure like this:
#Bean
public ThreadPoolTaskScheduler taskScheduler() {
ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
threadPoolTaskScheduler.setPoolSize(5);
threadPoolTaskScheduler.setThreadNamePrefix("async_thread_");
threadPoolTaskScheduler.initialize();
return threadPoolTaskScheduler;
}
If you give another executor name such as taskScheduler1 then you should provide executor name as an attribute in #Async("taskScheduler1").
By this implementation you already have answers for your questions 1 & 2.
About your 3rd question, if all threads in your thread pool are busy then your schedulers do nothing until a thread released then it executes the scheduling task. You may think to give the thread pool a bigger size.
Last but not least, only usage of #Scheduled is enough as it works asynchronously.
Related
My Spring batch is triggered by a Rest end point. I am looking for the solution to run only one instance of a job at a time.
You can implement a listener overriding beforeJob method, that will run before the job starts and check if another instance of the same job is running. If another instance is already running, it will stop the current run.
#Component
public class MultipleCheckListener implements JobExecutionListener {
#Autowired
private JobExplorer explorer;
#Override
public void beforeJob(JobExecution jobExecution) {
String jobName = jobExecution.getJobInstance().getJobName();
Set<JobExecution> executions = explorer.findRunningJobExecutions(jobName);
if(executions.size() > 1) {
jobExecution.stop();
}
}
}
i am working in the spring-batch where i am having one situation
we have 2 set of schedulers
public class SchedulerA {
#Autowired
private Job a;
#Autowired
private Job b;
#Autowired
private Job c;
#Autowired
private Job d;
#Autowired
private SpringBatchJobHandler springBatchJobHandler;
#Autowired
private JobHandler jobHandler;
private List<String> jobName = new ArrayList<>();
#Bean
public void SchedulerLoad() {
jobName.add(a.getName());
jobName.add(b.getName());
jobName.add(c.getName());
jobName.add(d.getName());
}
#Scheduled(fixedDelay = 300000)
private void jobScheduler() throws Exception {
for (String job : jobName) {
if (!jobHandler.isJobForceStopped()) {
springBatchJobHandler.runJob(job);
}
}
}
and this is the second set of scheduler
public class SchedulerB {
#Autowired
private Job q;
#Autowired
private Job w;
#Autowired
private Job e;
#Autowired
private Job r;
#Autowired
private SpringBatchJobHandler springBatchJobHandler;
#Autowired
private JobHandler jobHandler;
private List<String> jobName = new ArrayList<>();
#Bean
public void SchedulerLoad() {
jobName.add(q.getName());
jobName.add(w.getName());
jobName.add(e.getName());
jobName.add(r.getName());
}
#Scheduled(fixedDelay = 300000)
private void jobScheduler() throws Exception {
for (String job : jobName) {
if (!jobHandler.isJobForceStopped()) {
springBatchJobHandler.runJob(job);
}
}
}
Now what we are trying is jobs within the each scheduler will run in sequential mode but both scheduler class will parallely.
Ex: Both schedulerA and schedulerB has to run at same time parallely, But the jobs within the respective classes has to be run in sequential mode only.
is that possible to achieve the above scenario?
i know above question will be so much confusing but we dont have any other choice..!!
please share your feedback for this situation..
There are two TaskExecutors involved in your scenario:
The one used by Spring Batch to run jobs when launched via a JobLauncher. By default, this uses a SyncTaskExecutor which runs submitted jobs in the calling thread. If you use the default JobLauncher in your SpringBatchJobHandler, then your jobs will be run in sequence since you submit them in a for loop.
The one used by Spring Boot to run scheduled tasks. By default, this one has a pool size of 1, see Spring Boot documentation: The thread pool uses one thread by default and those settings can be fine-tuned using the spring.task.scheduling namespace. This means if you schedule two tasks to run at the same time as in your use case, they will be run in sequence by the same thread.
Now If you want to run those scheduled tasks in parallels using different threads, you need to increase the pool size of the TaskExecutor configured by Spring Boot to 2 or more. You can do that by setting the following property:
spring.task.scheduling.pool.size=2
I had written rest services in spring that is running perfectly fine.
Now, I need to add perform some db transactions before returning response to user.
This db transaction is independent to response retrieved.
For example,
#PostMapping("login")
public TransactionResponse loginAuthentication(#Valid #RequestBody LoginRequestBody loginRequest) {
TransactionResponse transactionResponse = new TransactionResponse();
try {
transactionResponse = loginService.validateUser(loginRequest);
//independent transaction needs to be executed in a separate thread
loginSerice.addLoginLog(transactionResponse);
//return below response without waiting to compelete above log transaction
return transactionResponse;
}
catch (Exception e) {
return CommonUtils.setErrorResponse(transactionResponse, e);
}
}
I read upon async controller in spring mvc link. Although controller
executes respective functionality in a separate thread but I don't want to wait for db transaction to be completed. After getting response from service layer, it should be forwarded to user without any delay.
Any Suggestions !!
Spring version is 4.3
I posted this answer to help the fellow developers with same kind of requirement (to execute a void function in a separate thread).
Since I am not experienced in multithreading/asynchronous environment, I want to keep it simple by using spring asynchronous methods.
So, First I created the Thread Pool
#Configuration
#EnableAsync
public class ThreadConfig {
#Bean
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(4);
executor.setMaxPoolSize(4);
executor.setThreadNamePrefix("WEBAPP");
executor.initialize();
return executor;
}
}
Then I created a service that will execute my code in a separate thread.
#Async
#Service
#Transactional
public class LoggingService {
public void logintransaction() throws Exception{
System.out.println("start login loggin");
Thread.sleep(5000);
System.out.println("exit");
}
}
Lastly, I called the above service on my controller. As I can see Total Time Taken is printed first, then "start login loggin" is printed. This means my new method is executed in a new thread.
#Autowired
private LoggingService loggingService;
#PostMapping("login")
public TransactionResponse loginAuthentication(#Valid #RequestBody LoginRequestBody loginRequest) {
long startTime = System.currentTimeMillis();
TransactionResponse transactionResponse = new TransactionResponse();
try {
transactionResponse = loginService.validateUser(loginRequest);
//independent transaction needs to be executed in a separate thread
//loginSerice.addLoginLog(transactionResponse);
loggingService.logintransaction();
//return below response without waiting to compelete above log transaction
System.err.println("Total Time Taken=>"+(System.currentTimeMillis() - startTime));
return transactionResponse;
}
catch (Exception e) {
return CommonUtils.setErrorResponse(transactionResponse, e);
}
}
Thanks
I have the following ThreadPoolTaskExecutor thats gets created with the expected core/max pool size configurations.
#Slf4j
#Configuration
public class ThreadPoolConfig {
#Value("${corePoolSize}")
private Integer corePoolSize;
#Value("${queueCapacity}")
private Integer queueCapacity;
#Value("${maxPoolSize}")
private Integer maxPoolSize;
#Bean(name="myThreadPoolTaskExecutor")
public ThreadPoolTaskExecutor myThreadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setBeanName("myThreadPoolTaskExecutor");
executor.setCorePoolSize(corePoolSize);
executor.setQueueCapacity(queueCapacity);
executor.setMaxPoolSize(maxPoolSize);
executor.setThreadNamePrefix("my_thread_");
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.initialize();
log.debug("threadPoolTaskExecutor CorePoolSize is : " + executor.getCorePoolSize());
log.debug("threadPoolTaskExecutor MaxPoolSize is : " + executor.getMaxPoolSize());
return executor;
}
}
When my #scheduled method runs the max pool size is set to the DEFAULT value of 2147483647 and I don't understand why it's not using the configured ThreadPoolTaskExecutor above:
#EnableScheduling
public class SchedulingConfig {
}
#Component
public class Scheduler {
#Autowired
#Qualifier("myThreadPoolTaskExecutor")
private ThreadPoolTaskExecutor threadPoolTaskExecutor;
#Scheduled(fixedRateString = "${fixedRate}")
public void invokeScheduledThread() {
while (threadPoolTaskExecutor.getActiveCount() <= threadPoolTaskExecutor.getMaxPoolSize()) {
log.debug("Active Thread Pool count is : " + threadPoolTaskExecutor.getActiveCount() + ", Max Thread Pool count is : " + threadPoolTaskExecutor.getMaxPoolSize() + " on the scheduled Thread : " + Thread.currentThread().getName());
//call a service to retrieve some items to process
threadPoolTaskExecutor.execute(Some Object that implements runnable);
}
}
}
Output:
Active Thread Pool count is : 0, Max Thread Pool count is : 2147483647 on the scheduled Thread : task-scheduler-1
I put a break point into the initialise() method of org.springframework.scheduling.concurrent.ExecutorConfigurationSupport
and it looks like the method is getting invoked 3 times, twice with a ThreadName Prefix of "my_thread_"
which is expected and finally once for a Bean called "taskScheduler" with a ThreadName Prefix of "task-scheduler-".
Does anyone know why I can't use my own ThreadPoolTaskExecutor within the Scheduler class?
I wanted to use a default #Scheduler to run on a single thread every x number of seconds and create X number of Threads using my own ThreadPoolTaskExecutor.
Use ThreadPoolTaskScheduler instead of ThreadPoolTaskExecutor.
For example:
#Configuration
public class SpringSchedulerConfig {
private static final int THREAD_POOL_SIZE = 5;
#Bean
public ThreadPoolTaskScheduler getScheduler() {
ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
//we want every Job in a separate Thread.
threadPoolTaskScheduler.setPoolSize(THREAD_POOL_SIZE);
return threadPoolTaskScheduler;
}
}
I have a Spring Batch application in which I want to schedule jobs calls.
The scheduling interval is not known at build so I can't just annotate my Job with #Scheduled.This led me to use a ThreadPoolTaskScheduler.
The thing is the method schedule takes a Runnable as a parameter. Is it possible to schedule jobs this way ?
I can call the job directly from the following service but I can't schedule it.
Here is my the background of my problem, I tried to make it simple :
#Service
public class ScheduledProcessor{
private final ThreadPoolTaskScheduler threadPoolTaskScheduler;
private Application application;
#Autowired
public ScheduledProcessor(ThreadPoolTaskScheduler threadPoolTaskScheduler, Application application){
this.threadPoolTaskScheduler = threadPoolTaskScheduler;
this.application = application;
scheduledTasks = new ArrayList();
Trigger trigger = new CronTrigger("0/6 * * * * *");
//Here I am trying to schedule my job.
//The following line is wrong because a Job can't be cast to a Runnable but I wanted to show the intended behaviour.
threadPoolTaskScheduler.schedule((Runnable) application.importUserjob, trigger);
System.out.println("Job launch !");
}
And here is the JobBuilderFactory :
#Bean
public Job importUserJob(JobBuilderFactory jobs, Step s1, Step s2) {
return jobs.get("importUserJob")
.incrementer(new RunIdIncrementer())
.flow(s1)
.end()
.build();
}
I understand (well, I'm even not sure about that) that I can't directly cast a Job to a Runnable but is it possible to convert it in any way ? Or can you give me some advice about what to use for being able to dynamically schedule spring batch jobs ?
In case that changes something, I also need to be able to restart / skip my steps, as I currently can with the threadPoolTaskScheduler.
Thank you in advance for any help or hint you could provide.
I finally got how to do it !
I created a class which implements Runnable (and for convenience, extends Thread, which avoid the need to implement all of Runnable classes).
#Component
public class MyRunnableJob extends Thread implements Runnable{
private Job job;
private JobParameters jobParameters;
private final JobOperator jobOperator;
#Autowired
public MyRunnableJob(JobOperator jobOperator) {
this.jobOperator = jobOperator;
}
public void setJob(Job job){
this.job=job;
}
#Override
public void run(){
try {
String dateParam = new Date().toString();
this.jobParameters = new JobParametersBuilder().addString("date", dateParam).toJobParameters();
System.out.println("jobName : "+job.getName()+" at "+dateParam);
jobOperator.start(job.getName(), jobParameters.toString());
} catch (NoSuchJobException | JobInstanceAlreadyExistsException | JobParametersInvalidException ex) {
Logger.getLogger(MyRunnableJob.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
In my ScheduledProcessor class, I set a Job to myRunnable class and then pass it as a parameter of the schedule method.
public class SchedulingProcessor {
//Autowired fields :
private final JobLauncher jobLauncher;
private final Job importUserJob;
private final ThreadPoolTaskScheduler threadPoolTaskScheduler;
private final MyRunnableJob myRunnableJob;
//Other fields :
private List<ScheduledFuture> scheduledTasks;
#Autowired
public SchedulingProcessor(JobLauncher jobLauncher, Job importUserJob, ThreadPoolTaskScheduler threadPoolTaskScheduler, MyRunnableJob myRunnableJob) throws Exception {
this.jobLauncher=jobLauncher;
this.importUserJob=importUserJob;
this.threadPoolTaskScheduler=threadPoolTaskScheduler;
this.myRunnableJob=myRunnableJob;
Trigger trigger = new CronTrigger("0/6 * * * * *");
myRunnableJob.setJob(this.importUserJob);
scheduledTasks = new ArrayList();
scheduledTasks.add(this.threadPoolTaskScheduler.schedule((Runnable) myRunnableJob, trigger));
}
}
The scheduledTasks list is just to keep a control over the tasks I just scheduled.
This trick enabled me to dynamically (thanks to ThreadPoolTaskScheduler) schedule Spring Batch Jobs encapsulated in a class implementing Runnable. I wish it can help someone in the same case as mine.
Heres another way to trigger them from your spring context.
Job emailJob = (Job) applicationContext.getBean("xyzJob");
JobLauncher launcher = (JobLauncher) applicationContext
.getBean("jobLauncher");
launcher.run(emailJob, new JobParameters());