How to Design a Spring-boot scheduler as Service, so that other micro-services can use it - spring-boot

I want to design a scheduler as service using spring-boot. My scheduler should be generic so that other microservices can use it as they want.
I tried normal spring boot examples.
/**
* This scheduler will run on every 20 Seconds.
*/
#Scheduled(fixedRate = 20 * 1000, initialDelay = 5000)
public void scheduleTaskWithInitialDelay() {
logger.info("Fixed Rate Task With Initail Delay 20 Seconds:: Execution Time - "+dateTimeFormatter.format(LocalDateTime.now()));
}
/**
* This scheduler will run on every 10 Seconds.
*/
#Scheduled(fixedRate = 10* 1000, initialDelay = 5000)
public void scheduleTaskWithInitialDelay1() {
logger.info("Fixed Rate Task With Initail Delay 10 Seconds:: Execution Time - "+dateTimeFormatter.format(LocalDateTime.now()));
}

You need to store other microservice's requests to schedule something in your persistent. So, you have an inventory that which microservice requested the scheduling service and with delay or cron or something else.
Now, you can read all the requested configuration from the database and start scheduler for them.
This is a common use case in enterprise applications when people choose to write custom code.
Your database table should contain all the detail + what to do if scheduler reached to given time (Push data/event to some URL or something else).
Some technical detail
You schedule service should allow to
Add Schedule
Start/Stop/Update existing schedule
Callback or some other operation when scheduler meet the time
Hope, this will help.

Related

spring boot #scheduled triggered after the time expected

I try to trigger a task every day at 7h45 in spring boot by this way
#Scheduled(cron = "45 7 * * * ?")
public void method() {....}
And I saw this morning that it triggered at around 9h09.
I checked that server time correspond to the time displayed on my computer. Furthermore the server is running on windows
So Why this difference of time ?
The first element in Spring cron expressions are seconds.
So I assume that it was run at 9:07:45
This would be the correct expression:
0 45 7 * * ?
Check out: https://spring.io/blog/2020/11/10/new-in-spring-5-3-improved-cron-expressions

Spring integration inboundChannelAdapter stops polling unexpectedly

In our project we need to retrieve prices from a remote ftp server. During the office hours this works fine, prices are retrieved and successfully processed. After office hours there are no new prices published on the ftp server, so as expected we don't find anything new.
Our problem is that after a few hours of not finding new prices, the poller just stops polling. No error in the logfiles (even when running on org.springframework.integration on debug level) and no exceptions. We are now using a separate TaskExecutor to isolate the issue, but still the poller just stops. In the mean time we adjusted the cron expression to match these hours, to limited the resource use, but still the poller just stops when it is supposed to run.
Any help to troubleshoot this issue is very much appreciated!
We use an #InboudChannelAdapter on a FtpStreamingMessageSource which is configured like this:
#Bean
#InboundChannelAdapter(
value = FTP_PRICES_INBOUND,
poller = [Poller(
maxMessagesPerPoll = "\${ftp.fetch.size}",
cron = "\${ftp.poll.cron}",
taskExecutor = "ftpTaskExecutor"
)],
autoStartup = "\${ftp.fetch.enabled:false}"
)
fun ftpInboundFlow(
#Value("\${ftp.remote.prices.dir}") pricesDir: String,
#Value("\${ftp.remote.prices.file.pattern}") remoteFilePattern: String,
#Value("\${ftp.fetch.size}") fetchSize: Int,
#Value("\${ftp.fetch.enabled:false}") fetchEnabled: Boolean,
clock: Clock,
remoteFileTemplate: RemoteFileTemplate<FTPFile>,
priceParseService: PriceParseService,
ftpFilterOnlyFilesFromMaxDurationAgo: FtpFilterOnlyFilesFromMaxDurationAgo
): FtpStreamingMessageSource {
val messageSource = FtpStreamingMessageSource(remoteFileTemplate, null)
messageSource.setRemoteDirectory(pricesDir)
messageSource.maxFetchSize = fetchSize
messageSource.setFilter(
inboundFilters(
remoteFilePattern,
ftpFilterOnlyFilesFromMaxDurationAgo
)
)
return messageSource;
}
The property values are:
poll.cron: "*/30 * 4-20 * * MON-FRI"
fetch.size: 10
fetch.enabled: true
We limit the poll.cron we used the retrieve every minute.
In the related DefaultFtpSessionFactory, the timeouts are set to 60 seconds to override the default value of -1 (which means no timeout at all):
sessionFactory.setDataTimeout(timeOut)
sessionFactory.setConnectTimeout(timeOut)
sessionFactory.setDefaultTimeout(timeOut)
Maybe my answer seems a bit too easy, bit is it because your cron expression states that it should schedule the job between 4 and 20 hour. After 8:00 PM it will not schedule the job anymore and it will start polling again at 4:00 AM.
It turned out that the processing took longer than the scheduled interval, so during processing a new task was already executed. So eventually multiple task were trying to accomplish the same thing.
We solved this by using a fixedDelay on the poller instead of a fixedRate.
The difference is that a fixedRate schedules on a regular interval independent if the task was finished and the fixedDelay schedules a delay after the task is finished.

Retry with 30 minutes delay

I need to call a external rest service, if it fails on first attempt then I have to call again after 30 minutes. Max 3 time I can call like this.
I know spring has RetryTemplate for the retry. But I feel, for my cases its not fit. I have to call like this for more than 1000 records.
Any idea How can I achieve this in Spring.
Use a TaskScheduler.
scheduler.schedule(() -> { ... },
new Date(System.currentTimeMillis() + (30 * 60_000));
Keep track of how many times and if not exhausted, re-schedule.

Spring Batch Job scheduler stops being executing after several runs

On a project we have to run a job that starts periodically (every 5 minutes on QA env now) that processes some assignments for 40k users.
We had decided to utilize Spring Batch because it fits perfectly and implemented it with pretty much default configuration (e.g. it uses SyncTaskExecutor under the hood).
Okay, so, there is a job that consists of a single step with:
out-of-box HibernatePagingItemReader
custom ItemProcessorthat performs lightweight calculations in memory
custom ItemWriter that persists data to the same PostgreSQL db via several JPQL and native queries.
The job itself is scheduled with #EnableScheduling and is being triggered every 5 mins by cron expression:
#Scheduled(cron = "${job.assignment-rules}")
void processAssignments() {
try {
log.debug("Running assignment processing job");
jobLauncher.run(assignmentProcessingJob, populateJobParameters());
} catch (JobExecutionException e) {
log.error("Job processing has failed", e);
}
}
Here is a cron expression from application.yml:
job:
assignment-rules: "0 0/5 * * * *"
The problem is that it stops being scheduled after several runs (different amount of runs every time). Let's take a look into the Spring Batch schema:
select ex.job_instance_id, ex.create_time, ex.start_time, ex.end_time, ex.status, ex.exit_code, ex.exit_message
from batch_job_execution ex inner join batch_job_instance bji on ex.job_instance_id = bji.job_instance_id
order by start_time desc, job_instance_id desc;
And then silence. Nothing special in logs.
The only thing I believe could make sense is that there are two more jobs running on that instance. And one of them is time consuming because it sends emails via SMTP.
The entire jobs schedule is:
job:
invitation-email: "0 0/10 * * * *"
assignment-rules: "0 0/5 * * * *"
rm-subordinates-count: "0 0/30 * * * *"
Colleagues, could anybody point me out the way this problem could be troubleshooted?
Thanks a lot in advance
Using the default SyncTaskExecutor to launch jobs is not safe in your use case as all jobs will be executed by a single thread. If one of the jobs takes more than 5 minutes to run, next jobs will pile up and fail to start at some point.
I would configure a JobLauncher with an asynchronous TaskExecutor implementation (like org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor) in your use case. You can find an example in the Configuring a JobLauncher section (See "Figure 3. Asynchronous Job Launcher Sequence").
Apart from the task executor configured to be used by the JobLauncher of Spring Batch, you need to make sure you have the right task executor used by Spring Boot to schedule tasks (since you are using #EnableScheduling). Please refer to the Task Execution and Scheduling section for more details.

DefaultJmsListenerContainerFactory / DefaultMessageListenerContainer Dynamic Scaling: How does it work?

I am trying to get my Spring 4 JMS application to dynamically scale down after a lot of messages were processed. I currently have a concurrency of 1-3 consumers and I am able to see a successful scaling up toward the maximum consumers, but once the consumers have been created, they don't go away. They stay there in a WAIT state. I would like to know if there is a setting or configuration that enables the consumers to shutdown after the load subsides. Does anyone know?
Thanks,
Juan
Take a look to this option:
/**
* Specify the limit for idle executions of a consumer task, not having
* received any message within its execution. If this limit is reached,
* the task will shut down and leave receiving to other executing tasks.
* <p>The default is 1, closing idle resources early once a task didn't
* receive a message. This applies to dynamic scheduling only; see the
* {#link #setMaxConcurrentConsumers "maxConcurrentConsumers"} setting.
* The minimum number of consumers
* (see {#link #setConcurrentConsumers "concurrentConsumers"})
* will be kept around until shutdown in any case.
* <p>Within each task execution, a number of message reception attempts
* (according to the "maxMessagesPerTask" setting) will each wait for an incoming
* message (according to the "receiveTimeout" setting). If all of those receive
* attempts in a given task return without a message, the task is considered
* idle with respect to received messages. Such a task may still be rescheduled;
* however, once it reached the specified "idleTaskExecutionLimit", it will
* shut down (in case of dynamic scaling).
* <p>Raise this limit if you encounter too frequent scaling up and down.
* With this limit being higher, an idle consumer will be kept around longer,
* avoiding the restart of a consumer once a new load of messages comes in.
* Alternatively, specify a higher "maxMessagesPerTask" and/or "receiveTimeout" value,
* which will also lead to idle consumers being kept around for a longer time
* (while also increasing the average execution time of each scheduled task).
* <p><b>This setting can be modified at runtime, for example through JMX.</b>
* #see #setMaxMessagesPerTask
* #see #setReceiveTimeout
*/
public void setIdleTaskExecutionLimit(int idleTaskExecutionLimit) {
Also you can study all other options of the DefaultMessageListenerContainer via their JavaDocs.

Resources