We have quartz scheduler to trigger the jenkins job which will be created on the fly through Jenkins create API.
So I have placed my create jenkins job API call inside the execute internal method.When multiple parallel requests are made, sometimes all the requests are picked up for execution and sometimes few are missed.
The count of missed executions differs.
How do I make this stop and make quartz to run all of my requests.
Tried increasing the thread count and misfire threshold but the issue exists
Seems like all you need is to set the correct misfire instruction based on your business logic and type of trigger
Trigger trigger = TriggerBuilder.newTrigger() .
withIdentity(changePoolNameTriggerKey).
startAt(new DateTime().plusSeconds(configuration.getInt(JobConstants.execution_latency)).
toDate().
build();
((MutableTrigger) trigger).setMisfireInstruction(MISFIRE_INSTRUCTION_FIRE_NOW)
Related
My requirement is whenever we call certain RestAPI from UI/Postman, At backend it should trigger JOB that perform several operations/task.
Example:
Assume some POST Rest API is invoked -
It should invoke "Identify-JOB"(performs several activities)- Based on certain condition, It should invoke PLANA-JOB or PLANB-JOB
1> Suppose PLANA-JOB is invoked, on the success of this JOB, it should trigger another JOB called "finish-JOB". On the failure it should not invoke another JOB "finish-JOB"
Can you please help here how can i do this?
You can use async processing and that'll trigger the first job and that task will trigger the next set of tasks.
You can build them like AWS step functions
You can use Rqueue to enqueue async task and that'll be processed by one of the listeners.
In my Spring Boot application, based on the Cron job(runs every 5 minutes) I need to process 2000 products in my database.
Right now the process time of these 2000 products takes more than 5 minutes. I ran into the issue where the second Cron job runs when the first one is not completed yet.
Is there in Spring/Cron out of the box functionality that will allow to synchronize these jobs and wait for the previous job completion before starting the next one?
Please advise how to properly implement such kind of system. Anyway, the following technologies are also available Neo4j, MongoDB, Kafka. Please advise how to properly design/implement this functionality using the Spring/Cron separately or even together with the mentioned technologies.
1) You may try to use #Scheduled(fixedDelay = 5*60*1000). It will guarantee that next invocation will happen strictly in 5 minutes after previous one is finished. But this may break your scheduling requirements
2) You can limit the underlying ThreadExecutor's pool size to 1 thread, so next invocation will have to wait until previous is finished, but this, again, can break the logic, since it would affect all periodic tasks invoked by #Scheduled
3) You can use Quartz instead of spring's native #Scheduled. It's more complicated to configure, but allows to achieve the desired behaviour via #DisallowConcurrentExecution annotation or via setting JobDetail::isConcurrentExectionDisallowed in your job details
We have the below requirement,
In spring xd, we have a job lets assume the job name as MyJob
which will be invoked by another process using the rest service of spring xd, lets assume process name as OutsideProcess (non-spring xd process).
OutsideJob invokes MyJob when ever a file added to a location (lets assume FILES_LOC) to which OutsideJob is listening.
In this scenario, lets assume that MyJob takes 5minutes to complete the job.
At 10:00 AM, there is a file copied to FILES_LOC, then OutsideProcess will trigger MyJob immediately. (approximately it will be completed at 10:05AM)
At 10:01 AM, another file copied to FILES_LOC, then OutsideProcess will trigger one more instance of MyJob at 10:01AM. But the second instance is getting queued and starts the execution once the first instance completes its execution (approximately at 10:05AM).
If we invoke the different jobs at the same time they are getting executed concurrenctly, but the same job multiple instances are not getting executed concurrenctly.
Please let me know how can I execute the same job with multiple instances concurrently.
Thanks in advance.
The only thing I can think of is dynamic deployment of the job and triggering it right away. You can use SpringXD Rest template to create the job definition on the fly and launch them after sleeping a few seconds. And make sure you undeploy/destroy the job when the job completes successfully.
Another solution could be to create a few module instances of your job with different names and use them as your slave processes. You can query status of these job module instances and launch the one that is finished or queue the one that is least recently launched.
Remember you can run jobs with partition support if applicable. This way you will finish your job faster and be able to run more jobs.
In my situation, clients are allowed to schedule a job. As I can see, quartz often use a cronExpression to perform a schedule. But there are many clients with many schedules, I can't write many trigger beans cuz I don't know how many schedules really are, it depends on clients. So, would some one help?
Quartz is designed to add and remove jobs and triggers at runtime. Spring is a degenerated case where all triggers and jobs are defined at startup time. In Quartz, when having an instance of Scheduler you can easily create, browse and delete triggers at your wish, example from How-To: Scheduling a Job:
// Define job instance
JobDetail job1 = newJob(ColorJob.class)
.withIdentity("job1", "group1")
.build();
// Define a Trigger that will fire "now", and not repeat
Trigger trigger = newTrigger()
.withIdentity("trigger1", "group1")
.startNow()
.build();
// Schedule the job with the trigger
schedulder.scheduleJob(job, trigger);
See the official documentation, especially the cookbook.
Also distinguish between jobs (a piece of code wrapped in a class that should be executed) and a trigger (time condition under which job is executed). Remember that by default when last trigger associated with the job (job can have several triggers) fires, the job is removed from the scheduler.
I'd like to find the best way to handle exceptions (failure of any steps) from an Oracle scheduler job chain (11gR2).
Say I have a chain that contains 20 steps. If at any point the chain exits with FAILURE, I'd like to do a set of actions. These actions are specific to that chain, not the individual steps (each step's procedure may be used outside of scheduler or in other chains).
Thanks to 11gR2, I can now setup an email notification on FAILURE of chain, but this is only 1 of several actions I need to do, so its only a partial solution for me.
The only thing I can think of is have another polling job check the status of my chain every x minutes and launch the failure actions when it sees the latest job of the chain exited with FAILURE status. But this is a hack at best imo.
What is the best way to handle exceptions for a given job chain?
thanks
The most flexible way to handle jobs exceptions in general is to use a job exception monitoring procedure and define the jobs to generate events upon job status changes. The job exception monitoring procedure should watch the scheduler event queue in a loop and react upon events in a way you define.
Doing so takes away the burden to have to create failure steps for about each and every job step in a chain. This is a very powerful mechanism.
by lack of time: in the book is a complete scenario of event based scheduling. Will dig one up later.