We would like to stop Sidetiq from scheduling new jobs based on some condition. In our use case, that there are no newer versions of the code running in another deployment/container)
How can we add logic at the point Sidetiq is scheduling the job and prevent it from doing so?
Related
We have quartz scheduler to trigger the jenkins job which will be created on the fly through Jenkins create API.
So I have placed my create jenkins job API call inside the execute internal method.When multiple parallel requests are made, sometimes all the requests are picked up for execution and sometimes few are missed.
The count of missed executions differs.
How do I make this stop and make quartz to run all of my requests.
Tried increasing the thread count and misfire threshold but the issue exists
Seems like all you need is to set the correct misfire instruction based on your business logic and type of trigger
Trigger trigger = TriggerBuilder.newTrigger() .
withIdentity(changePoolNameTriggerKey).
startAt(new DateTime().plusSeconds(configuration.getInt(JobConstants.execution_latency)).
toDate().
build();
((MutableTrigger) trigger).setMisfireInstruction(MISFIRE_INSTRUCTION_FIRE_NOW)
According to the business logic of my Spring Boot application with Quartz Scheduling and MongoDB as Job persistent storage, every user of the system can create the postponed job that must be executed at some point in time. The user chooses the time when it must be executed.
Right now I'm thinking about the approach where every user will create a dedicated JobDetail for every postponed job, something like this:
schedulerFactoryBean.getScheduler().addJob(jobDetail(), true, true);
The issue I can potentially see here, that with this approach I can quickly create thousands of jobs in Quartz scheduler. Previously I never scheduled such amount of jobs in Spring Scheduling with Quartz and don't know how the system will handle it. Is it a good idea to implement the system in such way and will Spring Scheduling Quartz handle such amount of jobs without problems?
Yes, Quartz itself can handle thousands of jobs and triggers without any issues.
If you are going to have many jobs executing concurrently, just make sure that you configure Quartz with a sufficient number of worker threads. The number of worker threads should be typically equal to the maximum number of jobs that can be running concurrently + some small buffer (10% or so) just in case.
From what you write I assume that your jobs will be one-off jobs, i.e. each job will be executed only once. If that is the case, Quartz can automatically discard your jobs as soon as they finish executing unless your jobs are marked as durable. Quartz automatically removes non-durable jobs if they are not scheduled to run in the future. This feature may help you reduce the total number of registered jobs.
I hope this helps. If not, please ask.
Nothing seems to prevent a perform method to use the Sidekiq API. It should be safe in read-only mode.
What if it calls a "write" methods ? Especially when this method acts on the current job itself.
We would like to reschedule a job without creating a new job because we need to track the job completion with the sidekiq-status gem from another worker.
Using MyWorker.perform_in or MyWorker.perform_at to reschedule the job from inside the worker creates a new job, making it difficult to track the total completion. We're thinking of using Sidekiq::ScheduledSet.new.find and the reschedule method but it seems awkward and potentially dangerous to reschedule a job that is about to complete.
Does Sidekiq and its API support this use case ?
You might be able to hack something together but it'll be really slow if you try to modify the Sets and Lists in Redis directly. They aren't designed to be used that way.
The official Sidekiq solution to this problem is a Batch.
https://github.com/mperham/sidekiq/wiki/Batches#status
You create a one-job batch. If the job needs to be rescheduled, it adds a new job to the Batch to be executed later. Your other worker just checks the status of the overall Batch and if it is 100% complete.
I have a sequence of mapreduce jobs that need to be run. I was wondering if there is any advantage of using Oozie for that, instead of having "one big driver" that will run that sequence?
I know that Oozie can be used to run multiple actions of different type, e.g. pig script, shell script, mr job, but I'm concretely interested should I split my two jobs and run them using Oozie, or have a single jar to do that?
Oozie is a scheduler - crude, poorly documented, but a scheduler.
If you don't need scheduling per se, or if CRON on an edge node is sufficient
if you want to handle your workflow logic by yourself (e.g. conditional
branching, parallel executions w/ waiting for stragglers, calling
generic sub-workflows w/ ad hoc parameters, e-mail alerts on errors,
<insert your pet feature here>) or don't need any fancy logic
if you handle your executions logs and state history by yourself, or don't care about history
... well, don't use a scheduler.
PS: you also have Luigi (Spotify) and Azkaban (LinkedIn) as alternative Hadoop schedulers.
[edit] extra point to consider: if your "driver" crashes for whatever reason, you may not have a chance to send an alert; but if run from Oozie, the crash will be detected eventually (may take as much as 30 min. in a corner case e.g. AM job self-destruction due to YARN RM failover)
In my situation, clients are allowed to schedule a job. As I can see, quartz often use a cronExpression to perform a schedule. But there are many clients with many schedules, I can't write many trigger beans cuz I don't know how many schedules really are, it depends on clients. So, would some one help?
Quartz is designed to add and remove jobs and triggers at runtime. Spring is a degenerated case where all triggers and jobs are defined at startup time. In Quartz, when having an instance of Scheduler you can easily create, browse and delete triggers at your wish, example from How-To: Scheduling a Job:
// Define job instance
JobDetail job1 = newJob(ColorJob.class)
.withIdentity("job1", "group1")
.build();
// Define a Trigger that will fire "now", and not repeat
Trigger trigger = newTrigger()
.withIdentity("trigger1", "group1")
.startNow()
.build();
// Schedule the job with the trigger
schedulder.scheduleJob(job, trigger);
See the official documentation, especially the cookbook.
Also distinguish between jobs (a piece of code wrapped in a class that should be executed) and a trigger (time condition under which job is executed). Remember that by default when last trigger associated with the job (job can have several triggers) fires, the job is removed from the scheduler.