I am using quartz library, mainly to schedule multiple tasks that will run forever at specific time. I have a customized executor (a retry executor that reschedule a task for a specified number of times in case of failure (retries are customizable). I want to know if there is a way to setup quartz to use this customized executor? Currently I am using the executor inside the Job i.e. call executor.execute() inside the Job's execute method.
From the Guide there is only explained how to configure your own ThreadPool.
Fine-tuning the Scheduler
You can also implement Quatz's interface ThreadExecutor or adjust your "RetryExecutor" implementation to do so.
Then it can be passed as component via setThreadExecutor to the QuartzSchedulerResources which in turn are used to configure QuartzScheduler - the heart of Quartz.
Keep the Job isolated
It's discouraged to modify scheduling or execute additional jobs from within a job's execute method. This control-flow is kept outside from the jobs by Quartz. It's part of the Scheduler's responsibilities. Thus your current solution:
using the executor inside the Job i.e. call executor.execute() inside the Job's execute method
can inflict the correct function of the Scheduler itself.
Retry controlled from within the job
There might be a couple of ways how to deal with retries in Quartz.
Take a search here in Stackoverflow for [quartz-scheduler] retry:
Automatic retry of failed jobs
Count-based retries, increasing delays between retries, etc.
This question explains some:
Quartz retry when failure
Related
how to start the dataflow I created without accessing the apache nifi interface. Is it possible to trigger run by running a .bat file? I am new in apache nifi and somewhat clueless on the limitation of apache nifi
I saved the dataflow as a template and want to start it without accessing apache nifi interface
There are several ways to start a processor.
Timer driven
This is the default mode. The Processor will be scheduled
to run on a regular interval. The interval at which the Processor is
run is defined by the 'Run Schedule' option (see below).
CRON driven
When using the CRON driven scheduling mode, the Processor is scheduled
to run periodically, similar to the Timer driven scheduling mode.
However, the CRON driven mode provides significantly more flexibility
at the expense of increasing the complexity of the configuration. The
CRON driven scheduling value is a string of six required fields and
one optional field, each separated by a space.
Event driven
When this mode is selected, the Processor will be triggered to run by
an event, and that event occurs when FlowFiles enter Connections
feeding this Processor. This mode is currently considered experimental
and is not supported by all Processors. When this mode is selected,
the 'Run Schedule' option is not configurable, as the Processor is not
triggered to run periodically but as the result of an event.
Additionally, this is the only mode for which the 'Concurrent Tasks'
option can be set to 0. In this case, the number of threads is limited
only by the size of the Event-Driven Thread Pool that the
administrator has configured.
You can read more about it in the Scheduling part of the NiFi User Guide.
If you specifically want to start a processor from a bat file, you can use cURL. For that your flow must start with either ListenHTTP or HandleHttpRequest. E.g. if ListenHTTP listens on port 8089 and your NiFi instance is accessible via my-nifi-intance.com, then you will have a webhook like my-nifi-intance.com:8089/webhook that will initiate the flow.
Since you are asking a very basic question, I encourage you to start with reading the Apache NiFi User Guide.
I am writing a SCDF spi implementation for supporting stream and task application. As part of this we need to perform some clean up operations once the task finishes.
Could someone provide info on whether SCDF will be getting a callback on task completion. If not then what are the alternative ways to perform cleanup.
A Task is a short-lived and finite operation. Depending on what you're trying to accomplish, you can do one of the following to invoke any custom cleanup routine.
1) A task running a batch-job and in that job, you can define "n" number of steps as part of a workflow, and upon successful upstream steps, the last step could invoke the cleanup routine.
2) You can have a stream in SCDF listening to Task-complete events (a batch-job example here), which can finally kickoff another task/job to invoke the cleanup routine.
3) You can define a composed-task graph (via Dashboard/shell) where each of the steps (aka tasks) can run its intended operation, and upon successful transition or failure event, you get the opportunity to kick off the cleanup routine.
I'm using quartz 1.8.6 in clustered mode with 4 instances. Now, I observed high contention on table QRTZ_LOCKS. My application also provide webservices for online clients. This webservices also do scheduling of new jobs. Now, I see timeout exceptions on those webservices, because when they want to schedule new job they wait too loooong to obtain lock on QRTZ_LOCKS table. It's important for me to establish 100% reliable operation for webservices (more important than quartz jobs operations). Is it possible to start quartz job runner on 1 instance only and other 3 instances configure with org.quartz.jobStore.isClustered=false to allow them perform scheduling WITHOUT getting lock on QRTZ_LOCKS?
update: Actually, if I plan to run only one instance with job runner and all others just allowed to add new jobs this won't be a cluster anymore. So, actual question would be: is it possible to configure org.quartz.jobStore.isClustered=false to all 4 instances, make only 1 instance run jobs, but allow all 4 to schedule new jobs to same jdbc storage?
Try to turn batch mode on, and set maximum batch count to the amount of threads, available for quartz scheduler.
http://www.ebaytechblog.com/2016/01/14/performance-tuning-on-quartz-scheduler/
I am trying to understand how the various components of Mesos work together, and found this excellent tutorial that contains the following architectural overview:
I have a few concerns about this that aren't made clear (either in the article or in the official Mesos docs):
Where are the Schedulers running? Are there "Scheduler nodes" where only the Schedulers should be running?
If I was writing my own Mesos framework, what Scheduler functionality would I need to implement? Is it just a binary yes/no or accept/reject for Offers sent by the Master? Any concrete examples?
If I was writing my own Mesos framework, what Executor functionality would I need to implement? Any concrete examples?
What's a concrete example of a Task that would be sent to an Executor?
Are Executors "pinned" (permanently installed on) Slaves, or do they float around in an "on demand" type fashion, being installed and executed dynamically/on-the-fly?
Great questions!
I believe it would be really helpful to have a look at a sample framework such as Rendler. This will probably answer most of your question and give you feeling for the framework internal.
Let me now try to answer the question which might be still be open after this.
Scheduler Location
Schedulers are not on on any special nodes, but keep in mind that schedulers can failover as well (as any part in a distributed system).
Scheduler functionality
Have a look at Rendler or at the framework development guide.
Executor functionality/Task
I believe Rendler is a good example to understand the Task/Executor relationship. Just start reading the README/description on the main github page.
Executor pinning
Executors are started on each node when the first Task requiring such executor is send to this node. After this it will remain on that node.
Hope this helped!
To add to js84's excellent response,
Scheduler Location: Many users like to launch the schedulers via another framework like Marathon to ensure that if the scheduler or its node dies, then it can be restarted elsewhere.
Scheduler functionality: After registering with Mesos, your scheduler will start getting resource offers in the resourceOffers() callback, in which your scheduler should launch (at least) one task on a subset (or all) of the resources being offered. You'll probably also want to implement the statusUpdate() callback to handle task completion/failure.
Note that you may not even need to implement your own scheduler if an existing framework like Marathon/Chronos/Aurora/Kubernetes could suffice.
Executor functionality: You usually don't need to create a custom executor if you just want to launch a linux process or docker container and know when it completes. You could just use the default mesos-executor (by specifying a CommandInfo directly in TaskInfo, instead of embedded inside an ExecutorInfo). If, however you want to build a custom executor, at minimum you need to implement launchTask(), and ideally also killTask().
Example Task: An example task could be a simple linux command like sleep 1000 or echo "Hello World", or a docker container (via ContainerInfo) like image : 'mysql'. Or, if you use a custom executor, then the executor defines what a task is and how to run it, so a task could instead be run as another thread in the executor's process, or just become an item in a queue in a single-threaded executor.
Executor pinning: The executor is distributed via CommandInfo URIs, just like any task binaries, so they do not need to be preinstalled on the nodes. Mesos will fetch and run it for you.
Schedulers: are some strategy to accept or reject the offer. Schedulers we can write our own or we can use some existing one like chronos. In scheduler we should evaluate the resources available and then either accept or reject.
Scheduler functionality: Example could be like suppose say u have a task which needs 8 cpus to run, but the offer from mesos may be 6 cpus which won't serve the need in this case u can reject.
Executor functionality : Executor handles state related information of your task. Set of APIs you need to implement like what is the status of assigned task in mesos slave. What is the num of cpus currently available in mesos slave where executor is running.
concrete example for executor : chronos
being installed and executed dynamically/on-the-fly : These are not possible, you need to pre configure the executors. However you can replicate the executors using autoscaling.
I am using JSF-2, Spring 4, hibernate 4 in my application. I have Spring type service layer, Dao Layers , Models and other thing. I want to schedule some of the services which should be automatically executed or called at specified time, usually these services or business logic would perform some kind of data-mapping from excel-file to database.
I want to perform these task without user-intervention and scheduler should take care all these data-mapping.
Note : I am calling these services from my view as well as these services also should be used in scheduler to perform data-mapping.
I am newbie at utmost level, never used any kind of scheduler or anything. So my question :
1)what should I have to use to schedule these task?
2)I am confused regarding Spring Batch and Spring-sheduler? are they both perform scheduling ,if no then what is actual use of sping-batch?
3)Can spring-scheduler itself sufficient enough to perform these scheduling
Any help would be highly considerable.
1)what should I have to schedule these task?
Basically you need the classes that support the operations that you want to do (excel creation from database queries), spring in both cases.
2)I am confused regarding Spring Batch and Spring-sheduler? are they both perform scheduling ,if know then what is actual use of sping-batch?
Spring Batch provides reusable functions that are essential in
processing large volumes of records, including logging/tracing,
transaction management, job processing statistics, job restart, skip,
and resource management. It also provides more advanced technical
services and features that will enable extremely high-volume and high
performance batch jobs though optimization and partitioning techniques
Spring scheduler just run any method at certain time, it is not so robust, and only execute the logic involve on a process, not statistic, not job restart, just start a process during predefined period of time (calling a method of a class)
3)Can spring-scheduler itself sufficient enough to perform these scheduling?
Yes it is, if you are not very related with spring-batch this will take more time that just call the methods you already have.
Scheduler A scheduler is a software product that allows an enterprise
to schedule and track computer batch tasks
Scheduler just ran the process.