Are there some alternative open source GUI developed to improve the management of jobs and job chains in Open Source Job Scheduler?
In 3+ months I didn't receive any answer, so I guess is time to answer myself "No. Seems like there aren't any alternative open source GUI developed for Job Scheduler".
I know this is tooooo old to answer. But yes, a graphical XML editor for job configuration and a web GUI for job management are available. JobScheduler can be controlled by the built-in web server's graphical user interface. JobScheduler uses an XML file for the configuration of executable files or shell scripts and to set the timing and frequency of job starts.
JobScheduler has two web based GUIs:
• Operations GUI
To operate the operations GUI the no web server is needed, because JobScheduler has its own built-in web server.
• Managed Jobs GUI
To operate the Managed Jobs interface at least an Apache or IIS is required. In addition, the Managed Jobs web interface needs PHP.
Read more here
Update 2018
Chronos is a Job scheduler. It run on top of Apache Mesos, so it's a best to install if you are not running a data center, but it gets the job done. Comes with a GUI as well. Stores job state in Zookeeper.
Related
I was reading about Spring Batch and I read the below:
Spring Batch is not a scheduling framework. There are many good
enterprise schedulers (such as Quartz, Tivoli, Control-M, etc.)
available in both the commercial and open source spaces. It is
intended to work in conjunction with a scheduler, not replace a
scheduler.
Source: https://docs.spring.io/spring-batch/docs/current/reference/html/spring-batch-intro.html#springBatchBackground
So what is the difference between Spring Batch and Tivoli?
Spring Batch is mainly designed to provide a runtime for java batch workload.
IBM Workload Scheduler (Tivoli) / HCL Workload Automation, like other schedulers, doesn't run the workload directly, but is used to triggers any kind of workload (jobs), including Spring Batch, on on-prem or hybrid and multi cloud environments, including Kubernetes.
It can trigger jobs based on calendar, time, considering free/working day, complex runcycles (e.g. 3 working days before the end of each month).
In addition it can trigger workload based on dependencies on other jobs, so that they can start as soon as the previous job (running on any other system) has completed successfully, or run jobs only if predecessor has completed with a specific RC or result. Or you can use logic resources and limits to control how many jobs using the same machine or resource can run at the same time.
It can be also used to trigger workload based on events, e.g. when a new file is uploaded.
In recent releases IBM Workload Scheduler / HCL Workload Automation also added built-in capabilities to transfer files.
IBM Workload Scheduler / HCL Workload Automation is also key to have a centralized monitoring and recovery of failures, to centralize security granting access to different teams only on their jobs, to have a centralized governance (e.g. auditing any change and recovery on jobs).
It's also able to forecast the job durations and when every job will run, and generate alerts if they are running too long or if based on predecessors they are expected to miss their deadline.
We are trying to revamp our batch job scheduling and monitoring process over the entire enterprise. Currently all our batch jobs are scheduled using Unix crontab and are monitored using log files generated by shell scripts.
This process has lot of disadvantages and as the number of applications grow this gets really complicated.
Two copies of applications need to be deployed one to App-Server and one as standalone(since business logic is shared between both). This is complicating our build process too.
There is no easy of use web-ui for us to see the status of jobs and manually run failed jobs remotely without getting onto the unix box.
There is no fail over or load balanced batch processing.
So I was thinking of using Quartz (with our existing Spring apps) in our applications and deploy them to App-Servers and no longer rely on the unix crontab.
Is there a way I can write a centralized web application from where I can schedule and monitor jobs running on different quartz schedulers on different app servers?
P.S: I know quartzdesk.com is one solution, but I don't want to enable RMI on my JVM.
You could use SpringBoot scheduler as an Orchestrator and call REST APIs for the remote (or local, if you are small) execution. This way, as your app grows you could easily leverage a load balancer.
If you have the possibility of using cloud services (like Amazon, Azure or Google Cloud), this could be done easily using their own load balancers. They also support docker and could take care of any peaks of utilization.
I am trying to understand how the various components of Mesos work together, and found this excellent tutorial that contains the following architectural overview:
I have a few concerns about this that aren't made clear (either in the article or in the official Mesos docs):
Where are the Schedulers running? Are there "Scheduler nodes" where only the Schedulers should be running?
If I was writing my own Mesos framework, what Scheduler functionality would I need to implement? Is it just a binary yes/no or accept/reject for Offers sent by the Master? Any concrete examples?
If I was writing my own Mesos framework, what Executor functionality would I need to implement? Any concrete examples?
What's a concrete example of a Task that would be sent to an Executor?
Are Executors "pinned" (permanently installed on) Slaves, or do they float around in an "on demand" type fashion, being installed and executed dynamically/on-the-fly?
Great questions!
I believe it would be really helpful to have a look at a sample framework such as Rendler. This will probably answer most of your question and give you feeling for the framework internal.
Let me now try to answer the question which might be still be open after this.
Scheduler Location
Schedulers are not on on any special nodes, but keep in mind that schedulers can failover as well (as any part in a distributed system).
Scheduler functionality
Have a look at Rendler or at the framework development guide.
Executor functionality/Task
I believe Rendler is a good example to understand the Task/Executor relationship. Just start reading the README/description on the main github page.
Executor pinning
Executors are started on each node when the first Task requiring such executor is send to this node. After this it will remain on that node.
Hope this helped!
To add to js84's excellent response,
Scheduler Location: Many users like to launch the schedulers via another framework like Marathon to ensure that if the scheduler or its node dies, then it can be restarted elsewhere.
Scheduler functionality: After registering with Mesos, your scheduler will start getting resource offers in the resourceOffers() callback, in which your scheduler should launch (at least) one task on a subset (or all) of the resources being offered. You'll probably also want to implement the statusUpdate() callback to handle task completion/failure.
Note that you may not even need to implement your own scheduler if an existing framework like Marathon/Chronos/Aurora/Kubernetes could suffice.
Executor functionality: You usually don't need to create a custom executor if you just want to launch a linux process or docker container and know when it completes. You could just use the default mesos-executor (by specifying a CommandInfo directly in TaskInfo, instead of embedded inside an ExecutorInfo). If, however you want to build a custom executor, at minimum you need to implement launchTask(), and ideally also killTask().
Example Task: An example task could be a simple linux command like sleep 1000 or echo "Hello World", or a docker container (via ContainerInfo) like image : 'mysql'. Or, if you use a custom executor, then the executor defines what a task is and how to run it, so a task could instead be run as another thread in the executor's process, or just become an item in a queue in a single-threaded executor.
Executor pinning: The executor is distributed via CommandInfo URIs, just like any task binaries, so they do not need to be preinstalled on the nodes. Mesos will fetch and run it for you.
Schedulers: are some strategy to accept or reject the offer. Schedulers we can write our own or we can use some existing one like chronos. In scheduler we should evaluate the resources available and then either accept or reject.
Scheduler functionality: Example could be like suppose say u have a task which needs 8 cpus to run, but the offer from mesos may be 6 cpus which won't serve the need in this case u can reject.
Executor functionality : Executor handles state related information of your task. Set of APIs you need to implement like what is the status of assigned task in mesos slave. What is the num of cpus currently available in mesos slave where executor is running.
concrete example for executor : chronos
being installed and executed dynamically/on-the-fly : These are not possible, you need to pre configure the executors. However you can replicate the executors using autoscaling.
I'm using Watin for browser testing, which has to run on the UI thread.
I want to schedule a job using Quartz.NET but can't work out a way to run it on the UI thread (using a WPF application)
Any ideas?
I'm not sure how you're running watin but you could:
1. Start the scheduler from inside wherever you are running watin and then connect to it via remoting from the UI thread to schedule the job.
2. Start the scheduler as a windows service and then connect to it via remoting from the UI thread to schedule the job.
3. Write a simple console app that starts the scheduler and exposes it via remoting. Then connect to it from the UI thread to schedule your job.
Take a look at this answer I wrote up earlier with some code samples:
https://stackoverflow.com/questions/1356789/quartz-net-with-asp-net/. Hopefully it will be useful.
I am not familiar with Quartz.NET but the Java version gives ThreadPool interface which can be implemented to make custom threadpool implementations. I don't understand why you would want to run it in a UI thread which is already dedicated to another task.
is there any way i can make my records in the database to be automatic. e.g i want a message to be sent to helpdesk if a requested service is not attended within 24 hours, without clicking anything.
technically it depends on the database you are using. if the database supports it, you could set up a scheduled job to scan the records and identify late services and email the helpdesk.
if the database doesn't support scheduled tasks then you could set up a client job on a timer to do the same thing.
This is what application software is for.
When the application saves to the database, the application also sends an email.
The traditional approach to this is to schedule a job (there are too many ways[1] to do that for me to go into details without knowing your server operating system, DBMS, and how much control you have to install or schedule programs on the server).
Your scheduled job would regularly check the database for records that have not been attended, and then take the appropriate action such as emailing the support team.
[1] Just so that this is not left completely unanswered; some DBMS (ex. SQL Server) have built in job scheduling facilities. You could run a Windows service on the server to do this. If not, you might consider running a Windows Service on one of your own servers to access the website (a great way to waste bandwidth).
Use a scheduler like this one, found on rufus site. You could program it to run, for instance, every hour, and make it do the job without human interaction.
I am a Java shop myself and I've been using quartz. It is quite good and usable if you can adjust to jruby.
I've never liked database or operating system based solutions, since you might not control them and often get asked to run on different environments.
Here's a very simple background job handler for Ruby:
codeforpeople.rubyforge.org/svn/bj/trunk/README
Easy to install and use. Fairly lightweight. It uses a SQL backend for managing concurrency. Runs on multiple machines simultaneously if you need it to.