I am working on a Spring Boot project which has a scheduler service using Quartz. This scheduler service runs a specific scheduled or cron job by calling a separate micro-service.
Now I am planning to deploy my micro-services on kubernetes and I want to make sure that my scheduler service is also clustered. I have noticed that Quartz supports clustering using JDBC-JobStore but I am not entirely sure if such a scenario can be deployed using kubernetes. Will it work if I have 3 pods of the Scheduler service managed and scaled using kubernetes?
Yes as long as you run the quartz in Cluster Mode and set org.quartz.scheduler.instanceId = AUTO it does work.
In case of scaling, just some pods will be added or get removed, Quartz will take care of it since each instance would have its distinct instance ID.
Related
Is there any way to schedule and monitor spring batch using IBM workload scheduler.
If your application exposes REST APIs, the easiest solution is to use RESTful job type running on dynamic or zcentric agent, the job connects remotely to the REST API, so the agent can run on a separate machine or container.
I want to host a Spring Cloud Data Flow Local server for Monitoring and executing my various Spring Boot Batch projects.
The issue or the infrastructure I want to achieve is that, I want my Spring Cloud Data Flow Server host on Server A which can be able to execute Spring Boot Batches/Tasks on Server B.
Is this a possible configuration I am trying to achieve ? If not how should I achieve this ? Since I have few Spring Boot batch applications which run on different server.
This is not how SCDF works. So I do no think it is possible. If you want to monitor your batch jobs then you need to register your jobs in the SCDF server.
It depends on how you launch and configure your batch applications. You can have a custom task application (call it batch-launcher) that launches your batch job on an external cluster. But, in terms of monitoring the application, SCDF can help monitoring the task application (the batch-launcher) that is used for launching your actual batch but not the actual job that is running on the external cluster (unless you have a mechanism to retrieve the metrics of the batch application into the batch-launcher).
Launching a Spark compute job on a Spark cluster using an SCDF task (using Spark client) is one such example. In this case, you would register the SCDF task and monitor only the Spark client task app via SCDF (not the Spark Compute Job).
I have two spring boot applications, where one features the creating and scheduling of the job. There is another Spring Boot application where I configured the Quartz Scheduler, which will prepare the job parameters using a shared database and launches the spring batch job.
I need to update the running Quartz Scheduler if the user updates or adds a new job from UI. Also if server restarts I need to restart the Scheduler and the Jobs.
How should I update my Quartz Scheduler object when there's the new job added or updated by the user? My Quartz Scheduler will always be in running condition. Can I use REST Template so that my UI application will notify my scheduler application for the jobs?
You will need to store the jobs to schedule in a database (using the share database) from the UI application, create a job and build a trigger for it.
Its hard to say without any code given, but there is a guide that does something like you want.
https://www.callicoder.com/spring-boot-quartz-scheduler-email-scheduling-example/
This is a dynamic scheduler/trigger
I have a springboot microservice application, deployed as docker image in a Kubernetes cluster.
This microservice is (also) an Ignite Client.
If for some reason Ignite Bean fails the startup (for example because there is no Ingnite Server service), the overall application is not able to startup.
I would like to make not mandatory the complete startup of Ignite Bean, so my service can be up & running, even if it cannot use Ignite.
Thanks in advance
We are planning to retire the existing legacy java batch applications and recreate it with the latest available batch framework.
Given that we have a large number of batch jobs to be modernised, we are looking for a framework or architecture that would allow us to
Develop a batch solution that would allow us to dynamically deploy a new batch as and when they are created, without disturbing the existing deployed applications. - Does Spring cloud Task provide any of this feature. Note: We are looking only to deploy the apps to our local server, and has nothing to do with cloud.
If Spring Batch/Boot can provide us the feature we typically expect from a batch application, what is the special value add to go for Spring Cloud Task? - I wasn't able to completely understand this from the Spring documentation available online.
From the documentation of the Spring Cloud Task, I was able to understand that it allows an application to have many tasks within it. What should I do if each of the tasks have their own library dependencies, which might contradict with the dependencies of other Tasks? So in that case, should each of these tasks moved to a new application or this there a work around for that?
To answer your questions:
Does Spring Cloud Task handle orchestration - No. Spring Cloud Task does not handle orchestration of tasks or jobs. The component in this ecosystem that handles the deployment/orchestration of tasks or jobs is really Spring Cloud Data Flow (which is why I asked if you use any type of cloud platform including YARN, Cloud Foundry, Kubernetes, or Mesos...the environments supported by Spring Cloud Data Flow).
What added value does Spring Cloud Task provide over Spring Boot/Spring Batch - Spring Cloud Task is designed to provide a few things:
Similar abilities to Spring Batch with regards to state management without needing to create a batch job. When running a Boot application on a cloud environment, there is no standard way of getting the results from environment to environment (YARN handles job results differently from tasks on Cloud Foundry which is different from jobs on Kubernetes, etc). Spring Batch provides this but now all short lived processes need the overhead of the Batch API so Spring Cloud Task provides a lighter touch to those use cases.
Automatically adds informational listeners. With Spring XD, when you ran a job in an XD container, the XD container automatically added a number of informational listeners that broadcast events that you could listen for. Spring Cloud Task brings the same functionality without the need for the XD container.
Integration with Spring Cloud Stream. Spring Cloud Task provides the ability to launch tasks from messages received from Spring Cloud Stream. Also, the informational messages previously mentioned (both Batch events as well as Task events) are sent via Spring Cloud Stream channels.
The DeployerPartitionHandler. When working in a cloud environment, this PartitionHandler implementation allows you to launch workers for a partitioned batch job as tasks. This allows for the dynamic scaling of partitioned batch jobs instead of the traditional option of pre-deploying workers that listen for work which wastes resources in a modern cloud environment.
How does the packaging of multiple tasks work with dependencies - In short, this is not recommended. The idea of a Spring Cloud Task is that the execution of the Spring Boot application is the Task. While you could package up multiple tasks and using different methods, have them execute based on different stimulus, that goes against the 12 factor application concepts which are essential for correct use of Spring Cloud Task.
My two cents
For the best option for a modern batch platform, you really need to look into some from of platform first and that begins at the Cloud Foundry/Kubernetes/Mesos/YARN layer. Without that, you end up building a large part of the infrastructure yourself. That is why Spring XD evolved into Spring Cloud Data Flow. The added complexity that lived in the containers of Spring XD is removed by requiring a modern platform to run on (since they all handle those guarantees themselves). Without that piece, you're going to spend a lot of time managing the deployment and orchestration of applications that most modern platforms handle for you.
From there, the choice becomes pretty easy IMHO with Spring Cloud Task for simple tasks, Spring Batch for batch jobs, and Spring Cloud Data Flow for orchestration.