Scenario: The system needs to check the Product table in the database DAILY to check every product's expiration date. The system needs to get a list of products with expiration dates matching the current date. Then, removes these products in the database.
Things to consider:
- Runs a single query to retrieve products that has matching expiration date.
- Remove these products in the database.
- We are talking about thousands of products here.
QUESTION: Is there a need for me to create a Spring Batch Job supported with Scheduler for this or just a Scheduled Job to do this efficiently?
Because in Scheduled Job, I can just schedule the checking and removing daily. And its done. At the same time, I can also do it using Spring Batch with Scheduler. But, what do you think is the more efficient way?
i think spring batch would be a wise decision, if you need to restart your job with more steps inside. otherwise, if it's really just one job, you could solve that restart functionality anyway manually...
all those spring-batch configuration aren't that heavy in your applicationcontext, but you need to create tables for the according repository. (perhaps you gotta stage those tables as well..)
Spring Batch:
if you need restart-functionality
Common Scheduling: Easy and fast, ain't need that much knowledge about the framework
Related
I read a lot about how to enable parallel processing and chunking of an individual job, using Master/Slave paradigm. Consider an already implemented Spring Batch solution that was intended to run on a standalone server. With minimal refactoring I would like to enable this to horizontally scale and be more resilient in production operation. Speed and efficiency is not a goal.
http://www.mkyong.com/spring-batch/spring-batch-hello-world-example/
In the following example a Job Repository is used that connects to an initializes a database schema for the Job Repository. Job initiation requests are fed to a message queue, that a single server, with a single Java process is listening on via Spring JMS. When encountering this it executes a new Java process that is the Spring Batch job. If the job has not been started according to the Job Repository it will begin. If the job had failed it will pick up where the job left off. If the job is in process it will ignore.
The single point of failure is the single server and single listening process for job initiation. I would like to increase resiliency by horizontally scaling identical server instances all competing for who can first grab the job initiation message when it first appears in the queue. That server instance will now attempt to run the job.
I was conceiving that all instances of the JobRepository would share the same schema, so they can all query for when the status is currently in process and decide what they will do. I am unsure though if this schema or JobRepository implementation is meant to be utilized by multiple instances.
Is there a risk in pursuing this that this approach could result in deadlocking the database? There are other constraints to where the Partition features of Spring Batch will not work for my application.
I decided to build a prototype to test if the condition that the Spring Batch Job Repository schema and SimpleJobRepository can be used in a load balanced way with multiple Spring Batch Java processes running concurrently. I was afraid that deadlock scenarios might have occurred at the database to where all running job processes get stuck.
My Test
I started with the mkyong Spring Batch HelloWorld example and made some changes to it where it could be packaged into a Jar that can be executed from the command line. I also removed the initialize database step defined in the database.config file and manually established a local MySQL server with the proper schema elements. I added a Job parameter for time to be the current time in millis so that each job instance would be unique.
Next, I wrote a separate Java main class that used Apache Commons Exec framework to create 50 sub processes with no wait between them. Each of these processes have a Thread.sleep for 1 second within their Processor objects as well so that a number of processes will all kick off at the same time and all attempt to access the database at the same time.
Results
After running this test a number of times in a row I see that all 50 Spring batch processes consistently complete successfully and update the same database schema correctly. I don't see any indication that if there were multiple Spring Batch job processes running on multiple servers connecting to the same database that they would interfere with each other on the schema nor do I see any indication that a deadlock could happen at this time.
So it sounds as if load balancing of Spring Batch jobs without the use of advanced Master/Slave and Step Partitioning approaches is a valid use case.
If anybody would like to comment on my test or suggest ways to improve it I would appreciate it.
Here is excerpt from
Spring Batch docs on how Spring Batch handles database updates for its repository:
Spring Batch employs an optimistic locking strategy when dealing with updates to the database. This means that each time a record is 'touched' (updated) the value in the version column is incremented by one. When the repository goes back to save the value, if the version number has changed it throws an OptimisticLockingFailureException, indicating there has been an error with concurrent access. This check is necessary, since, even though different batch jobs may be running in different machines, they all use the same database tables.
Problem
Users can submit data to generate a report, which triggers a spring-batch job. If the same data is submitted (by the same user or another user) the same report should be generated such that Spring Batch doesn't start a new job, under the premise that the report has already been generated.
To make matters a little more complicated, generated reports expire after 90 days. The idea behind this is that the data gleaned from various web services used to build the report is likely out of date. Therefore, after 90 days the report should be re-generated using new data from those web services.
Questions
When a job has already run, how can I discover the job execution id for that job? This id is used in the URL to uniquely identify a report. JobExplorer is severely limited in querying Spring Batch data.
How can I trigger another instance of the job only after 90 days? The issue is that given duplicate job parameters, a JobInstanceAlreadyCompleteException will be thrown. Must I encode the 90 days has an extra identifying parameter, or is there an easier way?
Clean up old jobs must be done using business methods as well as for expired reports.
After this premise you can try a different path to solve your problem:
Every user launch a different job, with same report properties but
an extra job-parameters to make every jobs different
First step is to check - using business method - if there is a running job for that report ; in this case notify user he have to wait or retry later (use a decider)
Second step is to check if there is a completed - not expired - report using a business method;if so retrieve it and show to user (use a decider as for step 2)
Generate report (delete old, if necessary)
Show report to user.
Of course, generated report metadata tables are different from SB tables and should be accessed using DAO related to domain context (the report in your case).
Can this a valid alternative?
I would like to be able to set all indexes to "Update on schedule" and then have them all update automatically like Magento says they should in the background. The problem is, this doesn't happen. There is no cron job that automatically reindexes (See this related question).
So, if I have to create my own cron job, how do I do this exactly in an efficient way? I don't want to run "php shell/indexer.php reindexall". That does full rebuilds of index tables. Sure, I could do that nightly, but that means that no changes will be reflected on the frontend until the next day. That's not an acceptable solution. If I run full reindexes throughout the day, I end up with the same problem that I have right now - table locks and slowness due to reindexing while people are working in the admin.
Magento's new "partial reindexing" should fix this right?
This is my understanding of how it works:
I edit an entity that has a related index (e.g. A product).
A database trigger adds a record to related change log tables.
Some process later reads the change log tables and reindexes these specific entities
Concrete example
I update a value in "catalog_product_entity_varchar".
The database trigger "trg_catalog_product_entity_varchar_after_update" flags this product as changed by inserting a new version into "catalog_product_flat_cl" and "catalogsearch_fulltext_cl".
A partial reindex process reads these change log tables and reindexes only the products mention to "catalog_product_flat" and "catalogsearch_fulltext" respectively.
If this were the case, the reindexing process would be minimal and could be run often. Even every minute to where indexing becomes almost unnoticeable to admin users. (I say every minute, because Magento tells us this is possible)
In this release, however, the flat catalog is updated for you — either every minute, or according to your Magento cron job.
Where is this mystical partial reindex? How do I call it instead of reindexing everything?
Is there a reindexPartial()?
The enterprise_refresh_index cron job appears to run this. It runs every time the Magento cron runs. See Enterprise_Index_Model_Observer::refreshIndex().
This is not intended to run manually because of the need to establish a lock file. It is easiest just to run the cron.php file if you need a manual reindex.
I believe I just have a project specific issue with this not running.
The partial reindexing is executed through the cron job operator built into Magento. You do not need to run the actual indexer.php file. Instead, you must setup Magento's built in Cron scheduler based on the documentation.
Documentation: http://www.magentocommerce.com/wiki/groups/227/setting_up_magento_in_cron
You simply execute the cron.php file, which will in turn call the partial reindexing process.
php5-cli -f /home/USERNAME/public_html/cron.php
How it works:
A change to the an entity is made and is flagged to be reindex.
A cronjob executes the cron.php file
Magento checks to see which cron tasks it will run, and runs the partial reindexing process
The indexing process will see the changed entity and update the index tables with the new values.
I would like to develop an application which listens a database table and process the data immediately. Batch does not execute for only one time. It listens the table regularly, for example with 10 seconds interval. Whenever new data are inserted in the table, the batch starts the process.
Is it possible to develop such an application with "Spring Batch". If yes, can you please give some advice for it?
Thank you.
This goal can be archived using quartz ; just launch the job every 10 seconds and process, in your step, the new data. I haven't tried myself and can't check now (I'm writing from my mobile) but there are a lot of example on SO.
A newbie question on Sprint Batch Admin.
My requirement is that the user should be able to schedule new jobs (passing some parameters for the job functionality) through a web UI. These jobs should be persistent, will be repetitive and could be cancelled or deleted. Also, a report could be generated for last run jobs and to list all the existing jobs with their next run dates.
Perhaps my most important requirement is that this should be possible "on the fly", not requiring redeploying the web-application or a server re-start.
Can this be done using Spring Batch Admin (I see that the guide talks about uploading an XML for adding a job but that seems tedious, if there is an API why shouldn't we be able to create a job on the fly through the Batch Admin Web UI)? Or does JDK Timer or Quartz support it?
Once a job has been created, it can't be deleted, but it can be stopped. Allowing deletion from DB is a risky operation, as Spring Batch might have already been started the job execution, but the DB has not been updated yet. If one removes the job at this moment, you have inconsistency.
Scheduling a new job is described in Launch Job. It is not possible to create new types of jobs, as jobs can generally have complicated configuration which is parsed only once when Spring Context is loaded.
Dynamic deployment (on the fly) of jobs and configurations, without requiring server restart, is a feature we implemented in Trooper Batch Profile - it is not exactly Spring Batch admin but builds on it. You continue to write your jobs using Spring batch, just the container changes for in Trooper you would use its Batch profile runtime. Screen shots and features are here : https://github.com/regunathb/Trooper/wiki/Writing-Batch-jobs-in-Trooper
I think we can deploy the each spring batch job by a SBA. I mean each batch job will be compiled as a war file. We deploy them together in server. In this way, we have the following visiting urls to monitor each jobs:
h t t p://bactchjobserver/job1
h t t p://bactchjobserver/job2
h t t p://bactchjobserver/job3
h t t p://bactchjobserver/job4
But the downside is that each war fill surely contains lib files, which make each war file like 10MB size.
At the same time, I tried to manually add new-job.xml to war-file\WEB-INF\classes\META-INF\spring\batch\jobs, and new-job.jar to war-file\WEB-INF\lib without stopping JBoss. It works. The new job can be showed in SBA UI and runnable.
But obviously this would lead much maintenance and trouble shooting. It is not implementable.