Our service will schedule some tasks in the future by TaskScheduler, these tasks will store in memory so to prevent tasks was lost after restarted, I stored it in database, and reload the tasks when startup. But if I have multiple instances of service, the tasks will loaded in every instances. How to deal with this problem ?
You could implement a lock system with your database. Add a new column to store the name of the instance which has already loaded tasks.
When an instance start, it requests the database to know if any other instance has already loaded tasks (instance name are set in DB). If the column is already set, the instance does nothing (jobs are already done by another instance). Else, the instance set his own name on DB and loads tasks.
Note that for updating the instance name, you'll need optimistic locking in case several instances try to acquire a lock at the same time (see: https://www.baeldung.com/jpa-optimistic-locking)
Related
I have a nginx loadbalancer in front of two tomcat instances each contains a spring boot application. Each spring boot application executes a batch that writes data in a database.
The batch executes every day at 1am.
The problem is that both instances execute the batch simultaniously which i don't want.
Is there a way to keep the batchs deployed in two instances and tell tomcat or nginx to start the batch in master server (and the slave server doesn't run the batch).
If one of the servers stops, the second server could start the batch on his behalf.
Is there a tool in nginx or tomcat (or some other technology) to do that ?
thank you in advance.
Here is a simplistic design approach.
Since you have two scheduled methods in the 2 VMs triggered at same time, add a random delay to both. This answer has many options on how to delay the trigger for a random duration. Spring #Scheduled annotation random delay
Inside the method run the job only if it is NOT already started (by the other VM). This could be done with a new table to track this.
Here is the pseudo code for this design:
#Scheduled(cron = "schedule expression")
public void batchUpdateMethod() {
//Check database for signs of job running now.
if (job is not running){
//update database table to indicate job is running
//Run the batch job
//update database table to indicate job is finished
}
}
The database, or some common file location, should be used as a lock to sync between the two runs, since the two VMs are independent of each other.
For a more robust design, consider Spring Batch
Spring Batch uses a database for its jobs (JobsRepository). By default an in memory datasource is used to keep track of running jobs and their status. In your setup, the 2 instances are (most likely) using their own in memory database.
Multiple instances of Spring Batch can coordinate with each other as a cluster and one can run jobs, while the other actasa backup, if the jobsRepository database is shared.
For this you need to configure the 2 instances to use a common datasource.
Here are some docs:
https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#jobrepository
https://docs.spring.io/spring-batch/docs/current/reference/html/job.html#configuringJobRepository
If you design two app server instances to run the same job at the same time, then by design, one will succeed to create a job instance and the other will fail (and this failure can be ignored). See Javadoc of JobRepository. This is one of the roles of the job repository: to act as a safeguard against duplicate job executions in a clustered environment.
If one of the servers stops, the second server could start the batch on his behalf. Is there a tool in nginx or tomcat (or some other technology) to do that ?
I believe there is no need for such tool or technology. If one of the servers is down at the time of the schedule, the other will be able to take things over and succeed in launching the job.
I did implement a simple BCM Server functionality, where all servers do register(create a Server-table entry) with their unique IP. The Servers need to register within a defined time(e.g. 10 sec). If a Server does not register within time(last update timestamp > 10 sec), then the Server gets de-registered(delete Server-table entry) by the Server, which do register.
At the end I do have a table with ordered Server entries and can define the task uniquely to the registered Servers.
The Implementation is very simple and Works perfectly.
Before I did also have in mind the Spring Batch Job Sharing functionality, but I wanted zu have a more lightweight and more flexible Solution.
Currently I use it in all my projects where I need to have Batch-Processing implemented.
We have our application deployed on Websphere application server. The application is running on clustered environment with 6 Nodes. EJB timer service is configured using custom scheduler with datasource pointing to Oracle database.So when the application is deployed on cluster it triggers the Ejb timer service on Node1 which is given in the Oracle database.
Some times the value in oracle database changes automatically to some
other nodes like node2 or node3 because of which EJB timer is getting
stopped.Any Suggestions or advice on why it gets changed automatically.
EJB timer configuration
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).datasourceJNDIName = jdbc/cdb_db
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).nonPersistentTimerRetryCount = -1
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).nonPersistentTimerRetryInterval = 300
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).numAlarmThreads = 1
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).numNPTimerThreads = 1
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).pollInterval = 300
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).tablePrefix = EJBTIMER_
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).uniqueTimerManagerForNP = false
As the first comment added to this question points out, it is the designed behavior of EJB Persistent Timers/Scheduler to have any one member run all of the tasks until that member isn't available or cannot respond quickly enough, in which case another member takes over.
If you don't like this behavior and want to change it so that your timer tasks can only run on a single member, you can accomplish that by stopping the scheduler poll daemon on all members except for the one that you want to run the tasks. Here is a knowledge center document which describes how to do that:
https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/scheduler/xmp/xsch_stopstart.html
Just be aware that if you do this, you will be losing out on the ability for the scheduler to automatically start running tasks on a different member should the member that you have designated to run them go down. In this case, tasks will not run at all until either of
1) the member that is allowed to run them comes back up, or
2) you manually use the aforementioned WASScheduler MBean to start the scheduler poll daemon on a different member, thus allowing tasks to run there
I saw Neo4j can run as Impermanent DB for unit testing porpouses, I'm not sure if this fits my needs. I have my data stored in neo4j the usual way (persistent) but, starts from my data, I want to let each user start an "experimental session": the users add/delete nodes and relationships, but NOT in permanent way, just experimenting with the data (after that session the edits should be lost). The edits shouldn't be saved and obiouvsly they shouldn't be visibile to the others. What's the best way to accomplish that?
Using impermanent database should work. You would
need to import the data to each new database
spring-data-neo4j is not able to connect to multiple databases (in current release), you would need to start multiple instances of your application, e.g. in a tomcat container
when your application stops (or crashes) you would obviously lose data
Or you could potentially use only 1 database with the base data being public (= visible to everyone) and then for all new nodes/relationships you can add owner property.
When querying the data you would check the property is either public or the current user.
At the end of the session you would just delete all nodes and relationships with given owner.
If you also want to edit existing data then it gets more complicated, you could create a copy of the node/relationship and somehow handle that, or if it's not too large copy whole dataset.
You can build a docker image from the neo4j base image (or build your own) and copy your graph.db into it.
Then you can have every user start a docker container from said image.
If that doesn't answer your question, more info is needed.
I read a lot about how to enable parallel processing and chunking of an individual job, using Master/Slave paradigm. Consider an already implemented Spring Batch solution that was intended to run on a standalone server. With minimal refactoring I would like to enable this to horizontally scale and be more resilient in production operation. Speed and efficiency is not a goal.
http://www.mkyong.com/spring-batch/spring-batch-hello-world-example/
In the following example a Job Repository is used that connects to an initializes a database schema for the Job Repository. Job initiation requests are fed to a message queue, that a single server, with a single Java process is listening on via Spring JMS. When encountering this it executes a new Java process that is the Spring Batch job. If the job has not been started according to the Job Repository it will begin. If the job had failed it will pick up where the job left off. If the job is in process it will ignore.
The single point of failure is the single server and single listening process for job initiation. I would like to increase resiliency by horizontally scaling identical server instances all competing for who can first grab the job initiation message when it first appears in the queue. That server instance will now attempt to run the job.
I was conceiving that all instances of the JobRepository would share the same schema, so they can all query for when the status is currently in process and decide what they will do. I am unsure though if this schema or JobRepository implementation is meant to be utilized by multiple instances.
Is there a risk in pursuing this that this approach could result in deadlocking the database? There are other constraints to where the Partition features of Spring Batch will not work for my application.
I decided to build a prototype to test if the condition that the Spring Batch Job Repository schema and SimpleJobRepository can be used in a load balanced way with multiple Spring Batch Java processes running concurrently. I was afraid that deadlock scenarios might have occurred at the database to where all running job processes get stuck.
My Test
I started with the mkyong Spring Batch HelloWorld example and made some changes to it where it could be packaged into a Jar that can be executed from the command line. I also removed the initialize database step defined in the database.config file and manually established a local MySQL server with the proper schema elements. I added a Job parameter for time to be the current time in millis so that each job instance would be unique.
Next, I wrote a separate Java main class that used Apache Commons Exec framework to create 50 sub processes with no wait between them. Each of these processes have a Thread.sleep for 1 second within their Processor objects as well so that a number of processes will all kick off at the same time and all attempt to access the database at the same time.
Results
After running this test a number of times in a row I see that all 50 Spring batch processes consistently complete successfully and update the same database schema correctly. I don't see any indication that if there were multiple Spring Batch job processes running on multiple servers connecting to the same database that they would interfere with each other on the schema nor do I see any indication that a deadlock could happen at this time.
So it sounds as if load balancing of Spring Batch jobs without the use of advanced Master/Slave and Step Partitioning approaches is a valid use case.
If anybody would like to comment on my test or suggest ways to improve it I would appreciate it.
Here is excerpt from
Spring Batch docs on how Spring Batch handles database updates for its repository:
Spring Batch employs an optimistic locking strategy when dealing with updates to the database. This means that each time a record is 'touched' (updated) the value in the version column is incremented by one. When the repository goes back to save the value, if the version number has changed it throws an OptimisticLockingFailureException, indicating there has been an error with concurrent access. This check is necessary, since, even though different batch jobs may be running in different machines, they all use the same database tables.
How do we define spring batch jobs to run against multiple tenants?
I have setup to run a sequence of jobs in order every night against one database schema currently. All the jobs currently read files from a location and insert to database.The batch configuration was very basic where I defined a data-source, transaction manager and mapped job-repository to it. My jobs will point to this repository and transaction manager. Also I am currently persisting batch meta data information in database.
My new requirement is to able to run the same jobs (executed in order) against multiple tenants. Each tenants data can live in same database server but different schema or even different database servers. My questions are
1) Do we store the batch specific metadata information for all the tenants in one common database or each tenant database should have its own?
2) My understanding is that we need a data-source per tenant so that jobs specific to this tenant will have access to database to store data read from files. Does spring batch repository should also point to current data-source when executing jobs for that tenant?
3) We are planning to start all tenants [jobs] paralleled meaning JOB1 can be running at the same time for the all the tenants. At this time I am still not sure how to manage job-repository , data-source, transaction manage when these tenants are running with each associated to a different data-source.
4) At the top of my head all I am thinking is to duplicate my existing configuration for each tenant with own job-repositoyy pointing to tenant specific data-source and transaction manager. This is the last thing I would implement if there is no other way to define the same dynamically with out duplicating.
If any body has solved or has any ideas on how to approach to a solution please share. A sample config should help.
I was involved in building a SaaS application where you needed to do something similar but not exactly using Spring Batch.
The main idea for you is to:
a. Define a master database where you would store all the configuration specific data say suppose you have a table which maps your tenant name, information and datasource configuration.
b. Start your application and read this data source and maintain a local cache at your server end with key as your tenant name and value as the tenant information (data source etc.)
c. Maintain a thread local with you, example:
public class TenantThreadLocalContext
{
public static final ThreadLocal<TenantInformation> threadLocal = new ThreadLocal<TenantInformation>();
public static void set(TenantInformation tenantInformation)
{
threadLocal.set(tenantInformation);
}
public static void unset()
{
threadLocal.remove();
}
public static TenantInformation get()
{
return threadLocal.get();
}
}
d. Whenever you are starting any thread to begin your processing (batch processing)set this thread local with the tenant information so that each of this thread knows that it is associated to which tenant.
e. Finally at the time of database processing you could see that the thread has what data source and you could use this data source to make a connection.
If you are using Hibernate then you are lucky as Hibernate 4 has done all this for you. Refer: this. If you need help in hibernate configurations, etc, then probably I could help you out there as well.