I have a spring boot application (which reads data from txt file --> insert into database --> execute few sql queries --> write to csv file). It will run using a cron scheduler and is deployed in namespace prod-1. I want to deploy the same application in one more namespace prod-2 for backup (disaster recovery). I want to implement a fail over mechanism such that whenever the job in prod-1 failed, it should automatically start job in prod-2 and pick up from where it failed. Any ideas regarding the approach would be appreciated.
Related
There are three instances of the application on the server. but we want to execute the cronjob schedular logic on one instance of the application.
Using Spring-data, couchbase repository with couchbase database. is there any simple solution to my problem please suggest me. thanks in advance . i suffer from this problem from many days.
Facing similar situations with multiple spring boot instances, we schedule the cron externally, this can be a simple cron script that executes a curl, or a specific external scheduler app/instance. Your load balancer will pick an instance to run on.
You could also consider using quartz or shedlock with couchbase to manage without external trigger.
I have a working Spring Boot application which embeds a Spring Batch Job. The job is not run on a schedule, instead we kick it with an endpoint. It is working as it should. The basics of the batch are
Kick the endpoint to start the job
Reader reads from input file
Processor reads from oracle database using jpa repository and simple spring datasource config
Writer writes to output file
However there are new requirements:
The schema of the repository database is from here on unknown on application startup. The tables are the same, it is just an unknown schema. This fact is out of our control and you might think it is stupid but there are reasons for it and this cant be changed. This means that with current functionality we need to reconfigure the datasource when we know the new schema name, and restart the application. This is a job that we will run for a number of times when migrating from one system to another, so it has a limited lifecycle and we just need a "quick fix" to be able to use it without rewriting the whole app. So what I would like to do is:
Send the schema name as a query param to the application, put it in job parameters and then - get a new datasource when the processor reads from the repository. Would this be doable at all using Spring Batch? Any help appreciated!
I have one #Scheduler job which runs on multiple servers in a clustered environment. However I want to restrict the job to run in only in one server and other servers should not run the same job once any other server has started it .
I have explored Spring Batch has lock mechanism using some Database table , but looking for any a solution only in spring task:scheduler.
I had the same problem and the solution what I implemented was a the Lock mechanism with Hazelcast and to made it easy to use I also added a proper annotation and a bit of spring AOP for that. So with this trick I was able to enforce a single schedule over the cluster done with a single annotation.
Spring Batch has this nice functionality that it would not run the job with same job arguments twice.
You can use this feature so that when a spring batch job kicks start in another server it does not run.
Usually people pass a timestamp as argument so it will by pass this logic, which you can change it.
We have Spring4 and Spring Batch 3 and our app consumes CSV files as input file. Currently we kick off the jobs manually from the command line, using CommandLineJobRunner with parms, including the name of the file to process.
I want to kick off a job to process asynchronously just as soon as the input file arrives in a monitored directory. How can we do that?
You may use java.nio.file.WatchService to monitor directory for a file.
Once file appears you may start (or kick off a job to process asynchronously) actual processing.
You may also use FileReadingMessageSource.WatchServiceDirectoryScanner from Spring Integration (https://docs.spring.io/spring-integration/reference/html/files.html#watch-service-directory-scanner)
Comparing release notes Spring Batch https://github.com/spring-projects/spring-batch/releases
to Spring Integration https://github.com/spring-projects/spring-integration/releases it looks that Spring Integration is released more often. It also has more features and Integration points.
In this case it looks like a overkill to bring Spring Integration if you just need to watch a directory for a file.
I would recommend using the powerful combination of Spring Batch with Spring Integration. For example, you can use a FileInboundChannelAdapter from Spring Integration to monitor a directory and start a Spring Batch Job as soon as the input file arrives.
There is a code example for this typical use case in the reference documentation of Spring Batch here: https://docs.spring.io/spring-batch/4.0.x/reference/html/spring-batch-integration.html#launching-batch-jobs-through-messages
I hope this helps.
I am able to import data from my MS sql to HDFS using JDBCHDFC Spring Batch jobs.But if that containers fails , the job does not shift to other container. How do I proceed to make the job fault tolerant.
I am using spring xd 1.0.1 release
You don't mention which version of Spring XD you're currently using so I can't verify the exact behavior. However, on a container failure with a batch job running in the current version, the job should be re-deployed to a new eligible container. That being said, it will not restart the job automatically. We are currently looking at options for how to allow a user to specify if they want it restarted (there are scenarios that fall into both camps so we need to allow a user to configure that).