Spring Batch - Running a particular job in an application with multiple jobs - spring-boot

I have just got into a new project, which is all about migrating COBOL jobs to spring batch.
Before I joined, my colleague started on this migration and created the spring batch application with 1 job. My colleague was/is able to run the spring batch job from the command line just by running the application's jar file.
After I joined, I followed the 1st spring batch job and developed the 2nd spring batch job. But my spring batch job is not running from the command line by running the application's jar file like the 1st spring batch job.
I tried to explicitly launch my job (2nd job) from the commandlinerunner's run method and it worked. (The 1st spring batch job didn't have to do this).
Once the jobs are developed and tested, each job will be run from the Tivoli workload scheduler scheduled at different times.
The command that I used from command line:
java -Dspring.profiles.active=dev -jar target/SPRINGBATCHJOBS-0.0.2-SNAPSHOT.jar job.name=JOB2 dateCode=$$$$
Package structure
App
src
main
java
com.test
BatchApplication.java
common
dao
...
entity
...
service
...
job1
config
...
core
...
dao
...
entity
...
job2
config
...
core
...
dao
...
entity
...
Code Snippet (on a very high-level)
public class BatchApplication {
public static void main(String[] args) {
SpringApplication.run(BatchApplication.class, args);
}
}
public class Job2Config {
public Job JOB2() {
return jobBuilderFactory.get("JOB2")
.preventRestart()
.incrementer(batchJobIncrementer)
.listener(jobExecutionListener())
.flow(job2Step())
.end()
.build();
}
public Step job2Step() {
return stepBuilderFactory.get("job2Step")
.<String, String>chunk(
Integer.parseInt(chunkSize))
.reader(job2ItemReader())
.writer(this.job2ItemWriter)
.faultTolerant()
.skipLimit(Integer.parseInt(this.skipLimit))
.skipPolicy(skipPolicy())
.retryPolicy(new NeverRetryPolicy())
.listener(chunkListener())
.allowStartIfComplete(true)
.build();
}
public JdbcCursorItemReader<String> job2ItemReader() {
return new JdbcCursorItemReaderBuilder<String>()
.dataSource(oracleDataSource)
.name("job2ItemReader")
.sql(this.sqlQuery)
.fetchSize(Integer.parseInt(this.fetchSize))
.rowMapper((resultSet, rowNum) -> resultSet.getString(1))
.build();
}
}
public class Job2CommandLineRunner implements CommandLineRunner {
#Autowired
private JobLauncher jobLauncher;
#Autowired
private Job JOB2;
public void run(String... args) throws Exception {
String jobName = params.get("job.name");
if (jobName != null && jobName.equalsIgnoreCase("JOB2")) {
JobParameters jobParameters =
new JobParametersBuilder()
.addLong("time", System.currentTimeMillis())
.addString("dateCode", params.get("dateCode"))
.toJobParameters();
this.jobLauncher.run(JOB2, jobParameters);
}
}
}
I have couple of questions:
The 2nd spring batch job simply followed the 1st spring batch job for the implementation but I'm not sure why the 2nd spring batch job didn't get invoked
from the command line like the 1st spring batch job. To invoke the 2nd spring batch job, I had to explicitly launch it from the command line runner.
When there're multiple jobs in a Spring Batch application, is invoking a particular job explicitly thru CommandLineRunner the right way of doing it? If it's not, what's the right way of invoking a particular job (of an application with multiple jobs) from command line (eventually the jobs need to be called thru Tivoli workload scheduler)
It will be great if someone can help me clarify my questions.
Thanks in advance.

Related

Launch spring cloud task with JVM args via DeployerPartitionHandler

I am planning to execute spring batch job on Pivotal cloud Foundry. The job executes fine on a single JVM with multiple threads(Local partitioning). I am looking to scale the job and the first option i considered is running the worker processes as a spring cloud task.
Before run it in PCF i am running it in local. I created a partition handler in the following manner.
#Bean
public PartitionHandler partitionHandler(TaskLauncher taskLauncher, JobExplorer jobExplorer, TaskRepository taskRepository) throws Exception {
Resource resource = this.resourceLoader
.getResource("file:I:/Project/target/batch.jar");
DeployerPartitionHandler partitionHandler =
new DeployerPartitionHandler(taskLauncher, jobExplorer, resource, "workerStep", taskRepository);
List<String> commandLineArgs = new ArrayList<>(3);
commandLineArgs.add("--spring.profiles.active=worker");
commandLineArgs.add("--spring.cloud.task.initialize-enabled=false");
commandLineArgs.add("--spring.batch.initializer.enabled=false");
commandLineArgs.add("--java.security.krb5.conf=I:/krb5.conf");
commandLineArgs.add("--java.security.auth.login.config=I:/jaas.conf");
partitionHandler
.setCommandLineArgsProvider(new PassThroughCommandLineArgsProvider(commandLineArgs));
partitionHandler
.setEnvironmentVariablesProvider(new SimpleEnvironmentVariablesProvider(this.environment));
partitionHandler.setMaxWorkers(2);
partitionHandler.setApplicationName("PartitionedBatchJobTask");
return partitionHandler;
}
#Bean
#Profile("worker")
public DeployerStepExecutionHandler stepExecutionHandler(JobExplorer jobExplorer) {
return new DeployerStepExecutionHandler(this.context, jobExplorer, this.jobRepository);
}
The task is started, but as mentioned in the args the jar has to run with certain JVM args. These args are not being properly sent to task to start the app as a task. (I am connecting to a DB using Kerberose. Need to send these properties as a JVM arg)
I see the below being executed as a task
Command to be executed: I:/java.exe -jar I:/Project/target/batch.jar --spring.profiles.active=worker --spring.cloud.task.initialize-enabled=false --spring.batch.initializer.enabled=false --java.security.krb5.conf=I:/krb5.conf --java.security.auth.login.config=I:/jaas.conf --jdk.tls.client.protocols=TLSv1.2 --spring.cloud.task.job-execution-id=1 --spring.cloud.task.step-execution-id=3 --spring.cloud.task.step-name=workerStep --spring.cloud.task.name=application-1_migrateProfileJob_migrateProfileFollowerStep:partition0 --spring.cloud.task.parentExecutionId=29 --spring.cloud.task.executionid=30
Since the JVM args are sent after the jar command the app is not able to recognize th eproperties.. Can any one please let me know what i am doing wrong

How to restrict multiple instances of spring batch job running?

My Spring batch is triggered by a Rest end point. I am looking for the solution to run only one instance of a job at a time.
You can implement a listener overriding beforeJob method, that will run before the job starts and check if another instance of the same job is running. If another instance is already running, it will stop the current run.
#Component
public class MultipleCheckListener implements JobExecutionListener {
#Autowired
private JobExplorer explorer;
#Override
public void beforeJob(JobExecution jobExecution) {
String jobName = jobExecution.getJobInstance().getJobName();
Set<JobExecution> executions = explorer.findRunningJobExecutions(jobName);
if(executions.size() > 1) {
jobExecution.stop();
}
}
}

Spring batch, check jobExecution status after running the job directly?

I have 2 jobs where i have to launch the job2 based on the the status of the job1.
is it right to make the following call:
JobExecution jobExecution1 = jobLauncher.run(job1, jobParameters1);
if(jobExecution1.getStatus()== BatchStatus.COMPLETED){
JobExecution jobExecution2 = jobLauncher.run(job2, jobParameters2);
}
i have a doubt that the initial jobexecution1 status might not be final when the condition is checked.
if anyone could explain more about yhis process it would be great
thanks in advance .
It depends on the TaskExecutor implementation that you configure in your JobLauncher:
If the task executor is synchronous, then the first job will be run until completion (either success or failure) before launching the second job. In this case, your code is correct.
If the task executor is asynchronous, then the call JobExecution jobExecution1 = jobLauncher.run(job1, jobParameters1); will return immediately and the status of jobExecution1 would be unknown at that point. This means it is incorrect to check the status of job1 right after launching it as a condition to launch job2 (job1 will be run in a separate thread and could still be running at that point).
By default, Spring Batch configures a synchronous task executor in the JobLauncher.
There are multiple ways. We are using below option by chaining jobs by decider. Below is the code snippet . Please let me know if you face any issues.
public class MyDecider implements JobExecutionDecider {
#Override
public FlowExecutionStatus decide(JobExecution jobExecution, StepExecution stepExecution) {
String exitCode = new Random().nextFloat() < .70f ? "CORRECT":"INCORRECT";
return new FlowExecutionStatus(exitCode);
}
}
#Bean
public JobExecutionDecider myDecider() {
return new MyDecider();
}
#Bean
public Job myTestJob() {
return this.jobBuilderFactory.get("myTestJob")
.start(step1())
.next(step2())
.on("FAILED").to(step3())
.from(step4())
.on("*").to(myDecider())
.on("PRESENT").to(myNextStepOrJob)
.next(myDecider()).on("CORRECT").to(myNextStepOrJob())
.from(myDecider()).on("INCORRECT").to(myNextStepOrJob())
.from(decider())
.on("NOT_PRESENT").to(myNextStepOrJob())
.end()
.build();
}

How to launch a Job

I'm currently trying to launch a Job programatically in my Spring application.
Here is my actual code:
public void launchJob(String[] args)
throws JobParametersInvalidException, JobExecutionAlreadyRunningException, JobRestartException,
JobInstanceAlreadyCompleteException {
String jobName = args[0];
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
Job job = context.getBean(jobName, Job.class);
JobParameters jobParameters = getJobParameters(args);
JobExecution jobExecution = jobLauncher.run(job, jobParameters);
BatchStatus batchStatus = jobExecution.getStatus();
}
And how I launch it :
String[] args = {"transformXML2CSV", "myFile.xml", "myFile.csv"};
springBatchJobRunner.launchJob(args);
But I have some troubles during the launch, the first is to retrieve the app context, my first try was to annotate my class with a #Service and use an #Autowired like this:
#Autowired
private ApplicationContext context;
But with this way my context is always null.
My second try was to get the context by this way:
ApplicationContext context = new AnnotationConfigApplicationContext(SpringBatchNoDatabaseConfiguration.class);
The SpringBatchNoDatabaseConfiguration is my #Configuration and the Job is inside it.
Using this my context is not null but I have a strange behavior and I can't understand why and how to prevent this:
I run the launchJob function from my processor class, then when I get the context by AnnotationConfigApplicationContext my processor class is rebuild and I have a NullPointerException in it stopping all the process.
I really don't understand the last part, why it's relaunching my processor class when I get the context ?
As indicated in comments above, you run a parent batch (with spring-batch) which at a moment needs your job to process an xml file.
I suggest you keep the same spring-batch context and run the process xml file job as a nested job of the parent job. You can do that using JobStep class and spring batch controlling step flow feature. As an example, here is what your parent job would like to :
public Job parentJob(){
JobParameters processXmlFileJobParameters = getJobParameters(String[]{"transformXML2CSV", "myFile.xml", "myFile.csv"});
return this.jobBuilderFactory.get("parentJob")
.start(firstStepOfParentJob())
.on("PROCESS_XML_FILE").to(processXmlFileStep(processXmlFileJobParameters)
.from(firstStepOfParentJob()).on("*").to(lastStepOfParentJob())
.from(processXmlFileStep(processXmlFileJobParameters))
.next(lastStepOfParentJob())
.end().build();
}
public Step firstStepOfParentJob(){
return stepBuilderFactory.get("firstStepOfParentJob")
// ... depends on your parent job's business
.build();
}
public Step lastStepOfParentJob(){
return stepBuilderFactory.get("lastStepOfParentJob")
// ... depends on your parent job's business
.build();
}
public Step processXmlFileStep(JobParameters processXmlFileJobParameters){
return stepBuilderFactory.get("processXmlFileStep")
.job(processXmlFileJob())
.parametersExtractor((job,exec)->processXmlFileJobParameters)
.build();
}
public Job processXmlFileJob(){
return jobBuilderFactory.get("processXmlFileJob")
// ... describe here your xml process job
.build();
}
It seems a second Spring context is initialized with the instruction new AnnotationConfigApplicationContext(SpringBatchNoDatabaseConfiguration.class) so Spring initialiezs your beans a second time.
I recommend you to use Spring boot to automatically launch your jobs at sartup
If you don't want to use Spring boot and launch your job manually, your main method should look like this :
public static void main(String[] args){
String[] localArgs = {"transformXML2CSV", "myFile.xml", "myFile.csv"};
ApplicationContext context = new AnnotationConfigApplicationContext(SpringBatchNoDatabaseConfiguration.class);
Job job = context.getBean(Job.class);
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
JobParameters jobParameters = getJobParameters(localArgs);
jobExecution = jobLauncher.run(job, jobParameters);
// Ohter code to do stuff after the job ends...
}
Notice that this code should be completed with your own needs. The class SpringBatchNoDatabaseConfigurationhas to be anotated with #Configuration and #EnableBatchProcessingand should define the beans of your job like this
You can also use the Spring batch class CommandLineJobRunner with your java config as explain here : how to launch Spring Batch Job using CommandLineJobRunner having Java configuration
It saves writing the code above

Rerunning a completed Job from Spring Framework?

I am developing a web application using Spring Framework that does some jobs as the application starts up, and these jobs primarily consist of loading data from CSVs and making Java objects out of them.
Currently, I am trying to build a RESTful API using Restlet and Spring framework and one of the queries is supposed to take in a job name as parameter and restart that job even if that job has been marked as COMPLETED, how do I accomplish a job restart? I have tried the spring frameworks' Joboperator interface's startNextInstance() method and have also tried to manually increment the JobParameters so that there is no jobinstancealrradyrunning exception?
Anyone has any code snippet or alternative idea on how to restart a Job in Spring Framework that has been marked as Completed?
Any help would be greatly appreciated, thanks!!
Because of the terms you're using i'm quite certain you're using Spring Batch
In Batch terms you cannot actually restart a COMPLETED instance or execution. Single job instance is identified by job parameters. If you need to run the job again with same parameters, one way would be to include some unique parameter, for example the current timestamp to the JobParameters before launching.
So restarting a completed job would mean starting a new instance of the job with similiar parameters. Here's an slightly modified snippet i've used before that uses JobLauncher and JobRegistry to launch a new job by name:
#Autowired
#Qualifier("asyncJobLauncher")
private JobLauncher asyncJobLauncher;
#Autowired
private JobRegistry jobRegistry;
...
public JobExecution startJob(String jobName) {
Job job;
try {
job = jobRegistry.getJob(jobName);
} catch (NoSuchJobException e) {
// handle invalid job name
}
JobParametersBuilder jobParams = new JobParametersBuilder();
jobParams.addLong("currentTime", System.currentTimeMillis());
// add other relevant parameters
try {
JobExecution jobExecution = asyncJobLauncher.run(job, jobParams.toJobParameters());
return jobExecution;
} catch (JobExecutionAlreadyRunningException e) {
// handle instance already running with given params
} catch (Exception e) {
// handle other errors
}
}
Hope it helps, here's some reading about the subject.

Resources