I developed two different batches, let's say they're named as batch1 and batch2.
In an integration test, I'm trying to prove that the execution of twice of the same batch (batch1 then batch1 for instance) isn't duplicating data saved in database in the writer.
The issue I face is that the first batch is running successfully, but the second isn't doing anything (reader, processor, writer are not called). However, the batch status is marked as COMPLETED.
Here's the code:
#Test
void testBatchWorksIfJobIsRanTwice()
throws JobInstanceAlreadyCompleteException, JobExecutionAlreadyRunningException, JobParametersInvalidException, JobRestartException {
// given
createData();
// when
JobExecution firstJobExecution = runJobAndGetExecution("batch1_id1", jobRepository, job);
assertEquals(BatchStatus.COMPLETED, firstJobExecution.getStatus());
createMoreData();
JobExecution secondJobExecution = runJobAndGetExecution("batch1_id2", jobRepository, job);
assertEquals(BatchStatus.COMPLETED, secondJobExecution.getStatus());
// then
// assertions on data
}
public static JobExecution runJobAndGetExecution(String jobId, JobRepository jobRepository, Job job)
throws JobInstanceAlreadyCompleteException, JobExecutionAlreadyRunningException, JobParametersInvalidException, JobRestartException {
JobParameters param = new JobParametersBuilder()
.addString("JobID", jobId)
.addLong("currentTime", System.currentTimeMillis())
.toJobParameters();
SimpleJobLauncher launcher = new SimpleJobLauncher();
launcher.setJobRepository(jobRepository);
launcher.setTaskExecutor(new SyncTaskExecutor());
return launcher.run(job, param);
}
Please note that I pass a timestamp param in the job parameters to ensure that params are different.
Please also note that running the two different batches one after the other (batch1 and then batch2 is working fine, but batch1 then batch1 or batch2 then batch2 is not)
Any idea why it seems to run ? (I put some breakpoints and everything seems to happen correctly, just the reader, processer and writer are not called).
I resolved it. It was a misconception of the reader I implemented that was never reinitializing data, because I missunderstood the way reader works. The reader is called only once even if we're running the job 10 times.
The comment from Mahmoud Ben Hassine on the post is complety right
Related
I have 2 jobs where i have to launch the job2 based on the the status of the job1.
is it right to make the following call:
JobExecution jobExecution1 = jobLauncher.run(job1, jobParameters1);
if(jobExecution1.getStatus()== BatchStatus.COMPLETED){
JobExecution jobExecution2 = jobLauncher.run(job2, jobParameters2);
}
i have a doubt that the initial jobexecution1 status might not be final when the condition is checked.
if anyone could explain more about yhis process it would be great
thanks in advance .
It depends on the TaskExecutor implementation that you configure in your JobLauncher:
If the task executor is synchronous, then the first job will be run until completion (either success or failure) before launching the second job. In this case, your code is correct.
If the task executor is asynchronous, then the call JobExecution jobExecution1 = jobLauncher.run(job1, jobParameters1); will return immediately and the status of jobExecution1 would be unknown at that point. This means it is incorrect to check the status of job1 right after launching it as a condition to launch job2 (job1 will be run in a separate thread and could still be running at that point).
By default, Spring Batch configures a synchronous task executor in the JobLauncher.
There are multiple ways. We are using below option by chaining jobs by decider. Below is the code snippet . Please let me know if you face any issues.
public class MyDecider implements JobExecutionDecider {
#Override
public FlowExecutionStatus decide(JobExecution jobExecution, StepExecution stepExecution) {
String exitCode = new Random().nextFloat() < .70f ? "CORRECT":"INCORRECT";
return new FlowExecutionStatus(exitCode);
}
}
#Bean
public JobExecutionDecider myDecider() {
return new MyDecider();
}
#Bean
public Job myTestJob() {
return this.jobBuilderFactory.get("myTestJob")
.start(step1())
.next(step2())
.on("FAILED").to(step3())
.from(step4())
.on("*").to(myDecider())
.on("PRESENT").to(myNextStepOrJob)
.next(myDecider()).on("CORRECT").to(myNextStepOrJob())
.from(myDecider()).on("INCORRECT").to(myNextStepOrJob())
.from(decider())
.on("NOT_PRESENT").to(myNextStepOrJob())
.end()
.build();
}
I'm currently trying to launch a Job programatically in my Spring application.
Here is my actual code:
public void launchJob(String[] args)
throws JobParametersInvalidException, JobExecutionAlreadyRunningException, JobRestartException,
JobInstanceAlreadyCompleteException {
String jobName = args[0];
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
Job job = context.getBean(jobName, Job.class);
JobParameters jobParameters = getJobParameters(args);
JobExecution jobExecution = jobLauncher.run(job, jobParameters);
BatchStatus batchStatus = jobExecution.getStatus();
}
And how I launch it :
String[] args = {"transformXML2CSV", "myFile.xml", "myFile.csv"};
springBatchJobRunner.launchJob(args);
But I have some troubles during the launch, the first is to retrieve the app context, my first try was to annotate my class with a #Service and use an #Autowired like this:
#Autowired
private ApplicationContext context;
But with this way my context is always null.
My second try was to get the context by this way:
ApplicationContext context = new AnnotationConfigApplicationContext(SpringBatchNoDatabaseConfiguration.class);
The SpringBatchNoDatabaseConfiguration is my #Configuration and the Job is inside it.
Using this my context is not null but I have a strange behavior and I can't understand why and how to prevent this:
I run the launchJob function from my processor class, then when I get the context by AnnotationConfigApplicationContext my processor class is rebuild and I have a NullPointerException in it stopping all the process.
I really don't understand the last part, why it's relaunching my processor class when I get the context ?
As indicated in comments above, you run a parent batch (with spring-batch) which at a moment needs your job to process an xml file.
I suggest you keep the same spring-batch context and run the process xml file job as a nested job of the parent job. You can do that using JobStep class and spring batch controlling step flow feature. As an example, here is what your parent job would like to :
public Job parentJob(){
JobParameters processXmlFileJobParameters = getJobParameters(String[]{"transformXML2CSV", "myFile.xml", "myFile.csv"});
return this.jobBuilderFactory.get("parentJob")
.start(firstStepOfParentJob())
.on("PROCESS_XML_FILE").to(processXmlFileStep(processXmlFileJobParameters)
.from(firstStepOfParentJob()).on("*").to(lastStepOfParentJob())
.from(processXmlFileStep(processXmlFileJobParameters))
.next(lastStepOfParentJob())
.end().build();
}
public Step firstStepOfParentJob(){
return stepBuilderFactory.get("firstStepOfParentJob")
// ... depends on your parent job's business
.build();
}
public Step lastStepOfParentJob(){
return stepBuilderFactory.get("lastStepOfParentJob")
// ... depends on your parent job's business
.build();
}
public Step processXmlFileStep(JobParameters processXmlFileJobParameters){
return stepBuilderFactory.get("processXmlFileStep")
.job(processXmlFileJob())
.parametersExtractor((job,exec)->processXmlFileJobParameters)
.build();
}
public Job processXmlFileJob(){
return jobBuilderFactory.get("processXmlFileJob")
// ... describe here your xml process job
.build();
}
It seems a second Spring context is initialized with the instruction new AnnotationConfigApplicationContext(SpringBatchNoDatabaseConfiguration.class) so Spring initialiezs your beans a second time.
I recommend you to use Spring boot to automatically launch your jobs at sartup
If you don't want to use Spring boot and launch your job manually, your main method should look like this :
public static void main(String[] args){
String[] localArgs = {"transformXML2CSV", "myFile.xml", "myFile.csv"};
ApplicationContext context = new AnnotationConfigApplicationContext(SpringBatchNoDatabaseConfiguration.class);
Job job = context.getBean(Job.class);
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
JobParameters jobParameters = getJobParameters(localArgs);
jobExecution = jobLauncher.run(job, jobParameters);
// Ohter code to do stuff after the job ends...
}
Notice that this code should be completed with your own needs. The class SpringBatchNoDatabaseConfigurationhas to be anotated with #Configuration and #EnableBatchProcessingand should define the beans of your job like this
You can also use the Spring batch class CommandLineJobRunner with your java config as explain here : how to launch Spring Batch Job using CommandLineJobRunner having Java configuration
It saves writing the code above
I'm trying to return a Spring Batch Job ID before the job completes.
My current implementation only returns the information once the job
completes.
I use a batch program controller and a batch service, posted below.
Thanks, I'm new to Spring Batch and after exhaustive searching
couldn't find much related to my question. There was one post with
someone using Apache Camel but I am not.
I know about SimpleAsyncTaskExecutor() but not sure how to apply it here
Controller
#RequestMapping(value = "/batch/{progName}", method = RequestMethod.GET)
public ResponseEntity<JobResults> process(#PathVariable("progName") String progName,
#RequestHeader("Accept") MediaType accept) throws Exception {
HttpHeaders responseHeaders = MccControllerUtils.createCacheDisabledHeaders();
responseHeaders.setContentType(MediaType.APPLICATION_XML);
if(!accept.toString().equals("*/*")) {
responseHeaders.setContentType(accept);
}
LOGGER.info("Running batch program " + progName);
JobResults response = batchService.processProgName(progName);
return new ResponseEntity<JobResults>(response, responseHeaders, HttpStatus.OK);
}
Service
#Override
public JobResults processProgName(String progName) throws Exception {
String jobName = "call" + progName.toUpperCase() + "Job";
JobExecution jobExecution = null;
String result = "";
Long jobId = null;
JobResults results = new JobResults();
BatchStatus jobStatus = null;
try {
// Launch the appropriate batch job.
Map<String, Job> jobs = applicationContext.getBeansOfType(Job.class);
LOGGER.info("BATCHSTART:Starting sync batch job:" + jobName);
JobParametersBuilder builder = new JobParametersBuilder();
// Pass in the runtime to ensure a fresh execution.
builder.addDate("Runtime", new Date());
jobExecution = jobLauncher.run(jobs.get(jobName), builder.toJobParameters());
jobId = jobExecution.getId();
jobStatus = jobExecution.getStatus();
results.setName(jobName);
results.setId(jobId);
results.setMessage("The job has finished.");
results.setStatus(jobStatus);
LOGGER.info("The job ID is " + jobId);
LOGGER.info("BATCHEND:Completed sync batch job:" + jobName);
LOGGER.info("Completion status for batch job " + jobName + " is " + jobExecution.getStatus().name());
List<Throwable> failures = jobExecution.getAllFailureExceptions();
if (failures.isEmpty()) {
result = jobExecution.getExecutionContext().getString(AbstractSetupTasklet.BATCH_PROGRAM_RESULT);
} else {
for (Throwable fail : failures) {
result += fail.getMessage() + "\n";
}
}
LOGGER.info("The job results are: " + result);
} catch (JobExecutionAlreadyRunningException | JobRestartException | JobInstanceAlreadyCompleteException
| JobParametersInvalidException e) {
throw new RuntimeException("An error occurred while attempting to execute a job. " + e.getMessage(), e);
}
return results;
}
Thanks again.
EDIT
I've added this to my batch configuration
#Bean(name = "AsyncJobLauncher")
public JobLauncher simpleJobLauncher(JobRepository jobRepository){
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
jobLauncher.setTaskExecutor(new SimpleAsyncTaskExecutor());
return jobLauncher;
}
EDIT
I solved the problem with the help of Mahmoud Ben Hassine's comment :)
I had to remove the result variable and information from the Service, as well as add the code seen above. When I hit my endpoint now I receive the job information immediately.
What you can do is to inject a JobLauncher implementation in your BatchService that is backed by an asynchronous task executor (for example SimpleAsyncTaskExecutor or ThreadPoolTaskExecutor). This will run your jobs asynchronously and will immediately return a job execution (from which you can get the id) without waiting for the job to complete.
So by injecting the right implementation of JobLauncher in your BatchService, you will be able to achieve your requirement without changing the code of the processProgName method.
You can find more details about how to run jobs synchronously or asynchronously here: https://docs.spring.io/spring-batch/4.0.x/reference/html/job.html#configuringJobLauncher
I am developing a web application using Spring Framework that does some jobs as the application starts up, and these jobs primarily consist of loading data from CSVs and making Java objects out of them.
Currently, I am trying to build a RESTful API using Restlet and Spring framework and one of the queries is supposed to take in a job name as parameter and restart that job even if that job has been marked as COMPLETED, how do I accomplish a job restart? I have tried the spring frameworks' Joboperator interface's startNextInstance() method and have also tried to manually increment the JobParameters so that there is no jobinstancealrradyrunning exception?
Anyone has any code snippet or alternative idea on how to restart a Job in Spring Framework that has been marked as Completed?
Any help would be greatly appreciated, thanks!!
Because of the terms you're using i'm quite certain you're using Spring Batch
In Batch terms you cannot actually restart a COMPLETED instance or execution. Single job instance is identified by job parameters. If you need to run the job again with same parameters, one way would be to include some unique parameter, for example the current timestamp to the JobParameters before launching.
So restarting a completed job would mean starting a new instance of the job with similiar parameters. Here's an slightly modified snippet i've used before that uses JobLauncher and JobRegistry to launch a new job by name:
#Autowired
#Qualifier("asyncJobLauncher")
private JobLauncher asyncJobLauncher;
#Autowired
private JobRegistry jobRegistry;
...
public JobExecution startJob(String jobName) {
Job job;
try {
job = jobRegistry.getJob(jobName);
} catch (NoSuchJobException e) {
// handle invalid job name
}
JobParametersBuilder jobParams = new JobParametersBuilder();
jobParams.addLong("currentTime", System.currentTimeMillis());
// add other relevant parameters
try {
JobExecution jobExecution = asyncJobLauncher.run(job, jobParams.toJobParameters());
return jobExecution;
} catch (JobExecutionAlreadyRunningException e) {
// handle instance already running with given params
} catch (Exception e) {
// handle other errors
}
}
Hope it helps, here's some reading about the subject.
I've developed a Spring Batch with 10 Step.
How can i execute a specific Step ?
My wish is to passed in JobParameter the Step to execute.
Only the Step specified in JobParameter must be executed.
I searched to use the Decider, but i'm not really satisfied.
There's a better Solution ?
Any idea ?
Thanks a lot.
cordially
The Decider is the correct option for the type of processing you're talking about. I'd be interested in why you were "not really satisfied" by that option?
I had similar use case. I wanted to pass job name and step name as the input and expectation was to execute only that particular step from the job.
I created a Rest API which accepts job name and step name as URL parameters.
Below is the code inside the Rest API. It requires few objects injected into the class.
jobRegistry - instance of MapJobRegistry
jobRepository - instance of jobRepository
jobLauncher - instance of JobLauncher
Job job = null;
Step step = null;
try{
Job job = jobRegistry.getJob(jobName);
if(job instanceof StepLocator){
Step = ((StepLocator)job).getStep(stepName);
}
}catch(NoSuchJobException ex){
throw new Exception("Invalid Job", ex);
}catch(NoSuchStepException ex){
throw new Exception("Invalid Step", ex);
}
if(null == step){
throw new Exception("invalid step");
}
JobBuilderFactory jobBuilder = new JobBuiderFactory(jobRepository);
Job newJob = jobBuilder.get("specific-step-job")
.start(step)
.build();
jobLauncher.run(newJob, jobParameters); //jobParameters relevant for this job
This will dynamically create a new job with name "specific-step-job" and add the specific step instance as the start/only step inside the new Job and executes this job.
Yes, you can test an individual step using JobLauncherTestUtils#launchStep.
Please have a look at section 10.3. Testing Individual Steps
Find a sample code here Spring Batch unit test example