I am developing a web application using Spring Framework that does some jobs as the application starts up, and these jobs primarily consist of loading data from CSVs and making Java objects out of them.
Currently, I am trying to build a RESTful API using Restlet and Spring framework and one of the queries is supposed to take in a job name as parameter and restart that job even if that job has been marked as COMPLETED, how do I accomplish a job restart? I have tried the spring frameworks' Joboperator interface's startNextInstance() method and have also tried to manually increment the JobParameters so that there is no jobinstancealrradyrunning exception?
Anyone has any code snippet or alternative idea on how to restart a Job in Spring Framework that has been marked as Completed?
Any help would be greatly appreciated, thanks!!
Because of the terms you're using i'm quite certain you're using Spring Batch
In Batch terms you cannot actually restart a COMPLETED instance or execution. Single job instance is identified by job parameters. If you need to run the job again with same parameters, one way would be to include some unique parameter, for example the current timestamp to the JobParameters before launching.
So restarting a completed job would mean starting a new instance of the job with similiar parameters. Here's an slightly modified snippet i've used before that uses JobLauncher and JobRegistry to launch a new job by name:
#Autowired
#Qualifier("asyncJobLauncher")
private JobLauncher asyncJobLauncher;
#Autowired
private JobRegistry jobRegistry;
...
public JobExecution startJob(String jobName) {
Job job;
try {
job = jobRegistry.getJob(jobName);
} catch (NoSuchJobException e) {
// handle invalid job name
}
JobParametersBuilder jobParams = new JobParametersBuilder();
jobParams.addLong("currentTime", System.currentTimeMillis());
// add other relevant parameters
try {
JobExecution jobExecution = asyncJobLauncher.run(job, jobParams.toJobParameters());
return jobExecution;
} catch (JobExecutionAlreadyRunningException e) {
// handle instance already running with given params
} catch (Exception e) {
// handle other errors
}
}
Hope it helps, here's some reading about the subject.
Related
I have just got into a new project, which is all about migrating COBOL jobs to spring batch.
Before I joined, my colleague started on this migration and created the spring batch application with 1 job. My colleague was/is able to run the spring batch job from the command line just by running the application's jar file.
After I joined, I followed the 1st spring batch job and developed the 2nd spring batch job. But my spring batch job is not running from the command line by running the application's jar file like the 1st spring batch job.
I tried to explicitly launch my job (2nd job) from the commandlinerunner's run method and it worked. (The 1st spring batch job didn't have to do this).
Once the jobs are developed and tested, each job will be run from the Tivoli workload scheduler scheduled at different times.
The command that I used from command line:
java -Dspring.profiles.active=dev -jar target/SPRINGBATCHJOBS-0.0.2-SNAPSHOT.jar job.name=JOB2 dateCode=$$$$
Package structure
App
src
main
java
com.test
BatchApplication.java
common
dao
...
entity
...
service
...
job1
config
...
core
...
dao
...
entity
...
job2
config
...
core
...
dao
...
entity
...
Code Snippet (on a very high-level)
public class BatchApplication {
public static void main(String[] args) {
SpringApplication.run(BatchApplication.class, args);
}
}
public class Job2Config {
public Job JOB2() {
return jobBuilderFactory.get("JOB2")
.preventRestart()
.incrementer(batchJobIncrementer)
.listener(jobExecutionListener())
.flow(job2Step())
.end()
.build();
}
public Step job2Step() {
return stepBuilderFactory.get("job2Step")
.<String, String>chunk(
Integer.parseInt(chunkSize))
.reader(job2ItemReader())
.writer(this.job2ItemWriter)
.faultTolerant()
.skipLimit(Integer.parseInt(this.skipLimit))
.skipPolicy(skipPolicy())
.retryPolicy(new NeverRetryPolicy())
.listener(chunkListener())
.allowStartIfComplete(true)
.build();
}
public JdbcCursorItemReader<String> job2ItemReader() {
return new JdbcCursorItemReaderBuilder<String>()
.dataSource(oracleDataSource)
.name("job2ItemReader")
.sql(this.sqlQuery)
.fetchSize(Integer.parseInt(this.fetchSize))
.rowMapper((resultSet, rowNum) -> resultSet.getString(1))
.build();
}
}
public class Job2CommandLineRunner implements CommandLineRunner {
#Autowired
private JobLauncher jobLauncher;
#Autowired
private Job JOB2;
public void run(String... args) throws Exception {
String jobName = params.get("job.name");
if (jobName != null && jobName.equalsIgnoreCase("JOB2")) {
JobParameters jobParameters =
new JobParametersBuilder()
.addLong("time", System.currentTimeMillis())
.addString("dateCode", params.get("dateCode"))
.toJobParameters();
this.jobLauncher.run(JOB2, jobParameters);
}
}
}
I have couple of questions:
The 2nd spring batch job simply followed the 1st spring batch job for the implementation but I'm not sure why the 2nd spring batch job didn't get invoked
from the command line like the 1st spring batch job. To invoke the 2nd spring batch job, I had to explicitly launch it from the command line runner.
When there're multiple jobs in a Spring Batch application, is invoking a particular job explicitly thru CommandLineRunner the right way of doing it? If it's not, what's the right way of invoking a particular job (of an application with multiple jobs) from command line (eventually the jobs need to be called thru Tivoli workload scheduler)
It will be great if someone can help me clarify my questions.
Thanks in advance.
I'm currently trying to launch a Job programatically in my Spring application.
Here is my actual code:
public void launchJob(String[] args)
throws JobParametersInvalidException, JobExecutionAlreadyRunningException, JobRestartException,
JobInstanceAlreadyCompleteException {
String jobName = args[0];
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
Job job = context.getBean(jobName, Job.class);
JobParameters jobParameters = getJobParameters(args);
JobExecution jobExecution = jobLauncher.run(job, jobParameters);
BatchStatus batchStatus = jobExecution.getStatus();
}
And how I launch it :
String[] args = {"transformXML2CSV", "myFile.xml", "myFile.csv"};
springBatchJobRunner.launchJob(args);
But I have some troubles during the launch, the first is to retrieve the app context, my first try was to annotate my class with a #Service and use an #Autowired like this:
#Autowired
private ApplicationContext context;
But with this way my context is always null.
My second try was to get the context by this way:
ApplicationContext context = new AnnotationConfigApplicationContext(SpringBatchNoDatabaseConfiguration.class);
The SpringBatchNoDatabaseConfiguration is my #Configuration and the Job is inside it.
Using this my context is not null but I have a strange behavior and I can't understand why and how to prevent this:
I run the launchJob function from my processor class, then when I get the context by AnnotationConfigApplicationContext my processor class is rebuild and I have a NullPointerException in it stopping all the process.
I really don't understand the last part, why it's relaunching my processor class when I get the context ?
As indicated in comments above, you run a parent batch (with spring-batch) which at a moment needs your job to process an xml file.
I suggest you keep the same spring-batch context and run the process xml file job as a nested job of the parent job. You can do that using JobStep class and spring batch controlling step flow feature. As an example, here is what your parent job would like to :
public Job parentJob(){
JobParameters processXmlFileJobParameters = getJobParameters(String[]{"transformXML2CSV", "myFile.xml", "myFile.csv"});
return this.jobBuilderFactory.get("parentJob")
.start(firstStepOfParentJob())
.on("PROCESS_XML_FILE").to(processXmlFileStep(processXmlFileJobParameters)
.from(firstStepOfParentJob()).on("*").to(lastStepOfParentJob())
.from(processXmlFileStep(processXmlFileJobParameters))
.next(lastStepOfParentJob())
.end().build();
}
public Step firstStepOfParentJob(){
return stepBuilderFactory.get("firstStepOfParentJob")
// ... depends on your parent job's business
.build();
}
public Step lastStepOfParentJob(){
return stepBuilderFactory.get("lastStepOfParentJob")
// ... depends on your parent job's business
.build();
}
public Step processXmlFileStep(JobParameters processXmlFileJobParameters){
return stepBuilderFactory.get("processXmlFileStep")
.job(processXmlFileJob())
.parametersExtractor((job,exec)->processXmlFileJobParameters)
.build();
}
public Job processXmlFileJob(){
return jobBuilderFactory.get("processXmlFileJob")
// ... describe here your xml process job
.build();
}
It seems a second Spring context is initialized with the instruction new AnnotationConfigApplicationContext(SpringBatchNoDatabaseConfiguration.class) so Spring initialiezs your beans a second time.
I recommend you to use Spring boot to automatically launch your jobs at sartup
If you don't want to use Spring boot and launch your job manually, your main method should look like this :
public static void main(String[] args){
String[] localArgs = {"transformXML2CSV", "myFile.xml", "myFile.csv"};
ApplicationContext context = new AnnotationConfigApplicationContext(SpringBatchNoDatabaseConfiguration.class);
Job job = context.getBean(Job.class);
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
JobParameters jobParameters = getJobParameters(localArgs);
jobExecution = jobLauncher.run(job, jobParameters);
// Ohter code to do stuff after the job ends...
}
Notice that this code should be completed with your own needs. The class SpringBatchNoDatabaseConfigurationhas to be anotated with #Configuration and #EnableBatchProcessingand should define the beans of your job like this
You can also use the Spring batch class CommandLineJobRunner with your java config as explain here : how to launch Spring Batch Job using CommandLineJobRunner having Java configuration
It saves writing the code above
I am writing various jobs using spring batch with java configuration.
I need to get the current state of the job
e.g.
which steps are currently running (I may have multiple steps running at the same time)
Which steps failed (the status and exit code)
etc.
The only examples I see online are of XML based spring batch and I want to use java config only.
Thanks.
Another option is to use JobExplorer
Entry point for browsing executions of running or historical jobs and steps. Since the data may be re-hydrated from persistent storage, it may not contain volatile fields that would have been present when the execution was active.
List<JobExecution> jobExecutions = jobExplorer.getJobExecutions(jobInstance);
for (JobExecution jobExecution : jobExecutions) {
jobExecution.getStepExecutions();
//read step info
}
And for create jobExplorer you have to use the factory:
import org.springframework.batch.core.explore.support.JobExplorerFactoryBean;
JobExplorerFactoryBean factory = new JobExplorerFactoryBean();
factory.setDataSource(dataSource);
factory.getObject();
I use these two queries on spring batch meta data tables to know about job progress and step details.
SELECT * FROM BATCH_JOB_EXECUTION ORDER BY START_TIME DESC;
SELECT * FROM BATCH_STEP_EXECUTION WHERE JOB_EXECUTION_ID=? ORDER BY STATUS;
With First query, I first find JOB_EXECUTION_ID corresponding to my job execution then use that id in second query to find details about specific steps.
Additionally, your config choice ( Java or XML ) has nothing to do with Spring Batch meta data. If you are persisting data then it doesn't matter if its XML config or Java Config.
For Java based monitoring- you can use JobExplorer & JobRepository beans to query jobs etc.
e.g. List<JobInstance> from jobExplorer.getJobInstances & jobExplorer.getJobExecutions(jobInstance) etc.
From JobExecutions you can get StepExecutions and so on.
You might have to set JobRegistryBeanPostProcessor bean like below for JobExplorer & JobRepository to work properly.
#Bean
public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(
JobRegistry jobRegistry) {
JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor = new JobRegistryBeanPostProcessor();
jobRegistryBeanPostProcessor.setJobRegistry(jobRegistry);
return jobRegistryBeanPostProcessor;
}
I have a requirement where I need to process files based on the rest call in which I get the name of the file, I am adding it to the job parameter and using it while creating the beans.
I am creating step scope Beans for (reader,writer) and using the job parameter.I am starting the job in a new thread as I am using asynchronus task exceutor to launch the job and my question is how will the beans be created by spring when we define #StepScope
jobParametersBuilder.addString("fileName", request.getFileName());
jobExecution = jobLauncher.run(job, jobParametersBuilder.toJobParameters());
#Bean
public JobLauncher jobLauncher() {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository());
jobLauncher.setTaskExecutor(asyncTaskExecutor());
return jobLauncher;
}
#Bean
#StepScope
public ItemWriter<Object> writer(#Value ("#{jobParameters['fileName']}"String fileName) {
JdbcBatchItemWriter<Object> writer = new JdbcBatchItemWriter<>();
writer.setItemSqlParameterSourceProvider(
new BeanPropertyItemSqlParameterSourceProvider<Object>());
writer.setSql(queryCollection.getquery());
writer.setDataSource(dataSource(fileName));
return writer;
}
A spring batch StepScope object is one which is unique to a specific step and not a singleton. As you probably know, the default bean scope in Spring is a singleton. But by specifying a spring batch component being StepScope means that Spring Batch will use the spring container to instantiate a new instance of that component for each step execution.
This is often useful for doing parameter late binding where a parameter may be specified either at the StepContext or the JobExecutionContext level and needs to be substituted for a placeholder, much like your example with the filename requirement.
Another useful reason to use StepScope is when you decide to reuse the same component in parallel steps. If the component manages any internal state, its important that it be StepScope based so that one thread does not impair the state managed by another thread (e.g, each thread of a given step has its own instance of the StepScope component).
I've developed a Spring Batch with 10 Step.
How can i execute a specific Step ?
My wish is to passed in JobParameter the Step to execute.
Only the Step specified in JobParameter must be executed.
I searched to use the Decider, but i'm not really satisfied.
There's a better Solution ?
Any idea ?
Thanks a lot.
cordially
The Decider is the correct option for the type of processing you're talking about. I'd be interested in why you were "not really satisfied" by that option?
I had similar use case. I wanted to pass job name and step name as the input and expectation was to execute only that particular step from the job.
I created a Rest API which accepts job name and step name as URL parameters.
Below is the code inside the Rest API. It requires few objects injected into the class.
jobRegistry - instance of MapJobRegistry
jobRepository - instance of jobRepository
jobLauncher - instance of JobLauncher
Job job = null;
Step step = null;
try{
Job job = jobRegistry.getJob(jobName);
if(job instanceof StepLocator){
Step = ((StepLocator)job).getStep(stepName);
}
}catch(NoSuchJobException ex){
throw new Exception("Invalid Job", ex);
}catch(NoSuchStepException ex){
throw new Exception("Invalid Step", ex);
}
if(null == step){
throw new Exception("invalid step");
}
JobBuilderFactory jobBuilder = new JobBuiderFactory(jobRepository);
Job newJob = jobBuilder.get("specific-step-job")
.start(step)
.build();
jobLauncher.run(newJob, jobParameters); //jobParameters relevant for this job
This will dynamically create a new job with name "specific-step-job" and add the specific step instance as the start/only step inside the new Job and executes this job.
Yes, you can test an individual step using JobLauncherTestUtils#launchStep.
Please have a look at section 10.3. Testing Individual Steps
Find a sample code here Spring Batch unit test example