Spring Batch - execute Steps based on decisions issue - spring

I'm developing Spring Batch where I've Steps-1 to Step-10 which runs sequentially. In my case for step-4 to Step-7, I've to make conditional decision based on which I've execute Step-X, Step-Y and , Step-Z etc.
Ex: Step-4 - Precondition, if Step-X gives any status other than error, then execute Step-4, if Step-X failed then execute failedStep() step.
Step-5 - Precondition, if Step-Y gives any status other than error, then execute Step-5, if Step-Y failed then execute failedStep() step.
Step-6 - Precondition, if Step-Z gives any status other than error, then execute Step-6, if Step-z failed then execute failedStep() step and the step-8 to step-10. I'm looking to do all in a single batch job only.
I've developed some code, but it looks like it's not liking the condition.
Error - The method on(String) is undefined for the type Step
#Configuration
public class JobConfig {
#Autowired
private JobBuilderFactory jobs;
#Autowired
private StepBuilderFactory steps;
#Autowired
private JdbcTemplate jdbcTemplate;
#Bean
public Step step1() {
return steps.get("step1")
.tasklet((StepContribution contribution, ChunkContext chunkContext) -> {
System.out.println("Step1");
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public Step step2() {
return steps.get("step2")
.tasklet((StepContribution contribution, ChunkContext chunkContext) -> {
System.out.println("step2");
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public Step step3() {
return steps.get("step3")
.tasklet((StepContribution contribution, ChunkContext chunkContext) -> {
System.out.println("step3");
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public Step step4() {
return steps.get("step4")
.tasklet((StepContribution contribution, ChunkContext chunkContext) -> {
System.out.println("step4");
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public Step step5() {
return steps.get("step5")
.tasklet((StepContribution contribution, ChunkContext chunkContext) -> {
System.out.println("step5");
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public Step stepX() {
return steps.get("stepX")
.tasklet((StepContribution contribution, ChunkContext chunkContext) -> {
System.out.println("step4Check");
// int update = jdbcTemplate.update("Query", "SBC"); // Perform some meaningful DB operations
int update = 1; // To simulate issue locally !
if(update == 1) {
chunkContext.getStepContext().getStepExecution().setExitStatus(new ExitStatus("COMPLETED"));
}else {
chunkContext.getStepContext().getStepExecution().setExitStatus(new ExitStatus("FAILED"));
}
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public Step stepY() {
return steps.get("stepY")
.tasklet((StepContribution contribution, ChunkContext chunkContext) -> {
System.out.println("step4Check");
// int update = jdbcTemplate.update("Query", "XYZ"); // Perform some meaningful DB operations
int update = 1; // To simulate issue locally !
if(update == 1) {
chunkContext.getStepContext().getStepExecution().setExitStatus(new ExitStatus("COMPLETED"));
}else {
chunkContext.getStepContext().getStepExecution().setExitStatus(new ExitStatus("FAILED"));
}
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public Step failedStep() {
return steps.get("failedStep")
.tasklet((contribution, chunkContext) -> {
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public Job job() {
return jobs.get("job")
.start(step1())
.next(step2())
.next(step3())
.next(stepX().on(ExitStatus.FAILED.getExitCode()).to(failedStep())
.from(stepX()).on("*").to(step4())
.from(stepY()).on(ExitStatus.FAILED.getExitCode()).to(failedStep())
.from(stepY()).on("*").to(step5())
.build();
}
}

I was able to solve the issue using following code - This works perfectly fine for me.
Approach-1
#Bean
public Job job() {
return jobs.get("job")
.start(step1())
.next(step2())
.next(step3()).on(ExitStatus.FAILED.getExitCode()).to(failedStep())
.from(step3()).on("*").to(stepX())
.from(stepX()).on(ExitStatus.FAILED.getExitCode()).to(failedStep())
.from(stepX()).on("*").to(step4())
.from(step4()).on(ExitStatus.FAILED.getExitCode()).to(failedStep())
.from(step4()).on("*").to(stepY())
.from(stepY()).on(ExitStatus.FAILED.getExitCode()).to(failedStep())
.from(stepY()).on("*").to(step5())
.next(step6())
.build()
.build();
}
Approach-2
#Bean
public Job job() {
return jobs.get("job")
.start(step1())
.next(step2())
.next(step3())
.next(stepX()).on(ExitStatus.FAILED.getExitCode()).to(failedStep())
.from(stepX()).on("*").to(step4())
.next(stepY()).on(ExitStatus.FAILED.getExitCode()).to(failedStep())
.from(stepY()).on("*").to(step5())
.next(step6())
.build()
.build();
}

Related

Spring Batch SkippListener not printing logs

I am not sure why the SkipListener is not printing the simple logs. I want to log the skipped records into a separate log file and I am thinking I can use MDC.put(). But I am not even able to print a log in console not sure what is happening. I think I am missing something. A help would be really appreciated. I even tried with generic Exception.class for testing but still not doing anything. Here is my code;
BtachConfiguration
#EnableBatchProcessing
public class BatchConfiguration {
static final Logger LOG = LogManager.getLogger(BatchConfiguration.class);
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public DataSource dataSource;
#Bean
public Job loadUserJob() {
return jobBuilderFactory.get("loadUserJob")
.incrementer(new RunIdIncrementer())
.start(loadUsersStep())
.listener(new JobLoggerListener())
.build();
}
#Bean
public Step loadUsersStep() {
return stepBuilderFactory.get("loadUsersStep")
.<UserInfo, UserInfoDTO> chunk(10)
.reader(reader())
.processor(UserInfoItemProcessor())
.writer(writer())
.faultTolerant()
.skipLimit(3)
.skip(Exception.class)
//.skip(UserNotFoundException.class)
.listener(new StepStartStopListener())
.listener(new LoadDataSkipListener())
.build();
}
#Bean
public FlatFileItemReader<UserInfo> reader() {
return new FlatFileItemReaderBuilder<UserInfo>()
.name("reader")
.delimited()
.names(new String[] {
"first_name",
"last_name",
"email"})
.targetType(UserInfo.class)
.resource(new ClassPathResource("userinfo_file.csv"))
.build();
}
#Bean
public UserInfoItemProcessor UserInfoItemProcessor() {
return new UserInfoItemProcessor();
}
#Bean
public JdbcBatchItemWriter<UserInfoDTO> writer() {
JdbcBatchItemWriter<UserInfoDTO> writer = new JdbcBatchItemWriter<UserInfoDTO>();
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>());
writer.setSql("INSERT INTO db1.userinfo "
+ "(first_name,last_name,email) " +
"VALUES (:firstName, :lastName,:email)");
writer.setDataSource(dataSource);
return writer;
}
}
SkipListener Class
public class LoadDataSkipListener implements SkipListener<UserInfoDTO, UserInfoDTO>{
public static final Logger LOG = LogManager.getLogger(LoadDataSkipListener.class);
#Override
public void onSkipInRead(Throwable t) {
LOG.info("ItemWriter: ");
}
#Override
public void onSkipInWrite(UserInfoDTO item, Throwable t) {
LOG.info(">>> onSkipInWrite <<<");
// MDC.put("skipped_users", String.valueOf(item.getUserId())); // I want to log skipped user ids on write
}
#Override
public void onSkipInProcess(UserInfoDTO item, Throwable t) {
LOG.info(">>> onSkipInProcess<<< ");
}
}
application.properties
spring.datasource.url=jdbc:mysql://localhost:3306/db1?useSSL=false
spring.datasource.username=username
spring.datasource.password=password
spring.datasource.schema=classpath:/org/springframework/batch/core/schema-mysql.sql
spring.batch.initialize-schema=always
spring.datasource.initialize=true
#logging.file={log-name}.log --> WIll this write into a new file ??
Just want to know
Why its not even printing in console ?
Can the commented MDC be used to write to a new file ?
Thanks in advance!

How create a spring batch to download files from api and insert data to database?

I have create a spring batch project to be able to download CSV files from an API and insert the data from those csv files to postreSQL database. The download has been done, but the insert didn't work when i started the project from the first time. I have two jobs, one for the download, and the other for the insert. And for each job has a step. Please, is there someone can help me ?
Thanks.
public class SpringBatchConfig {
private JobBuilderFactory jobBuilderFactory;
private StepBuilderFactory stepBuilderFactory;
private DataSource dataSource;
private JobExplorer jobExplorer;
#Value("${filePath}")
private String filePath;
#Value("classpath:insertions_*.csv")
private Resource[] inputResources;
#Autowired
public SpringBatchConfig(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory, DataSource dataSource, JobExplorer jobExplorer) {
this.jobBuilderFactory = jobBuilderFactory;
this.stepBuilderFactory = stepBuilderFactory;
this.dataSource = dataSource;
this.jobExplorer = jobExplorer;
}
#Bean
public FlatFileItemReader<SharadarFundamentalsDto> readerInsert() {
FlatFileItemReader<SharadarFundamentalsDto> reader = new FlatFileItemReader<>();
reader.setLinesToSkip(1);
reader.setLineMapper(new DefaultLineMapper<SharadarFundamentalsDto>() {{
setLineTokenizer(new DelimitedLineTokenizer() {{
setNames("ticker","dimension","calendardate","datekey","reportperiod");
}});
setFieldSetMapper(new BeanWrapperFieldSetMapper() {{
setTargetType(SharadarFundamentalsDto.class);
}});
}});
return reader;
}
#Bean
public MultiResourceItemReader<SharadarFundamentalsDto> multiResourceItemReader() {
MultiResourceItemReader<SharadarFundamentalsDto> resourceItemReader = new MultiResourceItemReader<>();
resourceItemReader.setResources(inputResources);
resourceItemReader.setDelegate(readerInsert());
return resourceItemReader;
}
#Bean
public SharadarFundamentalsProcessor processor() {
return new SharadarFundamentalsProcessor();
}
#Bean
public JdbcBatchItemWriter<SharadarFundamentals> writerInsert() {
JdbcBatchItemWriter<SharadarFundamentals> writer = new JdbcBatchItemWriter<>();
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>());
writer.setSql("INSERT INTO public.sharadar_fundamentals(ticker, dimension, calendardate, datekey, reportperiod)" +
"VALUES (:ticker, :dimension, :calendardate, :datekey, :reportperiod)");
writer.setDataSource(dataSource);
return writer;
}
#Bean
public Job downloadFiles(JobListener listener) {
return jobBuilderFactory.get("downloadFiles")
.incrementer(new RunIdIncrementer())
.listener(listener)
.start(step1())
.build();
}
#Bean
public Job insertJob(JobListener listener) {
return jobBuilderFactory.get("insertJob")
.incrementer(new RunIdIncrementer())
.listener(listener)
.start(step2())
.build();
}
#Bean
public DownloadTasklet downloadTasklet() {
return new DownloadTasklet();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.tasklet(downloadTasklet())
.build();
}
#Bean
public Step step2() {
return stepBuilderFactory.get("step2")
.<SharadarFundamentalsDto, SharadarFundamentals> chunk(10)
.reader(multiResourceItemReader())
.faultTolerant()
.skipLimit(1)
.skip(FlatFileParseException.class)
.processor(processor())
.writer(writerInsert())
.build();
}
}
What you want is a single job that performs 2 tasks synchronously.
#Bean
public Job downloadFilesAndInsertData(JobListener listener) {
return jobBuilderFactory.get("downloadFiles")
.incrementer(new RunIdIncrementer())
.listener(listener)
.start(step1())
.next(step2())
.build();
}

MongoDB Spring Batch job using Gradle

I want to create an item reader, writer and processor for a Spring batch job using Gradle. I am having trouble with a few things. The delimited() portion is giving me an error. I am trying to read two fields for now: rxFname, rxLname.
Here is my code:
#Configuration
#EnableBatchProcessing
public class PaymentPortalJob {
private static final Logger LOG =
LoggerFactory.getLogger(PaymentPortalJob.class);
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean
MongoItemReader <PaymentAudit> fileReader(){
return new MongoItemReaderBuilder <PaymentAudit>()
.name("file-reader")
.targetType(PaymentAudit.class)
.delimited().delimiter(",").names(new String [] {"rxFname" , "rxLname"})
.build();
}
#Bean
public ItemProcessor<PaymentAudit, PaymentAudit> processor() {
return new PaymentAuditItemProcessor();
}
#Bean
public ItemWriter<PaymentAudit> writer() {
return new MongoItemWriterBuilder()<PaymentAudit>();
try {
writer.setTemplate(mongoTemplate());
}catch (Exception e) {
LOG.error(e.toString());
}
writer.setCollection("paymentAudit");
return writer;
}
#Bean
Job job (JobBuilderFactory jbf,
StepBuilderFactory sbf,
ItemReader<? extends PaymentAudit> ir,
ItemWriter<? super PaymentAudit> iw) {
Step s1 = sbf.get("file-db")
.<PaymentAudit, PaymentAudit>chunk(100)
.reader(ir)
.writer(iw)
.build();
return jbf.get("etl")
.incrementer(new RunIdIncrementer())
.start(s1)
.build();
}
#Bean
public MongoDbFactory mongoDbFactory() throws Exception {
return new SimpleMongoDbFactory(new MongoClient(), "db-name");
}
#Bean
public MongoTemplate mongoTemplate() throws Exception {
MongoTemplate mongoTemplate = new MongoTemplate(mongoDbFactory());
return mongoTemplate;
}
}

method annotation with #BeforeStep not getting called

my goal is passing some value from a tasklet to another step which is consist of itemreader, processor and writer. based on the reference, i should use ExecutionContextPromotionListener. however, for some reason #BeforeStep is not being called. this is what I have.
Tasklet
#Component
public class RequestTasklet implements Tasklet {
#Autowired
private HistoryRepository historyRepository;
#Override
public RepeatStatus execute(StepContribution sc, ChunkContext cc) throws Exception {
List<History> requests
= historyRepository.findHistory();
ExecutionContext stepContext = cc.getStepContext().getStepExecution().getJobExecution().getExecutionContext();
stepContext.put("someKey", requests);
return RepeatStatus.FINISHED;
}
}
ItemReader
#Component
public class RequestReader implements ItemReader<History> {
private List<History> requests;
#Override
public History read() throws UnexpectedInputException,
ParseException,
NonTransientResourceException {
System.out.println("requests====>" + requests);
if (CollectionUtils.isNotEmpty(requests)) {
History request = requests.remove(0);
return request;
}
return null;
}
#BeforeStep
public void beforeStep(StepExecution stepExecution) {
System.out.println("====================here======================");
JobExecution jobExecution = stepExecution.getJobExecution();
ExecutionContext jobContext = jobExecution.getExecutionContext();
this.requests = (List<History>) jobContext.get("someKey");
}
}
Configuration
#Bean(name = "update-job")
public Job updateUserAttributes(
JobBuilderFactory jbf,
StepBuilderFactory sbf,
ExecutionContextPromotionListener promoListener,
HistoryProcessor processor,
RequestReader reader,
HistoryWriter writer,
RequestTasklet loadRequestTasklet) {
Step preStep = sbf.get("load-request-tasklet")
.tasklet(loadRequestTasklet)
.listener(promoListener)
.build();
Step step = sbf.get("update-step")
.<History, History>chunk(2)
.reader(reader)
.processor(processor)
.writer(writer)
.taskExecutor(taskExecutor())
.build();
return jbf.get("update-job")
.incrementer(new RunIdIncrementer())
.start(preStep).next(step).
build();
}
#Bean
public ExecutionContextPromotionListener promoListener() {
ExecutionContextPromotionListener listener = new ExecutionContextPromotionListener();
listener.setKeys(new String[]{
"someKey",
});
return listener;
}
I also tried to extend StepExecutionListenerSupport in ItemReader but got the same result.
I googled around and looked at this question answered due to spring proxy which is not my case beforestep issue.
little more information as i am testing. I set listener.setStrict(Boolean.TRUE); to see if keys are set. but i am getting
java.lang.IllegalArgumentException: The key [someKey] was not found in the Step's ExecutionContext.
at org.springframework.batch.core.listener.ExecutionContextPromotionListener.afterStep(ExecutionContextPromotionListener.java:61) ~[spring-batch-core-4.0.0.M1.jar:4.0.0.M1]
appreciate any help.

Is there a bug in Spring Batch Step flow function?

In the below piece of code, when StepA fails only StepB and StepC should execute but what actually happens is that all the 3 steps are getting executed! I want to split a spring batch job depending upon whether a step passes or not. I know that there are other ways of doing this by using JobDecider, setting some job parameter, etc but I wanted to know I was doing wrongly here?
#Configuration
#EnableBatchProcessing
public class JobConfig {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Bean
public PlatformTransactionManager transactionManager() {
return new ResourcelessTransactionManager();
}
#Bean
public JobRepository jobRepository() {
try {
return new MapJobRepositoryFactoryBean(transactionManager())
.getJobRepository();
} catch (Exception e) {
return null;
}
}
#Bean
public JobLauncher jobLauncher() {
final SimpleJobLauncher launcher = new SimpleJobLauncher();
launcher.setJobRepository(jobRepository());
return launcher;
}
#Bean
public Job job() {
return jobBuilderFactory.get("job").
flow(stepA()).on("FAILED").to(stepC()).next(stepD()).
from(stepA()).on("*").to(stepB()).next(stepC()).end().build();
}
#Bean
public Step stepA() {
return stepBuilderFactory.get("stepA")
.tasklet(new RandomFailTasket("stepA")).build();
}
#Bean
public Step stepB() {
return stepBuilderFactory.get("stepB")
.tasklet(new PrintTextTasklet("stepB")).build();
}
#Bean
public Step stepC() {
return stepBuilderFactory.get("stepC")
.tasklet(new PrintTextTasklet("stepC")).build();
}
#Bean
public Step stepD() {
return stepBuilderFactory.get("stepD")
.tasklet(new PrintTextTasklet("stepD")).build();
}
#SuppressWarnings("resource")
public static void main(String[] args) {
// create spring application context
final ApplicationContext appContext = new AnnotationConfigApplicationContext(
JobConfig.class);
// get the job config bean (i.e this bean)
final JobConfig jobConfig = appContext.getBean(JobConfig.class);
// get the job launcher
JobLauncher launcher = jobConfig.jobLauncher();
try {
// launch the job
JobExecution execution = launcher.run(jobConfig.job(), new JobParameters());
System.out.println(execution.getJobInstance().toString());
} catch (JobExecutionAlreadyRunningException e) {
e.printStackTrace();
} catch (JobRestartException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (JobInstanceAlreadyCompleteException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (JobParametersInvalidException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
StepA: is a dummy job which fails i.e it throws some exception
public class RandomFailTasket extends PrintTextTasklet {
public RandomFailTasket(String text) {
super(text);
}
public RepeatStatus execute(StepContribution arg0, ChunkContext arg1)
throws Exception {
if (Math.random() < 0.5){
throw new Exception("fail");
}
return RepeatStatus.FINISHED;
}
}
StepB, StepC, StepD are also dummy tasklets:
public class PrintTextTasklet implements Tasklet {
private final String text;
public PrintTextTasklet(String text){
this.text = text;
}
public RepeatStatus execute(StepContribution arg0, ChunkContext arg1)
throws Exception {
System.out.println(text);
return RepeatStatus.FINISHED;
}
}
need to have a look at the xml structure that you are using.
Try using Step listener - and then in the after step method you can check the Step status and then you can implement your logic to call the next step or not

Resources