Spring Batch: Retrying a tasklet using #Retryable and #EnableRetry annotation - spring

I have this tasklet which uploads a file to Amazon S3. Now, I want to retry the tasklet execution whenever an AmazonClientException is thrown. I figured using #Retryable annotation will do the job.
Tasklet:
#Component
#StepScope
#Retryable(value=AmazonClientException.class, stateful=true, backoff=#Backoff(2000))
public class S3UploadTasklet extends ArgsSupport implements Tasklet {
#Autowired
private S3Client s3Client;
#Autowired
private S3Properties s3Properties;
private static final Logger LOGGER = LoggerFactory.getLogger(S3UploadTasklet.class);
private static final String FILE_EXTENSION = ".gpg";
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
try {
String localFilename = getTempOutputFilename() + FILE_EXTENSION;
String s3Filename = s3Properties.getReportPath() + getS3OutputFilename() + FILE_EXTENSION;
File f = new File(localFilename);
if(f.exists()) {
LOGGER.info("Uploading " + localFilename + " to s3...");
s3Client.upload(localFilename, s3Filename, s3Properties.getBucketName());
LOGGER.info("Uploading done!");
} else {
throw new RuntimeException("Encrypted file not found! Encryption process might have failed.");
}
} catch(AmazonClientException e) {
LOGGER.error("Problems uploading to S3. " + e.getMessage(), e);
throw e;
} catch(RuntimeException e) {
LOGGER.error("Runtime error occured. " + e.getMessage(), e);
throw e;
}
return RepeatStatus.FINISHED;
}
}
Job configuration:
#Configuration
#EnableBatchProcessing
#EnableRetry
public class BatchConfiguration {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private Step generateReport;
#Autowired
private Step encrypt;
#Autowired
private Step upload;
#Autowired
private Step cleanUp;
#Bean
#Transactional(value="appTransactionManager", isolation=Isolation.READ_COMMITTED)
public Job generateReconJob() {
return jobBuilderFactory.get("reconJob")
.incrementer(new RunIdIncrementer())
.start(generateReport)
.on("COMPLETED").to(encrypt)
.from(generateReport)
.on("NOOP").end()
.from(generateReport)
.on("FAILED").to(cleanUp)
.from(encrypt)
.on("COMPLETED").to(upload)
.from(encrypt)
.on("FAILED").to(cleanUp)
.from(upload)
.on("*").to(cleanUp)
.from(cleanUp)
.on("*").end()
.end()
.build();
}
}
However, it doesn't do what it is supposed to do. The batch job still doesn't retry the tasklet when the exception is thrown.
Any thoughts?
Here's the config also
#Configuration
public class ReportConfiguration {
...
#Autowired
private S3UploadTasklet s3UploadTasklet;
...
#Bean
public Step upload() {
return stepBuilderFactory.get("upload")
.tasklet(s3UploadTasklet)
.build();
}
}

The #Retryable(value=AmazonClientException.class, stateful=true, backoff=#Backoff(2000)) annotation should be on the method you want to retry, not the class.

Related

If exception thrown, spring batch should not go to last method

#Configuration
public class BookStorePoliciesJobConfiguration {
private static final String CRON_EXPRESSION_FOR_JOB = "0 0/10 * 1/1 * ? *";//change
private static final String JOB_NAME = "bookStorePolicyJob";
private static final String JOB_STEP_NAME = JOB_NAME + "Step";
#Autowired
private DataSource nonXAcrmDataSource;
#Autowired
private JtaTransactionManager transactionManager;
#Bean
public CanerScheduledJobFactoryBean bookStorePoliciesScheduledJob() {
MafScheduledJobFactoryBean bean = new MafScheduledJobFactoryBean();
bean.setBatchJobName(JOB_NAME);
bean.setCronExp(CRON_EXPRESSION_FOR_JOB);
bean.setNonXAcrmDataSource(this.nonXAcrmDataSource);
return bean;
}
#Bean
public Job bookStorePolicyJob(#Autowired BookStorePoliciesJobTasklet tasklet,
#Autowired JobRepository batchJobRepository) {
SimpleJob job = new SimpleJob(JOB_NAME);
job.setSteps(Collections.singletonList(bookStorePolicyJobStep(tasklet, batchJobRepository)));
job.setJobRepository(batchJobRepository);
return job;
}
public Step bookStorePolicyJobStep(BookStoreJobTasklet tasklet, JobRepository batchJobRepository) {
TaskletStep step = new TaskletStep(JOB_STEP_NAME);
step.setTasklet(tasklet);
step.setJobRepository(batchJobRepository);
transactionManager.setAllowCustomIsolationLevels(true);
step.setTransactionManager(transactionManager);
return step;
}
}
This is the job:
#Component
public class BookStoreJobTasklet extends CanerTasklet {
#Override
public RepeatStatus doExecute(StepContribution contribution, ChunkContext chunkContext) {
///
//
sendFileBySftp(fileName, file);//if this throws, should not go down line, should exit but it goes
updateParameters();//if it is successfull only!
return RepeatStatus.FINISHED;
}
And the method:
private void sendFileBySftp(String fileName, File file) {
//
//
try{
//
} catch (JSchException | SftpException | IOException e) {
logger.error("error in sendFileFTP {}", e.getMessage());//it comes here and continues. it should exit?
}
logger.info("Session connection closed....");
}
What should i do to stop batch?
i look at here
Make a spring-batch job exit with non-zero code if an exception is thrown
Should i return RepeatStatus for each method so i can check if failed?
Maybe a boolean value and for each catch block, putting it to false?

Spring Batch Multiple JobExecutionListener - not working

I have a Spring boot batch application.
I have two job and two different JobExecutionListener. But always only one JobExecutionListener is called from both the job.
Job One
#Configuration
public class TestDriveJob {
#Autowired
JobLauncher jobLauncher;
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
TestDriveTasklet testDriveTasklet;
#Autowired
BatchJobMgmt batchJobMgmt;
#Autowired
TestDriveJobExecutionListener testDriveJobExecutionListener;
private static final String JOB_NAME = "test-drive-job";
#Scheduled(fixedDelay = 100 * 1000, initialDelay = 5000)
public void schedule() throws Exception {
if (batchJobMgmt.isJobAllowed(JOB_NAME)) {
JobParameters param = new JobParametersBuilder().addString("JobID", "TEST" + String.valueOf(System.currentTimeMillis())).toJobParameters();
jobLauncher.run(job(), param);
}
}
#Bean
public Job job() {
return jobBuilderFactory.get(JOB_NAME).listener(testDriveJobExecutionListener).incrementer(new RunIdIncrementer()).start(testDriveStep()).build();
}
#Bean
public Step testDriveStep() {
return stepBuilderFactory.get("test-drive-step").tasklet(testDriveTasklet).listener(testDriveJobExecutionListener).build();
}
}
Job Two
#Configuration
public class ExtendedWarrantyJob {
private static final Logger LOGGER = LoggerFactory.getLogger(ExtendedWarrantyJob.class);
#Autowired
JobLauncher jobLauncher;
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
ExtendedWarrantyTasklet extendedWarrantyTasklet;
#Autowired
BatchJobMgmt batchJobMgmt;
#Autowired
ExtendedWarrantyJobExecutionListener extendedWarrantyJobExecutionListener;
private static final String JOB_NAME = "extended-warranty-job";
#Scheduled(fixedDelay = 900 * 1000, initialDelay = 20000)
public void schedule() throws Exception {
if (batchJobMgmt.isJobAllowed(JOB_NAME)) {
LOGGER.debug("Extended Warranty Job Started at :" + new Date());
JobParameters param = new JobParametersBuilder().addString("JobID", String.valueOf(System.currentTimeMillis())).toJobParameters();
JobExecution execution = jobLauncher.run(job(), param);
LOGGER.debug("Extended Warranty Job finished with status :" + execution.getStatus());
}
}
#Bean
public Job job() {
return jobBuilderFactory.get(JOB_NAME).listener(extendedWarrantyJobExecutionListener).incrementer(new RunIdIncrementer()).start(extendedWarrantyStep())
.build();
}
#Bean
public Step extendedWarrantyStep() {
return stepBuilderFactory.get("extended-warranty-step").tasklet(extendedWarrantyTasklet).build();
}
}
Job one Listener
#Service
public class TestDriveJobExecutionListener implements JobExecutionListener {
private static final Logger LOGGER = LoggerFactory.getLogger(TestDriveJobExecutionListener.class);
#Override
public void beforeJob(JobExecution jobExecution) {
LOGGER.debug("{} Started at : {}", jobExecution.getJobInstance().getJobName(), new Date());
}
#Override
public void afterJob(JobExecution jobExecution) {
LOGGER.debug("{} Completed at : {}", jobExecution.getJobInstance().getJobName(), new Date());
}
}
Job Two Listener
#Service
public class ExtendedWarrantyJobExecutionListener implements JobExecutionListener {
private static final Logger LOGGER = LoggerFactory.getLogger(ExtendedWarrantyJobExecutionListener.class);
#Override
public void beforeJob(JobExecution jobExecution) {
LOGGER.debug("{} Started=== at : {}", jobExecution.getJobInstance().getJobName(), new Date());
}
#Override
public void afterJob(JobExecution jobExecution) {
LOGGER.debug("{} Completed at : {}", jobExecution.getJobInstance().getJobName(), new Date());
}
}
Both the jobs are executing without any issue. But Always the job one listener is called. Job Two Listener is not working.
I tried adding #Qualifier to the ##Autowired annotation, but it did not make any sense.
any help is appreciated.

MongoDB Spring Batch job using Gradle

I want to create an item reader, writer and processor for a Spring batch job using Gradle. I am having trouble with a few things. The delimited() portion is giving me an error. I am trying to read two fields for now: rxFname, rxLname.
Here is my code:
#Configuration
#EnableBatchProcessing
public class PaymentPortalJob {
private static final Logger LOG =
LoggerFactory.getLogger(PaymentPortalJob.class);
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean
MongoItemReader <PaymentAudit> fileReader(){
return new MongoItemReaderBuilder <PaymentAudit>()
.name("file-reader")
.targetType(PaymentAudit.class)
.delimited().delimiter(",").names(new String [] {"rxFname" , "rxLname"})
.build();
}
#Bean
public ItemProcessor<PaymentAudit, PaymentAudit> processor() {
return new PaymentAuditItemProcessor();
}
#Bean
public ItemWriter<PaymentAudit> writer() {
return new MongoItemWriterBuilder()<PaymentAudit>();
try {
writer.setTemplate(mongoTemplate());
}catch (Exception e) {
LOG.error(e.toString());
}
writer.setCollection("paymentAudit");
return writer;
}
#Bean
Job job (JobBuilderFactory jbf,
StepBuilderFactory sbf,
ItemReader<? extends PaymentAudit> ir,
ItemWriter<? super PaymentAudit> iw) {
Step s1 = sbf.get("file-db")
.<PaymentAudit, PaymentAudit>chunk(100)
.reader(ir)
.writer(iw)
.build();
return jbf.get("etl")
.incrementer(new RunIdIncrementer())
.start(s1)
.build();
}
#Bean
public MongoDbFactory mongoDbFactory() throws Exception {
return new SimpleMongoDbFactory(new MongoClient(), "db-name");
}
#Bean
public MongoTemplate mongoTemplate() throws Exception {
MongoTemplate mongoTemplate = new MongoTemplate(mongoDbFactory());
return mongoTemplate;
}
}

Spring batch execute dynamically generated steps in a tasklet

I have a spring batch job that does the following...
Step 1. Creates a list of objects that need to be processed
Step 2. Creates a list of steps depending on how many items are in the list of objects created in step 1.
Step 3. Tries to executes the steps from the list of steps created in step 2.
The executing x steps is done below in executeDynamicStepsTasklet(). While the code runs without any errors it does not seem to be doing anything. Does what I have in that method look correct?
thanks
/*
*
*/
#Configuration
public class ExportMasterListCsvJobConfig {
public static final String JOB_NAME = "exportMasterListCsv";
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Value("${exportMasterListCsv.generateMasterListRows.chunkSize}")
public int chunkSize;
#Value("${exportMasterListCsv.generateMasterListRows.masterListSql}")
public String masterListSql;
#Autowired
public DataSource onlineStagingDb;
#Value("${out.dir}")
public String outDir;
#Value("${exportMasterListCsv.generatePromoStartDateEndDateGroupings.promoStartDateEndDateSql}")
private String promoStartDateEndDateSql;
private List<DivisionIdPromoCompStartDtEndDtGrouping> divisionIdPromoCompStartDtEndDtGrouping;
private List<Step> dynamicSteps = Collections.synchronizedList(new ArrayList<Step>()) ;
#Bean
public Job exportMasterListCsvJob(
#Qualifier("createJobDatesStep") Step createJobDatesStep,
#Qualifier("createDynamicStepsStep") Step createDynamicStepsStep,
#Qualifier("executeDynamicStepsStep") Step executeDynamicStepsStep) {
return jobBuilderFactory.get(JOB_NAME)
.flow(createJobDatesStep)
.next(createDynamicStepsStep)
.next(executeDynamicStepsStep)
.end().build();
}
#Bean
public Step executeDynamicStepsStep(
#Qualifier("executeDynamicStepsTasklet") Tasklet executeDynamicStepsTasklet) {
return stepBuilderFactory
.get("executeDynamicStepsStep")
.tasklet(executeDynamicStepsTasklet)
.build();
}
#Bean
public Tasklet executeDynamicStepsTasklet() {
return new Tasklet() {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
FlowStep flowStep = new FlowStep(createParallelFlow());
SimpleJobBuilder jobBuilder = jobBuilderFactory.get("myNewJob").start(flowStep);
return RepeatStatus.FINISHED;
}
};
}
public Flow createParallelFlow() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setConcurrencyLimit(1);
List<Flow> flows = dynamicSteps.stream()
.map(step -> new FlowBuilder<Flow>("flow_" + step.getName()).start(step).build())
.collect(Collectors.toList());
return new FlowBuilder<SimpleFlow>("parallelStepsFlow")
.split(taskExecutor)
.add(flows.toArray(new Flow[flows.size()]))
.build();
}
#Bean
public Step createDynamicStepsStep(
#Qualifier("createDynamicStepsTasklet") Tasklet createDynamicStepsTasklet) {
return stepBuilderFactory
.get("createDynamicStepsStep")
.tasklet(createDynamicStepsTasklet)
.build();
}
#Bean
#JobScope
public Tasklet createDynamicStepsTasklet() {
return new Tasklet() {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
for (DivisionIdPromoCompStartDtEndDtGrouping grp: divisionIdPromoCompStartDtEndDtGrouping){
System.err.println("grp: " + grp);
String stepName = "stp_" + grp;
String fileName = grp + FlatFileConstants.EXTENSION_CSV;
Step dynamicStep =
stepBuilderFactory.get(stepName)
.<MasterList,MasterList> chunk(10)
.reader(queryStagingDbReader(
grp.getDivisionId(),
grp.getRpmPromoCompDetailStartDate(),
grp.getRpmPromoCompDetailEndDate()))
.writer(masterListFileWriter(fileName))
.build();
dynamicSteps.add(dynamicStep);
}
System.err.println("createDynamicStepsTasklet dynamicSteps: " + dynamicSteps);
return RepeatStatus.FINISHED;
}
};
}
public FlatFileItemWriter<MasterList> masterListFileWriter(String fileName) {
FlatFileItemWriter<MasterList> writer = new FlatFileItemWriter<>();
writer.setResource(new FileSystemResource(new File(outDir, fileName )));
writer.setHeaderCallback(masterListFlatFileHeaderCallback());
writer.setLineAggregator(masterListFormatterLineAggregator());
return writer;
}
So now I have a list of dynamic steps that need to be executed and I believe that they are in StepScope. Can someone advise me on how to execute them
This will not work. Your Tasklet just creates a job with a FlowStep as first Step. Using the jobBuilderfactory just creates the job. it does not launch it. The methodname "start" may be misleading, since this only defines the first step. But it does not launch the job.
You cannot change the structure of a job (its steps and substeps) once it is started. Therefore, it is not possible to configure a flowstep in step 2 based on things that are calculated in step 1. (of course you could do some hacking deeper inside the springbatch structure and directly modify the beans and so ... but you don't want to do that).
I suggest, that you use a kind of "SetupBean" with an appropriate postConstruct method which is injected into your class that configures your job. This "SetupBean" is responsible to calculate the list of objects being processed.
#Component
public class SetUpBean {
private List<Object> myObjects;
#PostConstruct
public afterPropertiesSet() {
myObjects = ...;
}
public List<Object> getMyObjects() {
return myObjects;
}
}
#Configuration
public class JobConfiguration {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private SetUpBean setup;
...
}

spring batch with annotations

I'm currently trying to create a batch with the spring annotations but the batch is never called. No error occurs, my batch isn't called. Its a simple batch that retrieves values from the database and add messages in a queue (rabbitmq).
The main configuration class:
#Configuration
#EnableBatchProcessing
public class BatchInfrastructureConfiguration {
#Bean
public JobLauncher getJobLauncher() throws Exception {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(getJobRepository());
jobLauncher.afterPropertiesSet();
return jobLauncher;
}
public JobRepository getJobRepository() throws Exception {
MapJobRepositoryFactoryBean factory = new MapJobRepositoryFactoryBean();
factory.setTransactionManager(new ResourcelessTransactionManager());
factory.afterPropertiesSet();
return (JobRepository) factory.getObject();
}
}
The configuration class specific to my batch
#Configuration
#Import(BatchInfrastructureConfiguration.class)
public class PurchaseStatusBatchConfiguration {
#Inject
private JobBuilderFactory jobBuilders;
#Inject
private StepBuilderFactory stepBuilders;
#Bean
public Job purchaseStatusJob(){
return jobBuilders.get("purchaseStatusJob")
.start(step())
.build();
}
#Bean
public Step step(){
return stepBuilders.get("purchaseStatusStep")
.tasklet(new PurchaseStatusBatch())
.build();
}
}
The batch class:
public class PurchaseStatusBatch implements Tasklet {
#Inject
private PurchaseRepository purchaseRepository;
#Inject
#Qualifier(ApplicationConst.BEAN_QUALIFIER_PURCHASE_QUEUE)
private RabbitTemplate rabbitTemplate;
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
PurchaseDto purchaseDto;
PurchaseMessage purchaseMessage;
List<Purchase> notVerifiedPurchase = purchaseRepository.findByVerified(false);
for (Purchase purchase : notVerifiedPurchase) {
purchaseDto = new PurchaseDto();
purchaseDto.setOrderId(purchase.getOrderId());
purchaseDto.setProductId(purchase.getProductId());
purchaseDto.setPurchaseToken(purchase.getPurchaseToken());
purchaseDto.setUserScrapbookKey(purchase.getUserScrapbookKey());
purchaseMessage = new PurchaseMessage();
purchaseMessage.setPurchaseDto(purchaseDto);
rabbitTemplate.convertAndSend(purchaseMessage);
}
return null;
}
}
The job runner (class calling the batch):
#Service
public class PurchaseStatusJobRunner {
#Inject
private JobLocator jobLocator;
#Inject
private JobLauncher jobLauncher;
//#Scheduled(fixedDelay = 3000L)
//#Scheduled(cron="* * * * *") // every 1 minute
#Scheduled(fixedDelay = 3000L)
public void runJob() throws JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException, JobParametersInvalidException, NoSuchJobException {
jobLauncher.run(jobLocator.getJob("purchaseStatusJob"), new JobParameters());
}
}
Short answer: The problem is that your PurchaseStatusBatch instance is not a spring bean... then all its attributes are null (they have not been injected)
Try this :
#Bean
public Step step(){
return stepBuilders.get("purchaseStatusStep")
.tasklet(purchaseStatusBatch())
.build();
}
#Bean
public PurchaseStatusBatch purchaseStatusBatch() {
return new PurchaseStatusBatch()
}
When you want to have feedback on your job execution, just use the JobExecution instance returned by the JobLauncher. You can get the ExitStatus, you can get all exceptions catched by the JobLauncher, and more information.
Another solution to have feedback would be providing a real database to your JobRepository, and then check execution statuses in it.

Resources