How can we get the JobId in the RetryContext? - spring

I am just extending my this question here - Spring Retry doesn't works when we use RetryTemplate?.
How can we get the JobId in the RetryContext ?
I went through link: Spring Batch how to configure retry period for failed jobs, but still did not know.
#Component
#Slf4j
public class RecoveryCallback implements RecoveryCallback<String>{
#Autowired
private NamedParameterJdbcTemplate namedJdbcTemplate;
#Autowired
private AbcService abcService;
#Value("#{stepExecution.jobExecution.jobId}")
private Long jobId;
#Override
public String recover(RetryContext context) throws Exception {
log.warn("RecoveryCallback | recover is executed ...");
ErrorLog errorLog = ErrorLog.builder()
.jobName("ABC")
.stepName("RETRY_STEP")
.stepType("RETRY")
....
....
....
.jobId(jobId)
.build();
abcService.updateErrLog(errorLog);
return "Batch Job Retried and exausted with all attemps";
}
}

Since you are injecting stepExecution.jobExecution.jobId in a field of a Spring bean, you need to make this bean Step scoped. With this approach, the RetryContext is not used.
If you want to use the retry context, then you need to put the jobId in the context in the retryable method first. From your linked question:
retryTemplate.execute(retryContext -> {
JobExecution jobExecution = jobLauncher.run(sampleAcctJob, pdfParams);
if(!jobExecution.getAllFailureExceptions().isEmpty()) {
log.error("============== sampleAcctJob Job failed, retrying.... ================");
throw jobExecution.getAllFailureExceptions().iterator().next();
}
logDetails(jobExecution);
// PUT JOB ID in retryContext
retryContext.setAttribute("jobId", jobExecution.getExecutionId());
return jobExecution;
});
With that, you can get the jobId from the context in the recover method:
#Override
public String recover(RetryContext context) throws Exception {
log.warn("RecoveryCallback | recover is executed ...");
ErrorLog errorLog = ErrorLog.builder()
.jobName("ABC")
.stepName("RETRY_STEP")
.stepType("RETRY")
....
....
.jobId(context.getAttribute("jobId"))
.build();
abcService.updateErrLog(errorLog);
return "Batch Job Retried and exausted with all attemps";
}

Related

Why is exception in Spring Batch AsycItemProcessor caught by SkipListener's onSkipInWrite method?

I'm writing a Spring Boot application that starts up, gathers and converts millions of database entries into a new streamlined JSON format, and then sends them all to a GCP PubSub topic. I'm attempting to use Spring Batch for this, but I'm running into trouble implementing fault tolerance for my process. The database is rife with data quality issues, and sometimes my conversions to JSON will fail. When failures occur, I don't want the job to immediately quit, I want it to continue processing as many records as it can and, before completion, to report which exact records failed so that I, and or my team, can examine these problematic database entries.
To achieve this, I've attempted to use Spring Batch's SkipListener interface. But I'm also using an AsyncItemProcessor and an AsyncItemWriter in my process, and even though the exceptions are occurring during the processing, the SkipListener's onSkipInWrite() method is catching them - rather than the onSkipInProcess() method. And unfortunately, the onSkipInWrite() method doesn't have access to the original database entity, so I can't store its ID in my list of problematic DB entries.
Have I misconfigured something? Is there any other way to gain access to the objects from the reader that failed the processing step of an AsynItemProcessor?
Here's what I've tried...
I have a singleton Spring Component where I store how many DB entries I've successfully processed along with up to 20 problematic database entries.
#Component
#Getter //lombok
public class ProcessStatus {
private int processed;
private int failureCount;
private final List<UnexpectedFailure> unexpectedFailures = new ArrayList<>();
public void incrementProgress { processed++; }
public void logUnexpectedFailure(UnexpectedFailure failure) {
failureCount++;
unexpectedFailure.add(failure);
}
#Getter
#AllArgsConstructor
public static class UnexpectedFailure {
private Throwable error;
private DBProjection dbData;
}
}
I have a Spring batch Skip Listener that's supposed to catch failures and update my status component accordingly:
#AllArgsConstructor
public class ConversionSkipListener implements SkipListener<DBProjection, Future<JsonMessage>> {
private ProcessStatus processStatus;
#Override
public void onSkipInRead(Throwable error) {}
#Override
public void onSkipInProcess(DBProjection dbData, Throwable error) {
processStatus.logUnexpectedFailure(new ProcessStatus.UnexpectedFailure(error, dbData));
}
#Override
public void onSkipInWrite(Future<JsonMessage> messageFuture, Throwable error) {
//This is getting called instead!! Even though the exception happened during processing :(
//But I have no access to the original DBProjection data here, and messageFuture.get() gives me null.
}
}
And then I've configured my job like this:
#Configuration
public class ConversionBatchJobConfig {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private TaskExecutor processThreadPool;
#Bean
public SimpleCompletionPolicy processChunkSize(#Value("${commit.chunk.size:100}") Integer chunkSize) {
return new SimpleCompletionPolicy(chunkSize);
}
#Bean
#StepScope
public ItemStreamReader<DbProjection> dbReader(
MyDomainRepository myDomainRepository,
#Value("#{jobParameters[pageSize]}") Integer pageSize,
#Value("#{jobParameters[limit]}") Integer limit) {
RepositoryItemReader<DbProjection> myDomainRepositoryReader = new RepositoryItemReader<>();
myDomainRepositoryReader.setRepository(myDomainRepository);
myDomainRepositoryReader.setMethodName("findActiveDbDomains"); //A native query
myDomainRepositoryReader.setArguments(new ArrayList<Object>() {{
add("ACTIVE");
}});
myDomainRepositoryReader.setSort(new HashMap<String, Sort.Direction>() {{
put("update_date", Sort.Direction.ASC);
}});
myDomainRepositoryReader.setPageSize(pageSize);
myDomainRepositoryReader.setMaxItemCount(limit);
// myDomainRepositoryReader.setSaveState(false); <== haven't figured out what this does yet
return myDomainRepositoryReader;
}
#Bean
#StepScope
public ItemProcessor<DbProjection, JsonMessage> dataConverter(DataRetrievalSerivice dataRetrievalService) {
//Sometimes throws exceptions when DB data is exceptionally weird, bad, or missing
return new DbProjectionToJsonMessageConverter(dataRetrievalService);
}
#Bean
#StepScope
public AsyncItemProcessor<DbProjection, JsonMessage> asyncDataConverter(
ItemProcessor<DbProjection, JsonMessage> dataConverter) throws Exception {
AsyncItemProcessor<DbProjection, JsonMessage> asyncDataConverter = new AsyncItemProcessor<>();
asyncDataConverter.setDelegate(dataConverter);
asyncDataConverter.setTaskExecutor(processThreadPool);
asyncDataConverter.afterPropertiesSet();
return asyncDataConverter;
}
#Bean
#StepScope
public ItemWriter<JsonMessage> jsonPublisher(GcpPubsubPublisherService publisherService) {
return new JsonMessageWriter(publisherService);
}
#Bean
#StepScope
public AsyncItemWriter<JsonMessage> asyncJsonPublisher(ItemWriter<JsonMessage> jsonPublisher) throws Exception {
AsyncItemWriter<JsonMessage> asyncJsonPublisher = new AsyncItemWriter<>();
asyncJsonPublisher.setDelegate(jsonPublisher);
asyncJsonPublisher.afterPropertiesSet();
return asyncJsonPublisher;
}
#Bean
public Step conversionProcess(SimpleCompletionPolicy processChunkSize,
ItemStreamReader<DbProjection> dbReader,
AsyncItemProcessor<DbProjection, JsonMessage> asyncDataConverter,
AsyncItemWriter<JsonMessage> asyncJsonPublisher,
ProcessStatus processStatus,
#Value("${conversion.failure.limit:20}") int maximumFailures) {
return stepBuilderFactory.get("conversionProcess")
.<DbProjection, Future<JsonMessage>>chunk(processChunkSize)
.reader(dbReader)
.processor(asyncDataConverter)
.writer(asyncJsonPublisher)
.faultTolerant()
.skipPolicy(new MyCustomConversionSkipPolicy(maximumFailures))
// ^ for now this returns true for everything until 20 failures
.listener(new ConversionSkipListener(processStatus))
.build();
}
#Bean
public Job conversionJob(Step conversionProcess) {
return jobBuilderFactory.get("conversionJob")
.start(conversionProcess)
.build();
}
}
This is because the future wrapped by the AsyncItemProcessor is only unwrapped in the AsyncItemWriter, so any exception that might occur at that time is seen as a write exception instead of a processing exception. That's why onSkipInWrite is called instead of onSkipInProcess.
This is actually a known limitation of this pattern which is documented in the Javadoc of the AsyncItemProcessor, here is an excerpt:
Because the Future is typically unwrapped in the ItemWriter,
there are lifecycle and stats limitations (since the framework doesn't know
what the result of the processor is).
While not an exhaustive list, things like StepExecution.filterCount will not
reflect the number of filtered items and
itemProcessListener.onProcessError(Object, Exception) will not be called.
The Javadoc states that the list is not exhaustive, and the side-effect regarding the SkipListener that you are experiencing is one these limitations.

Writing unit tests for camel routes in a SpringBoot application - getting messageCount 0

I am trying to write unit tests for a camel route - its for importing and processing a file
from(fullImportFTP)
.routeId(STUB_FILE_DOWNLOAD_ROUTE_ID)
.onException(Exception.class)
.handled(false)
.log(LoggingLevel.ERROR, STUB_IMPORT_ERROR_CODE)
.end()
.log("Processing Stub file:[${header.CamelFileName}]")
.to(ROUTE_TO_MACE);
from(ROUTE_TO_MACE)
.routeId(STUB_FILE_IMPORT_ROUTE_ID)
.onException(Exception.class)
.handled(false)
.log(LoggingLevel.ERROR, STUB_IMPORT_ERROR_CODE)
.end()
.onException(IOException.class)
.maximumRedeliveries(routesConfig.camelMaximumRetries).redeliveryDelay(routesConfig.camelRedeliveryDelay)
.handled(false)
.log(LoggingLevel.ERROR, STUB_IMPORT_ERROR_CODE)
.end()
.split().tokenizeXML(ITEM).streaming()
.process(processor)
.to("log:route.StubRoute?level=DEBUG")
.end()
.log("Stub file sucessfully processed:[${header.CamelFileName}]");
And below is the unit test:
#RunWith(CamelSpringBootRunner.class)
#SpringBootTest
#DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD)
public class CamelRouteTest {
#EndpointInject(uri = "mock:success_result")
private MockEndpoint successResultEndpoint;
#EndpointInject(uri = "direct:mock-import-stub-download")
private FluentProducerTemplate producer;
#Autowired
private CamelContext camelContext;
#MockBean
RestTemplate restTemplate;
private static final String MOCK_IMPORT_STUB_DOWNLOAD = "direct:mock-import-stub-download";
private static final String TEST_STUB_FILE_LOCATION = "src/test/resources";
#Before
public void setup() throws Exception {
camelContext.getRouteDefinition(STUB_FILE_DOWNLOAD_ROUTE_ID).autoStartup(true).adviceWith(camelContext,
new AdviceWithRouteBuilder() {
#Override
public void configure() throws Exception {
replaceFromWith(MOCK_IMPORT_STUB_DOWNLOAD);
interceptSendToEndpoint("log:route.StubRoute?level=DEBUG").skipSendToOriginalEndpoint().to(successResultEndpoint);
}
});
camelContext.start();
}
#Test
public void testFileDownloadRouter() throws Exception {
File file = new File(TEST_STUB_FILE_LOCATION + "/Stub_11092018_162149_59642501.xml");
successResultEndpoint.expectedMessageCount(1);
producer.withBody(file).withHeader(Exchange.FILE_NAME, "Stub_24102018_162149_59642501.xml").send();
successResultEndpoint.assertIsSatisfied();
}
I always get the message count as 0. Here is the ERROR
java.lang.AssertionError: mock://success_result Received message
count. Expected: <1> but was: <0> Expected :<1> Actual :<0>
What am I doing wrong here? I have 2 routes as you can see - the first one actually goes to the second one, so in the unit tests should I have 2 routes too? I haven't added 2 routes because if I debug I can see that it actually goes through the processor and returning the correct result.
First of all: you are using AdviceWith, so you should put the annotation #UseAdviceWith on your testclass. Otherwise the automatic start of the Camel context and the route advice could overlap.
For the missing message on the Mock: Perhaps your test just asserts too early. I guess the producer does not block while the message is processed, but the MockEndpoint assert follows immediately after sending the message. Right after sending the message the received message count is still 0.
To check if waiting helps, you could insert a Thread.sleep(). If it works, you should get rid of the Thread.sleep() and replace it with a Camel NotifyBuilder
Just saw another point. The final to() in your interceptSendToEndpoint chain points to the MockEndpoint instance variable. I think this should point to the MockEndpoint URI, i.e. .to("mock:success_result")
And even one more: you get the first route to advice with getRouteDefinition(STUB_FILE_DOWNLOAD_ROUTE_ID) but in this advice block you advice both routes. That is probably the reason for your problem. The second route is not adviced and therefore your mock is not in place. You have to advice the second route in its own advice block.
#Before
public void setup() throws Exception {
camelContext.getRouteDefinition(STUB_FILE_DOWNLOAD_ROUTE_ID).autoStartup(true).adviceWith(camelContext, new AdviceWithRouteBuilder() {
#Override
public void configure() throws Exception {
replaceFromWith(MOCK_IMPORT_STUB_DOWNLOAD);
}
});
camelContext.getRouteDefinition(STUB_FILE_IMPORT_ROUTE_ID).autoStartup(true).adviceWith(camelContext, new AdviceWithRouteBuilder() {
#Override
public void configure() throws Exception {
interceptSendToEndpoint("log:route.StubRoute?level=DEBUG").skipSendToOriginalEndpoint().to("mock:success_result");
}
});
camelContext.start();
}

Spring batch stop a job

How can I stop a job in spring batch ? I tried to use this method using the code below:
public class jobListener implements JobExecutionListener{
#Override
public void beforeJob(JobExecution jobExecution) {
jobExecution.setExitStatus(ExitStatus.STOPPED);
}
#Override
public void afterJob(JobExecution jobExecution) {
// TODO Auto-generated method stub
}
}
I tried also COMPLETED,FAILED but this method doesn't work and the job continues to execute. Any solution?
You can use JobOperator along with JobExplorer to stop a job from outside the job (see https://docs.spring.io/spring-batch/reference/html/configureJob.html#JobOperator). The method is stop(long executionId) You would have to use JobExplorer to find the correct executionId to stop.
Also from within a job flow config you can configure a job to stop after a steps execution based on exit status (see https://docs.spring.io/spring-batch/trunk/reference/html/configureStep.html#stopElement).
I assume you want to stop a job by a given name.
Here is the code.
String jobName = jobExecution.getJobInstance().getJobName(); // in most cases
DataSource dataSource = ... //#Autowire it in your code
JobOperator jobOperator = ... //#Autowire it in your code
JobExplorerFactoryBean factory = new JobExplorerFactoryBean();
factory.setDataSource(dataSource);
factory.setJdbcOperations(new JdbcTemplate(dataSource));
JobExplorer jobExplorer = factory.getObject();
Set<JobExecution> jobExecutions = jobExplorer.findRunningJobExecutions(jobName);
jobExecutions.forEach(jobExecution -> jobOperator.stop(jobExecution.getId()));

Return job id "immediately" for spring batch job before it completes

I am working on a project where we are using Spring Boot, Spring Batch and Camel.
The batch process is started by a call to a rest endpoint. The rest controller starts a camel route that starts the spring batch job flow (via spring batch camel component).
I have no control over the external application that calls my application. My application is part of a bigger nightly work flow.
The batch job can take a long time to complete and therefore the external application periodically polls my batch job via another rest endpoint asking if the job is complete. It does this by polling a status rest endpoint with the id of the jobExecution it wants a status on.
To accomplish this flow I have implemented a rest controller that starts the camel route via a ProducerTemplate. My problem is returning the job execution id immediately after starting the camel route. I don't want the rest call to wait until the job is complete to return.
startJobViaRestCall ------> createBatchJob ----> runBatchJobUntilDone
|
|
Return jobExecutionData |
<----------------------------------
I have tried using async calls and futures, but with no luck. I have also tried to use Camels wiretap to no avail. The problem is that there is only "onComplete" events. I need an hook that returns as soon as the job has been created, but not run.
For example, the following code waits until the batch job is done before returning the JobExecution data I want to send back (as json). It makes sense as extractFutureBody will wait until the response is ready.
#RestController
#Slf4j
public class BatchJobController {
#Autowired
ProducerTemplate producerTemplate;
#RequestMapping(value = "/batch/job/start", method = RequestMethod.GET)
#ResponseBody
public String startBatchJob() {
log.info("BatchJob start called...");
String jobExecution = producerTemplate.extractFutureBody(producerTemplate.asyncRequestBody(BatchRoute.ENDPOINT_JOB_START, ""), String.class);
return jobExecution;
}
}
The camel route is a simple call to the spring-batch-component
public class BatchRoute<I, O> extends BaseRoute {
private static final String ROUTE_START_BATCH = "spring-batch:springBatchJob";
#Override
public void configure() {
super.configure();
from(ENDPOINT_JOB_START).to(ROUTE_START_BATCH);
}
}
Any ideas as to how I can return the JobExecution data as soon as it is available?
Not sure How you could do it in Camel, but here is sample Job execution using spring-rest.
#RestController
public class KpRest {
private static final Logger LOG = LoggerFactory.getLogger(KpRest.class);
private static String RUN_ID_KEY = "run.id";
#Autowired
private JobLauncher launcher;
private final AtomicLong incrementer = new AtomicLong();
#Autowired
private Job job;
#RequestMapping("/hello")
public String sayHello(){
try {
JobParameters parameters = new JobParametersBuilder().addLong(RUN_ID_KEY, incrementer.incrementAndGet()).toJobParameters();
JobExecution execution = launcher.run(job, parameters);
LOG.info("JobId {}, JobStatus {}", execution.getJobId(), execution.getStatus().getBatchStatus());
return String.valueOf(execution.getJobId());
} catch (JobExecutionAlreadyRunningException | JobRestartException | JobInstanceAlreadyCompleteException
| JobParametersInvalidException e) {
LOG.info("Job execution failed, {}", e);
}
return "Some Error";
}
}
You can make the Job async by modifying JobLauncher.
#Bean
public JobLauncher simpleJobLauncher(JobRepository jobRepository){
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
jobLauncher.setTaskExecutor(new SimpleAsyncTaskExecutor());
return jobLauncher;
}
Refer the documentation for more info

Spring rabbit retries to deliver rejected message..is it OK?

I have the following configuration
spring.rabbitmq.listener.prefetch=1
spring.rabbitmq.listener.concurrency=1
spring.rabbitmq.listener.retry.enabled=true
spring.rabbitmq.listener.retry.max-attempts=3
spring.rabbitmq.listener.retry.max-interval=1000
spring.rabbitmq.listener.default-requeue-rejected=false //I have also changed it to true but the same behavior still happens
and in my listener I throw the exception AmqpRejectAndDontRequeueException to reject the message and enforce rabbit not to try to redeliver it...But rabbit redilvers it for 3 times then finally route it to dead letter queue.
Is that the standard behavior according to my provided configuration or do I miss something?
You have to configure the retry policy to not retry for that exception.
You can't do that with properties, you have to configure the retry advice yourself.
I'll post an example later if you need help with that.
requeue-rejected is at the container level (below retry on the stack).
EDIT
#SpringBootApplication
public class So39853762Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = SpringApplication.run(So39853762Application.class, args);
Thread.sleep(60000);
context.close();
}
#RabbitListener(queues = "foo")
public void foo(String foo) {
System.out.println(foo);
if ("foo".equals(foo)) {
throw new AmqpRejectAndDontRequeueException("foo"); // won't be retried.
}
else {
throw new IllegalStateException("bar"); // will be retried
}
}
#Bean
public ListenerRetryAdviceCustomizer retryCustomizer(SimpleRabbitListenerContainerFactory containerFactory,
RabbitProperties rabbitPropeties) {
return new ListenerRetryAdviceCustomizer(containerFactory, rabbitPropeties);
}
public static class ListenerRetryAdviceCustomizer implements InitializingBean {
private final SimpleRabbitListenerContainerFactory containerFactory;
private final RabbitProperties rabbitPropeties;
public ListenerRetryAdviceCustomizer(SimpleRabbitListenerContainerFactory containerFactory,
RabbitProperties rabbitPropeties) {
this.containerFactory = containerFactory;
this.rabbitPropeties = rabbitPropeties;
}
#Override
public void afterPropertiesSet() throws Exception {
ListenerRetry retryConfig = this.rabbitPropeties.getListener().getRetry();
if (retryConfig.isEnabled()) {
RetryInterceptorBuilder<?> builder = (retryConfig.isStateless()
? RetryInterceptorBuilder.stateless()
: RetryInterceptorBuilder.stateful());
Map<Class<? extends Throwable>, Boolean> retryableExceptions = new HashMap<>();
retryableExceptions.put(AmqpRejectAndDontRequeueException.class, false);
retryableExceptions.put(IllegalStateException.class, true);
SimpleRetryPolicy policy =
new SimpleRetryPolicy(retryConfig.getMaxAttempts(), retryableExceptions, true);
ExponentialBackOffPolicy backOff = new ExponentialBackOffPolicy();
backOff.setInitialInterval(retryConfig.getInitialInterval());
backOff.setMultiplier(retryConfig.getMultiplier());
backOff.setMaxInterval(retryConfig.getMaxInterval());
builder.retryPolicy(policy)
.backOffPolicy(backOff)
.recoverer(new RejectAndDontRequeueRecoverer());
this.containerFactory.setAdviceChain(builder.build());
}
}
}
}
NOTE: You cannot currently configure the policy to retry all exceptions, "except" this one - you have to classify all exceptions you want retried (and they can't be a superclass of AmqpRejectAndDontRequeueException). I have opened an issue to support this.
The other answers posted here didn't work me when using Spring Boot 2.3.5 and Spring AMQP Starter 2.2.12, but for these versions I was able to customize the retry policy to not retry AmqpRejectAndDontRequeueException exceptions:
#Configuration
public class RabbitConfiguration {
#Bean
public RabbitRetryTemplateCustomizer customizeRetryPolicy(
#Value("${spring.rabbitmq.listener.simple.retry.max-attempts}") int maxAttempts) {
SimpleRetryPolicy policy = new SimpleRetryPolicy(maxAttempts, Map.of(AmqpRejectAndDontRequeueException.class, false), true, true);
return (target, retryTemplate) -> retryTemplate.setRetryPolicy(policy);
}
}
This lets the retry policy skip retries for AmqpRejectAndDontRequeueExceptions but retries all other exceptions as usual.
Configured this way, it traverses the causes of an exception, and skips retries if it finds an AmqpRejectAndDontRequeueException.
Traversing the causes is needed as org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter#invokeHandler wraps all exceptions as a ListenerExecutionFailedException

Resources