SkipListener processing events out of order - spring

I have a spring batch job which uses a SkipListener to output any errors to an errors file. There are various types of errors that can occur during a step (read, process, write) and the skip listener knows how to handle them. This is working just fine.
However, I have noticed that the errors are being written to the output out of order. Meaning it appears that errors from the processor are written before any errors from the reader.
This causes me great concern because I need the errors to be written out in the same order in which they are read (you will notice that the skip limit is insanely high and the commit-interval is 1 so it's not a matter of chunking). For all practical purposes this is a very mundane job and doesn't do anything that would lead me to believe it's something within the actual reader or processor that is causing this.
This whole thing has me baffled; it's quite odd. I'm hoping someone out there could point me in the right direction on how to resolve this. Even if the answer is "That's the way it is, and you can't change that."
Here is the relevant portions of the job definition:
<job id="devJob" xmlns="http://www.springframework.org/schema/batch">
... some steps ...
<step id="devProcessStep" next="REMOVED_FOR_SO">
<tasklet>
<chunk reader="devFileReader" processor="devItemProcessor" writer="devOutputWriter" commit-interval="1" skip-limit="999999">
<streams>
<stream ref="devInputReaderStream" />
<stream ref="devErrorOutputWriterStream" />
<stream ref="devOutputWriterStream" />
</streams>
<skippable-exception-classes>
<include class="org.springframework.batch.item.ItemReaderException"/>
... others, but removed since it doesn't matter for this question ...
<exclude class="org.springframework.batch.item.NonTransientResourceException"/>
</skippable-exception-classes>
</chunk>
<listeners>
<listener ref="devSkipListener" />
</listeners>
</tasklet>
</step>
... more steps ...
</job>
...
<bean id="devSkipListener" class="you.have.to.guess.DevSkipListener" scope="job">
<property name="writer" ref="devErrorOutputWriterStream" />
... other properties that are based on job parameters ...
</bean>
...
<bean id="devErrorOutputWriter"
class="org.springframework.batch.item.file.FlatFileItemWriter" scope="job">
... a bunch of properties based on job parameters etc.
</bean>
And here is the SkipListener, with only the relevant parts included (note I am using Groovy not Java, since this is a Grails application)
package you.have.to.guess
// bunch of imports (removed for SO)
class DevSkipListener implements SkipListener {
private static def log = LogFactory.getLog(this)
ItemWriter writer
// some other properties, not important
#Override
void onSkipInRead(Throwable e) {
log.debug("Skip error in read", e)
// some stuff here to figure out the lines
writer.write(lines)
}
#Override
void onSkipInProcess(Object item, Throwable e) {
log.debug("Skip error in process", e)
// some stuff here to figure out the lines
writer.write(lines)
}
#Override
void onSkipInWrite(Object item, Throwable e) {
log.debug("Skip error in write", e)
// some stuff here to figure out the lines
writer.write(lines)
}
}

In short, the SkipListener's contract does not guarantee the order of the items to be processed. Only that the methods on the listener will be called once per skip.

Related

How to make Execution context in Spring batch Partitioner to run in sequence

I have a requirement where first I have to select no of MasterRecords from table and then for each MasterRecords I will have to fetch no of child rows and for each child rows process and write chunk wise.
To do this I used Partitioner in spring batch and created master and slave steps to achieve this. Now code is working fine if I dont need to run slave step in same sequence it was added to Execution context.
But my requirement is to run slave step for each execution context in same sequence it was added in partitioner. Because until I process parent record I cannot process child records.
Using partitioner slave step is not running in same sequence. Please help me how to maintain same sequence for slave step run ?????
Is there any other way to achieve this using spring batch. any help is welcomed.
<job id="EPICSDBJob" xmlns="http://www.springframework.org/schema/batch">
<!-- Create Order Master Start -->
<step id="populateNewOrdersMasterStep" allow-start-if-complete="false"
next="populateLineItemMasterStep">
<partition step="populateNewOrders" partitioner="pdcReadPartitioner">
<handler grid-size="1" task-executor="taskExecutor" />
</partition>
<batch:listeners>
<batch:listener ref="partitionerStepListner" />
</batch:listeners>
</step>
<!-- Create Order Master End -->
<listeners>
<listener ref="epicsPimsJobListner" />
</listeners>
</job>
<step id="populateNewOrders" xmlns="http://www.springframework.org/schema/batch">
<tasklet allow-start-if-complete="true">
<chunk reader="epicsDBReader" processor="epicsPimsProcessor"
writer="pimsWriter" commit-interval="10">
</chunk>
</tasklet>
<batch:listeners>
<batch:listener ref="stepJobListner" />
</batch:listeners>
</step>
<bean id="epicsDBReader" class="com.cat.epics.sf.batch.reader.EPICSDBReader" scope="step" >
<property name="sfObjName" value="#{stepExecutionContext[sfParentObjNm]}" />
<property name="readChunkCount" value="10" />
<property name="readerDao" ref="readerDao" />
<property name="configDao" ref="configDao" />
<property name="dBReaderService" ref="dBReaderService" />
</bean>
Partitioner Method:
#Override
public Map<String, ExecutionContext> partition(int arg0) {
Map<String, ExecutionContext> result = new LinkedHashMap<String, ExecutionContext>();
List<String> sfMappingObjectNames = configDao.getSFMappingObjNames();
int i=1;
for(String sfMappingObjectName: sfMappingObjectNames){
ExecutionContext value = new ExecutionContext();
value.putString("sfParentObjNm", sfMappingObjectName);
result.put("partition:"+i, value);
i++;
}
return result;
}
There isn't a way to guarantee order within Spring Batch's partitioning model. The fact that the partitions are executed in parallel means that, by definition, there will be no ordering to the records processed. I think this is a case where restructuring the job a bit may help.
If your requirement is to execute the parent then execute the children, using a driving query pattern along with the partitioning would work. You'd partition along the parent records (which it looks like you're doing), then in the worker step, you'd use the parent record to drive queries and processing for the children records. That would guarantee that the child records are processed after the master one.

Spring integration chain error handling

Need help in error handling in a chain during splitter and aggregator flow for a synchronous channel.
Below is the Use case and it will be synchronous channel. So in the chain there will be a set of service activator to perform the business logic. Now if there is any exception in the service activator present in the chain, I want that to be handled in the chain itself and continue with the other splitted messages.
Inorder to do that, I have tried adding header enricher for error handler in the chain.But did not work. Any suggestion.
Object1 contains List < Object2 >
Flow:
List < Object1 > --> Splitter1 (for Object1) --> Splitter2 (for Object2) --> Chain --> Aggregator --> Aggregator
Code
<int:chain input-channel="ch3" output-channel="ch10" >
<int:header-enricher>
<int:error-channel ref="exception1" ></int:error-channel>
</int:header-enricher>
<int:service-activator ref="myService" method="method1"></int:service-activator>
<int:service-activator ref="myService" method="method2"></int:service-activator>
<int:service-activator ref="myService" method="method3"></int:service-activator>
<int:service-activator ref="myService" method="method4"></int:service-activator>
</int:chain>
<!-- Exception channel for chain and the output should go to the chain output channel -->
<int:chain input-channel="exception1" output-channel="ch10" >
<int:service-activator ref="exp" method="myException"></int:service-activator>
</int:chain>
Unfortunately it doesn't work that way. The error-channel header is for the asynchronous cases, too. It allows to override the default behavior in the MessagePublishingErrorHandler for PollableChannel and channels with Executor. In other words when we really can't do try...catch if talk in raw Java words.
So, to fix your requirements you really should rely on the try...catch function for that particular <service-activator>. It is called like ExpressionEvaluatingRequestHandlerAdvice and must be configured as on the <request-handler-advice-chain>.
In your case you should configure that Advice like:
<bean class="ExpressionEvaluatingRequestHandlerAdvice">
<property name="trapException" value="true"/>
<property name="onFailureExpression" value="#exception"/>
<property name="failureChannel" value="myErrorChannel"/>
</bean>
The trapException="true" allows do not re-throw the exception to the top level. In your case to the <splitter>.
The onFailureExpression says what to send to the failureChannel from the catch block.
The failureChannel is your desired error-channel to handle <service-activator> failures.
The source code looks like, by the way:
try {
Object result = callback.execute();
if (this.onSuccessExpression != null) {
this.evaluateSuccessExpression(message);
}
return result;
}
catch (Exception e) {
Exception actualException = this.unwrapExceptionIfNecessary(e);
if (this.onFailureExpression != null) {
Object evalResult = this.evaluateFailureExpression(message, actualException);
if (this.returnFailureExpressionResult) {
return evalResult;
}
}
if (!this.trapException) {
throw actualException;
}
return null;
}
Since we prevent re-throw with trapException="true", we end up on the return null. And having <service-activator> with null-payload loyalty we allow our <splitter> to go ahead with other messages.
HTH

how to control repeat and next step execution in spring batch

Hi I have below like xml for executing Job
<batch:job id="Job1" restartable="false" xmlns="http://www.springframework.org/schema/batch">
<step id="step1" next="step2">
<tasklet ref="automate" />
</step>
<step id="step2">
<tasklet ref="drive" />
<next on="COMPLETED" to="step3"></next>
</step>
<step id="step3">
<tasklet ref="generate_file" />
</step>
</batch:job>
For this I have write a tasklet to execute a script. Now I want that if script execution failed three times then next step will not execute . But from Tasklet I am able to return only Finished which move the flow to next step and continuable which continue the process. What should I do in this.
you can write your own decider to decide wheather to goto next step or to end the job.
if you are able to handle the failures you can also handle the flow of a job
<decision id="validationDecision" decider="validationDecider">
<next on="FAILED" to="abcStep" />
<next on="COMPLETE" to="xyzstep" />
</decision>
config is
<bean id="validationDecider" class="com.xyz.StepFlowController" />
class is
public class StepFlowController implements JobExecutionDecider{
#Override
public FlowExecutionStatus decide(JobExecution jobExecution, StepExecution stepExecution) {
FlowExecutionStatus status = null;
try {
if (failure) {
status = new FlowExecutionStatus("FAILED");
}else {
status = new FlowExecutionStatus("COMPLETE");
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return status;
}
This can be achieved by specifying a custom "chunk-completion-policy" in that step and count the number of failures. Take a look at "Stopping a Job Manually for Business Reasons" and this example for custom chunk completion policy. Hope this helps.
EDIT: you can put the number of failures in step execution context like in your step logic and then retrieve it in your completion policy class:
stepExecution.getJobExecution().getExecutionContext().put("ERROR_COUNT", noOfErrors);

Mockito Spy SystemCommandTasklet in a Spring Batch Step

I am re-factoring a troublesome test I have inherited for a Spring Batch job that calls an external script to perform a task. The original version of the test substituted the real script with a simple move file script and started the job, then called that script and tested that a file was moved correctly (which served to verify that the script was called with the correct parameters).
My goal for the new version is to eliminate the need to call a real script for this test, instead stubbing and verifying the execution method of the tasklet being used to call the script, relying on Mockito's verify to ensure that the correct parameters are used.
The configuration the job is as follows:
<flow id="job-flow">
<step id="preprocess" parent="preprocess-base" next="process">
<tasklet>
<beans:bean class="com.company.project.main.package.PreprocessTasklet">
<beans:property name="doMove" value="false" />
</beans:bean>
</tasklet>
</step>
<step id="process" next="postprocess">
<tasklet ref="commandTasklet" />
</step>
<step id="postprocess" parent="postprocess-base" />
</flow>
<bean id="commandTasklet" class="org.springframework.batch.core.step.tasklet.SystemCommandTasklet" scope="step">
<property name="command" value="${a.dir}/job_script.sh" />
<property name="environmentParams" value="working_dir=#{jobExecutionContext['job.dir']}" />
<property name="workingDirectory" value="${b.dir}" />
<property name="timeout" value="3600000"/>
</bean>
<batch:job id="run-my-script" parent="base-job" incrementer="defaultIncrementer">
<batch:flow id="script-job" parent="job-flow" />
</batch:job>
In order to prevent commandTasklet from invoking the actual shell script, I use a BeanPostProcessor to replace it with a spy and stub the execute method.
public class SpiedCommandTaskletPostProcessor implements BeanPostProcessor {
#Override
public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
if (SystemCommandTasklet.class.isAssignableFrom(bean.getClass())) {
SystemCommandTasklet spiedTasklet = Mockito.spy((SystemCommandTasklet) bean);
try {
Mockito.doReturn(RepeatStatus.FINISHED).when(spiedTasklet)
.execute(Mockito.any(StepContribution.class), Mockito.any(ChunkContext.class));
} catch (Exception e) {
e.printStackTrace();
}
bean = spiedTasklet;
}
return bean;
}
#Override
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
return bean;
}
}
Configuration for the test:
<bean class="com.company.test.package.postprocessor.SpiedCommandTaskletPostProcessor" />
This part works like a charm. However, this leaves me with two questions: firstly, by the time the PostProcessor runs, the properties (i.e. the parameters for the SystemCommandTasklet I'm trying to verify) have already been presumably set; would I even be able to use Mockito.verify(...) to check for a desired value?
Secondly, since the SystemCommandTasklet is step-scoped, I can't use autowire to get a hold of it in my test, and trying to look it up in the ApplicationContext just creates a new (and real) instance of the Tasklet outside of the step. How can I access a step scoped tasklet like this in the context of a JUnit test?
#Test
public void testLaunch() throws Exception {
JobExecution jobExecution = getJobLauncher().launchJob(jobParms);
//Mockito.verify(spiedTasklet).execute(Mockito.any(StepContribution.class), Mockito.any(ChunkContext.class)); spiedTasklet not wired to anything. :(
assertEquals(ExitStatus.COMPLETED, jobExecution.getExitStatus());
}
Is there a better approach that accomplishes testing that the script is executed with the correct parameters without actually running a script?

how can i confirm that the retry is working in spring batch impelmentation

I got spring batch inplace and i want to configure a retry logic i got the following configuration:
<step id="roundUpMonthlyJobFetchUsers">
<tasklet>
<chunk reader="roundUpMonthlyItemReader" processor="roundUpMonthlyItemProcessor"
writer="roundUpMonthlyItemWriter" commit-interval="1" retry-limit="3" >
<retryable-exception-classes>
<include class="java.io.IOException" />
</retryable-exception-classes>
</chunk>
</tasklet>
<end on="COMPLETED"/>
<fail on="FAILED"/>
</step>
How can cofirm that this its actually trying to do the operation for atleast 3 times when it encounters the IOException
Change this class roundUpMonthlyItemReader
To log something on entry and then throw the IOException everytime, then check the logs :)
log.info("Reading Monthly round up");
throw new IOException("Dummy IOException");
If you wish to do this in your unit tests, you should be able to use Mockito to
Mock the reader or writer to throw IOException
Ensure it is called 3 times

Resources