I got spring batch inplace and i want to configure a retry logic i got the following configuration:
<step id="roundUpMonthlyJobFetchUsers">
<tasklet>
<chunk reader="roundUpMonthlyItemReader" processor="roundUpMonthlyItemProcessor"
writer="roundUpMonthlyItemWriter" commit-interval="1" retry-limit="3" >
<retryable-exception-classes>
<include class="java.io.IOException" />
</retryable-exception-classes>
</chunk>
</tasklet>
<end on="COMPLETED"/>
<fail on="FAILED"/>
</step>
How can cofirm that this its actually trying to do the operation for atleast 3 times when it encounters the IOException
Change this class roundUpMonthlyItemReader
To log something on entry and then throw the IOException everytime, then check the logs :)
log.info("Reading Monthly round up");
throw new IOException("Dummy IOException");
If you wish to do this in your unit tests, you should be able to use Mockito to
Mock the reader or writer to throw IOException
Ensure it is called 3 times
Related
I am defining my MultiResourceItemReader on this way:
<bean id="multiDataItemReader" class="org.springframework.batch.item.file.MultiResourceItemReader" scope="step">
<property name="resources" value="#{jobExecutionContext['filesResource']}"/>
<property name="delegate" ref="dataItemReader"/>
</bean>
How you can see I want read from the jobExecutionContext the "filesResource" value.
Note: I changed some names to keep the "code privacy". This is executing, Is somebody wants more info please tell me.
I am saving this value in my first step and I am using the reader in the second step, Should I have access to it?
I am saving it in the final lines from my step1 tasklet:
ExecutionContext jobContext = context.getStepContext().getStepExecution().getJobExecution().getExecutionContext();
jobContext.put("filesResource", resourceString);
<batch:job id="myJob">
<batch:step id="step1" next="step2">
<batch:tasklet ref="moveFilesFromTasklet" />
</batch:step>
<batch:step id="step2">
<tasklet>
<chunk commit-interval="500"
reader="multiDataItemReader"
processor="dataItemProcessor"
writer="dataItemWriter" />
</tasklet>
</batch:step>
</batch:job>
I am not really sure what I am forgetting to get the value. The error that I am getting is:
20190714 19:49:08.120 WARN org.springframework.batch.item.file.MultiResourceItemReader [[ # ]] - No resources to read. Set strict=true if this should be an error condition.
I see nothing wrong with your config. The value of resourceString should be an array of org.springframework.core.io.Resource as this is the parameter type of the resources attribute of MultiResourceItemReader.
You can pass an array or a list of String with the absolute path to each resource and it should work. Here is a quick example:
class MyTasklet implements Tasklet {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) {
List<String> resources = Arrays.asList(
"/full/path/to/resource1",
"/full/path/to/resource2");
chunkContext.getStepContext().getStepExecution().getJobExecution().getExecutionContext()
.put("filesResource", resources);
return RepeatStatus.FINISHED;
}
}
I have a requirement where first I have to select no of MasterRecords from table and then for each MasterRecords I will have to fetch no of child rows and for each child rows process and write chunk wise.
To do this I used Partitioner in spring batch and created master and slave steps to achieve this. Now code is working fine if I dont need to run slave step in same sequence it was added to Execution context.
But my requirement is to run slave step for each execution context in same sequence it was added in partitioner. Because until I process parent record I cannot process child records.
Using partitioner slave step is not running in same sequence. Please help me how to maintain same sequence for slave step run ?????
Is there any other way to achieve this using spring batch. any help is welcomed.
<job id="EPICSDBJob" xmlns="http://www.springframework.org/schema/batch">
<!-- Create Order Master Start -->
<step id="populateNewOrdersMasterStep" allow-start-if-complete="false"
next="populateLineItemMasterStep">
<partition step="populateNewOrders" partitioner="pdcReadPartitioner">
<handler grid-size="1" task-executor="taskExecutor" />
</partition>
<batch:listeners>
<batch:listener ref="partitionerStepListner" />
</batch:listeners>
</step>
<!-- Create Order Master End -->
<listeners>
<listener ref="epicsPimsJobListner" />
</listeners>
</job>
<step id="populateNewOrders" xmlns="http://www.springframework.org/schema/batch">
<tasklet allow-start-if-complete="true">
<chunk reader="epicsDBReader" processor="epicsPimsProcessor"
writer="pimsWriter" commit-interval="10">
</chunk>
</tasklet>
<batch:listeners>
<batch:listener ref="stepJobListner" />
</batch:listeners>
</step>
<bean id="epicsDBReader" class="com.cat.epics.sf.batch.reader.EPICSDBReader" scope="step" >
<property name="sfObjName" value="#{stepExecutionContext[sfParentObjNm]}" />
<property name="readChunkCount" value="10" />
<property name="readerDao" ref="readerDao" />
<property name="configDao" ref="configDao" />
<property name="dBReaderService" ref="dBReaderService" />
</bean>
Partitioner Method:
#Override
public Map<String, ExecutionContext> partition(int arg0) {
Map<String, ExecutionContext> result = new LinkedHashMap<String, ExecutionContext>();
List<String> sfMappingObjectNames = configDao.getSFMappingObjNames();
int i=1;
for(String sfMappingObjectName: sfMappingObjectNames){
ExecutionContext value = new ExecutionContext();
value.putString("sfParentObjNm", sfMappingObjectName);
result.put("partition:"+i, value);
i++;
}
return result;
}
There isn't a way to guarantee order within Spring Batch's partitioning model. The fact that the partitions are executed in parallel means that, by definition, there will be no ordering to the records processed. I think this is a case where restructuring the job a bit may help.
If your requirement is to execute the parent then execute the children, using a driving query pattern along with the partitioning would work. You'd partition along the parent records (which it looks like you're doing), then in the worker step, you'd use the parent record to drive queries and processing for the children records. That would guarantee that the child records are processed after the master one.
Is there any way in Spring Batch to stop the batch from proceeding further if validation count exceeds pre-defined count.
I am currently throwing the validation exception from item processor . But it just skips the item from sending to item writer not stopping the batch .
Thanks
If you are using the fluent api (StepBuilder) you can set it as follows:
#Bean
public Step step(){
return stepBuilderFactory.get("step")
.<... , ...>chunk(1)
.reader(reader())
.processor(processor())
.writer(writer())
.faultTolerant()
.skipLimit(10)
.build();
}
This will instantiate as LimitCheckingItemSkipPoliciy.
You can also instantiate or define your own SkipPolicy utilizing the skipPolicy method
...
.faultTolerant()
.skipPolicy(mySkipPolicy)
...
If you want to set the skiplimit in an XML configuration:
<step id="step1">
<tasklet>
<chunk reader="flatFileItemReader" writer="itemWriter"
commit-interval="10"
skip-limit="10">
<skippable-exception-classes>
<include class="..."/>
</skippable-exception-classes>
</chunk>
</tasklet>
</step>
Hi I have below like xml for executing Job
<batch:job id="Job1" restartable="false" xmlns="http://www.springframework.org/schema/batch">
<step id="step1" next="step2">
<tasklet ref="automate" />
</step>
<step id="step2">
<tasklet ref="drive" />
<next on="COMPLETED" to="step3"></next>
</step>
<step id="step3">
<tasklet ref="generate_file" />
</step>
</batch:job>
For this I have write a tasklet to execute a script. Now I want that if script execution failed three times then next step will not execute . But from Tasklet I am able to return only Finished which move the flow to next step and continuable which continue the process. What should I do in this.
you can write your own decider to decide wheather to goto next step or to end the job.
if you are able to handle the failures you can also handle the flow of a job
<decision id="validationDecision" decider="validationDecider">
<next on="FAILED" to="abcStep" />
<next on="COMPLETE" to="xyzstep" />
</decision>
config is
<bean id="validationDecider" class="com.xyz.StepFlowController" />
class is
public class StepFlowController implements JobExecutionDecider{
#Override
public FlowExecutionStatus decide(JobExecution jobExecution, StepExecution stepExecution) {
FlowExecutionStatus status = null;
try {
if (failure) {
status = new FlowExecutionStatus("FAILED");
}else {
status = new FlowExecutionStatus("COMPLETE");
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return status;
}
This can be achieved by specifying a custom "chunk-completion-policy" in that step and count the number of failures. Take a look at "Stopping a Job Manually for Business Reasons" and this example for custom chunk completion policy. Hope this helps.
EDIT: you can put the number of failures in step execution context like in your step logic and then retrieve it in your completion policy class:
stepExecution.getJobExecution().getExecutionContext().put("ERROR_COUNT", noOfErrors);
I have a spring batch job which uses a SkipListener to output any errors to an errors file. There are various types of errors that can occur during a step (read, process, write) and the skip listener knows how to handle them. This is working just fine.
However, I have noticed that the errors are being written to the output out of order. Meaning it appears that errors from the processor are written before any errors from the reader.
This causes me great concern because I need the errors to be written out in the same order in which they are read (you will notice that the skip limit is insanely high and the commit-interval is 1 so it's not a matter of chunking). For all practical purposes this is a very mundane job and doesn't do anything that would lead me to believe it's something within the actual reader or processor that is causing this.
This whole thing has me baffled; it's quite odd. I'm hoping someone out there could point me in the right direction on how to resolve this. Even if the answer is "That's the way it is, and you can't change that."
Here is the relevant portions of the job definition:
<job id="devJob" xmlns="http://www.springframework.org/schema/batch">
... some steps ...
<step id="devProcessStep" next="REMOVED_FOR_SO">
<tasklet>
<chunk reader="devFileReader" processor="devItemProcessor" writer="devOutputWriter" commit-interval="1" skip-limit="999999">
<streams>
<stream ref="devInputReaderStream" />
<stream ref="devErrorOutputWriterStream" />
<stream ref="devOutputWriterStream" />
</streams>
<skippable-exception-classes>
<include class="org.springframework.batch.item.ItemReaderException"/>
... others, but removed since it doesn't matter for this question ...
<exclude class="org.springframework.batch.item.NonTransientResourceException"/>
</skippable-exception-classes>
</chunk>
<listeners>
<listener ref="devSkipListener" />
</listeners>
</tasklet>
</step>
... more steps ...
</job>
...
<bean id="devSkipListener" class="you.have.to.guess.DevSkipListener" scope="job">
<property name="writer" ref="devErrorOutputWriterStream" />
... other properties that are based on job parameters ...
</bean>
...
<bean id="devErrorOutputWriter"
class="org.springframework.batch.item.file.FlatFileItemWriter" scope="job">
... a bunch of properties based on job parameters etc.
</bean>
And here is the SkipListener, with only the relevant parts included (note I am using Groovy not Java, since this is a Grails application)
package you.have.to.guess
// bunch of imports (removed for SO)
class DevSkipListener implements SkipListener {
private static def log = LogFactory.getLog(this)
ItemWriter writer
// some other properties, not important
#Override
void onSkipInRead(Throwable e) {
log.debug("Skip error in read", e)
// some stuff here to figure out the lines
writer.write(lines)
}
#Override
void onSkipInProcess(Object item, Throwable e) {
log.debug("Skip error in process", e)
// some stuff here to figure out the lines
writer.write(lines)
}
#Override
void onSkipInWrite(Object item, Throwable e) {
log.debug("Skip error in write", e)
// some stuff here to figure out the lines
writer.write(lines)
}
}
In short, the SkipListener's contract does not guarantee the order of the items to be processed. Only that the methods on the listener will be called once per skip.