Spring Batch - RepositoryItemReader read param from ExecutionContext - spring

I have Spring Batch job where I am passing some values between two stpes. I set the value in Job Context in Step1 and now trying to read from RepositoryItemReader in Step2. There is #BeforeStep method where I am able to read value set in context. But I am setting up my repository along with method name and args in #PostConstruct annotated method which is executed before #BeforeStep annotated method.
What is the best way to read param in ReposiotryItem from JobExecution Context?
#Component
#JobScope
public class MyItemReader extends RepositoryItemReader<Scan> {
#Autowired
private MyRepository repository;
private Integer lastIdPulled = null;
public MyItemReader() {
super();
}
#BeforeStep
public void initializeValues(StepExecution stepExecution) {
Integer value = stepExecution.getJobExecution().getExecutionContext().getInt("lastIdPulled");
System.out.println(">>>>>>>> last_pulled_id = " + value);
}
#PostConstruct
protected void init() {
final Map<String, Sort.Direction> sorts = new HashMap<>();
sorts.put("id", Direction.ASC);
this.setRepository(this.repository);
this.setSort(sorts);
this.setMethodName("findByGreaterThanId"); // You should sepcify the method which
//spring batch should call in your repository to fetch
// data and the arguments it needs needs to be
//specified with the below method.
List<Object> methodArgs = new ArrayList<Object>();
if(lastIdPulled== null || lastIdPulled<=0 ){
lastScanIdPulled = 0;
}
methodArgs.add(lastIdPulled);
this.setArguments(methodArgs);
}
}

Your reader needs to be #StepScoped instead of #JobScoped. Even though you're accessing the job context, the value is not available in the context until the previous step finishes. If you #StepScope your reader then it won't initialize until the step it is part of starts up and the value is available.
Another option is to construct the reader as a #Bean definition in a #Configuration file but the idea is the same. This uses SpEL for late binding.
#Configuration
public class JobConfig {
// Your item reader will get autowired into this method
// so you don't have to call it
#Bean
public Step myStep(MyItemReader myItemReader) {
//build your step
}
#Bean
#StepScope
public MyItemReader myItemReader(#Value("#{jobExecutionContext[partitionKey]}") Integer lastIdPulled) {
MyItemReader reader = new MyItemReader();
// Perform #PostConstruct tasks
return reader;
}
}

I was able to figure out how to solve your problem, and mine, without having to create a #Bean definition in #Configuration file and without using #PostConstruct.
Instead of using #PostConstruct, just set them in the constructor of the class and in your #BeforeStep set the arguments as shown below:
#Component
#StepScope
public class MyItemReader extends RepositoryItemReader<Scan> {
#Autowired
public MyItemReader(MyRepository myRepository) {
super();
this.setRepository(MyRepository myRepository);
this.setMethodName("findByGreaterThanId");
final Map<String, Sort.Direction> sorts = new HashMap<>();
sorts.put("id", Direction.ASC);
this.setSort(sorts);
}
#BeforeStep
public void initializeValues(StepExecution stepExecution) {
Integer value = stepExecution.getJobExecution().getExecutionContext().getInt("lastIdPulled");
System.out.println(">>>>>>>> last_pulled_id = " + value);
List<Object> methodArgs = new ArrayList<Object>();
if(lastIdPulled== null || lastIdPulled<=0 ){
lastScanIdPulled = 0;
}
methodArgs.add(lastIdPulled);
this.setArguments(methodArgs);
}
}

Related

Access Job Parameter in Custom ItemProcessor

I am implementing a custom ItemProcessor<I, O> in spring batch for processing data from a Rest api .
I want access some values from jobParameter inside my ItemProcessor class .
Any suggestion on how to do that ?
In Tasklet we can access JobParameter but not sure how to do in ItemProcessor .
MyItemProcessor.java
#Component
public class MyItemProcessor implements ItemProcessor<User, UserDetails> {
#Override
public UserDetails process(User user) throws Exception {
// access values from job parameter here
return null;
}
}
You can make your item processor step-scoped and inject job parameters in it. The following is one way of doing that:
#Component
#StepScope
public class MyItemProcessor implements ItemProcessor<User, UserDetails> {
#Value("#{jobParameters}")
private JobParameters jobParameters;
#Override
public UserDetails process(User user) throws Exception {
// access values from job parameter here
return null;
}
}
You could also inject a specific parameter if you want with something like the following:
#Component
#StepScope
public class MyItemProcessor implements ItemProcessor<User, UserDetails> {
#Value("#{jobParameters['myParameter']}")
private String myParameter;
#Override
public UserDetails process(User user) throws Exception {
// use myParameter as needed here
return null;
}
}
Since field injection is not recommended, you can inject job parameters in your item processor when you define it as a bean, something like:
// Note how nothing related to Spring is used here, and the processor can be unit tested as a regular Java class
public class MyItemProcessor implements ItemProcessor<User, UserDetails> {
private String myParameter;
public MyItemProcessor(String myParameter) {
this.myParameter = myParameter;
}
#Override
public UserDetails process(User user) throws Exception {
// use this.myParameter as needed here
return null;
}
}
Once that in place, you can declare your item processor bean as follows:
#Bean
#StepScope
public MyItemProcessor itemProcessor(#Value("#{jobParameters['myParameter']}") String myParameter) {
return new MyItemProcessor(myParameter);
}
Fore more details about scoped beans, please check the documentation here: Late Binding of Job and Step attributes.

How to pass List from StepExecutionListener to MultiResourceItemReader

During the execution of StepExecutionListener.beforeStep() I am initiating a List of resources with data from the database.
#Component
public class DailyExportStepExecutionListener implements StepExecutionListener {
#Autowired
private JdbcTemplate jdbcTemplate;
#Autowired
private ResourceLoader resourceLoader;
private List<Resource> listResource;
public DailyExportStepExecutionListener() {
listResource = new ArrayList<Resource>();
}
public List<Resource> getListResource() {
return listResource;
}
#Override
public void beforeStep(StepExecution stepExecution) {
jdbcTemplate.query("SELECT FullPath FROM DailyExportMetadata",
(rs, row) -> listResource.add(resourceLoader.getResource(rs.getString(1))));
}
Eventually what I would like to do is to use the list of resources for MultiResourceItemReader
#Bean
public MultiResourceItemReader<DailyExport> multiResourceItemReader(FieldSetMapper<DailyExport> testClassRowMapper) {
MultiResourceItemReader<DailyExport> multiResourceItemReader = new MultiResourceItemReader<>();
multiResourceItemReader.setName("dailyExportMultiReader");
multiResourceItemReader.setDelegate(reader(testClassRowMapper));
multiResourceItemReader.setStrict(true);
multiResourceItemReader.setResources(??);
return multiResourceItemReader;
}
How can I pass the ArrayList between the components ?
Thank you
The purpose of the method named multiResourceItemReader is to build the item reader which will be used in the job so it executed before the job starts and thus before the beforeStepis executed.
If you can, you should execute your sql to populate the listResource at configuration time : for example in a method annoted #Bean method :
#Bean
public List<Resource> listResource(JdbcTemplate jdbcTemplate, ResourceLoader resourceLoader) {
List<Resource> listResource;
jdbcTemplate.query("SELECT FullPath FROM DailyExportMetadata",
(rs, row) -> listResource.add(resourceLoader.getResource(rs.getString(1))));
return listResource;
}
And the multiResourceItemReader method would be like :
#Bean
public MultiResourceItemReader<DailyExport> multiResourceItemReader(FieldSetMapper<DailyExport> testClassRowMapper, List<Resource> listResource) {
MultiResourceItemReader<DailyExport> multiResourceItemReader = new MultiResourceItemReader<>();
multiResourceItemReader.setName("dailyExportMultiReader");
multiResourceItemReader.setDelegate(reader(testClassRowMapper));
multiResourceItemReader.setStrict(true);
multiResourceItemReader.setResources(listResource);
return multiResourceItemReader;
}
If you cannot initialize the listResource before starting the job, you should do as described in this post (Set IteamReader input from ExecutionContext). The beforeStep method should put in step execution context the listResource like this :
stepExecution.getExecutionContext().put("listResource", listResource);
"listResource" is the key to use to get the list in the #Value statement in the multiResourceItemReader method

#MOCK in Junit 5 returns NULL when using #Value in constructor

I am writing test cases for a project which uses File inpurstream at multiple places. Due to the issues with PowerMock and Junit5, I ended up using a Constructor with #Value to pass file paths to the class. Sample code will show.
Now When I use the constructor with #Value. The test cases for File inputStream work perfectly. But #MOck for any other class returns NULL.
I comment the constructor and #Mock starts working.
any One faced this before?
Class to be tested:
#Component
public class test {
TestUtils testUtil;
String nodeConfigFilePath;
String genConfigFilePath;
public test (#Value("${node.config.path}") String nodeConfigFilePath,
#Value("${genConfig.path}", #Autowired TestUtils testUtil) String genConfigFilePath) {
this.nodeConfigFilePath=nodeConfigFilePath;
this.genConfigFilePath= genConfigFilePath;
this.testUtil = testUtil;
}
public boolean filterConfig(JSONObject Json) {
if(testUtil.iterateJsonForPatternMatch(Json))
return true;
else
return false;
}
public void loadConfigFromJson() {
ObjectMapper mapper = new ObjectMapper();
TypeReference<Map<String, String>> typeRef = new TypeReference<Map<String, String>>() {
};
try {
InputStream inputStream = new FileInputStream(nodeConfigFilePath);
Map<String, String> nodeConfig = new HashMap<String, String>();
nodeConfig = mapper.readValue(inputStream,typeRef);
<do something>
}
}
catch(Exception e) {
}
}
}
Test Class:
#ExtendWith(MockitoExtension.class)
public class test {
#InjectMocks
test test;
#Mock
TestUtils testUtil;
#Test
public void testFilterConfig_success() throws Exception{
JSONObject jsonObj = new JSONObject();
when(testUtil.iterateJsonForPatternMatch(jsonObj)).thenReturn(true);
assertEquals(true, test.filterConfig(jsonObj));
}
#Test
public void testloadConfigFromJson_success() throws Exception{
test = new NotificationFilter(nodeConfigFile.getAbsolutePath(),genConfigJsonFile.getAbsolutePath(), testUtil);
test.loadConfigFromJson();
}
}
FYI: I changed the method names and created a dummy class for demo purpose.
So If Now I remove the #Value constructor, It starts working and no Null pointer for testUtil. But testloadConfigFromJson_success() starts to fail. And Vice versa. Can anyone explain why this is the case?
UPDATE: moved autoiring to constructor and passed the TestUtil instance via constructor in Test class. It seems to do the trick. I am not sure if its correct

preprocessing in spring boot batch with multiple threads

I have a Spring batch with multi threads. In my processor I want to use global variables say a map. The map contains some values which is to be queried from a table and is to be used by the processor. How can I achieve this? If i write the logic to set the map in the processor, the query will be executed for every record fetched by the item reader, which would be millions in numbers. Is there a way to do this?
You can intercept step execution
Spring Batch - Reference Documentation section 5.1.10. Intercepting Step Execution
For example, you can implement the StepExecutionListener interface
#Component
#JobScope
public class Processor implements ItemProcessor<Integer,Integer>, StepExecutionListener {
private final Map<String,String> map = new HashMap<>();
#Override
public void beforeStep(StepExecution stepExecution) {
// initialize a variable once before step
map.put("KEY","VALUE");
}
#Override
public Integer process(Integer item) throws Exception {
// use a variable for each iteration
final String key = map.get("KEY");
// ...
}
// ....
}
or use the #BeforeStep annotation
#Component
#JobScope
public class Processor implements ItemProcessor<Integer,Integer>{
private final Map<String,String> map = new HashMap<>();
#BeforeStep
public void beforeStep(StepExecution stepExecution) {
// initialize a variable once before step
map.put("KEY","VALUE");
}
#Override
public Integer process(Integer item) throws Exception {
// use a variable for each iteration
final String key = map.get("KEY");
//...
}
}

Spring Batch -#BeforeStep not getting invoked in Partitioner

We are trying to implement a batch job using spring batch partitioning.In this in "step 2" is a partitioned step where I need some data from step 1 for processing.I used StepExecutionContext which will be promoted to job Execution Context at step1 to store this data.
I tried to use #BeforeStep annotation in partitioner class to get the stepExecutionContext
from which I can extract the data stored previously and put it in ExecutionContext of the partitioner .But the method with #BeforeStep annotation is not getting invoked in the partitioner.
Is there any other way to achieve this.
Partitioner Implementation
public class NtfnPartitioner implements Partitioner {
private int index = 0;
String prev_job_time = null;
String curr_job_time = null;
private StepExecution stepExecution ;
ExecutionContext executionContext ;
#Override
public Map<String, ExecutionContext> partition(int gridSize)
{
System.out.println("Entered Partitioner");
List<Integer> referencIds = new ArrayList<Integer>();
for (int i = 0; i < gridSize;i++) {
referencIds.add(index++);
}
Map<String, ExecutionContext> results = new LinkedHashMap<String,ExecutionContext>();
for (int referencId : referencIds) {
ExecutionContext context = new ExecutionContext();
context.put("referenceId", referencId);
context.put(NtfnConstants.PREVIOUS_JOB_TIME, prev_job_time);
context.put(NtfnConstants.JOB_START_TIME, curr_job_time);
results.put("partition." + referencId, context);
}
return results;
}
#BeforeStep
public void beforeStep(StepExecution stepExecution) {
// TODO Auto-generated method stub
System.out.println("Entered Before step in partion");
JobExecution jobExecution = stepExecution.getJobExecution();
ExecutionContext jobContext = jobExecution.getExecutionContext();
System.out.println("ExecutionContext"+jobContext);
String prev_job_time = (String) jobContext.get(NtfnConstants.PREVIOUS_JOB_TIME);
String curr_job_time = (String) jobContext.get(NtfnConstants.JOB_START_TIME);
}
The bean should be step scoped.
Java, annotate class:
#StepScope
XML, in bean definition:
scope="step"
Also look at this answer regarding proxied bean (not sure if this applies to you since no other code than the partitioner was provided). In this case you can still add your partitioner as a listener explicitly during step building:
#Autowired
private NtfnPartitioner partitioner;
...
final Step masterStep = stepBuilderFactory.get("master")
.listener(partitioner)
.partitioner("slave", partitioner)
.step(slave)
...
Or if your partitioner is not bean (e.g. you are creating it based on something dynamic), you can still add it as a listener:
final NtfnPartitioner partitioner = new NtfnPartitioner();
final Step masterStep = stepBuilderFactory.get("master")
.listener(partitioner)
.partitioner("slave", partitioner)
.step(slave)
...
To get the handle of Job Parameters you can implement StepExecutionListener to your NtfnPartitioner Class to make use of Overridden methods beforeStep and afterStep. Please make sure that it should be of StepScoped.
public class NtfnPartitioner implements Partitioner, StepExecutionListener {
String prev_job_time = null;
String curr_job_time = null;
....
#Override
public Map<String, ExecutionContext> partition(int gridSize)
{
....
/* Please use prev_job_time and curr_job_time in this
method which was fetched from context */
....
}
#BeforeStep
public void beforeStep(StepExecution stepExecution) {
System.out.println("Entered Before step in partion");
ExecutionContext jobContext = stepExecution.getJobExecution().getExecutionContext();
System.out.println("ExecutionContext"+jobContext);
String prev_job_time = (String) jobContext.get(NtfnConstants.PREVIOUS_JOB_TIME);
String curr_job_time = (String) jobContext.get(NtfnConstants.JOB_START_TIME);
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
if (stepExecution.getStatus() == BatchStatus.COMPLETED) {
return ExitStatus.COMPLETED;
}
return ExitStatus.FAILED;
}
}

Resources