I have a spring batch JdbcCursorItemReader. It is defined as #JobScope. See method signature below.
#Bean
#JobScope
public JdbcCursorItemReader<MasterList> queryStagingDbReader(
#Value("#{jobParameters['" + JobParamConstants.PARAM_FROM_DATE + "']}") Date jobFromDate,
#Value("#{jobParameters['" + JobParamConstants.PARAM_TO_DATE + "']}") Date jobToDate) {
This JdbcCursorItemReader is part of step 2 in my job.
In step 1 of my job I have only a tasklet. Inside this tasklet I am building a list of dates which I would like my JdbcCursorItemReader in step 2 to know about.
My initial thought was to add my list of dates to the stepExecutionContext in my tasklet like so.
#Bean
#JobScope
public Tasklet createJobDatesTasklet(
#Value("#{jobParameters['" + JobParamConstants.PARAM_FROM_DATE + "']}") Date jobFromDate,
#Value("#{jobParameters['" + JobParamConstants.PARAM_TO_DATE + "']}") Date jobToDate) {
return new Tasklet() {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
LocalDate start = jobFromDate.toInstant().atZone(ZoneId.systemDefault()).toLocalDate();
LocalDate end = jobToDate.toInstant().atZone(ZoneId.systemDefault()).toLocalDate();
List<LocalDate> jobDates = new ArrayList<>();
while (!start.isAfter(end)) {
jobDates.add(start);
start = start.plusDays(1);
}
//ADDING TO CONTEXT HERE
chunkContext.getStepContext().getStepExecution().getExecutionContext().put("jobDates", jobDates);
return RepeatStatus.FINISHED;
}
};
}
And then to grab the list of dates from my JdbcCursorItemReader. But when I try to grab the step execution context inside my JdbcCursorItemReader it tells me that it cannot wire it in. And I think that is because my bean is #JobScope.
What can I do to grab my list of dates from the StepExecutionContext or alternatively can I do anything else to make this work for me?
thanks in advance
You cannot write it to the step context, rather write it to the Job Context, so that all the steps of the Job have the access to the data as below.
StepContext stepContext = chunkContext.getStepContext();
StepExecution stepExecution = stepContext.getStepExecution();
JobExecution jobExecution = stepExecution.getJobExecution();
ExecutionContext jobContext = jobExecution.getExecutionContext();
jobContext.put("jobDates", jobDates);
The below blog speaks about it
http://techie-mixture.blogspot.com/2016/07/passing-values-between-spring-batch.html
Related
I have to set the line of a record in a csv file within the record object when processing it before passing it to the writer .
What i want to achieve is to declare a global variable lets say line = 1 and each time an itemProcessor runs it should assign its value to the current item , and increment its value by 1 line++.
How can i share a variable over multiple itemProcessor runs ?
You can leverage the ExecutionContext to save the line variable of either the StepExecution if you only need the ExecutionContext in the single Step or JobExecution if you need the ExecutionContext in other Steps of the Job.
Inject either execution in your ItemProcessor:
#Value("#{stepExecution.jobExecution}") JobExecution jobExecution
or
#Value("#{stepExecution}") StepExecution stepExecution
Taking JobExecution as an example:
#StepScope
#Bean
public MyItemProcessor myItemProcessor(#Value("#{stepExecution.jobExecution}") JobExecution jobExecution) {
return new MyItemProcessor(jobExecution);
}
#Bean
public Step myStep() {
return stepBuilderFactory.get("myStep")
...
.processor(myItemProcessor(null))
...
.build()
}
public class MyItemProcessor implements ItemProcessor<MyItem, MyItem> {
private ExecutionContext executionContext;
public MyItemProcessor() {
this.executionContext = jobExecution.getExecutionContext();
}
#Override
public MyItem process(MyItem item) throws Exception {
// get the line from previous item processors, if exists, otherwise start with 0
int line = executionContext.getInt("myLineKey", 0);
item.setLine(line++);
// save the line for other item processors
item.put("myLineKey", line);
return item;
}
}
I am trying to pass a hashmap from one step to another step and use the map to create query and execute in next step. I am getting datasource must not be null while doing same.
Below is my code where I am trying to retrieve value and run query. I would have not retrieved and dynamically passed it yet. But I will be replacing this query dynamically.
#Autowired
DataSource dataSource;
#Override
public void afterPropertiesSet() throws Exception{
JobExecution jobExecution = stepExecution.getJobExecution();
ExecutionContext jobContext = jobExecution.getExecutionContext();
#SuppressWarnings("unchecked")
List<HashMap<String,String>> mapList = (List<HashMap<String, String>>) jobContext.get("mapList");
System.out.println("size of map received:::::::"+ mapList.size());
setSql("select count(*) as countValue from table where id=578");
setRowMapper(new dbMapper());
setDataSource(dataSource);
super.afterPropertiesSet();
}
#BeforeStep
public void saveStepExecution(final StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
Where am I going wrong?
This should probably be a comment, but I don't have enough reputation to add one yet. Does the class that this sample is from already have a setter for dataSource? If so, you need to change setDataSource(dataSource); to super.setDataSource(dataSource);.
I am reading from Oracle and writing to a CSV file. I have one step which reads and writes to the CSV file. I implemented a ChunkListener so I know how many records were written.
I want to be able to write a file trailer showing the number of records written to my file. I implemented FlatFileFooterCallback but cannot figure out how to get the data from StepExecution (the "readCount") to my FlatFileFooterCallback.
I guess I am struggling with how to get access to Job, Step scopes in my write.
Any examples, or links would be helpful. I am using [Spring Batch / Boot] so I am all annotated. I can find xml examples, so maybe this annotated stuff is more complicated.
ItemWriter<Object> databaseCsvItemWriter() {
FlatFileItemWriter<Object> csvFileWriter = new FlatFileItemWriter<>();
String exportFileHeader = "one,two,three";
StringHeaderWriter headerWriter = new StringHeaderWriter(exportFileHeader);
csvFileWriter.setHeaderCallback(headerWriter);
String exportFilePath = "/tmp/students.csv";
csvFileWriter.setResource(new FileSystemResource(exportFilePath));
LineAggregator<McsendRequest> lineAggregator = createRequestLineAggregator();
csvFileWriter.setLineAggregator(lineAggregator);
csvFileWriter.setFooterCallback(headerWriter);
return csvFileWriter;
}
You can implement CustomFooterCallback as follows:
public class CustomFooterCallback implements FlatFileFooterCallback {
#Value("#{StepExecution}")
private StepExecution stepExecution;
#Override
public void writeFooter(Writer writer) throws IOException {
writer.write("footer - number of items read: " + stepExecution.getReadCount());
writer.write("footer - number of items written: " + stepExecution.getWriteCount());
}
}
Then in a #Configuration class:
#Bean
#StepScope
public FlatFileFooterCallback customFooterCallback() {
return new CustomFooterCallback();
}
And use in the Writer:
csvFileWriter.setFooterCallback(customFooterCallback());
This way, you have access to StepExecution in order to read data as needed.
I created a simple Boot/Spring Batch 3.0.8.RELEASE job. I created a simple class that implements JobParametersIncrementer to go to the database, look up how many days the query should look for and puts those into the JobParameters object.
I need that value in my JdbcCursorItemReader, as it is selecting data based upon one of the looked up JobParameters, but I cannot figure this out via Java annotations. XML examples plenty, not so much for Java.
Below is my BatchConfiguration class that runs job.
`
#Autowired
SendJobParms jobParms; // this guy queries DB and puts data into JobParameters
#Bean
public Job job(#Qualifier("step1") Step step1, #Qualifier("step2") Step step2) {
return jobs.get("DW_Send").incrementer(jobParms).start(step1).next(step2).build();
}
#Bean
protected Step step2(ItemReader<McsendRequest> reader,
ItemWriter<McsendRequest> writer) {
return steps.get("step2")
.<McsendRequest, McsendRequest> chunk(5000)
.reader(reader)
.writer(writer)
.build();
}
#Bean
public JdbcCursorItemReader reader() {
JdbcCursorItemReader<McsendRequest> itemReader = new JdbcCursorItemReader<McsendRequest>();
itemReader.setDataSource(dataSource);
// want to get access to JobParameter here so I can pull values out for my sql query.
itemReader.setSql("select xxxx where rownum <= JobParameter.getCount()");
itemReader.setRowMapper(new McsendRequestMapper());
return itemReader;
}
`
Change reader definition as follows (example for parameter of type Long and name paramCount):
#Bean
#StepScope
public JdbcCursorItemReader reader(#Value("#{jobParameters[paramCount]}") Long paramCount) {
JdbcCursorItemReader<McsendRequest> itemReader = new JdbcCursorItemReader<McsendRequest>();
itemReader.setDataSource(dataSource);
itemReader.setSql("select xxxx where rownum <= ?");
ListPreparedStatementSetter listPreparedStatementSetter = new ListPreparedStatementSetter();
listPreparedStatementSetter.setParameters(Arrays.asList(paramCount));
itemReader.setPreparedStatementSetter(listPreparedStatementSetter);
itemReader.setRowMapper(new McsendRequestMapper());
return itemReader;
}
I'm having trouble getting a conditional spring batch flow to work using java config. The samples I've seen in spring batch samples, or spring batch's test code, or on stack overflow tend to show a conditional where a single step needs to be executed on condition, or it's the final step, or both. That's not the case I need to solve.
In procedural pseudo code, I want it to behave like
initStep()
if decision1()
subflow1()
middleStep()
if decision2()
subflow2()
lastStep()
So, subflow1 and 2 are conditional, but init, middle and last always execute. Here's my stripped down test case. In the current configuration, it just quits after executing subflow1.
public class FlowJobTest {
private JobBuilderFactory jobBuilderFactory;
private JobRepository jobRepository;
private JobExecution execution;
#BeforeMethod
public void setUp() throws Exception {
jobRepository = new MapJobRepositoryFactoryBean().getObject();
jobBuilderFactory = new JobBuilderFactory(jobRepository);
execution = jobRepository.createJobExecution("flow", new JobParameters());
}
#Test
public void figureOutFlowJobs() throws Exception {
JobExecutionDecider subflow1Decider = decider(true);
JobExecutionDecider subflow2Decider = decider(false);
Flow subflow1 = new FlowBuilder<Flow>("subflow-1").start(echo("subflow-1-Step-1")).next(echo("subflow-1-Step-2")).end();
Flow subflow2 = new FlowBuilder<Flow>("subflow-2").start(echo("subflow-2-Step-1")).next(echo("subflow-2-Step-2")).end();
Job job = jobBuilderFactory.get("testJob")
.start(echo("init"))
.next(subflow1Decider)
.on("YES").to(subflow1)
.from(subflow1Decider)
.on("*").to(echo("middle"))
.next(subflow2Decider)
.on("YES").to(subflow2)
.from(subflow2Decider)
.on("*").to(echo("last"))
.next(echo("last"))
.build().preventRestart().build();
job.execute(execution);
assertEquals(execution.getStatus(), BatchStatus.COMPLETED);
assertEquals(execution.getStepExecutions().size(), 5);
}
private Step echo(String stepName) {
return new AbstractStep() {
{
setName(stepName);
setJobRepository(jobRepository);
}
#Override
protected void doExecute(StepExecution stepExecution) throws Exception {
System.out.println("step: " + stepName);
stepExecution.upgradeStatus(BatchStatus.COMPLETED);
stepExecution.setExitStatus(ExitStatus.COMPLETED);
jobRepository.update(stepExecution);
}
};
}
private JobExecutionDecider decider(boolean decision) {
return (jobExecution, stepExecution) -> new FlowExecutionStatus(decision ? "YES" : "NO");
}
}
Your original job definition should work, as well, with only a small tweak. The test was failing because the job finished (with status COMPLETED) after the first sub flow. If you instruct it to continue to the middle step instead, it should work as intended. Similar adjustment for the second flow.
Job job = jobBuilderFactory.get("testJob")
.start(echo("init"))
.next(subflow1Decider)
.on("YES").to(subflow1).next(echo("middle"))
.from(subflow1Decider)
.on("*").to(echo("middle"))
.next(subflow2Decider)
.on("YES").to(subflow2).next(echo("last"))
.from(subflow2Decider)
.on("*").to(echo("last"))
.build().preventRestart().build();
The approach I used to make this work was to break my conditional flows into flow steps.
public void figureOutFlowJobsWithFlowStep(boolean decider1, boolean decider2, int expectedSteps) throws Exception {
JobExecutionDecider subflow1Decider = decider(decider1);
JobExecutionDecider subflow2Decider = decider(decider2);
Flow subFlow1 = new FlowBuilder<Flow>("sub-1")
.start(subflow1Decider)
.on("YES")
.to(echo("sub-1-1")).next(echo("sub-1-2"))
.from(subflow1Decider)
.on("*").end()
.end();
Flow subFlow2 = new FlowBuilder<Flow>("sub-2")
.start(subflow2Decider)
.on("YES").to(echo("sub-2-1")).next(echo("sub-2-2"))
.from(subflow2Decider)
.on("*").end()
.end();
Step subFlowStep1 = new StepBuilder("sub1step").flow(subFlow1).repository(jobRepository).build();
Step subFlowStep2 = new StepBuilder("sub2step").flow(subFlow2).repository(jobRepository).build();
Job job = jobBuilderFactory.get("testJob")
.start(echo("init"))
.next(subFlowStep1)
.next(echo("middle"))
.next(subFlowStep2)
.next(echo("last"))
.preventRestart().build();
job.execute(execution);
assertEquals(execution.getStatus(), BatchStatus.COMPLETED);
assertEquals(execution.getStepExecutions().size(), expectedSteps);
}