I am reading from Oracle and writing to a CSV file. I have one step which reads and writes to the CSV file. I implemented a ChunkListener so I know how many records were written.
I want to be able to write a file trailer showing the number of records written to my file. I implemented FlatFileFooterCallback but cannot figure out how to get the data from StepExecution (the "readCount") to my FlatFileFooterCallback.
I guess I am struggling with how to get access to Job, Step scopes in my write.
Any examples, or links would be helpful. I am using [Spring Batch / Boot] so I am all annotated. I can find xml examples, so maybe this annotated stuff is more complicated.
ItemWriter<Object> databaseCsvItemWriter() {
FlatFileItemWriter<Object> csvFileWriter = new FlatFileItemWriter<>();
String exportFileHeader = "one,two,three";
StringHeaderWriter headerWriter = new StringHeaderWriter(exportFileHeader);
csvFileWriter.setHeaderCallback(headerWriter);
String exportFilePath = "/tmp/students.csv";
csvFileWriter.setResource(new FileSystemResource(exportFilePath));
LineAggregator<McsendRequest> lineAggregator = createRequestLineAggregator();
csvFileWriter.setLineAggregator(lineAggregator);
csvFileWriter.setFooterCallback(headerWriter);
return csvFileWriter;
}
You can implement CustomFooterCallback as follows:
public class CustomFooterCallback implements FlatFileFooterCallback {
#Value("#{StepExecution}")
private StepExecution stepExecution;
#Override
public void writeFooter(Writer writer) throws IOException {
writer.write("footer - number of items read: " + stepExecution.getReadCount());
writer.write("footer - number of items written: " + stepExecution.getWriteCount());
}
}
Then in a #Configuration class:
#Bean
#StepScope
public FlatFileFooterCallback customFooterCallback() {
return new CustomFooterCallback();
}
And use in the Writer:
csvFileWriter.setFooterCallback(customFooterCallback());
This way, you have access to StepExecution in order to read data as needed.
Related
I get the reader not open exception when tried to read JdbcCursorItemReader in the Item Reader implementation. I checked the stack overflow, but couldnt get a discussion on jdbc item reader.
Here is the code of Batch config and Item reader implementation. Added only required code.
public class BatchConfig extends DefaultBatchConfigurer {
#Bean
public ItemReader<Allocation> allocationReader() {
return new AllocationReader(dataSource);
}
#Bean
public Step step() {
return stepBuilderFactory.get("step").<Allocation, Allocation>chunk (1).reader(allocationReader()).processor(allocationProcessor()).writer(allocationWriter()).build();
}
}
public class AllocationReader implements ItemReader<Allocation> {
private DataSource ds;
private string block;
public AllocationReader(DataSource ds) {
this.ds = ds;
}
#BeforeStep
public void readStep(StepExecution StepExecution) {
this.stepExecution = StepExecution;
block = StepExecution.getJobExecution().get ExecutionContext().get("blocks");
}
#Override
public Allocation read() {
JdbcCursorItemReader<Allocation> reader = new JdbcCursorItemReader<Allocation>();
reader.setSql("select * from blocks where block_ref = + block);
reader.setDataSource(this.ds);
reader.setRowMapper(new AllocationMapper());
return reader.read();
}}
I could not write Item reader as bean in batch config as i need to call before step in item reader to access stepexecution.
If the Item reader read() function return type is changed to jdbccursorItemReader type, it throws type exception in Step reader() .
Let me know what am i missing or any other code snippet is required .
You are creating a JdbcCursorItemReader instance in the read method of AllocationReader. This is not correct. The code of this method should be the implementation of the actual read operation, and not for creating an item reader.
I could not write Item reader as bean in batch config as i need to call before step in item reader to access step execution.
For this use case, you can define the reader as a step-scoped bean and inject attributes from the job execution context as needed. This is explained in the reference documentation here: Late Binding of Job and Step Attributes. In your case, the reader could be defined like this:
#Bean
#StepScoped
public JdbcCursorItemReader<Allocation> itemReader(#Value("#{jobExecutionContext['block']}") String block) {
// use "block" as needed to define the reader
JdbcCursorItemReader<Allocation> reader = new JdbcCursorItemReader<Allocation>();
reader.setSql("select * from blocks where block_ref = " + block);
reader.setDataSource(this.ds);
reader.setRowMapper(new AllocationMapper());
return reader;
}
When you define an item reader bean that is an ItemStream, you need to make the bean definition method return at least ItemStreamReader (or the actual implementation type), so that Spring Batch correctly scopes the bean and calls open/update/close appropriately during the step. Otherwise, the open method will not be called and therefore you will get that ReaderNotOpenException.
I have following the FlatFileItemReader configuration for my step:
#Bean
#StepScope
public FlatFileItemReader<RawInput> reader(FieldSetMapper<RawInput> fieldSetMapper, #Value("#{jobParameters['files.location']}") Resource resource) {
var reader = new FlatFileItemReader<RawInput>();
reader.setName("my-reader");
reader.setResource(resource);
var mapper = new DefaultLineMapper<RawInput>();
mapper.setLineTokenizer(crmCsvLineTokenizer());
mapper.setFieldSetMapper(fieldSetMapper);
mapper.afterPropertiesSet();
reader.setLineMapper(mapper);
return reader;
}
RawInput contains 1 field so it allows me to read csv with single column. For now requirements were changes and now I have to be able to read any csv file with any amount of rows thus instead of RawInput I need to pass array somehow. is it possible with FlatFileItemReader or maybe I should change implementation ?
It works:
var reader = new FlatFileItemReader<List<String>>();
reader.setName("reader");
reader.setResource(resource);
//line mapper
var lineMapper = new DefaultLineMapper<List<String>>();
lineMapper.setLineTokenizer(new DelimitedLineTokenizer());
lineMapper.setFieldSetMapper(myFieldSetMapper); // see implementation below
lineMapper.afterPropertiesSet();
reader.setLineMapper(lineMapper);
return reader;
#Component
public class MyFieldSetMapper implements FieldSetMapper<List<String>> {
#NonNull
#Override
public List<String> mapFieldSet(#NonNull FieldSet fieldSet) {
return Arrays.stream(fieldSet.getValues())
.map(StringUtils::lowerCase) // optional
.map(StringUtils::trimToNull) // optional
.collect(Collectors.toList());
}
}
I created a simple Boot/Spring Batch 3.0.8.RELEASE job. I created a simple class that implements JobParametersIncrementer to go to the database, look up how many days the query should look for and puts those into the JobParameters object.
I need that value in my JdbcCursorItemReader, as it is selecting data based upon one of the looked up JobParameters, but I cannot figure this out via Java annotations. XML examples plenty, not so much for Java.
Below is my BatchConfiguration class that runs job.
`
#Autowired
SendJobParms jobParms; // this guy queries DB and puts data into JobParameters
#Bean
public Job job(#Qualifier("step1") Step step1, #Qualifier("step2") Step step2) {
return jobs.get("DW_Send").incrementer(jobParms).start(step1).next(step2).build();
}
#Bean
protected Step step2(ItemReader<McsendRequest> reader,
ItemWriter<McsendRequest> writer) {
return steps.get("step2")
.<McsendRequest, McsendRequest> chunk(5000)
.reader(reader)
.writer(writer)
.build();
}
#Bean
public JdbcCursorItemReader reader() {
JdbcCursorItemReader<McsendRequest> itemReader = new JdbcCursorItemReader<McsendRequest>();
itemReader.setDataSource(dataSource);
// want to get access to JobParameter here so I can pull values out for my sql query.
itemReader.setSql("select xxxx where rownum <= JobParameter.getCount()");
itemReader.setRowMapper(new McsendRequestMapper());
return itemReader;
}
`
Change reader definition as follows (example for parameter of type Long and name paramCount):
#Bean
#StepScope
public JdbcCursorItemReader reader(#Value("#{jobParameters[paramCount]}") Long paramCount) {
JdbcCursorItemReader<McsendRequest> itemReader = new JdbcCursorItemReader<McsendRequest>();
itemReader.setDataSource(dataSource);
itemReader.setSql("select xxxx where rownum <= ?");
ListPreparedStatementSetter listPreparedStatementSetter = new ListPreparedStatementSetter();
listPreparedStatementSetter.setParameters(Arrays.asList(paramCount));
itemReader.setPreparedStatementSetter(listPreparedStatementSetter);
itemReader.setRowMapper(new McsendRequestMapper());
return itemReader;
}
So I've created a batch job which generates reports (csv files). I have been able to generate the the files seamlessly using FlatFileItemWriter but my end goal is to create an InputStream to call a rest service which will store the document or a byte array to store it in the database.
public class CustomWriter implements ItemWriter<Report> {
#Override
public void write(List<? extends Report> reports) throws Exception {
reports.forEach(this::writeDataToFile);
}
private void writeDataToFile(final Report data) throws Exception {
FlatFileItemWriter writer = new FlatFileItemWriter();
writer.setResource(new FileSystemResource("C:/reports/test-report.csv"));
writer.setLineAggregator(getLineAggregator();
writer.afterPropertiesSet();
writer.open(new ExecutionContext());
writer.write(data);
writer.close();
}
private DelimitedLineAggregator<Report> getLineAggregator(final Report report) {
DelimitedLineAggregator<Report> delimitedLineAgg = new DelimitedLineAggregator<Report>();
delimitedLineAgg.setDelimiter(",");
delimitedLineAgg.setFieldExtractor(getFieldExtractor());
return delimitedLineAgg;
}
private FieldExtractor<Report> getFieldExtractor() {
BeanWrapperFieldExtractor<Report> fieldExtractor = new BeanWrapperFieldExtractor<Report>();
fieldExtractor.setNames(COLUMN_HEADERS.toArray(new String[0]));
return fieldExtractor;
}
}
One way I could do this is to store the file locally temporarily and create a new step to pick the generated files up and do the sending/storing but I would really like to skip this step and send/store it in the first step.
How do I go about doing this?
I have a spring batch job which reads from a db then outputs to a multiple csv's. Inside my db I have a special column named divisionId. A CSV file should exist for every distinct value of divisionId. I split out the data using a ClassifierCompositeItemWriter.
At the moment I have an ItemWriter bean defined for every distinct value of divisionId. The beans are the same, it's only the file name that is different.
How can I change the configuration below to create a file with the divisionId automatically pre-pended to the file name without having to register a new ItemWriter for each divisionId?
I've been playing around with #JobScope and #StepScope annotations but can't get it right.
Thanks in advance.
#Bean
public Step readStgDbAndExportMasterListStep() {
return commonJobConfig.stepBuilderFactory
.get("readStgDbAndExportMasterListStep")
.<MasterList,MasterList>chunk(commonJobConfig.chunkSize)
.reader(commonJobConfig.queryStagingDbReader())
.processor(masterListOutputProcessor())
.writer(masterListFileWriter())
.stream((ItemStream) divisionMasterListFileWriter45())
.stream((ItemStream) divisionMasterListFileWriter90())
.build();
}
#Bean
public ItemWriter<MasterList> masterListFileWriter() {
BackToBackPatternClassifier classifier = new BackToBackPatternClassifier();
classifier.setRouterDelegate(new DivisionClassifier());
classifier.setMatcherMap(new HashMap<String, ItemWriter<? extends MasterList>>() {{
put("45", divisionMasterListFileWriter45());
put("90", divisionMasterListFileWriter90());
}});
ClassifierCompositeItemWriter<MasterList> writer = new ClassifierCompositeItemWriter<MasterList>();
writer.setClassifier(classifier);
return writer;
}
#Bean
public ItemWriter<MasterList> divisionMasterListFileWriter45() {
FlatFileItemWriter<MasterList> writer = new FlatFileItemWriter<>();
writer.setResource(new FileSystemResource(new File(commonJobConfig.outDir, "45_masterList" + "" + ".csv")));
writer.setHeaderCallback(masterListFlatFileHeaderCallback());
writer.setLineAggregator(masterListFormatterLineAggregator());
return writer;
}
#Bean
public ItemWriter<MasterList> divisionMasterListFileWriter90() {
FlatFileItemWriter<MasterList> writer = new FlatFileItemWriter<>();
writer.setResource(new FileSystemResource(new File(commonJobConfig.outDir, "90_masterList" + "" + ".csv")));
writer.setHeaderCallback(masterListFlatFileHeaderCallback());
writer.setLineAggregator(masterListFormatterLineAggregator());
return writer;
}
I came up with a pretty complex way of doing this. I followed a tutorial at https://github.com/langmi/spring-batch-examples/wiki/Rename-Files.
The premise is to use the step execution context to place the file name in it.