reader -> injest a class named Person
processor -> injest Person, and returns a wrapper object, that has 3 fields, all of them are of type Person (1. inputPerson, 2. outputPerson, 3. output)
writer -> injest this wrapper and should write first two fields in one file, and the third one in a second file as xml.
That is the code, that I have written for this problem:
#Bean
public CompositeItemWriter<Wrapper> compositeItemWriter(){
CompositeItemWriter writer = new CompositeItemWriter();
writer.setDelegates(Arrays.asList(firstTwoWriters(), thirdWriter));
return writer;
}
#Bean
public StaxEventItemWriter<Wrapper> firstTwoWriters() {
StaxEventItemWriter<Wrapper> xmlFileWriter = new StaxEventItemWriter<>();
xmlFileWriter.setRootTagName("something");
String outputName = applicationArguments.getOptionValues("output").get(0);
FileSystemResource outputResource = new FileSystemResource(outputName);
xmlFileWriter.setResource(outputResource);
Jaxb2Marshaller personMarshaller = new Jaxb2Marshaller();
scoringMapMarshaller.setClassesToBeBound(Person.class);
xmlFileWriter.setMarshaller(personMarshaller );
}
The problem is, that i cannot choose which field (inputPerson, outputPerson or output) should be used by this writer(which field should be converted to xml).
Any ideas, how can I do this? (if possible with an example)
Related
I have following the FlatFileItemReader configuration for my step:
#Bean
#StepScope
public FlatFileItemReader<RawInput> reader(FieldSetMapper<RawInput> fieldSetMapper, #Value("#{jobParameters['files.location']}") Resource resource) {
var reader = new FlatFileItemReader<RawInput>();
reader.setName("my-reader");
reader.setResource(resource);
var mapper = new DefaultLineMapper<RawInput>();
mapper.setLineTokenizer(crmCsvLineTokenizer());
mapper.setFieldSetMapper(fieldSetMapper);
mapper.afterPropertiesSet();
reader.setLineMapper(mapper);
return reader;
}
RawInput contains 1 field so it allows me to read csv with single column. For now requirements were changes and now I have to be able to read any csv file with any amount of rows thus instead of RawInput I need to pass array somehow. is it possible with FlatFileItemReader or maybe I should change implementation ?
It works:
var reader = new FlatFileItemReader<List<String>>();
reader.setName("reader");
reader.setResource(resource);
//line mapper
var lineMapper = new DefaultLineMapper<List<String>>();
lineMapper.setLineTokenizer(new DelimitedLineTokenizer());
lineMapper.setFieldSetMapper(myFieldSetMapper); // see implementation below
lineMapper.afterPropertiesSet();
reader.setLineMapper(lineMapper);
return reader;
#Component
public class MyFieldSetMapper implements FieldSetMapper<List<String>> {
#NonNull
#Override
public List<String> mapFieldSet(#NonNull FieldSet fieldSet) {
return Arrays.stream(fieldSet.getValues())
.map(StringUtils::lowerCase) // optional
.map(StringUtils::trimToNull) // optional
.collect(Collectors.toList());
}
}
I have a flat file containing different records(header, record and footer)
HR,...
RD,...
FR,...
ItemReader
#Bean
#StepScope
public FlatFileItemReader reader(#Value("#{jobParameters['inputFileName']}") String inputFileName) {
FlatFileItemReader reader = new FlatFileItemReader();
reader.setResource(new FileSystemResource(inputFileName));
reader.setLineMapper(patternLineMapper());
return reader;
}
#Bean
public LineMapper patternLineMapper() {
PatternMatchingCompositeLineMapper patternLineMapper = new PatternMatchingCompositeLineMapper<>();
tokenizers = new HashMap<String, LineTokenizer>();
try {
tokenizers.put("HR*", headerLineTokenizer());
tokenizers.put("RD*", recordLineTokenizer());
tokenizers.put("FR*", footerLineTokenizer());
} catch (Exception e) {
e.printStackTrace();
}
fieldSetMappers = new HashMap<String, FieldSetMapper>();
fieldSetMappers.put("HR*", new HeaderFieldSetMapper());
fieldSetMappers.put("RD*", new RecordFieldSetMapper());
fieldSetMappers.put("FR*", new FooterFieldSetMapper());
patternLineMapper.setTokenizers(tokenizers);
patternLineMapper.setFieldSetMappers(fieldSetMappers);
return patternLineMapper;
}
They are working fine and spring batch calls the appropriate reader for each record the problem is when it comes to item processor I want to use the same approach I get java.lang.ClassCastException cuz spring batch try to map domain object [returned from reader] to java.lang.String
ItemProcessor
#Bean
#StepScope
public ItemProcessor processor() {
ClassifierCompositeItemProcessor processor = new ClassifierCompositeItemProcessor();
PatternMatchingClassifier<ItemProcessor> classifier = new PatternMatchingClassifier<>();
Map<String, ItemProcessor> patternMap = new HashMap<>();
patternMap.put("HR*", new HeaderItemProcessor());
patternMap.put("RD*", new RecordItemProcessor());
patternMap.put("FR*", new FooterItemProcessor());
classifier.setPatternMap(patternMap);
processor.setClassifier(classifier);
return processor;
}
I also used BackToBackPatternClassifier but it turns out it has a bug and when I use generics like ItemWriter<Object> I get an exception Couldn't Open File. the question is
How can I make ItemProcessor that handles different record types returned from Reader??
Your issue is that the classifier you use in the ClassifierCompositeItemProcessor is based on a String pattern and not a type. What really should happen is something like:
The reader returns a specific type of items based on the input pattern, something like:
HR* -> HRType
RD* -> RDType
FR* -> FRType
This is what you have basically done on the reader side. Now on the processing side, the processor will receive objects of type HRType, RDType and FRType. So the classifier should not be based on String as input type, but on the item type, something like:
Map<Object, ItemProcessor> patternMap = new HashMap<>();
patternMap.put(HRType.class, new HeaderItemProcessor());
patternMap.put(RDType.class, new RecordItemProcessor());
patternMap.put(FRType.class, new FooterItemProcessor());
This classifier uses Object type because your ItemReader returns a raw type. I would not recommend using raw types and Object type in the classifier. What you should do is:
create a base class of your items and a specific class for each type
Make the reader return items of type <? extends BaseClass>
Use a org.springframework.classify.SubclassClassifier in your ClassifierCompositeItemProcessor
My approach so far:
#Bean
FlatFileItemReader<Blub> flatFileItemReader() {
FlatFileItemReader<Blub> reader = new FlatFileItemReader<>();
reader.setResource(new FileSystemResource("test.json"));
JsonLineMapper lineMapper = new JsonLineMapper();
reader.setLineMapper(lineMapper);
return reader;
}
The challenge is: reader.setLineMapper() cannot use the JsonLineMapper. How to use the JsonLineMapper properly?
create a class BlubJsonLineMapper
public class BlubJsonLineMapper implements LineMapper<Blub> {
private ObjectMapper mapper = new ObjectMapper();
/**
* Interpret the line as a Json object and create a Blub Entity from it.
*
* #see LineMapper#mapLine(String, int)
*/
#Override
public Blub mapLine(String line, int lineNumber) throws Exception {
return mapper.readValue(line, Blub.class);
}
}
then you can set in the FlatFileItemReader
#Bean
FlatFileItemReader<Blub> flatFileItemReader() {
FlatFileItemReader<Blub> reader = new FlatFileItemReader<>();
reader.setResource(new FileSystemResource("test.json"));
BlubJsonLineMapper lineMapper = new BlubJsonLineMapper();
reader.setLineMapper(lineMapper);
return reader;
}
How to setup a FlatFileItemReader to read a json file?
It depends on the format of your json file:
1. Each line is a json object (known as NDJson)
For example:
{object1}
{object2}
then you have two options:
1.1 Use the JsonLineMapper which returns a Map<String, Object>. In this case, your reader should also return Map<String, Object> and you can use an item processor to transform items from Map<String, Object> to Blub (BTW, transforming data from one type to another is a typical use case for an item processor)
1.2 Use a custom implementation of LineMapper<Blub> based on Jackson or Gson or any other library (as shown in the answer by #clevertension)
2. Lines are wrapped in a json array
For example:
[
{object1},
{object2}
]
then you can use the new JsonItemReader that we introduced in version 4.1.0.M1 (See example in the blog post here: https://spring.io/blog/2018/05/31/spring-batch-4-1-0-m1-released#add-a-new-json-item-reader).
There are similar questions to this one, I'm adding them here for reference:
How to read a complex JSON in spring batch?
Json Array reader file with spring batch
Is there a bug in the new Spring JSON reader or am I doing something wrong?
I have build a small demo for Json. If you need any more than it, let me know I can build another example for you
https://github.com/bigzidane/spring-batch-jsonListItem-reader
In Spring batch I configure a file write as such:
#Bean
public FlatFileItemWriter<MyObject> flatFileItemWriter() throws Exception{
FlatFileItemWriter<MyObject> itemWriter = new FlatFileItemWriter();
// pass through aggregator just calls toString on any item pass in.
itemWriter.setLineAggregator(new PassThroughLineAggregator<>());
String outputPath = File.createTempFile("output", ".out").getAbsolutePath();
System.out.println(">>output path=" + outputPath);
itemWriter.setResource(new FileSystemResource(outputPath));
itemWriter.afterPropertiesSet();
return itemWriter;
}
What happens if MyObject is a complex structure that can vary depending on configuration settings etc and I want to generate different parts of that structure to different files.
How do I do this?
Have you looked at CompositeItemWriter? You may need to have CompositeLineMapper in your reader as well as ClassifierCompositeItemProcessor depending on your needs.
Below is example of a CompositeItemWriter
#Bean
public ItemWriter fileWriter() {
CompositeItemWriter compWriter = new CompositeItemWriter();
FlatFileItemWriter<MyObject_data> dataWriter = new FlatFileItemWriter<MyObject_data>();
FlatFileItemWriter<MyObject_otherdata> otherWriter = new FlatFileItemWriter<MyObject_otherdata>();
List<ItemWriter> iList = new ArrayList<ItemWriter>();
iList.add(dataWriter);
iList.add(otherWriter);
compWriter.setDelegates(iList);
return compWriter;
}
I've written a spring batch job to read from a database and then write to a csv.
The job works but unfortunately in my output CSV file it just puts whatever is in the toString method of my Domain Object.
What I am really after is all the values in the bean separated by a comma. Which is why in my ItemWriter below I put in a DelimitedLineAggregator.
But I think my understanding of that DelimitedLineAggregator is wrong. I thought that the LineAggregator was used for the output but now I think it's used for the input data.
#Bean
#StepScope
public ItemWriter<MasterList> masterListFileWriter(
FileSystemResource masterListFile,
#Value("#{stepExecutionContext}")Map<String, Object> executionContext) {
FlatFileItemWriter<MasterList> writer = new FlatFileItemWriter<>();
writer.setResource(masterListFile);
DelimitedLineAggregator<MasterList> lineAggregator = new DelimitedLineAggregator<>();
lineAggregator.setDelimiter(";");
writer.setLineAggregator(lineAggregator);
writer.setForceSync(true);
writer.open(new ExecutionContext(executionContext));
return writer;
}
Two things.
What can I change to output all the values of my MasterList domain object separated by a comma? Is changing the toString method the only way?
Also can someone clarify the use of the LineAggregator in the writer. I'm now thinking it's used to specify how you want to aggregate lines coming from your Reader. Is that right?
Thanks in advance
I worked this out by adding a BeanWrapperFieldExtractor to the writer.
#Bean
#StepScope
public ItemWriter<MasterList> masterListFileWriter(
FileSystemResource masterListFile,
#Value("#{stepExecutionContext}")Map<String, Object> executionContext) {
FlatFileItemWriter<MasterList> writer = new FlatFileItemWriter<>();
writer.setResource(masterListFile);
DelimitedLineAggregator<MasterList> lineAggregator = new DelimitedLineAggregator<>();
lineAggregator.setDelimiter(",");
BeanWrapperFieldExtractor<MasterList> extractor = new BeanWrapperFieldExtractor<MasterList>();
extractor.setNames(new String[] { "l2", "l2Name"});
lineAggregator.setFieldExtractor(extractor);
writer.setLineAggregator(lineAggregator);
writer.setForceSync(true);
writer.open(new ExecutionContext(executionContext));
return writer;
}