Spring Batch - Read multiple files from S3 - spring

Question

Like for reading single file in spring batch from s3, we use
#Bean
public FlatFileItemReader<Map<String, Object>> itemReader() {
FlatFileItemReader<Map<String, Object>> reader = new FlatFileItemReader<>();
reader.setLineMapper(new JsonLineMapper());
reader.setRecordSeparatorPolicy(new JsonRecordSeparatorPolicy());
reader.setResource(resourceLoader.getResource("s3://" + amazonS3Bucket + "/" + file));
return reader;
}
But if i want to read all the files from some specific folder/key then is there something to MultiResourceItemReader, like below(which we use for local filesystem)
MultiResourceItemReader<UserData> reader = new MultiResourceItemReader<>();
reader.setResources(resources);

Answer 1

No, it is up to you to create the Resource array and pass it to the MultiResourceItemReader.

Answer 2

Create a MultiResourceItemReader like this,
#Autowired
private AmazonS3 s3;
#Autowired
private ResourceLoader resourceLoader;
public MultiResourceItemReader<String> fileItemReader() throws Exception {
List<Resource> resourceList = new ArrayList<>();
String s3ResponseFilePath = "s3://bucket/path/"; //put your s3 path here
//TODO: warn: this functn can only return max 1000 objects
s3objects = s3.listObjects("bucket", s3ResponseFilePath).getObjectSummaries();
for(S3ObjectSummary it:s3objects)
resourceList.add(resourceLoader.getResource( "s3://" + s3Config.getBucket() + "/" + it.getKey()));
Resource[] resources = resourceList.toArray(new Resource[resourceList.size()]);
MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
reader.setResources(resources);
reader.setDelegate(flatFileItemReader());
return reader;
}
This reader will need a delegate and lineMapper, you can implement it like this,
private FlatFileItemReader<String> flatFileItemReader() throws Exception {
FlatFileItemReader<String> reader = new FlatFileItemReader<>();
JsonLineMapper lineMapper = new JsonLineMapper();
reader.setLineMapper(lineMapper);
reader.afterPropertiesSet();
return reader;
}
public class JsonLineMapper implements LineMapper<String> {
private ObjectMapper mapper = new ObjectMapper();
#Override
public String mapLine(String s, int i) throws Exception {
return s;
}
}