AWS S3 with Java - Reactive: S3 upload doesn't work - spring

Question

I have a Spring Batch that should get a gzipped file from a rest service and then save it in a S3 bucket in the first step. In the second step it should retrieve this file from S3 and elaborate it. I have tried to follow this guide ("Single File Upload" case).
I'm able to save the file in S3, but I get "NoSuchKeyException: The specified key does not exist." at the beginning of the second step.
Looking at S3 bucket, the file has been written just 2 seconds after the second step started. It seems that the first step said "file created" even if its process was not completed. Infact this log:
log.info("S3 saving response: {}", response.sdkHttpResponse().isSuccessful());
is not present at all in the logs.
Thanks in advance!
The tasklet of the first step (simplified):
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
...
Mono<ResponseEntity<Flux<DataBuffer>>> response = myService.getAnalyticsDataBuffer2(authToken);
response.log().map(e -> {
DataBufferMapper dataBufferMapper = new DataBufferMapper(myFileManager, analyticTripFilename);
return dataBufferMapper.apply(e);
}).log().block();
return RepeatStatus.FINISHED;
}
The method to retrieve the gzipped file:
public Mono<ResponseEntity<Flux<DataBuffer>>> getAnalyticsDataBuffer2(String correlationId, String token) {
return webClient
.get()
.uri(
uriBuilder ->
uriBuilder.path(ANALITYCS)
.build()
)
.headers(h -> h.setBearerAuth(token))
.header("Business-User-Id", clientId)
.header("Connection", "close")
.accept(MediaType.APPLICATION_OCTET_STREAM)
.retrieve()
.toEntityFlux(DataBuffer.class)
.name(METRICS_NAME)
.tag(TAG_API_NAME, API_NAME)
.metrics();
}
The mapper Functional interface:
public class DataBufferMapper implements Function<ResponseEntity<Flux<DataBuffer>>, Mono> {
private final MyFileManager myFileManager;
private final String analyticTripFilename;
public DataBufferMapper(MyFileManager myFileManager, String analyticTripFilename) {
this.myFileManager = myFileManager;
this.analyticTripFilename = analyticTripFilename;
}
#Override
public Mono apply(ResponseEntity<Flux<DataBuffer>> response) {
return myFileManager.saveFile(analyticTripFilename, response.getBody(), response.getHeaders().getContentLength());
}
}
The implementation of the S3 upload.
public class S3FileManager implements MyFileManager {
#Autowired
private S3AsyncClient s3AsyncClient;
#Autowired
private S3ClientConfig s3ClientConfig;
#Override
public Mono saveFile(String fileName, Flux<DataBuffer> body, long contentLength) {
log.info("S3FileManager saving file/content length{}/{}", fileName, contentLength);
Flux<java.nio.ByteBuffer> buffers = body.map(DataBuffer::asByteBuffer);
CompletableFuture<PutObjectResponse> future = s3AsyncClient. putObject(PutObjectRequest.builder()
.bucket(s3ClientConfig.getBucket())
.contentLength(contentLength)
.key(fileName)
.contentType(MediaType.APPLICATION_OCTET_STREAM.toString())
.build(), AsyncRequestBody.fromPublisher(buffers));
return Mono.fromFuture(future)
.map(response -> {
checkResult(response);
log.info("S3 saving response: {}", response.sdkHttpResponse().isSuccessful());
return response;
});
}
Update
To isolate the problem, I have added reactor logs and added a 'local' implementation for MyFileManager like this:
#Override
public Mono saveFile(String fileName, Flux<DataBuffer> dataBuffer, long contentLength) {
String filePath = localFileManagerPath + fileName;
final Path path = FileSystems.getDefault().getPath(filePath);
log.info("LocalFileManager saving {}", filePath);
return DataBufferUtils.write(dataBuffer, path, CREATE_NEW).log();
}
Now the error is clearly turned into 'Input resource must exist' and the log says:
[ INFO] demo.service.batchservice.BatchService: Starting the runAnalyticTripsJob job
[ INFO] org.springframework.batch.core.launch.support.SimpleJobLauncher: Job: [SimpleJob: [name=analyticTripsLoaderJob]] launched with the following parameters: [{analyticTripFilename=ANALYTIC_TRIPS-30052022-175840.858.csv, time=T15:58:40.858902Z, isNewAnalyticTripNeeded=true}]
[ INFO] org.springframework.batch.core.job.SimpleStepHandler: Executing step: [retrievingDataStep]
[ INFO] demo.service.demo.DemoDataRetriever: analyticTripFilename = ANALYTIC_TRIPS-30052022-175840.858.csv, isNewAnalyticTripNeeded = true
[ INFO] demo.client.DemoAuthClient: Successfully accrued demo token
[ INFO] reactor.Mono.OnAssembly.1: | onSubscribe([Fuseable] FluxOnAssembly.OnAssemblySubscriber)
[ INFO] reactor.Mono.OnAssembly.2: | onSubscribe([Fuseable] FluxOnAssembly.OnAssemblySubscriber)
[ INFO] reactor.Mono.OnAssembly.3: | onSubscribe([Fuseable] FluxOnAssembly.OnAssemblySubscriber)
[ INFO] reactor.Mono.OnAssembly.3: | request(unbounded)
[ INFO] reactor.Mono.OnAssembly.2: | request(unbounded)
[ INFO] reactor.Mono.OnAssembly.1: | request(unbounded)
[ INFO] reactor.Mono.OnAssembly.1: | onNext(<200,Flux.onErrorResume ⇢ at org.springframework.web.reactive.function.client.DefaultWebClient$DefaultResponseSpec.handlerEntityFlux(DefaultWebClient.java:635),[access-control-allow-origin:"*", x-request-id:"xyz", content-disposition:"attachment; filename="data.csv.gz"", ... function-execution-id:"j", x-cloud-trace-context:"xxx;o=1", Connection:"close", Content-Length:"68", Content-Type:"application/octet-stream", Via:"1.1 google"]>)
[ INFO] reactor.Mono.OnAssembly.2: | onNext(<200,Flux.onErrorResume ⇢ at org.springframework.web.reactive.function.client.DefaultWebClient$DefaultResponseSpec.handlerEntityFlux(DefaultWebClient.java:635),[access-control-allow-origin:"*", x-request-id:"xyz", content-disposition:"attachment; filename="data.csv.gz"", ... function-execution-id:"j", x-cloud-trace-context:"xxx;o=1", Connection:"close", Content-Length:"68", Content-Type:"application/octet-stream", Via:"1.1 google"]>)
[ INFO] demo.service.demo.LocalFileManager: LocalFileManager saving ./tmp/ANALYTIC_TRIPS-30052022-175840.858.csv
[ INFO] reactor.Mono.OnAssembly.3: | onNext(Mono.log ⇢ at demo.service.demo.DemoDataRetriever.lambda$execute$0(DemoDataRetriever.java:45))
[ INFO] demo.service.demo.DemoDataRetriever: LocalFileManager saved ANALYTIC_TRIPS-30052022-175840.858.csv
[ INFO] reactor.Mono.OnAssembly.2: | onComplete()
[ INFO] reactor.Mono.OnAssembly.3: | onComplete()
[ INFO] reactor.Mono.OnAssembly.1: | onComplete()
[ INFO] org.springframework.batch.core.step.AbstractStep: Step: [retrievingDataStep] executed in 2s257ms
[ INFO] org.springframework.batch.core.job.SimpleStepHandler: Executing step: [processingDataStep]
[ INFO] demo.config.BatchConfiguration: analyticTripFilename = ANALYTIC_TRIPS-30052022-175840.858.csv
[ INFO] demo.service.demo.LocalFileManager: LocalFileManager getFileResource ./tmp/ANALYTIC_TRIPS-30052022-175840.858.csv
[ERROR] org.springframework.batch.core.step.AbstractStep: Encountered an error executing step processingDataStep in job analyticTripsLoaderJob
org.springframework.batch.item.ItemStreamException: Failed to initialize the reader
Thanks in advance!

Answer 1

Finally solved...it was just a matter of turning map into flatMap in the tasklet:
Mono<ResponseEntity<Flux<DataBuffer>>> response = myService.getAnalyticsDataBuffer2(authToken);
response.flatMap(e -> {
DataBufferMapper dataBufferMapper = new DataBufferMapper(myFileManager, analyticTripFilename);
return dataBufferMapper.apply(e);
}).block();
Now it works both locally and on S3.