I am downloading files with a GET request. Some of them are quite large, so I want to get them as a stream and read the bytes in chunks as I can process them, never reading the whole file in memory.
org.springframework.web.reactive.function.client.WebClient seemed like a good fit, but I am running into "UnsupportedMediaTypeException: Content type 'application/octet-stream' not supported.
Here is some short sample code.
#Autowired WebClient.Builder webClientBuilder;
....
ClientResponse clientResponse = webClientBuilder.clientConnector(this.connector)
.build()
.get()
.uri(uri)
.accept(MediaType.APPLICATION_OCTET_STREAM)
.exhange()
.block(Duration.of(1, ChronoUnit.MINUTES));
// blows up here, inside of the body call
InputStream responseInputStream = clientResponse.body(BodyExtractors.toMono(InputStream.class)).block(Duration.of(1, ChronoUnit.MINUTES));
Here is a chunk of the stack trace.
org.springframework.web.reactive.function.UnsupportedMediaTypeException: Content type 'application/octet-stream' not supported
at org.springframework.web.reactive.function.BodyExtractors.lambda$readWithMessageReaders$20(BodyExtractors.java:254)
at java.util.Optional.orElseGet(Optional.java:267)
at org.springframework.web.reactive.function.BodyExtractors.readWithMessageReaders(BodyExtractors.java:250)
at org.springframework.web.reactive.function.BodyExtractors.lambda$toMono$2(BodyExtractors.java:92)
......
I am on spring-webflux 5.0.7.
I am sure spring webclient must support something beyond JSON. I just don't know how to do that. Help?
Not an expert, but instead of an InputStream, you can get a Flux<byte[]> where each published array will contain a slice of the response body), using
.get()
.uri(uri)
.accept(MediaType.APPLICATION_OCTET_STREAM)
.retrieve()
.bodyToFlux(byte[].class)
You can also do the same with ByteBuffer instead of byte[] if you prefer.
Related
This is the API call (blocking) I am making to external application concurrently (max of 100 calls at the same time)
ClientResponse response = webclient.get()
.uri("https-url-here-with-query-params-here")
.header(HttpHeaders.AUTHORIZATION, "abcdefg")
.exchange()
.timeout(Duration.ofSeconds(300))
.block();
if (response.statusCode().is2xxSuccessful()) {
MyPojoEntity myPojo = response.bodyToMono(MyPojoEntity.class)
.timeout(Duration.ofSeconds(300))
.block();
}
What I observe is the call to response.bodyToMono(MyPojoEntity.class).timeout(Duration.ofSeconds(300)) is timing out after 5 minutes. My understanding is ClientResponse already has response body from the server and response.bodyToMono method just unmarshalling to pojo entity class. The payload is very small and shouldn't take more than few seconds to unmarshall it. May be it is still reading the response from server and timing out due to API issue on the server? If that is the case, then what does if (response.statusCode().is2xxSuccessful()) mean? I expect when response status is success, payload also to be there especially when i retrieve ClientResponse in a blocking way. Please help me to understand what is going on here.
I decided to use spring webClient in order to get large files.
Here is my method
suspend fun downloadDocument(
documentId: UUID
): InputStream =
webClient.get()
.uri("<some uri>")
.header(HttpHeaders.ACCEPT, MediaType.APPLICATION_OCTET_STREAM_VALUE)
.retrieve()
.bodyToFlux<DataBuffer>()
.awaitLast()
.asInputStream(true)
Unfortunately, it does't work as I expected. I receive only small part of data and that's it.
Where is an issue?
The aim of my code is to retrieve an image from a third-party service.
I struggled a little for endpoint of download to work and only partially succeeded. When I call the endpoint via postman the answer is a .bin file, but what I need is to have a .png file. The greatest success is being able to get a .png file being able to customize the name as well. But personalization of the is not strictly necessary.
The project is built with the initializer and has the following dependencies:
spring-boot-starter-web;
lombok
spring-boot-starter-webflux
reactor-spring
Below is the source code of my endpoint:
#GetMapping("/retrieve-image")
public Mono<byte[]> retrieveImage(ImageRequest request) throws ExecutionException, InterruptedException, IOException {
MultiValueMap<String, String> queryParams = new LinkedMultiValueMap<>();
queryParams.add("attribute", request.getAttribute()); // fake for query string setting.
Mono<byte[]> image = webClient
.get()
.uri(uriBuilder -> uriBuilder
.path(Endpoint.THIRD_PARTY_SERVICE_URI)
.queryParams(queryParams)
.build())
.accept(MediaType.valueOf(String.valueOf(MediaType.IMAGE_PNG)))
.exchange()
.flatMap(clientResponse -> clientResponse.bodyToMono(byte[].class)
.doOnSuccess(body -> {
if (clientResponse.statusCode().isError()) {
log.error("HttpStatusCode = {}", clientResponse.statusCode());
log.error("HttpHeaders = {}", clientResponse.headers().asHttpHeaders());
log.error("ResponseBody = {}", body);
}
}));
return image;
}
You can also add the mime type of the file to the produces section of the #GetMapping annotation, it should look something like this:
#GetMapping(path = "/retrieve-image",
produces = "image/png")
Additionally, instead of returning a Mono<byte[]>, you can wrap your response in a ResponseEntity<Resource>. This gives you the possibility to add Headers and tell the browser the name of your file. For example:
HttpHeaders header = new HttpHeaders();
header.add(HttpHeaders.CONTENT_DISPOSITION,
"attachment; filename=image.png");
header.add("Access-Control-Expose-Headers", "Content-Disposition");
return ResponseEntity.ok().
.headers(header)
.contentLength(Files.size(path))
.body(<<YOUR_FILE_HERE>>);
One last thought: If you add both spring-boot-starter-web and spring-boot-starter-webflux to your dependencies, the app will work, but it doesn't use Netty from Webflux, instead the usual Tomcat. So you don't benefit from the reactive features.
I've been trying to follow the simplest tutorials out there for how to use WebClient, which I understand to be the next greatest thing compared to RestTemplate.
For example, https://www.baeldung.com/spring-5-webclient#4-getting-a-response
So when I try to do the same thing with https://petstore.swagger.io/v2/pet/findByStatus?status=available which is supposed to return some json,
WebClient webClient = WebClient.create();
webClient.get().uri("https://petstore.swagger.io/v2/pet/findByStatus?status=available").exchange().block();
I have absolutely no idea how to proceed from the resultant DefaultClientResponse object. It shouldn't be this convoluted to arrive at the physical response body, but I digress.
How do I get the response body with the code I provided?
In the form you currently have it, and explaining the behaviour..
WebClient webClient = WebClient.create();
webClient.get()
.uri("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.exchange()
.block();
the block() starts the request by internally synchronously subscribing to the Mono, and returns the resulting ClientResponse. You could also handle this asynchronously by calling subscribe() on the Mono returned by the exchange() method, instead of block().
In this current form, after the block() you now have all the metadata (ie. from the response header) about the response in a ClientResponse object, including the success status. This does not mean that the response body has finished coming through. If you don't care about the response payload, you could confirm the success and leave it at that.
If you further want to look at the response body, you need to convert the response body stream into some class. A this point you can decide whether you want to read everything into a single Mono with bodyToMono or into a stream of objects (Flux) with bodyToFlux, such as in the case where the response is a JSON array that can be parsed into individual separate Java objects.
However, in your case, you just want to see the JSON as-is. So converting to a String is sufficient. You would just use bodyToMono which would return a Mono object.
WebClient webClient = WebClient.create();
String responseJson = webClient.get()
.uri("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.exchange()
.block()
.bodyToMono(String.class)
.block();
Here you use block() to wait for the response payload to arrive and be parsed into a String, but you could also subscribe to the Mono to receive it reactively when it is complete.
One thing to note is that retrieve() can be used instead of exchange() to shortcut the ClientResponse. In this case you let default behavior handle error responses. Using exchange() puts all the responsibility on the application for responding to error responses on the ClientResponse. Read more in the Javadoc. The retrieve() version would look as follows. No need to block() as you only care about the response data.
WebClient webClient = WebClient.create();
String responseJson = webClient.get()
.uri("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.retrieve()
.bodyToMono(String.class)
.block();
Here is how you make a request with RestTemplate
String json = new RestTemplate()
.getForEntity("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.getBody();
Here is how you make a request with requests
import requests
json = requests.get("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.content
Here is how you make a request with WebClient
String json = WebClient.create()
.get()
.uri("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.exchange()
.block()
.bodyToMono(String.class)
.block();
I'm using WebClient (SpringBoot 2.0.2.RELEASE) to send a POST with SOAP request, but it is missing "Content-Length" header required by the legacy API.
Is it possible to configure WebClient to include "Content-Length" header?
There is an Spring Framework Issue resolved and introduced for EncoderHttpMessageWriter in SpringBoot 2.0.1, but it seems not to work for JAXB.
I tried to use BodyInserters:
webClient.post().body(BodyInserters.fromObject(request)).exchange();
and syncBody:
webClient.post().syncBody(request).exchange();
None of them worked for WebClient. Though, when RestTemplate is used, Content-Length is set and API responds with success
I am struggling with the same problem, as an ugly work-around I am manually serializing the request (JSON in my case) and setting the length (Kotlin code):
open class PostRetrieverWith411ErrorFix(
private val objectMapper: ObjectMapper
) {
protected fun <T : Any> post(webClient: WebClient, body: Any, responseClass: Class<T>): Mono<T> {
val bodyJson = objectMapper.writeValueAsString(body)
return webClient.post()
.contentType(MediaType.APPLICATION_JSON_UTF8)
.contentLength(bodyJson.toByteArray(Charset.forName("UTF-8")).size.toLong())
.syncBody(bodyJson)
.retrieve()
.bodyToMono(responseClass)
}
}
If you apply Sven's colleague(Max) solution like we did you can also adapt it for cases like your body being a custom obj but you have to serialize it once:
String req = objectMapper.writeValueAsString(requestObject)
and passed that to
webClient.syncBody(req)
Keep in mind that with SpringBoot 2.0.3.RELEASE, if you'll pass a String to webClient as a request, it will put as ContentType header MediaType.TEXT_PLAIN and that made our integration with other service to fail. We fixed that by setting specifically content type header like this:
httpHeaders.setContentType(MediaType.APPLICATION_JSON);
WebClient is a streaming client and it's kind of difficult to set the content length until the stream has finished. By then the headers are long gone. If you work with legacy, you can re-use your mono (Mono/Flux can be reused, Java streams not) and check the length.
public void post() {
Mono<String> mono = Mono.just("HELLO WORLDZ");
final String response = WebClient.create("http://httpbin.org")
.post()
.uri("/post")
.header(HttpHeaders.CONTENT_LENGTH,
mono.map(s -> String.valueOf(s.getBytes(StandardCharsets.UTF_8).length)).block())
.body(BodyInserters.fromPublisher(mono, String.class))
.retrieve()
.bodyToMono(String.class)
.block();
System.out.println(response);
}
A colleague (well done Max!) of mine came up with cleaner solution, I added some wrapping code so it can be tested:
Mono<String> my = Mono.just("HELLO WORLDZZ")
.flatMap(body -> WebClient.create("http://httpbin.org")
.post()
.uri("/post")
.header(HttpHeaders.CONTENT_LENGTH,
String.valueOf(body.getBytes(StandardCharsets.UTF_8).length))
.syncBody(body)
.retrieve()
.bodyToMono(String.class));
System.out.println(my.block());