I decided to use spring webClient in order to get large files.
Here is my method
suspend fun downloadDocument(
documentId: UUID
): InputStream =
webClient.get()
.uri("<some uri>")
.header(HttpHeaders.ACCEPT, MediaType.APPLICATION_OCTET_STREAM_VALUE)
.retrieve()
.bodyToFlux<DataBuffer>()
.awaitLast()
.asInputStream(true)
Unfortunately, it does't work as I expected. I receive only small part of data and that's it.
Where is an issue?
Related
I am trying to retrieve the data from strapi latest versio 4 using webclient in spring boot.
The data is one level deeper, as per document following URL will give data.
/api/customer?filters[state][$eq]=karnataka&populate=%2A
/api/customer?filters[state][$eq]=karnataka&populate=* -> This will work for webclient
I tried to use webclient but its not working, if http client then it works.
Below pasted both the approaches, http one works but webclient doesnt give next level data at all
Can u please help me with this?
HttpRequest request =
HttpRequest.newBuilder().GET().uri(URI.create(urlToUse)).build();
java.net.http.HttpClient client =
java.net.http.HttpClient.newBuilder().build(); HttpResponse<String> response
= client.send(request, HttpResponse.BodyHandlers.ofString());
output= new ObjectMapper()
.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
.readValue(response.body(), Customer.class);
Mono<String> mon = webClient.get()
.uri(urlToUse)
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(String.class);
String resp = mon.block(Duration.ofMillis(cmsConfig.getTimeOut()));
output= new ObjectMapper()
.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
.readValue(resp, Customer.class);
Update on observation:
I did capture packets using wireshark:
With webclient packet i can see request as below:
?filters%5Bstate%5D%5B$eq%5D=karnataka&populate=%252A
With httpclient packet:
?filters[state][$eq]=Karnataka&populate=%2A
%25 is appended to 2A, as % ASCII is %25, how to avoid it? Is this causing an issue?
I've been trying to follow the simplest tutorials out there for how to use WebClient, which I understand to be the next greatest thing compared to RestTemplate.
For example, https://www.baeldung.com/spring-5-webclient#4-getting-a-response
So when I try to do the same thing with https://petstore.swagger.io/v2/pet/findByStatus?status=available which is supposed to return some json,
WebClient webClient = WebClient.create();
webClient.get().uri("https://petstore.swagger.io/v2/pet/findByStatus?status=available").exchange().block();
I have absolutely no idea how to proceed from the resultant DefaultClientResponse object. It shouldn't be this convoluted to arrive at the physical response body, but I digress.
How do I get the response body with the code I provided?
In the form you currently have it, and explaining the behaviour..
WebClient webClient = WebClient.create();
webClient.get()
.uri("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.exchange()
.block();
the block() starts the request by internally synchronously subscribing to the Mono, and returns the resulting ClientResponse. You could also handle this asynchronously by calling subscribe() on the Mono returned by the exchange() method, instead of block().
In this current form, after the block() you now have all the metadata (ie. from the response header) about the response in a ClientResponse object, including the success status. This does not mean that the response body has finished coming through. If you don't care about the response payload, you could confirm the success and leave it at that.
If you further want to look at the response body, you need to convert the response body stream into some class. A this point you can decide whether you want to read everything into a single Mono with bodyToMono or into a stream of objects (Flux) with bodyToFlux, such as in the case where the response is a JSON array that can be parsed into individual separate Java objects.
However, in your case, you just want to see the JSON as-is. So converting to a String is sufficient. You would just use bodyToMono which would return a Mono object.
WebClient webClient = WebClient.create();
String responseJson = webClient.get()
.uri("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.exchange()
.block()
.bodyToMono(String.class)
.block();
Here you use block() to wait for the response payload to arrive and be parsed into a String, but you could also subscribe to the Mono to receive it reactively when it is complete.
One thing to note is that retrieve() can be used instead of exchange() to shortcut the ClientResponse. In this case you let default behavior handle error responses. Using exchange() puts all the responsibility on the application for responding to error responses on the ClientResponse. Read more in the Javadoc. The retrieve() version would look as follows. No need to block() as you only care about the response data.
WebClient webClient = WebClient.create();
String responseJson = webClient.get()
.uri("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.retrieve()
.bodyToMono(String.class)
.block();
Here is how you make a request with RestTemplate
String json = new RestTemplate()
.getForEntity("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.getBody();
Here is how you make a request with requests
import requests
json = requests.get("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.content
Here is how you make a request with WebClient
String json = WebClient.create()
.get()
.uri("https://petstore.swagger.io/v2/pet/findByStatus?status=available")
.exchange()
.block()
.bodyToMono(String.class)
.block();
I am downloading files with a GET request. Some of them are quite large, so I want to get them as a stream and read the bytes in chunks as I can process them, never reading the whole file in memory.
org.springframework.web.reactive.function.client.WebClient seemed like a good fit, but I am running into "UnsupportedMediaTypeException: Content type 'application/octet-stream' not supported.
Here is some short sample code.
#Autowired WebClient.Builder webClientBuilder;
....
ClientResponse clientResponse = webClientBuilder.clientConnector(this.connector)
.build()
.get()
.uri(uri)
.accept(MediaType.APPLICATION_OCTET_STREAM)
.exhange()
.block(Duration.of(1, ChronoUnit.MINUTES));
// blows up here, inside of the body call
InputStream responseInputStream = clientResponse.body(BodyExtractors.toMono(InputStream.class)).block(Duration.of(1, ChronoUnit.MINUTES));
Here is a chunk of the stack trace.
org.springframework.web.reactive.function.UnsupportedMediaTypeException: Content type 'application/octet-stream' not supported
at org.springframework.web.reactive.function.BodyExtractors.lambda$readWithMessageReaders$20(BodyExtractors.java:254)
at java.util.Optional.orElseGet(Optional.java:267)
at org.springframework.web.reactive.function.BodyExtractors.readWithMessageReaders(BodyExtractors.java:250)
at org.springframework.web.reactive.function.BodyExtractors.lambda$toMono$2(BodyExtractors.java:92)
......
I am on spring-webflux 5.0.7.
I am sure spring webclient must support something beyond JSON. I just don't know how to do that. Help?
Not an expert, but instead of an InputStream, you can get a Flux<byte[]> where each published array will contain a slice of the response body), using
.get()
.uri(uri)
.accept(MediaType.APPLICATION_OCTET_STREAM)
.retrieve()
.bodyToFlux(byte[].class)
You can also do the same with ByteBuffer instead of byte[] if you prefer.
I am working in a Spring #Component class and I am trying to get the HTTP status code of a particular URL for further processing. I have a function as follows:
fun getStatus() : String
{
val webClient = WebClient.create("https://stackoverflow.com")
val result = webClient.get()
.exchange().map { res -> res.rawStatusCode() }
println(result)
return "statusGotten"
}
However, rather than getting the Int value of the status code (e.g. 200, or 401), I am simply getting: "MonoMap".
I am new to both Spring and Web Programming in general, so I'm a little confused how to proceed from here. I'm aware that "result" is being returned as a "Mono", but I'm less clear about what a "Mono" is, or how I might transform it into something with more scrutable properties, as even looking at "result" in the debugger doesn't shed any light as to whether the HTTP request was actually sent or was successful:
Am I calling the webclient incorrectly? Or merely failing to parse the resultant data in a meaningful way? Any suggestions on how or where I might learn more about the underlying topics would be much appreciated as well.
If you need a blocking way to do this is easy just
#Test
public void myTest(){
WebClient client = WebClient.builder().baseUrl("https://stackoverflow.com/").build();
ClientResponse resp = client
.get()
.uri("questions/")
.exchange()
.block();
System.out.println("Status code response is: "+resp.statusCode());
}
But for this you can use directly the RestTemplate instead the webclient... the recomended way to do this is non blocking what means you should return a Mono with the status and consume outside your method like for example:
public Mono<HttpStatus> myMethod(){
WebClient client = WebClient.builder().baseUrl("https://stackoverflow.com/").build();
return client
.get()
.uri("questions/")
.exchange()
.map( clientResp -> clientResp.statusCode());
}
The way of consume this Mono depends of your code...
I'm using WebClient (SpringBoot 2.0.2.RELEASE) to send a POST with SOAP request, but it is missing "Content-Length" header required by the legacy API.
Is it possible to configure WebClient to include "Content-Length" header?
There is an Spring Framework Issue resolved and introduced for EncoderHttpMessageWriter in SpringBoot 2.0.1, but it seems not to work for JAXB.
I tried to use BodyInserters:
webClient.post().body(BodyInserters.fromObject(request)).exchange();
and syncBody:
webClient.post().syncBody(request).exchange();
None of them worked for WebClient. Though, when RestTemplate is used, Content-Length is set and API responds with success
I am struggling with the same problem, as an ugly work-around I am manually serializing the request (JSON in my case) and setting the length (Kotlin code):
open class PostRetrieverWith411ErrorFix(
private val objectMapper: ObjectMapper
) {
protected fun <T : Any> post(webClient: WebClient, body: Any, responseClass: Class<T>): Mono<T> {
val bodyJson = objectMapper.writeValueAsString(body)
return webClient.post()
.contentType(MediaType.APPLICATION_JSON_UTF8)
.contentLength(bodyJson.toByteArray(Charset.forName("UTF-8")).size.toLong())
.syncBody(bodyJson)
.retrieve()
.bodyToMono(responseClass)
}
}
If you apply Sven's colleague(Max) solution like we did you can also adapt it for cases like your body being a custom obj but you have to serialize it once:
String req = objectMapper.writeValueAsString(requestObject)
and passed that to
webClient.syncBody(req)
Keep in mind that with SpringBoot 2.0.3.RELEASE, if you'll pass a String to webClient as a request, it will put as ContentType header MediaType.TEXT_PLAIN and that made our integration with other service to fail. We fixed that by setting specifically content type header like this:
httpHeaders.setContentType(MediaType.APPLICATION_JSON);
WebClient is a streaming client and it's kind of difficult to set the content length until the stream has finished. By then the headers are long gone. If you work with legacy, you can re-use your mono (Mono/Flux can be reused, Java streams not) and check the length.
public void post() {
Mono<String> mono = Mono.just("HELLO WORLDZ");
final String response = WebClient.create("http://httpbin.org")
.post()
.uri("/post")
.header(HttpHeaders.CONTENT_LENGTH,
mono.map(s -> String.valueOf(s.getBytes(StandardCharsets.UTF_8).length)).block())
.body(BodyInserters.fromPublisher(mono, String.class))
.retrieve()
.bodyToMono(String.class)
.block();
System.out.println(response);
}
A colleague (well done Max!) of mine came up with cleaner solution, I added some wrapping code so it can be tested:
Mono<String> my = Mono.just("HELLO WORLDZZ")
.flatMap(body -> WebClient.create("http://httpbin.org")
.post()
.uri("/post")
.header(HttpHeaders.CONTENT_LENGTH,
String.valueOf(body.getBytes(StandardCharsets.UTF_8).length))
.syncBody(body)
.retrieve()
.bodyToMono(String.class));
System.out.println(my.block());