I'm using Spring WebFlux 5.3.6's WebClient to stream a response from a REST endpoint that generates text/csv content.
I can use retrieve() and responseSpec.bodyToFlux to stream the body only like this:
WebClient.ResponseSpec responseSpec = headersSpec.retrieve();
Flux<DataBuffer> dataBufferFlux = responseSpec.bodyToFlux(DataBuffer.class);
DataBufferUtils
.write(dataBufferFlux, outputStream)
.blockLast(Duration.of(20, ChronoUnit.SECONDS));
but I want to get hold of the content-type header and validate it as part of the test. The above code provides access to the response body only, and not the headers.
I've tried to instead use exchangeToFlux() to get more control, and access to the response headers, but what I'm seeing is that the HTTP request is never made. If I add a breakpoint to myResponse.setStatus(clientResponse.rawStatusCode()); it is never hit.
A fuller code sample is below. I've struggled to find any examples of exchangeToFlux that use DataBuffer to stream the result back.
HttpClient httpClient = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 5000)
.responseTimeout(Duration.ofMillis(5000))
.doOnConnected(conn ->
conn.addHandlerLast(new ReadTimeoutHandler(5000, TimeUnit.MILLISECONDS))
.addHandlerLast(new WriteTimeoutHandler(5000, TimeUnit.MILLISECONDS)));
WebClient webClient = WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
WebClient.RequestHeadersSpec<?> headersSpec = webClient
.get()
.uri("http://localhost:8080/v1/users")
.header(CONTENT_TYPE, "text/csv");
MyResponse<T> myResponse = new MyResponse<>();
CountDownLatch latch = new CountDownLatch(1);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
headersSpec.exchangeToFlux(clientResponse -> {
// Never enters here!
myResponse.setStatus(clientResponse.rawStatusCode());
myResponse.setContentType(clientResponse.headers().contentType());
latch.countDown();
if (clientResponse.statusCode() == HttpStatus.OK) {
Flux<DataBuffer> dataBufferFlux = clientResponse.bodyToFlux(DataBuffer.class);
DataBufferUtils
.write(dataBufferFlux, outputStream)
.blockLast(Duration.of(20, ChronoUnit.SECONDS));
return dataBufferFlux;
}
return Flux.empty();
});
latch.await();
return myResponse;
Seem's like you are not subscribing to the Flux returned from headersSpec.exchangeToFlux.
headersSpec.exchangeToFlux(clientResponse -> {
// Never enters here!
myResponse.setStatus(clientResponse.rawStatusCode());
myResponse.setContentType(clientResponse.headers().contentType());
latch.countDown();
if (clientResponse.statusCode() == HttpStatus.OK) {
Flux<DataBuffer> dataBufferFlux = clientResponse.bodyToFlux(DataBuffer.class);
DataBufferUtils
.write(dataBufferFlux, outputStream)
.blockLast(Duration.of(20, ChronoUnit.SECONDS));
return dataBufferFlux;
}
return Flux.empty();
})
.subscribe(); // <- Subscribe is missing.
Related
I am sending an ArrayList object through WebClient in an HTTP POST request after converting it into an InputStream:
List<MyObject> myObjectList = // some data
ByteArrayOutputStream byteOutStream = new ByteArrayOutputStream();
ObjectOutputStream objectOutStream = new ObjectOutputStream(byteOutStream);
objectOutStream.writeObject(myObjectList );
objectOutStream.flush();
objectOutStream.close();
byte[] byteArray = byteOutStream.toByteArray();
InputStream inputStream = new ByteArrayInputStream(byteArray);
WebClient client = WebClient.builder()
.baseUrl(URL)
.exchangeStrategies(ExchangeStrategies.builder()
.codecs(configure -> configure.defaultCodecs().maxInMemorySize(64 * 1024 * 1024))
.build())
.build();
Mono<HttpStatus> response = client
.post()
.uri(URI)
.header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_OCTET_STREAM_VALUE)
.body(BodyInserters.fromResource(new InputStreamResource(inputStream)))
.exchangeToMono(clientResponse -> Mono.just(clientResponse.statusCode()));
HttpStatus status = response.block();
And at the server side, I am handling this request this way:
#PostMapping(value = "/data", consumes = MediaType.APPLICATION_OCTET_STREAM_VALUE)
public ResponseEntity<Void> saveData(#RequestBody InputStream inputStream) {
try (ObjectInputStream objectInputStream = new ObjectInputStream(inputStream)) {
List<MyObject> myObjectList = (List<MyObject>) objectInputStream.readObject();
LOGGER.info("Payload received : {}", myObjectList);
return new ResponseEntity<>(HttpStatus.OK);
} catch (Exception e) {
LOGGER.error(e.getMessage());
return new ResponseEntity<>(HttpStatus.BAD_REQUEST);
}
}
But after implementing this, I am getting this error:
[org.springframework.web.HttpMediaTypeNotSupportedException: Content type 'application/octet-stream;charset=UTF-8' not supported]
PS: I am converting the ArrayList into an InputStream due to it's large size which is causing java.lang.OutOfMemoryError: Direct buffer memory
First, I tried sending the entire List<> in a request but got OutOfMemoryError. Secondly, using the streaming approach I am getting this error
[org.springframework.web.HttpMediaTypeNotSupportedException: Content type 'application/octet-stream;charset=UTF-8' not supported]
Below is my code:
public static void main(String[] args) {
TcpClient tcpClient = TcpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 100_000)
.proxy(p ->
p.type(ProxyProvider.Proxy.HTTP)
.host("myServerHost.com")
.port(8080))
.doOnConnected ( connection ->
connection.addHandlerLast(new ReadTimeoutHandler(60000, TimeUnit.MILLISECONDS))
.addHandlerLast(new ReadTimeoutHandler(60000, TimeUnit.MILLISECONDS))
);
WebClient myWebClient = WebClient
.builder()
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(tcpClient)))
.baseUrl("https://baseurl.com")
.build();
Mono<String> rs = myWebClient.post()
.uri("uriForRestEndPoints")
.header(HttpHeaders.CONTENT_TYPE.toString(), MediaType.valueOf("text/plain;charset=UTF-8").toString())
.retrieve()
.bodyToMono(String.class)
.doOnEach(t -> {System.out.println("doOnEach: " + t.toString()); })
.doOnSuccess(x -> { System.out.println("Success: ") ;})
.doOnError(x -> { System.out.println("Error: ") ; });
System.out.println("got response "+ rs.block() + "... " );
}
In this sample POC for REST API project - and I am getting 'IllegalStateException' hence I have tried everything from internet. - Like having timeout, proxy, content headers etc.
But still getting same issue "java.lang.IllegalStateException: The underlying HTTP client completed without emitting a response."
Also doOnSuccess, doOnError, doOnEach - is not printing anything.
Can anyone please help me on this.
I am getting intermittent ReadTimeOut from netty with the below error:
The connection observed an error","logger_name":"reactor.netty.http.client.HttpClientConnect","thread_name":"reactor-http-epoll-3","level":"WARN","level_value":30000,"stack_trace":"io.netty.handler.timeout.ReadTimeoutException: null
One observation we made is this particular endpoint for which we are getting this issue is a POST with no request body. I am now sending a dummy json in body now which the downstream system ignores and now I don't see this error anymore at all.
Below is my code:
protected <T, S Mono<S sendMonoRequest (HttpMethod method,
HttpHeaders headers,
T requestBody,
URI uri, Class < S responseClass)
throws ApiException, IOException {
log.info("Calling {} {} {} {}", method.toString(), uri.toString(), headers.toString(),
mapper.writeValueAsString(requestBody));
WebClient.RequestBodySpec requestBodySpec = getWebClient().method(method).uri(uri);
headers.keySet().stream().forEach(headerKey -> headers.get(headerKey).stream().
forEach(headerValue -> requestBodySpec.header(headerKey, headerValue)));
return requestBodySpec
.body(BodyInserters.fromObject(requestBody != null ? requestBody : ""))
.retrieve()
.onStatus(HttpStatus::is4xxClientError, this::doOn4xxError)
.onStatus(HttpStatus::is5xxServerError, this::doOn5xxError)
.onStatus(HttpStatus::isError, this::doOnError)
.bodyToMono(responseClass);
}
protected WebClient getWebClient () {
HttpClient httpClient = HttpClient.create().tcpConfiguration(
client -> client.option(ChannelOption.CONNECT_TIMEOUT_MILLIS,
20000).doOnConnected(conn - conn
.addHandlerLast(new ReadTimeoutHandler(20)).addHandlerLast(new WriteTimeoutHandler(20))));
ClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);
return WebClient.builder().clientConnector(connector)
.filter(logResponse())
.build();
}
To resolve the intemrittent timeouts, I have to send a dummy pojo to sendMonoRequest() for request body. Any ideas ?
I want to call a microservice from another service using webclient in spring flux. But, I am not able to write the code properly. Can you please suggest how to call another service. Please find my code as below.
I need to call the below service
public Mono<ServerResponse> load(ServerRequest res){
String c1name = res.pathVariable("cust");
String c2name = res.queryParam("cl").orElse("");
String oname = res.queryParam("ol").orElse("");
return res.body()
}
public Mono<ResponseEntity<Void>> ftpFileSend(MultipartFile fileData, String cust, MultiValueMap<String,String) qpar {
MultiValueMap<String,String> qpar=new LinkedMultiValueMap<String,String>();
qpar.add("name","spring");
MultiValueMap<String,Object> body=new LinkedMultiValueMap<String,Object>();
String url="http://localhost:8088/"+ cust+"/load";
try {
body.add("file", fileData.getBytes());
} catch (IOException e) {
return Mono.error(e); // <-- note how to create an error signal
}
return webClient
.post()
.uri(uriBuilder -> uriBuilder.path(url).queryParams(qpar).build() )
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(BodyInserters.fromMultipartData(body))
.retrieve()
.toBodilessEntity();
}
Hmm it would be great if you have provided some error logs or so. Anyway if you want to create a multipart body there is a builder, MultipartBodyBuilder (in org.springframework.http.client.MultipartBodyBuilder).
Example usage is as follows,
MultipartBodyBuilder builder = new MultipartBodyBuilder();
builder.part("file", new MultipartFileResource(fileData));
MultiValueMap<String, HttpEntity<?>> multipartBody = builder.build();
Then use this multipartBody in webClient call.
return webClient
...
.body(BodyInserters.fromMultipartData(multipartBody))
.retrieve()
.toBodilessEntity();
I am trying to post via WebClient to get microsoft token:
public WebClient getWebclient() {
TcpClient client = TcpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000)
.doOnConnected(connection -> connection.addHandlerLast(new ReadTimeoutHandler(15)).addHandlerLast(new WriteTimeoutHandler(15)));
ExchangeStrategies strategies = ExchangeStrategies.builder()
.codecs(configurer -> {
configurer.registerDefaults(true);
FormHttpMessageReader formHttpMessageReader = new FormHttpMessageReader();
formHttpMessageReader.setEnableLoggingRequestDetails(true);
configurer.customCodecs().reader(formHttpMessageReader);
})
.build();
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(client).followRedirect(true)))
.exchangeStrategies(strategies)
.filter(logRequest())
.filter(logResponse())
.build();
}
MultiValueMap<String, String> credentials = new LinkedMultiValueMap<>();
credentials.add("grant_type", "password");
credentials.add("client_id", oauthClientId);
credentials.add("resource", oauthResource);
credentials.add("scope", oauthScope);
credentials.add("username", oauthUsername);
credentials.add("password", oauthPassword);
Mono<MicrosoftToken> response = webClientService.getWebclient().post()
.uri(oauthUrl)
.contentType(MediaType.APPLICATION_FORM_URLENCODED)
.body(BodyInserters.fromFormData(credentials))
.retrieve()
.onStatus(HttpStatus::is4xxClientError, clientResponse ->
Mono.error(new WebClientException(clientResponse.bodyToMono(String.class), clientResponse.statusCode())))
.bodyToMono(MicrosoftToken.class);
this.cachedToken = response.block();
The problem ist, that microsoft cannot handle a Content-type: application/x-www-form-urlencoded;charset=UTF-8.
Spring is automatically adding the charset=UTF-8 to the request. I need to get rid of this additional charset. I need a Content-Type: application/x-www-form-urlencoded. Is this possible? Otherwise i need to downgrade my spring version to 2.0.0 where the charset is not automatically be added.
My Debug Logs print:
2019-03-14 10:08:42 DEBUG [reactor.netty.channel.ChannelOperationsHandler]:
[id: 0x5d6effce, L:/192.168.148.14:52285 -
R:login.microsoftonline.de/51.4.136.42:443] Writing object
DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /common/oauth2/token HTTP/1.1
user-agent: ReactorNetty/0.8.4.RELEASE
host: login.microsoftonline.de
Content-Type: application/x-www-form-urlencoded;charset=UTF-8
Content-Length: 205
2019-03-14 10:08:42 DEBUG [reactor.netty.channel.ChannelOperationsHandler]:
[id: 0x5d6effce, L:/192.168.148.14:52285 -
R:login.microsoftonline.de/51.4.136.42:443] Writing object
I tested this with spring version 2.0.0 and there the charset is not added as in the new version:
POST /common/oauth2/token HTTP/1.1
user-agent: ReactorNetty/0.7.5.RELEASE
host: login.microsoftonline.de
accept-encoding: gzip
Content-Type: application/x-www-form-urlencoded
Content-Length: 205
This took me the best part of a morning to find out, but I finally managed. The problem is that Webflux BodyInserters.fromFormData always sets the content type to application/x-www-form-urlencoded;charset=... regardless of what you set in the headers.
To solve this, first define this method:
/**
* This method is unfortunately necessary because of Spring Webflux's propensity to add {#code ";charset=..."}
* to the {#code Content-Type} header, which the Generic Chinese Device doesn't handle properly.
*
* #return a {#link FormInserter} that doesn't add the character set to the content type header
*/
private FormInserter<String> formInserter() {
return new FormInserter<String>() {
private final MultiValueMap<String, String> data = new LinkedMultiValueMap<>();
#Override public FormInserter<String> with(final String key, final String value) {
data.add(key, value);
return this;
}
#Override public FormInserter<String> with(final MultiValueMap<String, String> values) {
data.addAll(values);
return this;
}
#Override public Mono<Void> insert(final ClientHttpRequest outputMessage, final Context context) {
final ResolvableType formDataType =
ResolvableType.forClassWithGenerics(MultiValueMap.class, String.class, String.class);
return new FormHttpMessageWriter() {
#Override protected MediaType getMediaType(final MediaType mediaType) {
if (MediaType.APPLICATION_FORM_URLENCODED.equals(mediaType)) {
return mediaType;
} else {
return super.getMediaType(mediaType);
}
}
}.write(Mono.just(this.data), formDataType,
MediaType.APPLICATION_FORM_URLENCODED,
outputMessage,
context.hints());
}
};
}
Then, to call your web service, do the following:
final SomeResponseObject response = WebClient
.builder()
.build()
.post()
.uri(someOrOtherUri)
.body(formInserter().with("param1", "value1")
.with("param2", "value2")
)
.retrieve()
.bodyToFlux(SomeReponseObject.class)
.blockLast();
Please note that the block above is mainly for demonstration purposes. You may or may not want to block and wait for the response.
Here's two ways to do it:
webClient
.mutate()
.defaultHeaders(headers -> {
headers.add("Content-Type", ContentType.APPLICATION_FORM_URLENCODED.getMimeType()
}).build()
. uri(uri)
...
OR
webClient
.post()
.uri(uri)
.body(body)
.headers(headers -> getHttpHeaders())
...
private HttpHeaders getHttpHeaders(){
HttpHeaders headers = new HttpHeaders();
headers.add("Content-Type", "application/x-www-form-urlencoded")
return headers;
}
Just a few ways you could utilize the headers consumer in .headers or .defaultHeaders..
But I don't think the charset is the issue to be honest. If you are getting application/json in your response it is probably because Microsoft is forwarding the request with that header through the redirect url you specified in your app registration.
The good news is this is probably desirable, since Microsoft returns the token fields as json, which allows you to call .bodyToMono(MicrosoftToken). I recall having issues with BodyInserters.fromFormData as it did not actually encode the values in the MultiValueMap.
This is what I'm using instead:
private BodyInserter<String, ReactiveHttpOutputMessage> getBodyInserter(Map<String,String> parameters) {
credentials.add("grant_type", encode("password"));
credentials.add("client_id", encode(oauthClientId));
credentials.add("resource", encode(oauthResource));
// and so on..
// note that parameters is a regular Map - not a MultiValueMap
BodyInserter<String, ReactiveHttpOutputMessage> bodyInserter = BodyInserters.fromObject(
parameters.entrySet().stream()
.map(entry -> entry.getKey().concat("=").concat(entry.getValue()))
.collect(Collectors.joining("&", "", "")));
return bodyInserter;
}
private String encode(String str) {
try {
return URLEncoder.encode(str, StandardCharsets.UTF_8.name());
} catch (UnsupportedEncodingException e) {
log.error("Error encoding req body", e);
}
}