Webflux Spring 2.1.2 customize Content-Type - spring

I am trying to post via WebClient to get microsoft token:
public WebClient getWebclient() {
TcpClient client = TcpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000)
.doOnConnected(connection -> connection.addHandlerLast(new ReadTimeoutHandler(15)).addHandlerLast(new WriteTimeoutHandler(15)));
ExchangeStrategies strategies = ExchangeStrategies.builder()
.codecs(configurer -> {
configurer.registerDefaults(true);
FormHttpMessageReader formHttpMessageReader = new FormHttpMessageReader();
formHttpMessageReader.setEnableLoggingRequestDetails(true);
configurer.customCodecs().reader(formHttpMessageReader);
})
.build();
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(client).followRedirect(true)))
.exchangeStrategies(strategies)
.filter(logRequest())
.filter(logResponse())
.build();
}
MultiValueMap<String, String> credentials = new LinkedMultiValueMap<>();
credentials.add("grant_type", "password");
credentials.add("client_id", oauthClientId);
credentials.add("resource", oauthResource);
credentials.add("scope", oauthScope);
credentials.add("username", oauthUsername);
credentials.add("password", oauthPassword);
Mono<MicrosoftToken> response = webClientService.getWebclient().post()
.uri(oauthUrl)
.contentType(MediaType.APPLICATION_FORM_URLENCODED)
.body(BodyInserters.fromFormData(credentials))
.retrieve()
.onStatus(HttpStatus::is4xxClientError, clientResponse ->
Mono.error(new WebClientException(clientResponse.bodyToMono(String.class), clientResponse.statusCode())))
.bodyToMono(MicrosoftToken.class);
this.cachedToken = response.block();
The problem ist, that microsoft cannot handle a Content-type: application/x-www-form-urlencoded;charset=UTF-8.
Spring is automatically adding the charset=UTF-8 to the request. I need to get rid of this additional charset. I need a Content-Type: application/x-www-form-urlencoded. Is this possible? Otherwise i need to downgrade my spring version to 2.0.0 where the charset is not automatically be added.
My Debug Logs print:
2019-03-14 10:08:42 DEBUG [reactor.netty.channel.ChannelOperationsHandler]:
[id: 0x5d6effce, L:/192.168.148.14:52285 -
R:login.microsoftonline.de/51.4.136.42:443] Writing object
DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /common/oauth2/token HTTP/1.1
user-agent: ReactorNetty/0.8.4.RELEASE
host: login.microsoftonline.de
Content-Type: application/x-www-form-urlencoded;charset=UTF-8
Content-Length: 205
2019-03-14 10:08:42 DEBUG [reactor.netty.channel.ChannelOperationsHandler]:
[id: 0x5d6effce, L:/192.168.148.14:52285 -
R:login.microsoftonline.de/51.4.136.42:443] Writing object
I tested this with spring version 2.0.0 and there the charset is not added as in the new version:
POST /common/oauth2/token HTTP/1.1
user-agent: ReactorNetty/0.7.5.RELEASE
host: login.microsoftonline.de
accept-encoding: gzip
Content-Type: application/x-www-form-urlencoded
Content-Length: 205

This took me the best part of a morning to find out, but I finally managed. The problem is that Webflux BodyInserters.fromFormData always sets the content type to application/x-www-form-urlencoded;charset=... regardless of what you set in the headers.
To solve this, first define this method:
/**
* This method is unfortunately necessary because of Spring Webflux's propensity to add {#code ";charset=..."}
* to the {#code Content-Type} header, which the Generic Chinese Device doesn't handle properly.
*
* #return a {#link FormInserter} that doesn't add the character set to the content type header
*/
private FormInserter<String> formInserter() {
return new FormInserter<String>() {
private final MultiValueMap<String, String> data = new LinkedMultiValueMap<>();
#Override public FormInserter<String> with(final String key, final String value) {
data.add(key, value);
return this;
}
#Override public FormInserter<String> with(final MultiValueMap<String, String> values) {
data.addAll(values);
return this;
}
#Override public Mono<Void> insert(final ClientHttpRequest outputMessage, final Context context) {
final ResolvableType formDataType =
ResolvableType.forClassWithGenerics(MultiValueMap.class, String.class, String.class);
return new FormHttpMessageWriter() {
#Override protected MediaType getMediaType(final MediaType mediaType) {
if (MediaType.APPLICATION_FORM_URLENCODED.equals(mediaType)) {
return mediaType;
} else {
return super.getMediaType(mediaType);
}
}
}.write(Mono.just(this.data), formDataType,
MediaType.APPLICATION_FORM_URLENCODED,
outputMessage,
context.hints());
}
};
}
Then, to call your web service, do the following:
final SomeResponseObject response = WebClient
.builder()
.build()
.post()
.uri(someOrOtherUri)
.body(formInserter().with("param1", "value1")
.with("param2", "value2")
)
.retrieve()
.bodyToFlux(SomeReponseObject.class)
.blockLast();
Please note that the block above is mainly for demonstration purposes. You may or may not want to block and wait for the response.

Here's two ways to do it:
webClient
.mutate()
.defaultHeaders(headers -> {
headers.add("Content-Type", ContentType.APPLICATION_FORM_URLENCODED.getMimeType()
}).build()
. uri(uri)
...
OR
webClient
.post()
.uri(uri)
.body(body)
.headers(headers -> getHttpHeaders())
...
private HttpHeaders getHttpHeaders(){
HttpHeaders headers = new HttpHeaders();
headers.add("Content-Type", "application/x-www-form-urlencoded")
return headers;
}
Just a few ways you could utilize the headers consumer in .headers or .defaultHeaders..
But I don't think the charset is the issue to be honest. If you are getting application/json in your response it is probably because Microsoft is forwarding the request with that header through the redirect url you specified in your app registration.
The good news is this is probably desirable, since Microsoft returns the token fields as json, which allows you to call .bodyToMono(MicrosoftToken). I recall having issues with BodyInserters.fromFormData as it did not actually encode the values in the MultiValueMap.
This is what I'm using instead:
private BodyInserter<String, ReactiveHttpOutputMessage> getBodyInserter(Map<String,String> parameters) {
credentials.add("grant_type", encode("password"));
credentials.add("client_id", encode(oauthClientId));
credentials.add("resource", encode(oauthResource));
// and so on..
// note that parameters is a regular Map - not a MultiValueMap
BodyInserter<String, ReactiveHttpOutputMessage> bodyInserter = BodyInserters.fromObject(
parameters.entrySet().stream()
.map(entry -> entry.getKey().concat("=").concat(entry.getValue()))
.collect(Collectors.joining("&", "", "")));
return bodyInserter;
}
private String encode(String str) {
try {
return URLEncoder.encode(str, StandardCharsets.UTF_8.name());
} catch (UnsupportedEncodingException e) {
log.error("Error encoding req body", e);
}
}

Related

Spring boot RestTemplate upload file to SharePoint online but file is corrupted

There is a RestController and I try to upload a MultiPartFile to SharePointOnline using, the SharePoint REST API I'm also using proxy due to corporate restrictions.
#Override
public ResponseEntity uploadFile(MultipartFile file) throws ApiException, IOException {
RestTemplate restTemplate = createBasicRestTemplate();
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file",file.getResource());
HttpHeaders header = new HttpHeaders();
header.setContentType(MediaType.MULTIPART_FORM_DATA);
HttpEntity<MultiValueMap<String, Object>> uploadBodyEntity = new HttpEntity<>(body, header);
ResponseEntity<String> response = restTemplate.exchange(BASE_URL, HttpMethod.POST,
uploadBodyEntity, String.class);
return response;
}
public RestTemplate createBasicRestTemplate() {
RestTemplate restTemplate = new RestTemplateBuilder(new ProxyCustomizer()).build();
return restTemplate;
}
#Override
public void customize(RestTemplate restTemplate) {
HttpHost proxy = new HttpHost(PROXY_HOST, PROXY_PORT);
HttpClient httpClient = HttpClientBuilder.create()
.setRoutePlanner(new DefaultProxyRoutePlanner(proxy) {
#Override
public HttpHost determineProxy(HttpHost target, HttpRequest request, HttpContext context) throws HttpException {
return super.determineProxy(target, request, context);
}
})
.build();
restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory(httpClient));
}
The file upload is success, but it's cannot be opened. For example if upload a txt it will looks like this:
--raF_ORlUJptia2_av7ppLBeeMcGf5BUr
Content-Disposition: form-data; name="file"; filename="test.txt"
Content-Type: text/plain
Content-Length: 159
--38dc5323d6b92b5c14c33fade0178306
Content-Disposition: form-data; name="file"; filename="test.txt"
blablalblalalal
--38dc5323d6b92b5c14c33fade0178306--
--raF_ORlUJptia2_av7ppLBeeMcGf5BUr--
If I upload an xlsx it's simply just not open, it shows 'File Format and Extension Don't Match' error.
I try to convert the MultiPartFile to simple File with this method:
public File convertFile(MultipartFile file) {
File convFile = new File(file.getOriginalFilename());
try {
convFile.createNewFile();
FileOutputStream fos = new FileOutputStream(convFile);
fos.write(file.getBytes());
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
return convFile;
}
and change the controller to:
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file",convertFile(file));
But the same thing happens.
How can I upload file with RestTemplate?
This is a sample request to the SharePoint REST API and based on documentation the endpoint should receive a array buffer
POST https://{site_url}/_api/web/GetFolderByServerRelativeUrl('/Folder Name')/Files/add(url='a.txt',overwrite=true)
Authorization: "Bearer " + accessToken
Content-Length: {length of request body as integer}
X-RequestDigest: "{form_digest_value}"
"Contents of file"
This is what i can see in the https log: http log
Solution was to remove MultiValueMap and replace with:
HttpEntity<byte[]> entity = new HttpEntity<>(file.getBytes(), spoHelperService.createAuthHeader(authToken));
ResponseEntity<SpoUploadResponse> response = restTemplate.exchange(uploadFileUrl, HttpMethod.POST,
entity, SpoUploadResponse.class);

Configuring AWS Signing in Reactive Elasticsearch Configuration

In one of our service I tried to configure AWS signing in Spring data Reactive Elasticsearch configuration.
Spring provides the configuring the webclient through webclientClientConfigurer
ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo("localhost:9200")
.usingSsl()
.withWebClientConfigurer(
webClient -> {
return webClient.mutate().filter(new AwsSigningInterceptor()).build();
})
. // ... other options to configure if required
.build();
through which we can configure to sign the requests but however AWS signing it requires url, queryparams, headers and request body(in case of POST,POST) to generate the signed headers.
Using this I created a simple exchange filter function to sign the request but in this function I was not able to access the request body and use it.
Below is the Filter function i was trying to use
#Component
public class AwsSigningInterceptor implements ExchangeFilterFunction
{
private final AwsHeaderSigner awsHeaderSigner;
public AwsSigningInterceptor(AwsHeaderSigner awsHeaderSigner)
{
this.awsHeaderSigner = awsHeaderSigner;
}
#Override
public Mono<ClientResponse> filter(ClientRequest request, ExchangeFunction next)
{
Map<String, List<String>> signingHeaders = awsHeaderSigner.createSigningHeaders(request, new byte[]{}, "es", "us-west-2"); // should pass request body bytes in place of new byte[]{}
ClientRequest.Builder requestBuilder = ClientRequest.from(request);
signingHeaders.forEach((key, value) -> requestBuilder.header(key, value.toArray(new String[0])));
return next.exchange(requestBuilder.build());
}
}
I also tried to access the request body inside ExchangeFilterFunction using below approach but once i get the request body using below approach.
ClientRequest.from(newRequest.build())
.body(
(outputMessage, context) -> {
ClientHttpRequestDecorator loggingOutputMessage =
new ClientHttpRequestDecorator(outputMessage) {
#Override
public Mono<Void> writeWith(Publisher<? extends DataBuffer> body) {
log.info("Inside write with method");
body =
DataBufferUtils.join(body)
.map(
content -> {
// Log request body using
// 'content.toString(StandardCharsets.UTF_8)'
String requestBody =
content.toString(StandardCharsets.UTF_8);
Map<String, Object> signedHeaders =
awsSigner.getSignedHeaders(
request.url().getPath(),
request.method().name(),
multimap,
requestHeadersMap,
Optional.of(
requestBody.getBytes(StandardCharsets.UTF_8)));
log.info("Signed Headers generated:{}", signedHeaders);
signedHeaders.forEach(
(key, value) -> {
newRequest.header(key, value.toString());
});
return content;
});
log.info("Before returning the body");
return super.writeWith(body);
}
#Override
public Mono<Void>
setComplete() { // This is for requests with no body (e.g. GET).
Map<String, Object> signedHeaders =
awsSigner.getSignedHeaders(
request.url().getPath(),
request.method().name(),
multimap,
requestHeadersMap,
Optional.of("".getBytes(StandardCharsets.UTF_8)));
log.info("Signed Headers generated:{}", signedHeaders);
signedHeaders.forEach(
(key, value) -> {
newRequest.header(key, value.toString());
});
return super.setComplete();
}
};
return originalBodyInserter.insert(loggingOutputMessage, context);
})
.build();
But with above approach I was not able to change the request headers as adding headers throws UnsupportedOperationException inside writewith method.
Has anyone used the spring data reactive elastic search and configured to sign with AWS signed headers?
Any help would be highly appreciated.

Spring Webflux Webclient timesout intermittently

I am getting intermittent ReadTimeOut from netty with the below error:
The connection observed an error","logger_name":"reactor.netty.http.client.HttpClientConnect","thread_name":"reactor-http-epoll-3","level":"WARN","level_value":30000,"stack_trace":"io.netty.handler.timeout.ReadTimeoutException: null
One observation we made is this particular endpoint for which we are getting this issue is a POST with no request body. I am now sending a dummy json in body now which the downstream system ignores and now I don't see this error anymore at all.
Below is my code:
protected <T, S Mono<S sendMonoRequest (HttpMethod method,
HttpHeaders headers,
T requestBody,
URI uri, Class < S responseClass)
throws ApiException, IOException {
log.info("Calling {} {} {} {}", method.toString(), uri.toString(), headers.toString(),
mapper.writeValueAsString(requestBody));
WebClient.RequestBodySpec requestBodySpec = getWebClient().method(method).uri(uri);
headers.keySet().stream().forEach(headerKey -> headers.get(headerKey).stream().
forEach(headerValue -> requestBodySpec.header(headerKey, headerValue)));
return requestBodySpec
.body(BodyInserters.fromObject(requestBody != null ? requestBody : ""))
.retrieve()
.onStatus(HttpStatus::is4xxClientError, this::doOn4xxError)
.onStatus(HttpStatus::is5xxServerError, this::doOn5xxError)
.onStatus(HttpStatus::isError, this::doOnError)
.bodyToMono(responseClass);
}
protected WebClient getWebClient () {
HttpClient httpClient = HttpClient.create().tcpConfiguration(
client -> client.option(ChannelOption.CONNECT_TIMEOUT_MILLIS,
20000).doOnConnected(conn - conn
.addHandlerLast(new ReadTimeoutHandler(20)).addHandlerLast(new WriteTimeoutHandler(20))));
ClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);
return WebClient.builder().clientConnector(connector)
.filter(logResponse())
.build();
}
To resolve the intemrittent timeouts, I have to send a dummy pojo to sendMonoRequest() for request body. Any ideas ?

How to send URL encoded data in spring webflux

I am writing a spring 5 web app and my requirement is to get a urlencoded form and in response send url encoded response back
This is Router Function code
#Configuration
public class AppRoute {
#Bean
public RouterFunction<ServerResponse> route(FormHandler formHandler) {
return RouterFunctions.route()
// .GET("/form", formHandler::sampleForm)
// .POST("/form", accept(MediaType.APPLICATION_FORM_URLENCODED), formHandler::displayFormData)
.POST("/formnew", accept(MediaType.APPLICATION_FORM_URLENCODED).and(contentType(MediaType.APPLICATION_FORM_URLENCODED)), formHandler::newForm)
.build();
}
}
and here's my Handler code
public Mono<ServerResponse> newForm(ServerRequest request) {
Mono<MultiValueMap<String, String>> formData = request.formData();
MultiValueMap<String, String> newFormData = new LinkedMultiValueMap<String, String>();
formData.subscribe(p -> newFormData.putAll(p));
newFormData.add("status", "success");
return ServerResponse.ok().contentType(MediaType.APPLICATION_FORM_URLENCODED)
.body(fromObject(newFormData));
}
Here's the error I get
2020-04-07 02:37:33.329 DEBUG 38688 --- [ctor-http-nio-3] org.springframework.web.HttpLogging : [07467aa5] Resolved [UnsupportedMediaTypeException: Content type 'application/x-www-form-urlencoded' not supported for bodyType=org.springframework.util.LinkedMultiValueMap] for HTTP POST /formnew
Whats the issue here. I couldn't find any way to write the url encoded response back.
Could anyone point what's the issue.
Try to refactor your code to functional style:
public Mono<ServerResponse> newForm(ServerRequest request) {
Mono<DataBuffer> resultMono = request.formData()
.map(formData -> new LinkedMultiValueMap(formData))
.doOnNext(newFormData -> newFormData.add("status", "success"))
.map(linkedMultiValueMap -> createBody(linkedMultiValueMap));
return ServerResponse.ok().contentType(MediaType.APPLICATION_FORM_URLENCODED)
.body(BodyInserters.fromDataBuffers(resultMono));
}
private DataBuffer createBody(MultiValueMap multiValueMap) {
try {
DefaultDataBufferFactory factory = new DefaultDataBufferFactory();
return factory.wrap(ByteBuffer.wrap(objectMapper.writeValueAsString(multiValueMap).getBytes(StandardCharsets.UTF_8)));
} catch (JsonProcessingException e) {
throw new IllegalArgumentException("incorrect body");
}
}

Issues moving from RestTemplate to WebClient (Spring Boot 2.0.0.M3)

Been stumped for a while on this one!
Moving from a regular MVC project to a reactive one, and am working with Spring Boot (new version 2.0.0.M3).
I've had zero issues with the library as a whole until this particular prblem arose.
While working with WebClient, I have a request that isn't working. It worked just fine previously with RestTemplate:
rt.getMessageConverters().add(new MappingJackson2HttpMessageConverter());
HttpHeaders headers = new HttpHeaders();
headers.add("Content-Type", "application/json");
headers.add("Authorization", "Basic REDACTED");
HttpEntity<OtherApiRequest> entity =
new HttpEntity<OtherApiRequest>(CrawlRequestBuilder.buildCrawlRequest(req), headers);
ResponseEntity<Void> response = rt.postForEntity("https://other_api/path",
entity,
Void.class);
System.out.println(response.getStatusCode());
My WebClient code:
client
.post()
.uri("https://other_api/path")
.header("Authorization", "Basic REDACTED")
.contentType(MediaType.APPLICATION_JSON)
.body(Mono.just(req), OtherApiRequest.class)
.exchange()
.then(res -> System.out.println(res.getStatusCode()));
I've also tried generating the body first:
ObjectMapper mapper = new ObjectMapper();
String body = mapper.writeValueAsString(
client
.post()
.uri("https://other_api/path")
.header("Authorization", "Basic REDACTED")
.contentType(MediaType.APPLICATION_JSON)
.body(body, String.class)
.exchange()
.then(res -> System.out.println(res.getStatusCode()));
Is there anything here that stands out as wrong? I can't see any issues between the two that would cause the second one to fail...
Edit:
The RestTemplate provides a response of 204. The WebClient provides a response of 400, saying the body is invalid JSON. Using the second example for WebClient above, I can print the body variable and see it is proper JSON.
Edit2: The POJO class I am serializing:
#JsonNaming(PropertyNamingStrategy.SnakeCaseStrategy.class)
public class OtherApiRequest {
private String app;
private String urllist;
private int maxDepth;
private int maxUrls;
public OtherApiRequest(String app, String urllist, int maxDepth, int maxUrls) {
this.app = app;
this.urllist = urllist;
this.maxDepth = maxDepth;
this.maxUrls = maxUrls;
}
public String getApp() {
return app;
}
public String getUrllist() {
return urllist;
}
public int getMaxDepth() {
return maxDepth;
}
public int getMaxUrls() {
return maxUrls;
}
public String toString() {
return "OtherApiRequest: {" +
"app: " + app + "," +
"urllist: " + urllist + "," +
"max_depth: " + maxDepth + "," +
"max_urls: " + maxUrls +
"}";
}
}
EDIT:
See better answers here
Missing Content-Length header sending POST request with WebClient (SpringBoot 2.0.2.RELEASE)
Bug Report
https://github.com/spring-projects/spring-framework/issues/21085
Fixed in 2.2
When I experienced "Invalid JSON Response" I looked at the WebClient Request via netcat and found out, that the actual payload, in this example 3.16 was enclosed in some kind of content information:
$ netcat -l 6500
PUT /value HTTP/1.1
user-agent: ReactorNetty/0.7.5.RELEASE
transfer-encoding: chunked
host: localhost:6500
accept-encoding: gzip
Content-Type: application/json
4
3.16
0
After I added contentLength() to the builder the preceeding 4 and the trailing 0 vanished.

Resources