Been stumped for a while on this one!
Moving from a regular MVC project to a reactive one, and am working with Spring Boot (new version 2.0.0.M3).
I've had zero issues with the library as a whole until this particular prblem arose.
While working with WebClient, I have a request that isn't working. It worked just fine previously with RestTemplate:
rt.getMessageConverters().add(new MappingJackson2HttpMessageConverter());
HttpHeaders headers = new HttpHeaders();
headers.add("Content-Type", "application/json");
headers.add("Authorization", "Basic REDACTED");
HttpEntity<OtherApiRequest> entity =
new HttpEntity<OtherApiRequest>(CrawlRequestBuilder.buildCrawlRequest(req), headers);
ResponseEntity<Void> response = rt.postForEntity("https://other_api/path",
entity,
Void.class);
System.out.println(response.getStatusCode());
My WebClient code:
client
.post()
.uri("https://other_api/path")
.header("Authorization", "Basic REDACTED")
.contentType(MediaType.APPLICATION_JSON)
.body(Mono.just(req), OtherApiRequest.class)
.exchange()
.then(res -> System.out.println(res.getStatusCode()));
I've also tried generating the body first:
ObjectMapper mapper = new ObjectMapper();
String body = mapper.writeValueAsString(
client
.post()
.uri("https://other_api/path")
.header("Authorization", "Basic REDACTED")
.contentType(MediaType.APPLICATION_JSON)
.body(body, String.class)
.exchange()
.then(res -> System.out.println(res.getStatusCode()));
Is there anything here that stands out as wrong? I can't see any issues between the two that would cause the second one to fail...
Edit:
The RestTemplate provides a response of 204. The WebClient provides a response of 400, saying the body is invalid JSON. Using the second example for WebClient above, I can print the body variable and see it is proper JSON.
Edit2: The POJO class I am serializing:
#JsonNaming(PropertyNamingStrategy.SnakeCaseStrategy.class)
public class OtherApiRequest {
private String app;
private String urllist;
private int maxDepth;
private int maxUrls;
public OtherApiRequest(String app, String urllist, int maxDepth, int maxUrls) {
this.app = app;
this.urllist = urllist;
this.maxDepth = maxDepth;
this.maxUrls = maxUrls;
}
public String getApp() {
return app;
}
public String getUrllist() {
return urllist;
}
public int getMaxDepth() {
return maxDepth;
}
public int getMaxUrls() {
return maxUrls;
}
public String toString() {
return "OtherApiRequest: {" +
"app: " + app + "," +
"urllist: " + urllist + "," +
"max_depth: " + maxDepth + "," +
"max_urls: " + maxUrls +
"}";
}
}
EDIT:
See better answers here
Missing Content-Length header sending POST request with WebClient (SpringBoot 2.0.2.RELEASE)
Bug Report
https://github.com/spring-projects/spring-framework/issues/21085
Fixed in 2.2
When I experienced "Invalid JSON Response" I looked at the WebClient Request via netcat and found out, that the actual payload, in this example 3.16 was enclosed in some kind of content information:
$ netcat -l 6500
PUT /value HTTP/1.1
user-agent: ReactorNetty/0.7.5.RELEASE
transfer-encoding: chunked
host: localhost:6500
accept-encoding: gzip
Content-Type: application/json
4
3.16
0
After I added contentLength() to the builder the preceeding 4 and the trailing 0 vanished.
Related
There is a RestController and I try to upload a MultiPartFile to SharePointOnline using, the SharePoint REST API I'm also using proxy due to corporate restrictions.
#Override
public ResponseEntity uploadFile(MultipartFile file) throws ApiException, IOException {
RestTemplate restTemplate = createBasicRestTemplate();
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file",file.getResource());
HttpHeaders header = new HttpHeaders();
header.setContentType(MediaType.MULTIPART_FORM_DATA);
HttpEntity<MultiValueMap<String, Object>> uploadBodyEntity = new HttpEntity<>(body, header);
ResponseEntity<String> response = restTemplate.exchange(BASE_URL, HttpMethod.POST,
uploadBodyEntity, String.class);
return response;
}
public RestTemplate createBasicRestTemplate() {
RestTemplate restTemplate = new RestTemplateBuilder(new ProxyCustomizer()).build();
return restTemplate;
}
#Override
public void customize(RestTemplate restTemplate) {
HttpHost proxy = new HttpHost(PROXY_HOST, PROXY_PORT);
HttpClient httpClient = HttpClientBuilder.create()
.setRoutePlanner(new DefaultProxyRoutePlanner(proxy) {
#Override
public HttpHost determineProxy(HttpHost target, HttpRequest request, HttpContext context) throws HttpException {
return super.determineProxy(target, request, context);
}
})
.build();
restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory(httpClient));
}
The file upload is success, but it's cannot be opened. For example if upload a txt it will looks like this:
--raF_ORlUJptia2_av7ppLBeeMcGf5BUr
Content-Disposition: form-data; name="file"; filename="test.txt"
Content-Type: text/plain
Content-Length: 159
--38dc5323d6b92b5c14c33fade0178306
Content-Disposition: form-data; name="file"; filename="test.txt"
blablalblalalal
--38dc5323d6b92b5c14c33fade0178306--
--raF_ORlUJptia2_av7ppLBeeMcGf5BUr--
If I upload an xlsx it's simply just not open, it shows 'File Format and Extension Don't Match' error.
I try to convert the MultiPartFile to simple File with this method:
public File convertFile(MultipartFile file) {
File convFile = new File(file.getOriginalFilename());
try {
convFile.createNewFile();
FileOutputStream fos = new FileOutputStream(convFile);
fos.write(file.getBytes());
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
return convFile;
}
and change the controller to:
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file",convertFile(file));
But the same thing happens.
How can I upload file with RestTemplate?
This is a sample request to the SharePoint REST API and based on documentation the endpoint should receive a array buffer
POST https://{site_url}/_api/web/GetFolderByServerRelativeUrl('/Folder Name')/Files/add(url='a.txt',overwrite=true)
Authorization: "Bearer " + accessToken
Content-Length: {length of request body as integer}
X-RequestDigest: "{form_digest_value}"
"Contents of file"
This is what i can see in the https log: http log
Solution was to remove MultiValueMap and replace with:
HttpEntity<byte[]> entity = new HttpEntity<>(file.getBytes(), spoHelperService.createAuthHeader(authToken));
ResponseEntity<SpoUploadResponse> response = restTemplate.exchange(uploadFileUrl, HttpMethod.POST,
entity, SpoUploadResponse.class);
i'm trying to send a POST request with body data as described here: https://scrapyrt.readthedocs.io/en/stable/api.html#post.
Here's what i've tried to do but it gives me HTTP code 500
String uri = "http://localhost:3000";
WebClient webClient = WebClient.builder()
.baseUrl(uri)
.build();
LinkedMultiValueMap map = new LinkedMultiValueMap();
String q = "\"url\": \"https://blog.trendmicro.com/trendlabs-security-intelligence\",\"meta\":{\"latestDate\" : \"18-05-2020\"}}";
map.add("request", q);
map.add("spider_name", "blog");
BodyInserter<MultiValueMap<String, Object>, ClientHttpRequest> inserter2
= BodyInserters.fromMultipartData(map);
Mono<ItemsList> result = webClient.post()
.uri(uriBuilder -> uriBuilder
.path("/crawl.json")
.build())
.body(inserter2)
.retrieve()
.bodyToMono(ItemsList.class);
ItemsList tempItems = result.block();
Here's what i've tried to do but it gives me HTTP code 500
Most likely because you're sending the wrong data in a mixture of wrong formats with the wrong type:
You're using multipart form data, not JSON
You're then setting the request parameter as a JSON string (q)
The JSON string you're using in q isn't even valid (it's at least missing an opening curly brace) - and handwriting JSON is almost universally a bad idea, leverage a framework to do it for you instead.
Instead, the normal thing to do would be to create a POJO structure that maps to your request, so:
public class CrawlRequest {
private CrawlInnerRequest request;
#JsonProperty("spider_name")
private String spiderName;
//....add the getters / setters
}
public class CrawlInnerRequest {
private String url;
private String callback;
#JsonProperty("dont_filter")
private String dontFilter;
//....add the getters / setters
}
...then simply create a CrawlRequest, set the values as you wish, then in your post call use:
.body(BodyInserters.fromValue(crawlRequest))
This is a rather fundamental, basic part of using a WebClient. I'd suggest reading around more widely to give yourself a better understanding of the fundamentals, it will help tremendously in the long run.
For me following code worked:
public String wcPost(){
Map<String, String> bodyMap = new HashMap();
bodyMap.put("key1","value1");
WebClient client = WebClient.builder()
.baseUrl("domainURL")
.build();
String responseSpec = client.post()
.uri("URI")
.headers(h -> h.setBearerAuth("token if any"))
.body(BodyInserters.fromValue(bodyMap))
.exchange()
.flatMap(clientResponse -> {
if (clientResponse.statusCode().is5xxServerError()) {
clientResponse.body((clientHttpResponse, context) -> {
return clientHttpResponse.getBody();
});
return clientResponse.bodyToMono(String.class);
}
else
return clientResponse.bodyToMono(String.class);
})
.block();
return responseSpec;
}
I am calling a POST service getOrder3 written in SpringBoot which is working fine (tested in Postman), but getting error when called via restTemplate.postForObject from another service. I tried 2 versions of the client service getOrderClient and getOrderClient2, but both are giving same error :
HttpClientErrorException$BadRequest: 400 : [no body]
Please find the details below. Any help is appreciated.
getOrder3
#PostMapping(value="/getOrder3/{month}",produces="application/json")
public ResponseEntity<OrderResponse> getOrder3(
#PathVariable("month") String month,
#RequestParam String parmRequestSource,
#RequestParam(required=false) String parmAudienceType,
#RequestBody OrderRequestForm orderRequestForm) {
OrderResponse orderResponse = new OrderResponse();
log.info("In getOrder3...parmRequestSource = " + parmRequestSource + " parmAudienceType = " + parmAudienceType);
try {
//validate JSON schema
//orderService.validateMessageAgainstJSONSchema(orderRequestForm);
//process order
orderResponse = orderService.processOrder(orderRequestForm);
orderResponse.setParmRequestSource(parmRequestSource);
orderResponse.setParmAudienceType(parmAudienceType);
orderResponse.setMonth(month);
}catch (Exception e) {
throw new OrderException("101", e.getMessage(), HttpStatus.BAD_REQUEST);
}
return new ResponseEntity<>(orderResponse,HttpStatus.OK);
}
The service is working fine , tested in postman
Now when I try to call via another microservice via restTemplate.postForObject, I get the error. Tried 2 versions of the client as below, getOrderClient and getOrderClient2
getOrderClient
#PostMapping(value="/getOrderClient/{month}",produces="application/json")
public OrderResponse getOrderClient(
#PathVariable("month") String month,
#RequestParam String parmRequestSource,
#RequestParam String parmAudienceType,
#RequestBody OrderRequestForm orderRequestForm) throws URISyntaxException, JsonProcessingException {
RestTemplate restTemplate = new RestTemplate();
URI uri = new URI("http://localhost:51001/orders/v1/getOrder/"+month+"?parmRequestSource="+parmRequestSource+"&parmAudienceType="+parmAudienceType);
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
String requestJson = new ObjectMapper().writeValueAsString(orderRequestForm);
HttpEntity<String> httpEntity = new HttpEntity<String>(requestJson,headers);
String response = restTemplate.postForObject(uri, httpEntity, String.class);
return new ObjectMapper().readValue(response, OrderResponse.class);
}
getOrderClient2
#PostMapping(value="/getOrderClient2/{month}",produces="application/json")
public OrderResponse getOrderClient2(
#PathVariable("month") String month,
#RequestParam String parmRequestSource,
#RequestParam String parmAudienceType,
#RequestBody OrderRequestForm orderRequestForm) throws URISyntaxException, JsonProcessingException {
RestTemplate restTemplate = new RestTemplate();
URI uri = new URI("http://localhost:51001/orders/v1/getOrder/"+month+"?parmRequestSource="+parmRequestSource+"&parmAudienceType="+parmAudienceType);
return restTemplate.postForObject(uri, orderRequestForm, OrderResponse.class);
}
Both are giving same error :
HttpClientErrorException$BadRequest: 400 : [no body]
Please suggest.
To improve the visibility of the solution, #astar fixed the issue by annotating the model object's properties with #JsonProperty.
In my application, I need to take data from another request and chain into a new one
I must use the exchange() method of RestTemplate because I have issue with jacksons lib and I cannot add/change the libs.
this is my code:
final RequestCallback requestCallback = new RequestCallback() {
#Override
public void doWithRequest(final ClientHttpRequest request) throws IOException {
request.getHeaders().add(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE);
// Add basic auth header
String auth = username + ":" + password;
byte[] encodedAuth = Base64Utils.encode(auth.getBytes(StandardCharsets.US_ASCII));
String authHeader = "Basic " + new String(encodedAuth);
request.getHeaders().add("Authorization", authHeader);
// Add Headers Request
Enumeration headerNamesReq = servletRequest.getHeaderNames();
while (headerNamesReq.hasMoreElements()) {
String headerName = (String) headerNamesReq.nextElement();
if (whiteListedHeaders.contains(headerName.toLowerCase())) {
String headerValue = servletRequest.getHeader(headerName);
request.getHeaders().add(headerName, headerValue);
}
}
request.getHeaders().forEach((name, value) -> {
log.info("RestExecutorMiddleware", "HEADERS ---\t" + name + ":" + value);
});
IOUtils.copy(new ByteArrayInputStream(payload.getBytes()), request.getBody());
}
};
// Factory for restTemplate
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);
ClientHttpResponse responsePost = restTemplate.execute(url, method, requestCallback, new ResponseFromHeadersExtractor());
But at the end, the endpoint cannot receive my JSON (receive data, but not JSON.)
Where I wrong?
Thanks
Very inaccuracy code. Make all steps one-to-one and it will work, you make optimization later ...
Basic Auth. Don't do unnecessary actions
var headers = new HttpHeaders();
headers.setBasicAuth(username, password);
That's all, Spring will take care of everything else - to apply Base64, add Basic: and set properly a header.
Set all required headers including headers.setContentType(MediaType.APPLICATION_JSON)
Get an entity/object which you need to send (set as a body) with a request.
Serialize your object. The most popular, proper and simple way is using fasterxml json framework, you can make serialization with mapper.writeBalueAsString(<your object>). If you really cannot use external libraries, HttpEntity should make it: var request = new HttpEntity<>(<object>, headers);
Make restTemplate request. In almost all cases more convenient methods are restTemplate.postForObject(), restTemplate.getForObject(), restTemplate.postForEntity(), etc.: restTemplate.postForObject(uri, request, ResponseObject.class)
I am trying to post via WebClient to get microsoft token:
public WebClient getWebclient() {
TcpClient client = TcpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000)
.doOnConnected(connection -> connection.addHandlerLast(new ReadTimeoutHandler(15)).addHandlerLast(new WriteTimeoutHandler(15)));
ExchangeStrategies strategies = ExchangeStrategies.builder()
.codecs(configurer -> {
configurer.registerDefaults(true);
FormHttpMessageReader formHttpMessageReader = new FormHttpMessageReader();
formHttpMessageReader.setEnableLoggingRequestDetails(true);
configurer.customCodecs().reader(formHttpMessageReader);
})
.build();
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(client).followRedirect(true)))
.exchangeStrategies(strategies)
.filter(logRequest())
.filter(logResponse())
.build();
}
MultiValueMap<String, String> credentials = new LinkedMultiValueMap<>();
credentials.add("grant_type", "password");
credentials.add("client_id", oauthClientId);
credentials.add("resource", oauthResource);
credentials.add("scope", oauthScope);
credentials.add("username", oauthUsername);
credentials.add("password", oauthPassword);
Mono<MicrosoftToken> response = webClientService.getWebclient().post()
.uri(oauthUrl)
.contentType(MediaType.APPLICATION_FORM_URLENCODED)
.body(BodyInserters.fromFormData(credentials))
.retrieve()
.onStatus(HttpStatus::is4xxClientError, clientResponse ->
Mono.error(new WebClientException(clientResponse.bodyToMono(String.class), clientResponse.statusCode())))
.bodyToMono(MicrosoftToken.class);
this.cachedToken = response.block();
The problem ist, that microsoft cannot handle a Content-type: application/x-www-form-urlencoded;charset=UTF-8.
Spring is automatically adding the charset=UTF-8 to the request. I need to get rid of this additional charset. I need a Content-Type: application/x-www-form-urlencoded. Is this possible? Otherwise i need to downgrade my spring version to 2.0.0 where the charset is not automatically be added.
My Debug Logs print:
2019-03-14 10:08:42 DEBUG [reactor.netty.channel.ChannelOperationsHandler]:
[id: 0x5d6effce, L:/192.168.148.14:52285 -
R:login.microsoftonline.de/51.4.136.42:443] Writing object
DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /common/oauth2/token HTTP/1.1
user-agent: ReactorNetty/0.8.4.RELEASE
host: login.microsoftonline.de
Content-Type: application/x-www-form-urlencoded;charset=UTF-8
Content-Length: 205
2019-03-14 10:08:42 DEBUG [reactor.netty.channel.ChannelOperationsHandler]:
[id: 0x5d6effce, L:/192.168.148.14:52285 -
R:login.microsoftonline.de/51.4.136.42:443] Writing object
I tested this with spring version 2.0.0 and there the charset is not added as in the new version:
POST /common/oauth2/token HTTP/1.1
user-agent: ReactorNetty/0.7.5.RELEASE
host: login.microsoftonline.de
accept-encoding: gzip
Content-Type: application/x-www-form-urlencoded
Content-Length: 205
This took me the best part of a morning to find out, but I finally managed. The problem is that Webflux BodyInserters.fromFormData always sets the content type to application/x-www-form-urlencoded;charset=... regardless of what you set in the headers.
To solve this, first define this method:
/**
* This method is unfortunately necessary because of Spring Webflux's propensity to add {#code ";charset=..."}
* to the {#code Content-Type} header, which the Generic Chinese Device doesn't handle properly.
*
* #return a {#link FormInserter} that doesn't add the character set to the content type header
*/
private FormInserter<String> formInserter() {
return new FormInserter<String>() {
private final MultiValueMap<String, String> data = new LinkedMultiValueMap<>();
#Override public FormInserter<String> with(final String key, final String value) {
data.add(key, value);
return this;
}
#Override public FormInserter<String> with(final MultiValueMap<String, String> values) {
data.addAll(values);
return this;
}
#Override public Mono<Void> insert(final ClientHttpRequest outputMessage, final Context context) {
final ResolvableType formDataType =
ResolvableType.forClassWithGenerics(MultiValueMap.class, String.class, String.class);
return new FormHttpMessageWriter() {
#Override protected MediaType getMediaType(final MediaType mediaType) {
if (MediaType.APPLICATION_FORM_URLENCODED.equals(mediaType)) {
return mediaType;
} else {
return super.getMediaType(mediaType);
}
}
}.write(Mono.just(this.data), formDataType,
MediaType.APPLICATION_FORM_URLENCODED,
outputMessage,
context.hints());
}
};
}
Then, to call your web service, do the following:
final SomeResponseObject response = WebClient
.builder()
.build()
.post()
.uri(someOrOtherUri)
.body(formInserter().with("param1", "value1")
.with("param2", "value2")
)
.retrieve()
.bodyToFlux(SomeReponseObject.class)
.blockLast();
Please note that the block above is mainly for demonstration purposes. You may or may not want to block and wait for the response.
Here's two ways to do it:
webClient
.mutate()
.defaultHeaders(headers -> {
headers.add("Content-Type", ContentType.APPLICATION_FORM_URLENCODED.getMimeType()
}).build()
. uri(uri)
...
OR
webClient
.post()
.uri(uri)
.body(body)
.headers(headers -> getHttpHeaders())
...
private HttpHeaders getHttpHeaders(){
HttpHeaders headers = new HttpHeaders();
headers.add("Content-Type", "application/x-www-form-urlencoded")
return headers;
}
Just a few ways you could utilize the headers consumer in .headers or .defaultHeaders..
But I don't think the charset is the issue to be honest. If you are getting application/json in your response it is probably because Microsoft is forwarding the request with that header through the redirect url you specified in your app registration.
The good news is this is probably desirable, since Microsoft returns the token fields as json, which allows you to call .bodyToMono(MicrosoftToken). I recall having issues with BodyInserters.fromFormData as it did not actually encode the values in the MultiValueMap.
This is what I'm using instead:
private BodyInserter<String, ReactiveHttpOutputMessage> getBodyInserter(Map<String,String> parameters) {
credentials.add("grant_type", encode("password"));
credentials.add("client_id", encode(oauthClientId));
credentials.add("resource", encode(oauthResource));
// and so on..
// note that parameters is a regular Map - not a MultiValueMap
BodyInserter<String, ReactiveHttpOutputMessage> bodyInserter = BodyInserters.fromObject(
parameters.entrySet().stream()
.map(entry -> entry.getKey().concat("=").concat(entry.getValue()))
.collect(Collectors.joining("&", "", "")));
return bodyInserter;
}
private String encode(String str) {
try {
return URLEncoder.encode(str, StandardCharsets.UTF_8.name());
} catch (UnsupportedEncodingException e) {
log.error("Error encoding req body", e);
}
}