Simple logging dump of WebClient request? - spring

I'm trying to use Spring WebClient to make some basic REST API calls. I'm getting an error that the request is malformed, but I can't tell exactly why. Is there any way to easily log the contents of the request (really, just the request body)? Everything I find online is super complicated. Here's what I have:
LinkedMultiValueMap params = new LinkedMultiValueMap();
params.add("app_id", getOneSignalAppId());
params.add("included_segments", inSegment);
params.add("content_available", true);
params.add("contents", new LinkedMultiValueMap() {{
add("en", inTitle);
}});
BodyInserters.MultipartInserter inserter = BodyInserters.fromMultipartData(params);
WebClient client = WebClient.builder()
.baseUrl("https://onesignal.com")
.defaultHeader(HttpHeaders.AUTHORIZATION, "Basic " + getOneSignalKey())
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.defaultHeader(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE)
.build();
Mono<NotificationResponse> result = client
.post()
.uri("/api/v1/notifications")
.body(inserter)
.retrieve()
.bodyToMono(NotificationResponse.class);
I just want a string of the JSON that will be inserted into the request body.

You can create your own wrapper/proxy class around the JSON encoder (assuming you're using JSON) and intercept the serialized body before it is sent into the intertubes.
If your request is going to send JSON.
Specifically, you would extend the encodeValue method (or encodeValues in case of streaming data) of Jackson2JsonEncoder (the default encoder). Then you can do with that data what you wish, such as logging etc. And you could even do this conditionally based on environment/profile.
This custom logging-encoder can be specified when creating the WebClient, by providing it as a codec:
CustomBodyLoggingEncoder bodyLoggingEncoder = new CustomBodyLoggingEncoder();
WebClient.builder()
.codecs(clientDefaultCodecsConfigurer -> {
clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonEncoder(bodyLoggingEncoder);
clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonDecoder(new Jackson2JsonDecoder(new ObjectMapper(), MediaType.APPLICATION_JSON));
})
...
I made a blog post about this. You might be able to find the encoder for Multipart data and apply similar principles
For completeness, the encoder might look something like this:
public class CustomBodyLoggingEncoder extends Jackson2JsonEncoder {
#Override
public DataBuffer encodeValue(final Object value, final DataBufferFactory bufferFactory,
final ResolvableType valueType, #Nullable final MimeType mimeType, #Nullable final Map<String, Object> hints) {
// Encode/Serialize data to JSON
final DataBuffer data = super.encodeValue(value, bufferFactory, valueType, mimeType, hints);
// This is your code:
SomethingAmazing.doItWithThisData(extractBytes(data));
// Return the data as normal
return data;
}
private byte[] extractBytes(final DataBuffer data) {
final byte[] bytes = new byte[data.readableByteCount()];
data.read(bytes);
// We've copied the data above to our array, but must reset the buffer for actual usage
data.readPosition(0);
return bytes;
}
}
Hope that helps somehow!

Related

Spring WebClient Post method Body

i'm trying to send a POST request with body data as described here: https://scrapyrt.readthedocs.io/en/stable/api.html#post.
Here's what i've tried to do but it gives me HTTP code 500
String uri = "http://localhost:3000";
WebClient webClient = WebClient.builder()
.baseUrl(uri)
.build();
LinkedMultiValueMap map = new LinkedMultiValueMap();
String q = "\"url\": \"https://blog.trendmicro.com/trendlabs-security-intelligence\",\"meta\":{\"latestDate\" : \"18-05-2020\"}}";
map.add("request", q);
map.add("spider_name", "blog");
BodyInserter<MultiValueMap<String, Object>, ClientHttpRequest> inserter2
= BodyInserters.fromMultipartData(map);
Mono<ItemsList> result = webClient.post()
.uri(uriBuilder -> uriBuilder
.path("/crawl.json")
.build())
.body(inserter2)
.retrieve()
.bodyToMono(ItemsList.class);
ItemsList tempItems = result.block();
Here's what i've tried to do but it gives me HTTP code 500
Most likely because you're sending the wrong data in a mixture of wrong formats with the wrong type:
You're using multipart form data, not JSON
You're then setting the request parameter as a JSON string (q)
The JSON string you're using in q isn't even valid (it's at least missing an opening curly brace) - and handwriting JSON is almost universally a bad idea, leverage a framework to do it for you instead.
Instead, the normal thing to do would be to create a POJO structure that maps to your request, so:
public class CrawlRequest {
private CrawlInnerRequest request;
#JsonProperty("spider_name")
private String spiderName;
//....add the getters / setters
}
public class CrawlInnerRequest {
private String url;
private String callback;
#JsonProperty("dont_filter")
private String dontFilter;
//....add the getters / setters
}
...then simply create a CrawlRequest, set the values as you wish, then in your post call use:
.body(BodyInserters.fromValue(crawlRequest))
This is a rather fundamental, basic part of using a WebClient. I'd suggest reading around more widely to give yourself a better understanding of the fundamentals, it will help tremendously in the long run.
For me following code worked:
public String wcPost(){
Map<String, String> bodyMap = new HashMap();
bodyMap.put("key1","value1");
WebClient client = WebClient.builder()
.baseUrl("domainURL")
.build();
String responseSpec = client.post()
.uri("URI")
.headers(h -> h.setBearerAuth("token if any"))
.body(BodyInserters.fromValue(bodyMap))
.exchange()
.flatMap(clientResponse -> {
if (clientResponse.statusCode().is5xxServerError()) {
clientResponse.body((clientHttpResponse, context) -> {
return clientHttpResponse.getBody();
});
return clientResponse.bodyToMono(String.class);
}
else
return clientResponse.bodyToMono(String.class);
})
.block();
return responseSpec;
}

RestTemplate execute() method cannot send JSON Payload

In my application, I need to take data from another request and chain into a new one
I must use the exchange() method of RestTemplate because I have issue with jacksons lib and I cannot add/change the libs.
this is my code:
final RequestCallback requestCallback = new RequestCallback() {
#Override
public void doWithRequest(final ClientHttpRequest request) throws IOException {
request.getHeaders().add(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE);
// Add basic auth header
String auth = username + ":" + password;
byte[] encodedAuth = Base64Utils.encode(auth.getBytes(StandardCharsets.US_ASCII));
String authHeader = "Basic " + new String(encodedAuth);
request.getHeaders().add("Authorization", authHeader);
// Add Headers Request
Enumeration headerNamesReq = servletRequest.getHeaderNames();
while (headerNamesReq.hasMoreElements()) {
String headerName = (String) headerNamesReq.nextElement();
if (whiteListedHeaders.contains(headerName.toLowerCase())) {
String headerValue = servletRequest.getHeader(headerName);
request.getHeaders().add(headerName, headerValue);
}
}
request.getHeaders().forEach((name, value) -> {
log.info("RestExecutorMiddleware", "HEADERS ---\t" + name + ":" + value);
});
IOUtils.copy(new ByteArrayInputStream(payload.getBytes()), request.getBody());
}
};
// Factory for restTemplate
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);
ClientHttpResponse responsePost = restTemplate.execute(url, method, requestCallback, new ResponseFromHeadersExtractor());
But at the end, the endpoint cannot receive my JSON (receive data, but not JSON.)
Where I wrong?
Thanks
Very inaccuracy code. Make all steps one-to-one and it will work, you make optimization later ...
Basic Auth. Don't do unnecessary actions
var headers = new HttpHeaders();
headers.setBasicAuth(username, password);
That's all, Spring will take care of everything else - to apply Base64, add Basic: and set properly a header.
Set all required headers including headers.setContentType(MediaType.APPLICATION_JSON)
Get an entity/object which you need to send (set as a body) with a request.
Serialize your object. The most popular, proper and simple way is using fasterxml json framework, you can make serialization with mapper.writeBalueAsString(<your object>). If you really cannot use external libraries, HttpEntity should make it: var request = new HttpEntity<>(<object>, headers);
Make restTemplate request. In almost all cases more convenient methods are restTemplate.postForObject(), restTemplate.getForObject(), restTemplate.postForEntity(), etc.: restTemplate.postForObject(uri, request, ResponseObject.class)

POST byte array in multipart using Spring RestTemplate

I'm trying to POST a multipart/form-data using Spring RestTemplate with a byte array as the file to upload and it keeps failing (Server rejects with different kinds of errors).
I'm using a MultiValueMap with ByteArrayResource. Is there something I'm missing?
Yes there is something missing.
I have found this article:
https://medium.com/#voziv/posting-a-byte-array-instead-of-a-file-using-spring-s-resttemplate-56268b45140b
The author mentions that in order to POST a byte array using Spring RestTemplate one needs to override getFileName() of the ByteArrayResource.
Here is the code example from the article:
private static void uploadWordDocument(byte[] fileContents, final String filename) {
RestTemplate restTemplate = new RestTemplate();
String fooResourceUrl = "http://localhost:8080/spring-rest/foos"; // Dummy URL.
MultiValueMap<String, Object> map = new LinkedMultiValueMap<String, Object>();
map.add("name", filename);
map.add("filename", filename);
// Here we
ByteArrayResource contentsAsResource = new ByteArrayResource(fileContents) {
#Override
public String getFilename() {
return filename; // Filename has to be returned in order to be able to post.
}
};
map.add("file", contentsAsResource);
// Now you can send your file along.
String result = restTemplate.postForObject(fooResourceUrl, map, String.class);
// Proceed as normal with your results.
}
I tried it and it works!
I added an issue to send a request from java client to Python service in FastApi and sending a ByteArrayResource instaead of simple byte[] fixed the issue.
FastAPI server returned: "Expected UploadFile, received: <class 'str'>","type":"value_error""

Spring Boot MVC to allow any kind of content-type in controller

I have a RestController that multiple partners use to send XML requests. However this is a legacy system that it was passed on to me and the original implementation was done in a very loose way in PHP.
This has allowed to clients, that now they refuse to change, to send different content-types (application/xml, text/xml, application/x-www-form-urlencoded) and it has left me with the need to support many MediaTypes to avoid returning 415 MediaType Not Supported Errors.
I have used the following code in a configuration class to allow many media types.
#Bean
public MarshallingHttpMessageConverter marshallingMessageConverter() {
MarshallingHttpMessageConverter converter = new MarshallingHttpMessageConverter();
converter.setMarshaller(jaxbMarshaller());
converter.setUnmarshaller(jaxbMarshaller());
converter.setSupportedMediaTypes(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.APPLICATION_XML,
MediaType.TEXT_XML, MediaType.TEXT_PLAIN, MediaType.APPLICATION_FORM_URLENCODED, MediaType.ALL));
return converter;
}
#Bean
public Jaxb2Marshaller jaxbMarshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setClassesToBeBound(CouponIssuedStatusDTO.class, CouponIssuedFailedDTO.class,
CouponIssuedSuccessDTO.class, RedemptionSuccessResultDTO.class, RedemptionResultHeaderDTO.class,
RedemptionFailResultDTO.class, RedemptionResultBodyDTO.class, RedemptionDTO.class, Param.class,
ChannelDTO.class, RedeemRequest.class);
Map<String, Object> props = new HashMap<>();
props.put(javax.xml.bind.Marshaller.JAXB_FORMATTED_OUTPUT, true);
marshaller.setMarshallerProperties(props);
return marshaller;
}
The controller method is this:
#PostMapping(value = "/request", produces = { "application/xml;charset=UTF-8" }, consumes = MediaType.ALL_VALUE)
public ResponseEntity<RedemptionResultDTO> request(
#RequestHeader(name = "Content-Type", required = false) String contentType,
#RequestBody String redeemRequest) {
return requestCustom(contentType, redeemRequest);
}
This endpoint is hit by all clients. It is only one last client giving me trouble. They are sending content-type = application/x-www-form-urlencoded; charset=65001 (UTF-8)": 65001 (UTF-8)
Due to the way the charset is sent, Spring Boot refuses to return anything but 415. Not even MediaType.ALL seems to have any effect.
Is there a way to make Spring allow this to reach me ignoring the content-type? Creating a filter and changing the content type was not feasible since the HttpServletRequest is not allowing to mutate the content-type. I am out of ideas but I really think there has to be a way to allow custom content-types.
UPDATE
If I remove the #RequestBody then I don't get the error 415 but I have no way to get the request body since the HttpServletRequest reaches the Controller action empty.
You best case is to remove the consumes argument from the RequestMapping constructor. The moment you have it added, spring will try to parse it into known type MediaType.parseMediaType(request.getContentType()) & which tries to create a new MimeType(type, subtype, parameters) and thus throws exception due to invalid charset format being passed.
However, if you remove the consumes, and you wanna validate/restrict the incoming Content-Type to certain type, you can inject HttpServletRequest in your method as parameter, and then check the value of request.getHeader(HttpHeaders.CONTENT_TYPE).
You also have to remove the #RequestBody annotation so Spring doesn't attempt to parse the content-type in attempt to unmarshall the body. If you directly attempt to read the request.getInputStream() or request.getReader() here, you will see null as the stream has already been read by Spring. So to get access to input content, use spring's ContentCachingRequestWrapper inject using Filter and then you can later repeatedly read the content as it's cached & not reading from original stream.
I am including some code snippet here for reference, however to see executable example, you can refer my github repo. Its a spring-boot project with maven, once you launch it, you can send your post request to http://localhost:3007/badmedia & it will reflect you back in response request content-type & body. Hope this helps.
#RestController
public class BadMediaController {
#PostMapping("/badmedia")
#ResponseBody
public Object reflect(HttpServletRequest request) throws IOException {
ObjectMapper mapper = new ObjectMapper();
JsonNode rootNode = mapper.createObjectNode();
((ObjectNode) rootNode).put("contentType", request.getHeader(HttpHeaders.CONTENT_TYPE));
String body = new String(((ContentCachingRequestWrapper) request).getContentAsByteArray(), StandardCharsets.UTF_8);
body = URLDecoder.decode(body, StandardCharsets.UTF_8.name());
((ObjectNode) rootNode).put("body", body);
return mapper.writerWithDefaultPrettyPrinter().writeValueAsString(rootNode);
}
}
#Component
public class CacheRequestFilter extends GenericFilterBean {
#Override
public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain chain)
throws IOException, ServletException {
HttpServletRequest cachedRequest
= new ContentCachingRequestWrapper((HttpServletRequest) servletRequest);
//invoke caching
cachedRequest.getParameterMap();
chain.doFilter(cachedRequest, servletResponse);
}
}

Sending request with headers to third parts api with WebClient

I really like the solution I have with RestTemplate but soon it will be depreciated with future spring releases. I am trying to send some text to a third party api using WebClient
String text = URLEncoder.encode(text,"UTF-8");
WebClient webClient = WebClient.builder()
.baseUrl(BASE_URL)
.defaultHeader("Key","af999-e99-4456-b556-4ef9947383d")
.defaultHeader("src", srcLang)
.defaultHeader("tgt", tgtLang)
.defaultHeader("text", text)
.build();
Then send a post here:
Mono<String> response = webClient.post().uri("/google/rtv/text")
.retrieve()
.bodyToMono(String.class);
Trying to parse based off of the legacy response:
private String parseJson( Mono<String> response) {
ObjectMapper mapper = new ObjectMapper();
JsonNode root = null;
JsonNode review = null;
//TODO: create an object and map it here. We need to save the original review too.
try {
root = mapper.readTree(response.toString());
review = root.path("message");
} catch (IOException e) {
e.printStackTrace();
}
return review.asText();
}
Later I need to parse the response but right now I am getting an error saying:
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'MonoFlatMap': was expecting ('true', 'false' or 'null')
at [Source: (String)"MonoFlatMap"; line: 1, column: 23]
and later:
java.lang.NullPointerException: null
What I am trying to accomplish is something like I have done with RestTemplate.
Like so:
UriComponentsBuilder builder = UriComponentsBuilder
.fromUriString(URL)
.queryParam("src", src)
.queryParam("tgt", tgt)
.queryParam("text", text);
ResponseEntity<String> response = restTemplate.exchange(builder.toUriString(), HttpMethod.GET, request, String.class);
then set my header for the subscription globally.
private ClientHttpResponse intercept(HttpRequest request, byte[] body,
ClientHttpRequestExecution execution) throws IOException {
request.getHeaders().add("Key","af999-e99-4456-b556-4ef9947383d");
ClientHttpResponse response = execution.execute(request, body);
return response;
}
#Bean
public RestTemplate restTemplate() {
RestTemplate restTemplate = new RestTemplate();
restTemplate.setInterceptors(Collections.singletonList(this::intercept));
return restTemplate;
}
Advice?
The problem happens here:
root = mapper.readTree(response.toString());
This code snippet is trying to serialize a Mono<String> as a String, when a Mono is a reactive type that can provide that String value eventually.
You could call response.block() and getting the resulting String, but this would be a blocking call and Reactor forbids that if in the middle of a reactive execution. This is done for good reasons, since this will block one of the few threads that your web application is using and can cause it to stop serving other requests.
You could instead have something like:
Mono<String> review = response.map(r -> parseJson(r);
And then reuse that new value down the line.
Note that WebClient natively supports JSON deserialization and you could deserialize the whole payload like so:
Mono<Review> review = webClient.post().uri("/google/rtv/text")
.retrieve()
.bodyToMono(Review.class);

Resources