I need to do a POST using WebClient and the server requires the body to be compressed. I've checked previous questions asked here and here, but none helped me understand what needs to be done.
My code looks something like this:
webClient.post()
.bodyValue(requestBody)
.retrieve()
.bodyToMono(Response.class)
I'd like to send the requestBody compressed using gzip. We were doing it with RestTemplate and a custom GZipFilter but I cannot see how to do it now with the WebClient.
I have implemented sample code to help you with this. You will need to clean this up and adapt to your needs, but I have tested this and it does work.
The first step is to implement an Encoder<T> where <T> is the type of the object you want to encode. In my example I am using JsonNode.
public class GzipEncoder extends AbstractEncoder<JsonNode> {
public GzipEncoder() {
super(MediaType.APPLICATION_JSON);
}
#Override
public boolean canEncode(ResolvableType elementType, MimeType mimeType) {
return MediaType.APPLICATION_JSON.equalsTypeAndSubtype(mimeType) && elementType.isAssignableFrom(JsonNode.class);
}
#Override
public Flux<DataBuffer> encode(Publisher<? extends JsonNode> inputStream, DataBufferFactory bufferFactory, ResolvableType elementType, MimeType mimeType, Map<String, Object> hints) {
return Flux.from(inputStream).map((JsonNode node) ->
encodeValue(node, bufferFactory, elementType, mimeType, hints));
}
#Override
public DataBuffer encodeValue(JsonNode node, DataBufferFactory bufferFactory, ResolvableType valueType, MimeType mimeType, Map<String, Object> hints) {
return bufferFactory.wrap(gzip(node.toString()));
}
private byte[] gzip(String value) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try (GZIPOutputStream gzipOutputStream = new GZIPOutputStream(baos)) {
gzipOutputStream.write(value.getBytes());
} catch (IOException e) {
throw new RuntimeException(e);
}
return baos.toByteArray();
}
}
Then you also have to implement an HttpMessageWriter
public class GzipHttpMessageWriter extends EncoderHttpMessageWriter {
public GzipHttpMessageWriter() {
super(new GzipEncoder());
}
#Override
public Mono<Void> write(Publisher inputStream, ResolvableType elementType, MediaType mediaType, ReactiveHttpOutputMessage message, Map hints) {
return super.write(inputStream, elementType, mediaType, updateContentEncoding(message), hints);
}
private ReactiveHttpOutputMessage updateContentEncoding(ReactiveHttpOutputMessage message) {
message.getHeaders().add("Content-Encoding", "gzip");
return message;
}
}
Now create your WebClient as follows (I have added wiretap to confirm gzip is working)
WebClient webclientGzip = WebClient.builder()
.codecs(clientCodecConfigurer -> clientCodecConfigurer.customCodecs().register(new GzipHttpMessageWriter()))
.clientConnector(new ReactorClientHttpConnector(HttpClient.create().wiretap(true)))
.build();
Now when I post a JsonNode body with the following, I can see the request going out encoded with gzip
webclientGzip.post().uri(uri)
.accept(MediaType.APPLICATION_JSON)
.contentType(MediaType.APPLICATION_JSON)
.body(Mono.just(body), JsonNode.class)
Related
im working in spring boot project where i want to instantiate a Restemplate Bean with Interceptors , my issue that i don't want to duplicate the code because there is just the header that changes for each conciguration. this is my code :
#Bean
#Qualifier("restTemplateOne")
public RestTemplate restTemplateWithAccessToken() {
return new RestTemplateBuilder()
.interceptors((HttpRequest request, byte[] body, ClientHttpRequestExecution execution) -> {
//this is the only header that i want to add for
request.getHeaders().set("MY_PARTICULAR_HEADER", "my value");
request.getHeaders().set(HttpHeaders.AUTHORIZATION,"my auth value");
return execution.execute(request, body);
}).build();
}
#Bean
#Qualifier("restTemplateTwo")
public RestTemplate restTemplateWithIdToken() {
return new RestTemplateBuilder()
.interceptors((HttpRequest request, byte[] body, ClientHttpRequestExecution execution) -> {
request.getHeaders().set(HttpHeaders.AUTHORIZATION,"my auth value");
return execution.execute(request, body);
}).build();
}
#Autowired
#Qualifier("restTemplateOne")
private RestTemplate restTemplateOne;
#Autowired
#Qualifier("restTemplateTwo")
private RestTemplate restTemplateTwo;
do you have any idea how i can optimize code to avoid duplication . something like adding a parameter to the method and adding the header or not according to the condition.
Thanks in advance.
Just extract and parameterize your interceptor:
#Bean
#Qualifier("restTemplateOne")
public RestTemplate restTemplateWithAccessToken() {
return new RestTemplateBuilder()
.interceptors(new CustomClientHttpRequestInterceptor(true))
.build();
}
#Bean
#Qualifier("restTemplateTwo")
public RestTemplate restTemplateWithIdToken() {
return new RestTemplateBuilder()
.interceptors(new CustomClientHttpRequestInterceptor(false))
.build();
}
private static class CustomClientHttpRequestInterceptor implements ClientHttpRequestInterceptor {
private boolean needParticularHeader;
public CustomClientHttpRequestInterceptor(boolean needParticularHeader) {
this.needParticularHeader = needParticularHeader;
}
#Override
public ClientHttpResponse intercept(HttpRequest request,
byte[] body,
ClientHttpRequestExecution execution) throws IOException {
if (needParticularHeader) {
//this is the only header that i want to add for
request.getHeaders().set("MY_PARTICULAR_HEADER", "my value");
}
request.getHeaders().set(HttpHeaders.AUTHORIZATION, "my auth value");
return execution.execute(request, body);
}
}
I need to secure REST API implemented with Spring Boot, WebFlux and spring security using HMAC of the request body. Simplifying a bit, on a high level - request comes with the header that has hashed value of the request body, so I have to read the header, read the body, calculate hash of the body and compare with the header value.
I think I should implement ServerAuthenticationConverter but all examples I was able to find so far only looking at the request headers, not the body and I'm not sure if I could just read the body, or should I wrap/mutate the request with cached body so it could be consumed by the underlying component second time?
Is it ok to use something along the lines of:
public class HttpHmacAuthenticationConverter implements ServerAuthenticationConverter {
#Override
public Mono<Authentication> convert(ServerWebExchange exchange) {
exchange.getRequest().getBody()
.next()
.flatMap(dataBuffer -> {
try {
return Mono.just(StreamUtils.copyToString(dataBuffer.asInputStream(), StandardCharsets.UTF_8));
} catch (IOException e) {
return Mono.error(e);
}
})
...
I'm getting a warning from the IDE on the copyToString line: Inappropriate blocking method call
Any guidelines or examples?
Thanks!
I have also tried:
#Override
public Mono<Authentication> convert(ServerWebExchange exchange) {
return Mono.justOrEmpty(exchange.getRequest().getHeaders().toSingleValueMap())
.zipWith(exchange.getRequest().getBody().next()
.flatMap(dataBuffer -> Mono.just(dataBuffer.asByteBuffer().array()))
)
.flatMap(tuple -> create(tuple.getT1(), tuple.getT2()));
But that doesn't work - code in the create() method on the last line is never executed.
I make it work. Posting my code for the reference.
Two components are required to make it work - WebFilter that would read and cache request body so it could be consumed multiple times and the ServerAuthenticationConverter that would calculate hash on a body and validate signature.
public class HttpRequestBodyCachingFilter implements WebFilter {
private static final byte[] EMPTY_BODY = new byte[0];
#Override
public Mono<Void> filter(ServerWebExchange exchange, WebFilterChain chain) {
// GET and DELETE don't have a body
HttpMethod method = exchange.getRequest().getMethod();
if (method == null || method.matches(HttpMethod.GET.name()) || method.matches(HttpMethod.DELETE.name())) {
return chain.filter(exchange);
}
return DataBufferUtils.join(exchange.getRequest().getBody())
.map(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
DataBufferUtils.release(dataBuffer);
return bytes;
})
.defaultIfEmpty(EMPTY_BODY)
.flatMap(bytes -> {
ServerHttpRequestDecorator decorator = new ServerHttpRequestDecorator(exchange.getRequest()) {
#Nonnull
#Override
public Flux<DataBuffer> getBody() {
if (bytes.length > 0) {
DataBufferFactory dataBufferFactory = exchange.getResponse().bufferFactory();
return Flux.just(dataBufferFactory.wrap(bytes));
}
return Flux.empty();
}
};
return chain.filter(exchange.mutate().request(decorator).build());
});
}
}
public class HttpJwsAuthenticationConverter implements ServerAuthenticationConverter {
private static final byte[] EMPTY_BODY = new byte[0];
#Override
public Mono<Authentication> convert(ServerWebExchange exchange) {
return DataBufferUtils.join(exchange.getRequest().getBody())
.map(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
DataBufferUtils.release(dataBuffer);
return bytes;
})
.defaultIfEmpty(EMPTY_BODY)
.flatMap(body -> create(
exchange.getRequest().getMethod(),
getFullRequestPath(exchange.getRequest()),
exchange.getRequest().getHeaders(),
body)
);
}
...
The create method in the Converter implements the logic to validate signature based on the request method, path, headers and the body. It returns an instance of the Authentication if successful or Mono.empty() if not.
The wiring up is done like this:
public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity http) {
http.authorizeExchange().pathMatchers(PATH_API).authenticated()
...
.and()
.addFilterBefore(new HttpRequestBodyCachingFilter(), SecurityWebFiltersOrder.AUTHENTICATION)
.addFilterAt(jwtAuthenticationFilter(...), SecurityWebFiltersOrder.AUTHENTICATION);
}
private AuthenticationWebFilter jwtAuthenticationFilter(ReactiveAuthenticationManager authManager) {
AuthenticationWebFilter authFilter = new AuthenticationWebFilter(authManager);
authFilter.setServerAuthenticationConverter(new HttpJwsAuthenticationConverter());
authFilter.setRequiresAuthenticationMatcher(ServerWebExchangeMatchers.pathMatchers(PATH_API));
return authFilter;
}
#Bean
public ReactiveAuthenticationManager reactiveAuthenticationManager() {
return Mono::just;
}
}
I am using spring cloud gateway as edge server.
This is the flow
If request has a header named 'x-foo' then find the header value, get a string from another server and send that string as response instead of actually proxying the request.
Here is code for Filter DSL
#Bean
public RouteLocator routes(RouteLocatorBuilder builder) {
return builder.routes()
.route("foo-filter", r -> r.header('x-foo').and().header("x-intercepted").negate()
.filters(f -> f.filter(fooFilter))
.uri("http://localhost:8081")) // 8081 is self port, there are other proxy related configurations too
.build();
}
Code for Foo filter
#Component
#Slf4j
public class FooFilter implements GatewayFilter {
#Autowired
private ReactiveRedisOperations<String, String> redisOps;
#Value("${header-name}")
private String headerName;
#Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
ServerHttpRequest request = exchange.getRequest();
var foo = request.getHeaders().getFirst(headerName);
return redisOps.opsForHash()
.get("foo:" + foo, "response")
.doOnSuccess(s -> {
log.info("data on success");
log.info(s.toString()); // I am getting proper response here
if (s != null) {
ServerHttpResponse response = exchange.getResponse();
response.setStatusCode(HttpStatus.OK);
response.getHeaders().set("x-intercepted", "true");
byte[] bytes = s.toString().getBytes(StandardCharsets.UTF_8);
DataBuffer buffer = response.bufferFactory().wrap(bytes);
response.writeWith(Mono.just(buffer));
response.setComplete();
}
})
.then(chain.filter(exchange));
}
}
The problem is, the response has the response is getting proper 200 code, the injected header is present on response but the data is not available in response.
This is how I got working.
Use flatMap instead of doOnSuccess
don't use then or switchIfEmpty instead use onErrorResume
Return the response.writeWith
#Component
#Slf4j
public class FooFilter implements GatewayFilter {
#Autowired
private ReactiveRedisOperations<String, String> redisOps;
#Value("${header-name}")
private String headerName;
#Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
ServerHttpRequest request = exchange.getRequest();
var foo = request.getHeaders().getFirst(headerName);
return redisOps.opsForHash()
.get("foo:" + foo, "response")
.flatMap(s -> {
log.info("data on success");
log.info(s.toString()); // I am getting proper response here
if (s != null) {
ServerHttpResponse response = exchange.getResponse();
response.setStatusCode(HttpStatus.OK);
response.getHeaders().set("x-intercepted", "true");
byte[] bytes = s.toString().getBytes(StandardCharsets.UTF_8);
DataBuffer buffer = response.bufferFactory().wrap(bytes);
return response.writeWith(Mono.just(buffer));
}else{ return chain.filter(exchange).then(Mono.fromRunnable(() -> {log.info("It was empty")} }
})
.onErrorResume(chain.filter(exchange));
}
}
While migrating my spring server from servlets to reactive I had to change all the filters in the code to WebFilter. One of the filters was decompressing gzipped content, but I couldn't do the same with the new WebFilter.
With servlets I wrapped the inputstream with a GzipInputStream. What is the best practice to do it with spring reactive?
Solution:
#Component
public class GzipFilter implements WebFilter {
private static final Logger LOG = LoggerFactory.getLogger(GzipFilter.class);
public static final String CONTENT_ENCODING = "content-encoding";
public static final String GZIP = "gzip";
public static final String UTF_8 = "UTF-8";
#Override
public Mono<Void> filter(ServerWebExchange exchange, WebFilterChain chain) {
ServerHttpRequest request = exchange.getRequest();
if (!isGzip(request)) {
return chain.filter(exchange);
}
else {
ServerHttpRequest mutatedRequest = new ServerHttpRequestWrapper(request);
ServerWebExchange mutatedExchange = exchange.mutate().request(mutatedRequest).build();
return chain.filter(mutatedExchange);
}
}
private boolean isGzip(ServerHttpRequest serverHttpRequest) {
String encoding = serverHttpRequest.getHeaders().getFirst(CONTENT_ENCODING);
return encoding != null && encoding.contains(GZIP);
}
private static class ServerHttpRequestWrapper implements ServerHttpRequest {
private ServerHttpRequest request;
public ServerHttpRequestWrapper(ServerHttpRequest request) {
this.request = request;
}
private static byte[] getDeflatedBytes(GZIPInputStream gzipInputStream) throws IOException {
StringWriter writer = new StringWriter();
IOUtils.copy(gzipInputStream, writer, UTF_8);
return writer.toString().getBytes();
}
#Override
public String getId() {
return request.getId();
}
#Override
public RequestPath getPath() {
return request.getPath();
}
#Override
public MultiValueMap<String, String> getQueryParams() {
return request.getQueryParams();
}
#Override
public MultiValueMap<String, HttpCookie> getCookies() {
return request.getCookies();
}
#Override
public String getMethodValue() {
return request.getMethodValue();
}
#Override
public URI getURI() {
return request.getURI();
}
#Override
public Flux<DataBuffer> getBody() {
Mono<DataBuffer> mono = request.getBody()
.map(dataBuffer -> dataBuffer.asInputStream(true))
.reduce(SequenceInputStream::new)
.map(inputStream -> {
try (GZIPInputStream gzipInputStream = new GZIPInputStream(inputStream)) {
byte[] targetArray = getDeflatedBytes(gzipInputStream);
return new DefaultDataBufferFactory().wrap(targetArray);
}
catch (IOException e) {
throw new IllegalGzipRequest(String.format("failed to decompress gzip content. Path: %s", request.getPath()));
}
});
return mono.flux();
}
#Override
public HttpHeaders getHeaders() {
return request.getHeaders();
}
}
}
love #Yuval's solution!
My original idea was to convert Flux to a local file, and then decompress the local file.
But getting a file downloaded in Spring Reactive is too challenging. I googled a lot, and most of them are blocking way to get file, (e.g. Spring WebClient: How to stream large byte[] to file? and How to correctly read Flux<DataBuffer> and convert it to a single inputStream , none of them works...) which makes no sense and will throw error when calling block() in a reactive flow.
#Yuval saved my day! It works well for me!
I migrated from Springfox Swagger to Springdoc OpenApi. I have added few lines in my configuration about springdoc:
springdoc:
pathsToMatch: /api/**
api-docs:
path: /api-docs
swagger-ui:
path: /swagger-ui.html
In configuration class MainConfig.kt I have following code:
val customGson: Gson = GsonBuilder()
.registerTypeAdapter(LocalDateTime::class.java, DateSerializer())
.registerTypeAdapter(ZonedDateTime::class.java, ZonedDateSerializer())
.addSerializationExclusionStrategy(AnnotationExclusionStrategy())
.enableComplexMapKeySerialization()
.setPrettyPrinting()
.create()
override fun configureMessageConverters(converters: MutableList<HttpMessageConverter<*>>) {
converters.add(GsonHttpMessageConverter(customGson))
}
When I go to http://localhost:8013/swagger-ui.html (in configuration I have server.port: 8013) the page is not redirect to swagger-ui/index.html?url=/api-docs&validatorUrl=. But this is not my main problem :). When I go to swagger-ui/index.html?url=/api-docs&validatorUrl= I got page with this information:
Unable to render this definition
The provided definition does not specify a valid version field.
Please indicate a valid Swagger or OpenAPI version field. Supported version fields are swagger: "2.0" and those that match openapi: 3.0.n (for example, openapi: 3.0.0).
But when I go to http://localhost:8013/api-docs I have this result:
"{\"openapi\":\"3.0.1\",\"info\":{(...)}}"
I tried using default config and I commented configureMessageConverters() method and result of \api-docs now looks like normal JSON:
// 20191218134933
// http://localhost:8013/api-docs
{
"openapi": "3.0.1",
"info": {(...)}
}
I remember when I was using Springfox there was something wrong with serialization and my customGson had additional line: .registerTypeAdapter(Json::class.java, JsonSerializer<Json> { src, _, _ -> JsonParser.parseString(src.value()) })
I was wondering that I should have special JsonSerializer. After debugging my first thought was leading to OpenApi class in io.swagger.v3.oas.models package. I added this code: .registerTypeAdapter(OpenAPI::class.java, JsonSerializer<OpenAPI> { _, _, _ -> JsonParser.parseString("") }) to customGson and nothing changed... So, I was digging deeper...
After when I ran my Swagger tests:
#EnableAutoConfiguration
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
#AutoConfigureMockMvc
#ExtendWith(SpringExtension::class)
#ActiveProfiles("test")
class SwaggerIntegrationTest(#Autowired private val mockMvc: MockMvc) {
#Test
fun `should display Swagger UI page`() {
val result = mockMvc.perform(MockMvcRequestBuilders.get("/swagger-ui/index.html"))
.andExpect(status().isOk)
.andReturn()
assertTrue(result.response.contentAsString.contains("Swagger UI"))
}
#Disabled("Redirect doesn't work. Check it later")
#Test
fun `should display Swagger UI page with redirect`() {
mockMvc.perform(MockMvcRequestBuilders.get("/swagger-ui.html"))
.andExpect(status().isOk)
.andExpect(MockMvcResultMatchers.content().contentTypeCompatibleWith(MediaType.TEXT_HTML))
}
#Test
fun `should get api docs`() {
mockMvc.perform(MockMvcRequestBuilders.get("/api-docs"))
.andExpect(status().isOk)
.andExpect(MockMvcResultMatchers.content().contentTypeCompatibleWith(MediaType.APPLICATION_JSON))
.andExpect(MockMvcResultMatchers.jsonPath("\$.openapi").exists())
}
}
I saw in console this:
MockHttpServletRequest:
HTTP Method = GET
Request URI = /api-docs
Parameters = {}
Headers = []
Body = null
Session Attrs = {}
Handler:
Type = org.springdoc.api.OpenApiResource
Method = org.springdoc.api.OpenApiResource#openapiJson(HttpServletRequest, String)
Next I check openapiJson in OpenApiResource and...
#Operation(hidden = true)
#GetMapping(value = API_DOCS_URL, produces = MediaType.APPLICATION_JSON_VALUE)
public String openapiJson(HttpServletRequest request, #Value(API_DOCS_URL) String apiDocsUrl)
throws JsonProcessingException {
calculateServerUrl(request, apiDocsUrl);
OpenAPI openAPI = this.getOpenApi();
return Json.mapper().writeValueAsString(openAPI);
}
OK, Jackson... I have disabled Jackson by #EnableAutoConfiguration(exclude = [(JacksonAutoConfiguration::class)]) because I (and my colleagues) prefer GSON, but it doesn't explain why serialization go wrong after adding custom GsonHttpMessageConverter. I have no idea what I made bad. This openapiJson() is endpoint and maybe it mess something... I don't know. I haven't any idea. Did you have a similar problem? Can you give some advice or hint?
PS. Sorry for my bad English :).
I had the same issue with a project written in Java, and I've just solved that by defining a filter to format my springdoc-openapi json documentation using Gson. I guess you can easily port this workaround to Kotlin.
#Override
public void doFilter(final ServletRequest request, final ServletResponse response, final FilterChain chain)
throws IOException, ServletException {
ByteResponseWrapper byteResponseWrapper = new ByteResponseWrapper((HttpServletResponse) response);
ByteRequestWrapper byteRequestWrapper = new ByteRequestWrapper((HttpServletRequest) request);
chain.doFilter(byteRequestWrapper, byteResponseWrapper);
String jsonResponse = new String(byteResponseWrapper.getBytes(), response.getCharacterEncoding());
response.getOutputStream().write((new com.google.gson.JsonParser().parse(jsonResponse).getAsString())
.getBytes(response.getCharacterEncoding()));
}
#Override
public void destroy() {
}
static class ByteResponseWrapper extends HttpServletResponseWrapper {
private PrintWriter writer;
private ByteOutputStream output;
public byte[] getBytes() {
writer.flush();
return output.getBytes();
}
public ByteResponseWrapper(HttpServletResponse response) {
super(response);
output = new ByteOutputStream();
writer = new PrintWriter(output);
}
#Override
public PrintWriter getWriter() {
return writer;
}
#Override
public ServletOutputStream getOutputStream() {
return output;
}
}
static class ByteRequestWrapper extends HttpServletRequestWrapper {
byte[] requestBytes = null;
private ByteInputStream byteInputStream;
public ByteRequestWrapper(HttpServletRequest request) throws IOException {
super(request);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
InputStream inputStream = request.getInputStream();
byte[] buffer = new byte[4096];
int read = 0;
while ((read = inputStream.read(buffer)) != -1) {
baos.write(buffer, 0, read);
}
replaceRequestPayload(baos.toByteArray());
}
#Override
public BufferedReader getReader() {
return new BufferedReader(new InputStreamReader(getInputStream()));
}
#Override
public ServletInputStream getInputStream() {
return byteInputStream;
}
public void replaceRequestPayload(byte[] newPayload) {
requestBytes = newPayload;
byteInputStream = new ByteInputStream(new ByteArrayInputStream(requestBytes));
}
}
static class ByteOutputStream extends ServletOutputStream {
private ByteArrayOutputStream bos = new ByteArrayOutputStream();
#Override
public void write(int b) {
bos.write(b);
}
public byte[] getBytes() {
return bos.toByteArray();
}
#Override
public boolean isReady() {
return false;
}
#Override
public void setWriteListener(WriteListener writeListener) {
}
}
static class ByteInputStream extends ServletInputStream {
private InputStream inputStream;
public ByteInputStream(final InputStream inputStream) {
this.inputStream = inputStream;
}
#Override
public int read() throws IOException {
return inputStream.read();
}
#Override
public boolean isFinished() {
return false;
}
#Override
public boolean isReady() {
return false;
}
#Override
public void setReadListener(ReadListener readListener) {
}
}
You will also have to register your filter only for your documentation url pattern.
#Bean
public FilterRegistrationBean<DocsFormatterFilter> loggingFilter() {
FilterRegistrationBean<DocsFormatterFilter> registrationBean = new FilterRegistrationBean<>();
registrationBean.setFilter(new DocsFormatterFilter());
registrationBean.addUrlPatterns("/v3/api-docs");
return registrationBean;
}