httpClient.post().uri(getSearchUrl())
.send(Mono.just(Unpooled.wrappedBuffer(bytes)))
.responseSingle((resp, buf) -> {
return buf;
})
.map(ByteBuf::retain)
.map(byteBuf -> {
response.setResponseBodyStream(new ByteBufInputStream(byteBuf, true));
return response;
});
In the method responseSingle,the respone is already uncompressed.
So how can I get the length of compressed response? Thanks.
Check the response headers. You probably have the header "Content-Length" that should tell you the response size.
Otherwise, you can get the body as a ByteArrayResource and check the size of the array of bytes.
.exchange()
.flatMap(clientResponse -> clientResponse.bodyToMono(ByteArrayResource.class)
.flatMap(responseBodyByteArray -> checkSize(responseBodyByteArray.getByteArray().length))
)
Related
I'm getting the DataBufferLimitException on receipt of a response to a HTTP request. I am using spring-boot-starter-parent:2.5.0, spring-cloud.version:2020.0.2.
I have tried practically all of the options described here(DataBufferLimitException: Exceeded limit on max bytes to buffer webflux error) and here(configure spring.codec.max-in-memory-size When using ReactiveElasticsearchClient), with no success. Is there anything else I can try? Here is my code to create the webclient:
private WebClient buildWebClient(long custId) {
return WebClient.builder()
.clientConnector(createWebClientWithTimeout())
// Add base url to all requests (callers only need to add the path and query params)
.baseUrl(baseUrl)
// Filter to add bearer token to auth header before sending request
.filter((request, next) -> getToken(custId).map(setAuthHeader(request)).flatMap(next::exchange))
// Filter to send the request, and try again if it has an auth error
.filter((request, next) -> next.exchange(request).flatMap(clientResponse -> {
if (clientResponse.statusCode() == HttpStatus.UNAUTHORIZED) {
logger.error("Received 401 Unauthorized from linkedin for request: {} {} with X-LI-UUID header: {}", request.method(), request.url(),
clientResponse.headers().header(LINKEDIN_HEADER));
// Retry once if auth failed
return clientResponse.bodyToMono(String.class)
.doOnNext(err -> logger.warn("Received 401 from linkedin; retrying once. Error body: {}", err))
.then(refreshToken(custId).map(setAuthHeader(request)).flatMap(next::exchange));
} else if (clientResponse.statusCode().isError()) {
logger.error("Received error status code: {} from linkedin for request: {} {} with X-LI-UUID header: {}", clientResponse.statusCode(), request.method(),
request.url(), clientResponse.headers().header(LINKEDIN_HEADER));
} else {
logger.debug("Received status code: {} from linkedin for request: {} {}", clientResponse.statusCode(), request.method(), request.url());
}
// If not a 401, just return the response
return Mono.just(clientResponse);
})).build();
}
Adding spring.codec.max-in-memory-size=16MB to the properties does not work, explicitly setting the value using ExchangeStrategies does not work, implementing WebFluxConfigurer does not work.
This code was working fine with spring-boot-starter-parent:2.1.6.RELEASE.
Any suggestions as to what I can try next?
Spring Boot preconfigures the WebClienter.Builder for you, which makes settings like spring.codec.max-in-memory-size work after all.
To make use of this preconfigured WebClient.Builder, you need to have it injected into your service, which does not look like what you are doing in the above example. You seem to use the WebClient.builder() method straight.
This code together with the application property spring.codec.max-in-memory-size could work:
#Service
public class Foo(WebClient.Builder injectedPreConfiguredBuilder) {
private WebClient buildWebClient(long custId) {
return injectedPreConfiguredBuilder
.clientConnector(createWebClientWithTimeout())
// Add base url to all requests (callers only need to add the path and query params)
.baseUrl(baseUrl)
// Filter to add bearer token to auth header before sending request
.filter((request, next) -> getToken(custId).map(setAuthHeader(request)).flatMap(next::exchange))
// Filter to send the request, and try again if it has an auth error
.filter((request, next) -> next.exchange(request).flatMap(clientResponse -> {
if (clientResponse.statusCode() == HttpStatus.UNAUTHORIZED) {
logger.error("Received 401 Unauthorized from linkedin for request: {} {} with X-LI-UUID header: {}", request.method(), request.url(),
clientResponse.headers().header(LINKEDIN_HEADER));
// Retry once if auth failed
return clientResponse.bodyToMono(String.class)
.doOnNext(err -> logger.warn("Received 401 from linkedin; retrying once. Error body: {}", err))
.then(refreshToken(custId).map(setAuthHeader(request)).flatMap(next::exchange));
} else if (clientResponse.statusCode().isError()) {
logger.error("Received error status code: {} from linkedin for request: {} {} with X-LI-UUID header: {}", clientResponse.statusCode(), request.method(),
request.url(), clientResponse.headers().header(LINKEDIN_HEADER));
} else {
logger.debug("Received status code: {} from linkedin for request: {} {}", clientResponse.statusCode(), request.method(), request.url());
}
// If not a 401, just return the response
return Mono.just(clientResponse);
})).build();
}
}
Also, you need to keep in mind that you need to use an injected / autowired WebClient.Builder in your tests for the property to work!
If you want to roll your own WebClient.Builder, it's possible to set the buffer size programmatically.
kotlin example:
val webClient = WebClient.builder()
.exchangeStrategies(
ExchangeStrategies.builder().codecs {
it.defaultCodecs().maxInMemorySize(1000000) // in bytes
}.build()
).build()
Java example:
WebClient.builder()
.exchangeStrategies(ExchangeStrategies.builder().codecs(
clientCodecConfigurer ->
clientCodecConfigurer.defaultCodecs().maxInMemorySize(1000000))
.build())
.baseUrl("https://stackoverflow.com/posts/68986553/")
.build();
I am using Spring WebClient to call REST API. I want to throw an error based on the response. For example, if there is an error (400) with body
{"error": "error message1 "}
then I want to throw an error with "error message1". Same way if there is an error(400) with the body
{"error_code": "100020"}
then I want to throw an error with error_cde 100020. I want to do it in a non-blocking way.
public Mono<Response> webclient1(...) {
webClient.post().uri(createUserUri).header(CONTENT_TYPE, APPLICATION_JSON)
.body(Mono.just(request), Request.class).retrieve()
.onStatus(HttpStatus::isError, clientResponse -> {
//Error Handling
}).bodyToMono(Response.class);
}
A body from ClientResponse should be extracted in a reactive way (javadoc) and lambda in onStatus method should return another Mono (javadoc). To sum up, take a look at below example
onStatus(HttpStatus::isError, response -> response
.bodyToMono(Map.class)
.flatMap(body -> {
var message = body.toString(); // here you should probably use some JSON mapper
return Mono.error(new Exception(message));
})
);
I have a controller proxy api endpoint where it receives different request payloads which are intended to different services. This controller validates payload and adds few headers based on certain rules. In this current context, i do not want to parse the received response from upstream services. proxy method should simply stream response to downstream clients so that it can scale well without going into any memory issues when dealing with large response payloads.
I have implemented method like this:
suspend fun proxyRequest(
url: String,
request: ServerHttpRequest,
customHeaders: HttpHeaders = HttpHeaders.EMPTY,
): ResponseEntity<String>? {
val modifiedReqHeaders = getHeadersWithoutOrigin(request, customHeaders)
val uri = URI.create(url)
val webClient = proxyClient.method(request.method!!)
.uri(uri)
.body(request.body)
modifiedReqHeaders.forEach {
val list = it.value.iterator().asSequence().toList()
val ar: Array<String> = list.toTypedArray()
#Suppress("SpreadOperator")
webClient.header(it.key, *ar)
}
return webClient.exchangeToMono { res ->
res.bodyToMono(String::class.java).map { b -> ResponseEntity.status(res.statusCode()).body(b) }
}.awaitFirstOrNull()
}
But this doesn't seems to be streaming. When i try to download large file, it is complaining failed to hold large data buffer.
Can someone help me in writing reactive streamed approach?
This is what i have done finally.
suspend fun proxyRequest(
url: String,
request: ServerHttpRequest,
response: ServerHttpResponse,
customHeaders: HttpHeaders = HttpHeaders.EMPTY,
): Void? {
val modifiedReqHeaders = getHeadersWithoutOrigin(request, customHeaders)
val uri = URI.create(url)
val webClient = proxyClient.method(request.method!!)
.uri(uri)
.body(request.body)
modifiedReqHeaders.forEach {
val list = it.value.iterator().asSequence().toList()
val ar: Array<String> = list.toTypedArray()
#Suppress("SpreadOperator")
webClient.header(it.key, *ar)
}
val respEntity = webClient
.retrieve()
.toEntityFlux<DataBuffer>()
.awaitSingle()
response.apply {
headers.putAll(respEntity.headers)
statusCode = respEntity.statusCode
}
return response.writeWith(respEntity.body ?: Flux.empty()).awaitFirstOrNull()
}
Let me know if this is truly sending data downstream and flushing?
Your first code snippet fails with memory issues because it is buffering in memory the whole response body as a String and forwards it after. If the response is quite large, you might fill the entire available memory.
The second approach also fails because instead of returning the entire Flux<DataBuffer> (so the entire response as buffers), you're only returning the first one. This fails because the response is incomplete.
Even if you manage to fix this particular issue, there are many other things to pay attention to:
it seems you're not returning the original response headers, effectively changing the response content type
you should not forward all the incoming response headers, as some of them are really up to the server (like transfer encoding)
what happens with security-related request/response headers?
how are you handling tracing and metrics?
You could take a look at the Spring Cloud Gateway project, which handles a lot of those subtleties and let you manipulate requests/responses.
I have the below code where I am able to log the headers and URL. But the body() method is returning an object of type BodyInserter. In the debug mode(STS), we can see the request body object. Is there any way to log the request?
[private ExchangeFilterFunction logRequest() {
return ExchangeFilterFunction.ofRequestProcessor(clientRequest -> { loggingService.info(clientRequest.url());
loggingService.info(clientRequest.headers());
BodyInserter<?, ? super ClientHttpRequest> bodyInserters= clientRequest.body();
return Mono.just(clientRequest);
});
}
I'm writing the Excel Workbook created using Apache POI to the response object directly as follows without creating a file:
val outputStream: ByteArrayOutputStream = new ByteArrayOutputStream()
workbook.write(outputStream)
ExcelOk(response.getOutputStream.write(outputStream.toByteArray))
But once the size of the response exceeds 8kB, it starts getting downloaded as zip file in Chrome and as octet-stream in FireFox.
My ExcelOk object looks like this:
object ExcelOk {
def apply(body: Any = Unit, headers: Map[String, String] = ExcelContentType, reason: String = "") = {
halt(ActionResult(responseStatus(200, reason), body, headers ))
}
}
and my ExcelContentType(i.e, response headers) is as below:
val ExcelContentType = Map(
"Access-Control-Allow-Credentials" -> "true",
"Access-Control-Allow-Methods" -> "GET, PUT, POST, DELETE, OPTIONS",
"Access-Control-Allow-Origin" -> "*",
"Access-Control-Max-Age" -> "1728000",
"Content-type" -> "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"Content-disposition" -> "attachment; filename=excel_report.xlsx"
)
I even tried adding "Transfer-Encoding" -> "chunked" to the header list but it doesn't work.
I added this snippet in my web.xml file as well but it didn't help either:
<mime-mapping>
<extension>xlsx</extension>
<mime-type>application/vnd.openxmlformats-officedocument.spreadsheetml.sheet</mime-type>
</mime-mapping>
Any help regarding this would be useful. Note that this behavior is observed only after response size exceeds certain threshold.
You have to set response headers before writing content to response output stream.
response.setHeader("Content-Type", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
response.setHeader("Content-disposition", "attachment; filename=excel_report.xlsx")
workbook.write(response.getOutputStream)