I'm using Spring Webclient in a Spring Boot project to call a remote API
I've observed a strange behaviour with Webclient. Whenever the Remote API timeouts increase, Webclient goes into a state with very few active connections (less than 5, even though maxConnections is set in the config as 3200), and all the incoming connections get added to the Pending Queue, due to which after a while almost all requests are rejected with a PoolAcquirePendingLimitException exception.
The expected behaviour is that Webclient should create new connections (max upto 3200) to handle the incoming traffic
Webclient Config is as follows:
#Bean
public WebClient webClient(WebClient.Builder builder)
{
TcpClient tcpClient = TcpClient.create(getConnectionProvider())
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000)
.wiretap(true)
.doOnConnected(connection ->
connection.addHandlerLast(new ReadTimeoutHandler(10000, TimeUnit.MILLISECONDS)));
ClientHttpConnector connector = new ReactorClientHttpConnector(HttpClient.from(tcpClient));
return builder.uriBuilderFactory(initUriTemplateHandler())
.clientConnector(connector)
.build();
}
private ConnectionProvider getConnectionProvider()
{
return ConnectionProvider.builder("fixed")
.maxConnections(3200)
.pendingAcquireTimeout(Duration.ofMillis(10000))
.pendingAcquireMaxCount(10000)
.maxIdleTime(Duration.ofMinutes(10))
.metrics(true)
.build();
}
private DefaultUriBuilderFactory initUriTemplateHandler()
{
DefaultUriBuilderFactory uriFactory = new DefaultUriBuilderFactory();
uriFactory.setEncodingMode(DefaultUriBuilderFactory.EncodingMode.NONE);
return uriFactory;
}
This is how I'm making the API calls
T response = webClient.get()
.uri(url)
.retrieve()
.bodyToMono(responseClass)
.timeout(Duration.ofMillis(requestTimeout)) // varies between 15-20ms
.block();
Below is a screenshot of the metrics[![enter image description here][2]][2]
This is a very high traffic application, and hence virtually it feels like the Pending Queue is stuck at 10000
Dependency Versions:
spring-boot-starter-webflux: 2.2.4.RELEASE
reactory-netty: 0.9.5.RELEASE
Related
I have an already existing application, and we are making calls to other application from this.
So, now, in our application we control the creation of WebClient object from a single place and then some other config are set on this like below :
ObjectMapper objectMapper = new ObjectMapper();
ExchangeStrategies exchangeStrategies = ExchangeStrategies.builder()
.codecs(configure ->
configure
.defaultCodecs()
.jackson2JsonDecoder(
new Jackson2JsonDecoder(objectMapper)))
.build();
HttpClient httpClient = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 12000)
.responseTimeout(Duration.ofMillis(60000));
return WebClient.builder()
.exchangeStrategies(exchangeStrategies)
.clientConnector(new ReactorClientHttpConnector(httpClient))
.baseUrl(modelDeployConfig.getHost())
.defaultHeaders( h -> {
h.setContentType(MediaType.APPLICATION_JSON);
h.setAccept(List.of(MediaType.APPLICATION_JSON));
})
.build();
Now, this webclient is build using this logic and used across the application. I want to impose some retry policies in our application which are common for all APIs.
But I dont find a way to do this at webclient level? any idea how to achieve this? any config settings available or any other way to suggest to achieve this where we can have some central retry policy which gets imposed on all APIs through this webclient object or by any other wrapper for webclient?
I'm using Citrus for perform integration tests based on http requests.
The test should perform steps as follows:
Sending a POST http request to external service with a callback url as body attribute
ex:
{... "callback": "http://url-to-be-called" ...}
Waiting for 3 sequential http requests from the external service
I've used Citrus httpClient to send the Post Request, and Citrus httpServer to exspose endpoints that the service will call using the callback url.
So,
httpClient --> sends POST to ext service
<-- ext service responds OK
(after several minutes)
<-- ext service invokes httpServer API
httpClient --> responds OK
(after several minutes)
<-- ext service invokes httpServer API
httpClient --> responds OK
(after several minutes)
<-- ext service invokes httpServer API
httpClient --> responds OK
My test code is
parallel().actions(
sequential().actions(
// wait for callback
http(httpActionBuilder -> httpActionBuilder
.server(httpServer)
.receive()
.post("/api/callback")
.contentType("application/json;charset=UTF-8")
.accept("text/plain,application/json,application/*+json,*/*")
),
// callback response
http(httpActionBuilder -> httpActionBuilder
.server(httpServer)
.send()
.response(HttpStatus.OK)
.contentType("application/json")
)),
sequential().actions(
http(httpActionBuilder -> httpActionBuilder
.client(httpClient)
.send()
.post("/externalapi)
.payload("{" +
"\"ret\": \"OK\"" +
"}")
.messageType(MessageType.JSON)
.contentType(ContentType.APPLICATION_JSON.getMimeType())
),
http(httpActionBuilder -> httpActionBuilder
.client(httpClient)
.receive()
.response(HttpStatus.OK)
.messageType(MessageType.JSON)
.payload("{" +
"\"ret\": \"OK\"" +
"}")
)
)
);
My configurations
#Bean
public HttpClient httpClient() {
return CitrusEndpoints
.http()
.client()
.requestUrl("http://localhost:8282")
.build();
}
#Bean
public HttpServer httpServer() throws Exception {
return CitrusEndpoints.http()
.server()
.port(8080)
.endpointAdapter(dispatchingEndpointAdapter())
.timeout(300000)
.autoStart(true)
.build();
}
#Bean
public RequestDispatchingEndpointAdapter dispatchingEndpointAdapter() {
RequestDispatchingEndpointAdapter dispatchingEndpointAdapter = new RequestDispatchingEndpointAdapter();
dispatchingEndpointAdapter.setMappingKeyExtractor(mappingKeyExtractor());
dispatchingEndpointAdapter.setMappingStrategy(mappingStrategy());
return dispatchingEndpointAdapter;
}
#Bean
public HeaderMappingKeyExtractor mappingKeyExtractor() {
HeaderMappingKeyExtractor mappingKeyExtractor = new HeaderMappingKeyExtractor();
mappingKeyExtractor.setHeaderName(HttpMessageHeaders.HTTP_REQUEST_URI);
return mappingKeyExtractor;
}
#Bean
public SimpleMappingStrategy mappingStrategy() {
SimpleMappingStrategy mappingStrategy = new SimpleMappingStrategy();
Map<String, EndpointAdapter> mappings = new HashMap<>();
mappings.put("/api/callback", callbackResponseAdapter());
mappingStrategy.setAdapterMappings(mappings);
return mappingStrategy;
}
#Bean
public EndpointAdapter callbackResponseAdapter() {
StaticResponseEndpointAdapter endpointAdapter = new StaticResponseEndpointAdapter();
endpointAdapter.setMessagePayload("{" +
"\"ret\": \"OK\"," +
"}");
return endpointAdapter;
}
The httpClient steps works fine, but when I add the HttpServer I get this error
com.consol.citrus.exceptions.TestCaseFailedException: Unable to create endpoint for static endpoint adapter type 'class com.consol.citrus.endpoint.adapter.RequestDispatchingEndpointAdapter'
In your http-server configuration you are using both static request dispatcher and test action based dynamic response generation. This is not valid.
Please choose only one approach at a time. Either remove the request dispatcher configuration completely or do not try to receive the incoming requests in your test via receive/send action.
By the way the parallel-sequential blocks you are using may not be required. You can use fork=true option on the 1st client request instead and sequentially wait for the Http requests to arrive. This should simplify the test a lot.
I'm trying to setup a WebClient connection in Spring Boot using a proxy. My implementation looks like the following:
final WebClient.Builder webclientBuilder = WebClient.builder();
final HttpClient httpClient = HttpClient.create();
httpClient.proxy(proxy -> proxy
.type(Proxy.HTTP)
.host(proxyName)
.port(Integer.parseInt(proxyPort)));
final ReactorClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);
webclientBuilder.clientConnector(connector);
final WebClient webClient = webclientBuilder
.baseUrl(baseUrl)
.build();
After running it and sending an API call, I receive a "Connection timed out: no further information". I should get back a Bad Request (in case my call is wrong), but I don't.
Is the implementation wrong?
the proxyName is written like this: "proxy.blabla.de"
After some trial and error and comparing I found a solution working for me:
String baseUrl = "https://mybaseurl";
String proxyName = "proxy.blabla.de";
int proxyPort = 1234;
public InitResponse addAccount() {
// for logging purposes, nothing to do with the proxy
LOGGER.info("LOGGER.info: addAccount()");
final InitRequest request = buildRequest();
HttpClient httpClient = HttpClient.create()
.proxy(proxy -> proxy.type(Proxy.HTTP)
.host(proxyName)
.port(proxyPort));
ReactorClientHttpConnector conn = new ReactorClientHttpConnector(httpClient);
WebClient webClient = WebClient.builder().clientConnector(conn).baseUrl(baseUrl).build();
I am facing connection error. Log entries are
readAddress(..) failed: Connection reset by peer; nested exception is io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
the connection observed an error
Pending acquire queue has reached its maximum size of 1000; nested exception is reactor.netty.internal.shaded.reactor.pool.PoolAcquirePendingLimitException
Webclient config is:
#Bean
public WebClient webClient(#Autowired ObjectMapperBean objectMapperBean) {
ConnectionProvider provider =
ConnectionProvider
.builder("custom")
.maxConnections(500)
.build();
HttpClient httpClient = HttpClient.create(provider);
ExchangeStrategies exchangeStrategies =
ExchangeStrategies
.builder()
.codecs(codecConfigurer -> codecConfigurer
.defaultCodecs()
.jackson2JsonEncoder(new Jackson2JsonEncoder(objectMapperBean.getObjectMapper(), MediaType.APPLICATION_JSON)))
.build();
return WebClient
.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.exchangeStrategies(exchangeStrategies)
.build();
}
I am not sure where the problem is. Can someone help me on this?
#springboot #webclient
By default the number of pending queue requests allowed for a reactor netty server is
2 * number of connections
if not provided explicitly. You need to take care of these configuration on the basis of your throughput expectations. You can modify this property as follows :
ConnectionProvider.builder(CONNECTION_NAME)
.maxConnections(<MaxConnectionThreads>)
.pendingAcquireTimeout(Duration.ofMillis(<PendingAcquireTimeout>))
.pendingAcquireMaxCount(<maxCount>)
.maxIdleTime(Duration.ofMillis(<MaxIdleTime>))
.build();
I'm aware of Spring 5 webflux how to set a timeout on Webclient but this configures the timeout globally for all requests. I'm looking for a way to configure the timeout on a per request basis. I.e. something like this (pseudo-code that doesn't work):
WebClient client = ...
// Call 1
client.timeout(5, TimeUnit.SECONDS).contentType(APPLICATION_JSON).syncBody(..).exchange(). ..
// Call 2
client.timeout(4, TimeUnit.SECONDS).contentType(APPLICATION_JSON).syncBody(..).exchange().
The timeout function is made-up to demonstrate what I'm after. How can I achieve this? It's also important that resources are cleaned up properly on timeout.
If it makes any difference I'm using Netty (reactor-netty 0.8.4.RELEASE):
HttpClient httpClient = HttpClient.create(). ...;
WebClient webClient = WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
These two answers clearly explains it.
set-timeout-in-spring-webflux-webclient
spring-5-webflux-how-to-set-a-timeout-on-webclient.
Additionally if you are looking to mutate the options,
you could do like below,
TcpClient tcpClient = TcpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 1000)
.doOnConnected(connection ->
connection.addHandlerLast(new ReadTimeoutHandler(10))
.addHandlerLast(new WriteTimeoutHandler(10)));
return this.webClient
.mutate()
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(tcpClient)))
.build()
.get()