I am facing connection error. Log entries are
readAddress(..) failed: Connection reset by peer; nested exception is io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
the connection observed an error
Pending acquire queue has reached its maximum size of 1000; nested exception is reactor.netty.internal.shaded.reactor.pool.PoolAcquirePendingLimitException
Webclient config is:
#Bean
public WebClient webClient(#Autowired ObjectMapperBean objectMapperBean) {
ConnectionProvider provider =
ConnectionProvider
.builder("custom")
.maxConnections(500)
.build();
HttpClient httpClient = HttpClient.create(provider);
ExchangeStrategies exchangeStrategies =
ExchangeStrategies
.builder()
.codecs(codecConfigurer -> codecConfigurer
.defaultCodecs()
.jackson2JsonEncoder(new Jackson2JsonEncoder(objectMapperBean.getObjectMapper(), MediaType.APPLICATION_JSON)))
.build();
return WebClient
.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.exchangeStrategies(exchangeStrategies)
.build();
}
I am not sure where the problem is. Can someone help me on this?
#springboot #webclient
By default the number of pending queue requests allowed for a reactor netty server is
2 * number of connections
if not provided explicitly. You need to take care of these configuration on the basis of your throughput expectations. You can modify this property as follows :
ConnectionProvider.builder(CONNECTION_NAME)
.maxConnections(<MaxConnectionThreads>)
.pendingAcquireTimeout(Duration.ofMillis(<PendingAcquireTimeout>))
.pendingAcquireMaxCount(<maxCount>)
.maxIdleTime(Duration.ofMillis(<MaxIdleTime>))
.build();
Related
I have setup WebClient with apache httpclient 5 connector. Here my code snippet
#Bean
public WebClient webClient() {
HttpAsyncClientBuilder clientBuilder = HttpAsyncClients.custom();
final PoolingAsyncClientConnectionManager connManager =
PoolingAsyncClientConnectionManagerBuilder
.create()
.build();
clientBuilder.setConnectionManager(connManager);
CloseableHttpAsyncClient client = clientBuilder.build();
ClientHttpConnector connector = new HttpComponentsClientHttpConnector(client);
return WebClient.builder().clientConnector(connector).build();
}
I did a loading test with my rest API which internally calls remote URL using WebClient configuration as mentioned above. I see many requests failed with error
Caused by: org.apache.hc.core5.util.DeadlineTimeoutException: Deadline: 2022-02-02T14:01:23.844+0530, -451 MILLISECONDS overdue
at org.apache.hc.core5.util.DeadlineTimeoutException.from(DeadlineTimeoutException.java:49)
at org.apache.hc.core5.pool.StrictConnPool.processPendingRequest(StrictConnPool.java:318)
at org.apache.hc.core5.pool.StrictConnPool.processNextPendingRequest(StrictConnPool.java:299)
I am not able to find what configuration should be added in order to fix the issue. Can anyone please help me to solve this issue?
I have rest template config similar to the following. I am trying to release a connection from the pool if I get a status code that does not equal 2XX (long story but need this code). Is here a way I can get the connection Manager and release a specific connection?
#Bean
public RestTemplate restTemplate() {
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(100);
connectionManager.setDefaultMaxPerRoute(20);
RequestConfig requestConfig = RequestConfig
.custom()
.setConnectionRequestTimeout(5000) // timeout to get connection from pool
.setSocketTimeout(5000) // standard connection timeout
.setConnectTimeout(5000) // standard connection timeout
.build();
HttpClient httpClient = HttpClientBuilder.create()
.setConnectionManager(connectionManager)
.setDefaultRequestConfig(requestConfig).build();
ClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(httpClient);
return new RestTemplate(requestFactory);
}
Looking for a way to accomplish something similar to the following
if(!httpStatusCode.substr(1).equals("2")) {
restTemplate.getConnectionPool().relase().thisConnection();
}
enter code here
I'm using Spring Webclient in a Spring Boot project to call a remote API
I've observed a strange behaviour with Webclient. Whenever the Remote API timeouts increase, Webclient goes into a state with very few active connections (less than 5, even though maxConnections is set in the config as 3200), and all the incoming connections get added to the Pending Queue, due to which after a while almost all requests are rejected with a PoolAcquirePendingLimitException exception.
The expected behaviour is that Webclient should create new connections (max upto 3200) to handle the incoming traffic
Webclient Config is as follows:
#Bean
public WebClient webClient(WebClient.Builder builder)
{
TcpClient tcpClient = TcpClient.create(getConnectionProvider())
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000)
.wiretap(true)
.doOnConnected(connection ->
connection.addHandlerLast(new ReadTimeoutHandler(10000, TimeUnit.MILLISECONDS)));
ClientHttpConnector connector = new ReactorClientHttpConnector(HttpClient.from(tcpClient));
return builder.uriBuilderFactory(initUriTemplateHandler())
.clientConnector(connector)
.build();
}
private ConnectionProvider getConnectionProvider()
{
return ConnectionProvider.builder("fixed")
.maxConnections(3200)
.pendingAcquireTimeout(Duration.ofMillis(10000))
.pendingAcquireMaxCount(10000)
.maxIdleTime(Duration.ofMinutes(10))
.metrics(true)
.build();
}
private DefaultUriBuilderFactory initUriTemplateHandler()
{
DefaultUriBuilderFactory uriFactory = new DefaultUriBuilderFactory();
uriFactory.setEncodingMode(DefaultUriBuilderFactory.EncodingMode.NONE);
return uriFactory;
}
This is how I'm making the API calls
T response = webClient.get()
.uri(url)
.retrieve()
.bodyToMono(responseClass)
.timeout(Duration.ofMillis(requestTimeout)) // varies between 15-20ms
.block();
Below is a screenshot of the metrics[![enter image description here][2]][2]
This is a very high traffic application, and hence virtually it feels like the Pending Queue is stuck at 10000
Dependency Versions:
spring-boot-starter-webflux: 2.2.4.RELEASE
reactory-netty: 0.9.5.RELEASE
I am using webClinet to consume OAuth2 secured services. When services takes time more than default timeout i get below error and request is retried.
io.netty.channel.unix.Errors$NativeEceptionIoException: readAddress failed: Connection reset by peer
The connection observed an error, the request will be retried.
This seem to be IO exception issue with netty.
How do I avoid retry in such scenario?
Here is my webclient configuration-
WebClient webclinet(OAuth2AuthorizedClientManager am){
ExchangeStrategies ex = ExchangeStrategies
.builder()
.codec(c-> c.defaultCodecs().maxInMemorySize(-1)).build();
ServletOAuth2AuthorizedClientExchangeFilterFunction oauth =
new ServletOAuth2AuthorizedClientExchangeFilterFunction(am);
SslContext ssl = new JdkSslContext(SSLContext.getDefault(),true,ClientAuth.REQUIRE);
ReactorClientHttpConnector clientHttpConnector = new ReactorClientHttpConnector(Http.create()
.secure(sslContextSpec-> sslContextSpec.sslContext(ssl)));
return WebClient.builder()
.exchangeStrategies(ex)
.clientConnector(clientHttpConnector)
.filter(oauth)
.build();
There is a property named sslContextSpec.sslContext(ssl).disableRetry(true) which can be used to disable retry functionality of it.
I'm aware of Spring 5 webflux how to set a timeout on Webclient but this configures the timeout globally for all requests. I'm looking for a way to configure the timeout on a per request basis. I.e. something like this (pseudo-code that doesn't work):
WebClient client = ...
// Call 1
client.timeout(5, TimeUnit.SECONDS).contentType(APPLICATION_JSON).syncBody(..).exchange(). ..
// Call 2
client.timeout(4, TimeUnit.SECONDS).contentType(APPLICATION_JSON).syncBody(..).exchange().
The timeout function is made-up to demonstrate what I'm after. How can I achieve this? It's also important that resources are cleaned up properly on timeout.
If it makes any difference I'm using Netty (reactor-netty 0.8.4.RELEASE):
HttpClient httpClient = HttpClient.create(). ...;
WebClient webClient = WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
These two answers clearly explains it.
set-timeout-in-spring-webflux-webclient
spring-5-webflux-how-to-set-a-timeout-on-webclient.
Additionally if you are looking to mutate the options,
you could do like below,
TcpClient tcpClient = TcpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 1000)
.doOnConnected(connection ->
connection.addHandlerLast(new ReadTimeoutHandler(10))
.addHandlerLast(new WriteTimeoutHandler(10)));
return this.webClient
.mutate()
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(tcpClient)))
.build()
.get()