I am facing connection error. Log entries are
readAddress(..) failed: Connection reset by peer; nested exception is io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
the connection observed an error
Pending acquire queue has reached its maximum size of 1000; nested exception is reactor.netty.internal.shaded.reactor.pool.PoolAcquirePendingLimitException
Webclient config is:
#Bean
public WebClient webClient(#Autowired ObjectMapperBean objectMapperBean) {
ConnectionProvider provider =
ConnectionProvider
.builder("custom")
.maxConnections(500)
.build();
HttpClient httpClient = HttpClient.create(provider);
ExchangeStrategies exchangeStrategies =
ExchangeStrategies
.builder()
.codecs(codecConfigurer -> codecConfigurer
.defaultCodecs()
.jackson2JsonEncoder(new Jackson2JsonEncoder(objectMapperBean.getObjectMapper(), MediaType.APPLICATION_JSON)))
.build();
return WebClient
.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.exchangeStrategies(exchangeStrategies)
.build();
}
I am not sure where the problem is. Can someone help me on this?
#springboot #webclient
By default the number of pending queue requests allowed for a reactor netty server is
2 * number of connections
if not provided explicitly. You need to take care of these configuration on the basis of your throughput expectations. You can modify this property as follows :
ConnectionProvider.builder(CONNECTION_NAME)
.maxConnections(<MaxConnectionThreads>)
.pendingAcquireTimeout(Duration.ofMillis(<PendingAcquireTimeout>))
.pendingAcquireMaxCount(<maxCount>)
.maxIdleTime(Duration.ofMillis(<MaxIdleTime>))
.build();
I am using webClinet to consume OAuth2 secured services. When services takes time more than default timeout i get below error and request is retried.
io.netty.channel.unix.Errors$NativeEceptionIoException: readAddress failed: Connection reset by peer
The connection observed an error, the request will be retried.
This seem to be IO exception issue with netty.
How do I avoid retry in such scenario?
Here is my webclient configuration-
WebClient webclinet(OAuth2AuthorizedClientManager am){
ExchangeStrategies ex = ExchangeStrategies
.builder()
.codec(c-> c.defaultCodecs().maxInMemorySize(-1)).build();
ServletOAuth2AuthorizedClientExchangeFilterFunction oauth =
new ServletOAuth2AuthorizedClientExchangeFilterFunction(am);
SslContext ssl = new JdkSslContext(SSLContext.getDefault(),true,ClientAuth.REQUIRE);
ReactorClientHttpConnector clientHttpConnector = new ReactorClientHttpConnector(Http.create()
.secure(sslContextSpec-> sslContextSpec.sslContext(ssl)));
return WebClient.builder()
.exchangeStrategies(ex)
.clientConnector(clientHttpConnector)
.filter(oauth)
.build();
There is a property named sslContextSpec.sslContext(ssl).disableRetry(true) which can be used to disable retry functionality of it.
I am using Spring Rest template along with apache's PoolingHttpClientConnectionManager for making API calls. The project in which I am working on requires setting custom timeout for each of the HTTP request I make via rest template. In order to achieve this, I am using CompletableFuture with a separate ExecutorService and calling get(Timeout) method.
try{
CompletableFuture<BidResponse> future = CompletableFuture.supplyAsync(() -> bidderService.getBid(), executorService);
bidResponse = future.get(bidderTimeout, TimeUnit.MILLISECONDS);
} catch (InterruptedException | TimeoutException | ExecutionException e) {
bidResponse = getTimeoutBidResponse();
}
Unfortunately, the problem with this approach is that in cases of timeout, the underlying thread keeps on working until the rest template finishes its call. So I am kind of losing out a thread from the thread pool, as well as a connection from the HTTP connection pool.
Is there a way to close the HTTP connection as soon as we receive a Timeout exception, and return the HTTP connection back to the pool ?
p.s. I also tried using Spring Webclient with Mono.timeout. Turns out it actually closes the HTTP connection immediately, but does not return it back to the HTTP pool.
#Bean
public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder)
{
return restTemplateBuilder
.setConnectTimeout(...)
.setReadTimeout(...)
.build();
}
I have spring reactive a micro-service which call another micro-service to get list of products, but it keep failing with the error
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed DURING response
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Assembly trace from producer [reactor.core.publisher.FluxLift] :
This is how i get WebClient object
public WebClient createWebClient(HttpRequestConfig config, String baseUri) {
Builder clientBuilder = webClientBuilder(config)
.clientConnector(new ReactorClientHttpConnector(HttpClient.newConnection().compress(true)))
.baseUrl(baseUri);
return clientBuilder.build();
}
Any idea about the issue?
I am working on a Spring application that serves up REST endpoints. One of the endpoints essentially acts as a proxy between the HTML client and a third party cloud storage provider. This endpoint retrieves files from the storage provider and proxies them back to the client. Something like the following (note there is a synchronous and asynchronous version of the same endpoint):
#Controller
public class CloudStorageController {
...
#RequestMapping(value = "/fetch-image/{id}", method = RequestMethod.GET, produces = MediaType.IMAGE_JPEG_VALUE)
public ResponseEntity<byte[]> fetchImageSynchronous(#PathVariable final Long id) {
final byte[] imageFileContents = this.fetchImage(id);
return ResponseEntity.ok().body(imageFileContents);
}
#RequestMapping(value = "/fetch-image-async/{id}", method = RequestMethod.GET, produces = MediaType.IMAGE_JPEG_VALUE)
public Callable<ResponseEntity<byte[]>> fetchImageAsynchronous(#PathVariable final Long id) {
return () -> {
final byte[] imageFileContents = this.fetchImage(id);
return ResponseEntity.ok().body(imageFileContents);
};
}
private byte[] fetchImage(final long id) {
// fetch the file from cloud storage and return as byte array
...
}
...
}
Due to the nature of the client app (HTML5 + ajax) and how this endpoint is used, user authentication is supplied to this endpoint differently that the other endpoints. To handle this, a HandlerInterceptor was developed to deal with authentication for this endpoint:
#Component("cloudStorageAuthenticationInterceptor")
public class CloudStorageAuthenticationInterceptor extends HandlerInterceptorAdapter {
#Override
public boolean preHandle(final HttpServletRequest request, final HttpServletResponse response, final Object handler) {
// examine the request for the authentication information and verify it
final Authentication authenticated = ...
if (authenticated == null) {
try {
pResponse.sendError(HttpServletResponse.SC_UNAUTHORIZED);
} catch (IOException e) {
throw new RuntimeException(e);
}
return false;
}
else {
try {
request.login(authenticated.getName(), (String) authenticated.getCredentials());
} catch (final ServletException e) {
throw new BadCredentialsException("Bad credentials");
}
}
return true;
}
}
The interceptor is registered like this:
#Configuration
#EnableWebMvc
public class ApiConfig extends WebMvcConfigurerAdapter {
#Autowired
#Qualifier("cloudStorageAuthenticationInterceptor")
private HandlerInterceptor cloudStorageAuthenticationInterceptor;
#Override
public void addInterceptors(final InterceptorRegistry registry) {
registry.addInterceptor(this.cloudStorageAuthenticationInterceptor)
.addPathPatterns(
"/fetch-image/**",
"/fetch-image-async/**"
);
}
#Override
public void configureAsyncSupport(final AsyncSupportConfigurer configurer) {
final ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(this.asyncThreadPoolCoreSize);
executor.setMaxPoolSize(this.asyncThreadPoolMaxSize);
executor.setQueueCapacity(this.asyncThreadPoolQueueCapacity);
executor.setThreadNamePrefix(this.asyncThreadPoolPrefix);
executor.initialize();
configurer.setTaskExecutor(executor);
super.configureAsyncSupport(configurer);
}
}
Ideally, the image fetching would be done asynchronously (using the /fetch-image-asyc/{id} endpoint) because it has to call a third party web service which could have some latency.
The synchronous endpoint (/fetch-image/{id}) works correctly for all browsers. However, if using the asynchronous endpoint (/fetch-image-async/{id}), Chrome and Firefox work as expect.
However, if the client is Microsoft IE or Microsoft Edge, we seem some strange behavior. The endpoint is called correctly and the response sent successfully (at least from the server's viewpoint). However, it seems that the browser is waiting for something additional. In the IE/Edge DevTools window, the network request for the image shows as pending for 30 seconds, then seems to timeout, updates to successful and the image is successfully display. It also seems the connection to the server is still open, as the server side resources like database connections are not released. In the other browsers, the async response is received and processed in a second or less.
If I remove the HandlerInterceptor and just hard-wire some credentials for debugging, the behavior goes away. So this seems to have something to with the interaction between the HandlerInterceptor and the asynchronous controller method, and is only exhibited for some clients.
Anyone have a suggestion on why the semantics of IE/Edge are causing this behavior?
Based on your description, there are some different behaviors when using IE or Edge
it seems that the browser is waiting for something additional
the connection seems still open
it works fine if remove HandlerInterceptor and use hard code in auth logic
For the first behavior, I would suggest you use fiddler to trace all http requests. It is better if you could compare two different actions via fiddler (1) run on chrome, 2) run on edge ). Check all http headers in requests and responses carefully to see whether there is some different part. For the other behaviors, I would suggest you write logs to find which part spend the most time. It will provide you useful information to troubleshot.
After much tracing on the server and reading through the JavaDocs comments for AsyncHandlerInterceptor, I was able to resolve the issue. For requests to asynchronous controller methods, the preHandle method of any interceptor is called twice. It is called before the request is handed off to the servlet handling the request and again after the servlet has handled the request. In my case, the interceptor was attempting to authenticate the request for both scenarios (pre and post request handling). The application's authentication provider checks credentials in a database. For some reason if the client is IE or Edge, the authentication provider was unable to get a database connection when called from preHandle in the interceptor after the servlet handled the request. The following exception would be thrown:
ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Could not open connection; nested exception is org.hibernate.exception.JDBCConnectionException: Could not open connection] with root cause
java.sql.SQLTransientConnectionException: HikariPool-0 - Connection is not available, request timed out after 30001ms.
So the servlet would successfully handle the request and send a response, but the filter would get hung up for 30 seconds waiting for the database connection to timeout on the post processing called to preHandle.
So for me, the simple solution was to add a check in preHandle if it is being called after the servlet has already handled the request. I updated the preHandle method as follows:
#Override
public boolean preHandle(final HttpServletRequest pRequest, final HttpServletResponse pResponse, final Object pHandler) {
if (pRequest.getDispatcherType().equals(DispatcherType.REQUEST)) {
... perform authentication ...
}
return true;
}
That solved the issue for me. It doesn't explain everything (i.e., why only IE/Edge would cause the issue), but it seems that preHandle should only do work before the servlet handles the request anyways.