I have a spring-boot application making a bunch of REST backend calls.
One of the Rest backend is throwing a ConnectionPoolTimeoutException.
After further investigation, I suspect that the connections are not closed for that backend when the response is an error or is null.
Tried debugging Spring RestTemplate code and saw a line:
finally {
if (response != null) {
response.close();
}
}
which looks like does not close connections if response was null.
Has anyone faced similar issue and what was the solution.
I am thinking of writing an interceptor which will always close the connection
Related
I have one issue in PRD. we recently released a springboot application and it has REST API exposed. Mobile/web APP call a legacy spring application which is in spring [not sprintboot] and it is a web applicationwhich then routes and makes a call to the these failing apis in new springboot. We are seeing timeout exception for these apis only .
there are lots of other OUTBOUND api calls made from spring legacy web application to other applications eg : login API [which has apis heavy traffic but these legacy apis work well and call other legacy applications.
There are no exception/error in logs in springboot application which has these REST API exposed. Infact we only see timeout in spring web application -meaning connection is exhausted but that does not explain why other apis OUTBOUND call are not failing which use same wrapper HTTPClient. Those which fail with timeout dont have request logs in springboot [ obviously because they dont leave spring web application tomcat JVM and die there due to timeout ]
So if we say connection pool is exhausted, the other heavey traffic OUTBOUnd calls should also face same issue but we dont see that.
All API call OUTWARD use HTTPCLient [apache.]
Not clear what is causing issue. I also explicitly defined below in new springboot for server side [I just did it to see if that makes difference but in vain]:
server:
tomcat:
connection-timeout: 10s
max-connections: 20000
max-threads: 500
min-spare-threads: 10
tomcat Log at spring web applicaiton [caller]:
org.apache.http.conn.ConnectionPoolTimeoutException
org.apache.http.conn.ConnectionPoolTimeoutException.Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:313)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
at
Any inputs?
Code snippet of Wrapper HTTPClient :
SSLContext sslContext = SSLContexts.createDefault();
HostnameVerifier hostnameVerifier = new DefaultHostnameVerifier();
SSLConnectionSocketFactory secureSSLConnectionSocketFactory = new SSLConnectionSocketFactory(
sslContext,
sslProtocolsArray,
ciphersArray,
hostnameVerifier);
ConnectionSocketFactory nonSecureConnectionSocketFactory = PlainConnectionSocketFactory.getSocketFactory();
Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder
.<ConnectionSocketFactory>create()
.register("https", secureSSLConnectionSocketFactory)
.register("http", nonSecureConnectionSocketFactory)
.build();
securePoolingConnectionManager = new PoolingHttpClientConnectionManager(socketFactoryRegistry);
securePoolingConnectionManager.setMaxTotal(this.connectionMgrMaxTotalSecure);
securePoolingConnectionManager.setDefaultMaxPerRoute(this.connectionMgrMaxPerRouteSecure);
SocketConfig secureSocketConfig = SocketConfig
.custom()
.setSoKeepAlive(true)
.setTcpNoDelay(true)
.build();
secureHttpsClient = HttpClients
.custom()
.setSSLSocketFactory(secureSSLConnectionSocketFactory)
.setConnectionManager(securePoolingConnectionManager)
.setDefaultRequestConfig(secureRequestConfig)
.setDefaultSocketConfig(secureSocketConfig)
.disableAutomaticRetries()
.build();
Stacktrace after above is just failing at wrapper HTTPClient method where call is invoked :
protected String execute(HttpClient httpclient, HttpRequestBase http) throws IOException {
String result;
ResponseHandler<String> responseHandler = new BasicResponseHandler();
result = httpclient.execute(http, responseHandler);
return result;
}
So I have to dig in another wrapper which was also using this HTTP pool and was being used in our legacy which was leaking. Closing this. Fortunately there was pool statistics api exposed so that I can see leased connection count which confirmed leaking. Since this second wrapper was used rerely and we had used in this release this was suspect and removing it solved the issue. It is another matter to dig that wrapper and find out how the pool was handled but the cause was caught.
I start a wiremock server in a integration test.
The IT pass in my local BUT some case failed in jenkins server, the error is
localhost:8089 failed to respond; nested exception is org.apache.http.NoHttpResponseException: localhost:8089 failed to respond
I try to add sleep(3000) in my test, that can fix the issue, But I don’t know the root cause of the issue, so the work around not a good idea
I also try to use #AutoConfigureWireMock(port=8089) to replace WireMockServer to start wiremock server, that could fix the problem, BUT I don't know how to do some configuration to the wiremock server using the annotation #AutoConfigureWireMock(port=8089).
Here my code to start a wiremock server, any suggestion to fix "NoHttpResponseException"?
#ContextConfiguration(
initializers = ConfigFileApplicationContextInitializer.class)
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT)
class BaseSpec extends Specification {
#Shared
WireMockServer wireMockServer
def setupSpec() {
wireMockServer = new WireMockServer(options().port(PORT).jettyHeaderBufferSize(12345)
.notifier(new ConsoleNotifier(new Boolean(System.getenv(“IT_WIREMOCK_LOG”) ?: ‘false’)))
.extensions(new ResponseTemplateTransformer(true)))
wireMockServer.start()
}
Apache HttpClient suffers from NoHttpResponseException from time to time. This is a very old problem.
Anyway, I guess in your case the problem might be caused by restarting the WireMock server between tests, and at the same time, Apache HttpClient pools HTTP connections and tries to reuse them between tests. If this is the case, there are two solutions:
Disable pooling HTTP connections in your tests. This makes sense because it's considered normal that the WireMock server can be restarted during tests execution. Alternatively, craft your WireMock stubs to always send "Connection": "close" among the headers. The outcome will be the same.
Switch from Apache HttpClient to Square OkHttp. OkHttp, although it pools http connections by default, is always able to gracefully recover from situations like a stale connection. Unfortunately the library from Apache is not so smart.
Coorect, as already written by G. Demecki it's not related to Wiremock.
It’s related to your application server, which is calling wiremock. Today it’s common, to reuse a connection to improve the performance in a micro service infrastructure. So connection-close-header, RequestScoped client, etc is not useful.
Check the apache http client:
httpclient-4.5.2 - PoolingHttpClientConnectionManager
documentation
The handling of stale connections was changed in version 4.4. Previously, the code would check every connection by default before re-using it. The code now only checks the connection if the elapsed time since the last use of the connection exceeds the timeout that has been set. The default timeout is set to 2000ms
Each time a wiremock-endpoint was destroyed and a new one was created for a new test class, it takes 2 seconds, until your application detects, that the previous connection is broken and a new one has to be opened.
If you don’t wait 2 seconds, such a NoHttpResponseException could be thrown, depends on the last check.
So a Thread.sleep(2000); looks ugly. But it's not so bad, as long we know why this is required.
Each time a wiremock endpoint is destroyed (because the wiremock server is restarted between tests) and a new one is created for a new test, it takes 2 seconds (as stated in documentation), until the application detects that the previous http connection is broken and a new one has to be opened.
The solution is to simply override the default keep-alive connection behaviour for every stub using .withHeader("Connection", "close"). Something like:
givenThat(get("/endpoint_path")
.withHeader("Authorization", equalTo(authHeader))
.willReturn(
ok()
.withBody(body)
.withHeader(HttpHeaders.CONNECTION, "close")
)
)
Also possible to do it globally using a transformer:
public class NoKeepAliveTransformer extends ResponseDefinitionTransformer {
#Override
public ResponseDefinition transform(Request request,
ResponseDefinition responseDefinition,
FileSource files,
Parameters parameters) {
return ResponseDefinitionBuilder
.like(responseDefinition)
.withHeader(CONNECTION, "close")
.build();
}
#Override
public String getName() {
return "keep-alive-disabler";
}
}
Then this transformer have to be registered when you create the wiremock server:
new WireMockServer(
options()
.port(port)
.extensions(NoKeepAliveTransformer.class)
)
Solution that worked for us in this situation was just adding retry to apache client:
#Configuration
public class FeignTestConfig {
#Bean
#Primary
public HttpClient testClient() {
return HttpClientBuilder.create().setRetryHandler((exception, executionCount, context) -> {
if (executionCount > 3) {
return false;
}
return exception instanceof org.apache.http.NoHttpResponseException || exception instanceof SocketException;
}).build();
}
}
Socket exception is there as well, because sometimes this exception is thrown instead of NoHttpResponse
AM trying to invoke a spring boot microservice from another application.
Here is my Server code, if any exception occurs
return ResponseEntity.status(HttpStatus.FORBIDDEN)
.body(new com.test.models.ResponseEntity("Tenant details
not found"));
Here is my client code,
ResponseEntity<Object> uploadResponse =
restTemplate.exchange("http://TESTAPPLICATION/v1/object",
HttpMethod.POST, requestEntity, Object.class);
Am getting a response as
Caused by: org.springframework.web.client.HttpClientErrorException: 403 null
If I hit my endpoint from PostMan, am getting proper response message.
But from my client-side application am getting response message as null. Please let me know what is going wrong.
UPDATE:
Screenshot.
Client Responds with INTERNAL_SERVER_ERROR. But from my server iam responding 403.
I created Spring Boot 2.0 demo application which contains two applications that communicate using WebClient. And I'm suffering that they often stop communicating when I use block() method of Flux from the WebClient's response. I want to use List not Flux by some reasons.
The server side application is like this. It just returns Flux object.
#GetMapping
public Flux<Item> findAll() {
return Flux.fromIterable(items);
}
And the client side (or BFF side) application is like this. I get Flux from the server and convert it to List by calling block() method.
#GetMapping
public List<Item> findBlock() {
return webClient.get()
.retrieve()
.bodyToFlux(Item.class)
.collectList()
.block(Duration.ofSeconds(10L));
}
While it works well at first, findBlock() won't respond and timeouts after several times access. When I modify the findBlock() method to return Flux deleting collectList() and block(), it works well. Then I assume that block() method cause this problem.
And, when I modify the findAll() method to return List, nothing changes.
Source code of the entire example application is here.
https://github.com/cero-t/webclient-example
"resource" is the server application, and "front" is the client application. After running both application, when I access to localhost:8080 it works well and I can reload any times, but when I access to localhost:8080/block it seems to work well but after several reloads it won't respond.
By the way, when I add "spring-boot-starter-web" dependency to the "front" applications's (not resource application's) pom.xml, which means I use tomcat, this problem never happens. Is this problem due to Netty server?
Any guidance would be greatly appreciated.
First, let me point that using Flux.fromIterable(items) is advised only if items has been fetched from memory, no I/O involved. Otherwise chances are you'd be using a blocking API to get it - and this can break your reactive application. In this case, this is an in-memory list, so no problem. Note that you can also go Flux.just(item1, item2, item3).
Using the following is the most efficient:
#GetMapping("/")
public Flux<Item> findFlux() {
return webClient.get()
.retrieve()
.bodyToFlux(Item.class);
}
Item instances will be read/written, decoded/encoded on the fly in a very efficient way.
On the other hand, this is not the preferred way:
#GetMapping("/block")
public List<Item> findBlock() {
return webClient.get()
.retrieve()
.bodyToFlux(Item.class)
.collectList()
.block(Duration.ofSeconds(10L));
}
In this case, your front application is buffering in memory the whole items list with collectList but is also blocking one of the few server threads available. This might cause very poor performance because your server might be blocked waiting for that data and can't service other requests at the same time.
In this particular case it's worse, since the application totally breaks.
Looking at the console, we can see the following:
WARN 3075 --- [ctor-http-nio-7] io.netty.util.concurrent.DefaultPromise : An exception was thrown by reactor.ipc.netty.channel.PooledClientContextHandler$$Lambda$532/356589024.operationComplete()
reactor.core.Exceptions$BubblingException: java.lang.IllegalArgumentException: Channel [id: 0xab15f050, L:/127.0.0.1:59350 - R:localhost/127.0.0.1:8081] was not acquired from this ChannelPool
at reactor.core.Exceptions.bubble(Exceptions.java:154) ~[reactor-core-3.1.3.RELEASE.jar:3.1.3.RELEASE]
This is probably linked to a reactor-netty client connection pool issue that should be fixed in 0.7.4.RELEASE. I don't know the specifics of this, but I suspect the whole connection pool gets corrupted as HTTP responses aren't properly read from the client connections.
Adding spring-boot-starter-web does make your application use Tomcat, but it mainly turns your Spring WebFlux application into a Spring MVC application (which now supports some reactive return types, but has a different runtime model). If you wish to test your application with Tomcat, you can add spring-boot-starter-tomcat to your POM and this will use Tomcat with Spring WebFlux.
I have an integration flow where I'm reading messages from a jms channel and sending them through a REST api call using http-outbound-gateway.
Now when I get a 5xx Http server error I'd like to log the the URL of the service along with the response message.
I see that the RestTemplate instance already logs the URL and calls a ResponseErrorHandler which can be a custom implementation.
My problem is: the ResponseErrorHandler only receives a ClientHttpResponse as a parameter and I can't know the URL from that instance. What I need is to log the server error like: 'Error URL - Response ...'
How can I do that? I can't override RestTemplate's error handling and the ResponseErrorHandler doesn't have enough info.
Thank you very much.
Well, what I see there.
The HttpRequestExecutingMessageHandler (<int-http:outbound-gateway>) catches all underlying exceptions:
catch (Exception e) {
throw new MessageHandlingException(requestMessage, "HTTP request execution failed for URI ["
+ (realUri == null ? uri : realUri.toString())
+ "]", e);
}
Including the result of that DefaultResponseErrorHandler from the RestTemplate.
And as you see we use here for the MessageHandlingException uri as well as the requestMessage.
Further this exception will be delegated to that jms channel.
Why isn't that enough for you?
From other side you can add <request-handler-advice-chain> to the <int-http:outbound-gateway> with ExpressionEvaluatingRequestHandlerAdvice to get deal with that MessageHandlingException just in place: http://docs.spring.io/spring-integration/docs/latest-ga/reference/html/messaging-endpoints-chapter.html#expression-advice