Spring Boot RestTemplate random ResourceAccessException: Connection reset errors - spring-boot

I have a system that implements 4 micro-services. The four services need to occasionally share information and they do it via RESTful requests using Spring's RestTemplate. Currently about 5%-10% of the requests fail with an exception like:
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://otherservice.com/path": Connection reset; nested exception is java.net.SocketException: Connection reset
Again, this appears random and only fails about 5%-10% of the time. I have tried multiple things but nothing seems to work. Currently I am trying this:
Configuration:
#Configuration
public class BeanConfiguration {
#Bean
public RestTemplate restTemplate() {
ClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(HttpClients.createDefault());
return new RestTemplate(requestFactory);
}
}
Service:
#Autowired
public MyService(RestTemplate restTemplate) {
this.restTemplate = restTemplate;
}
public Contact addContact(Contact contact) {
HttpEntity<Contact> entity = new HttpEntity<>(contact, authenticationTokenInfo.setTokenHeaders());
ResponseEntity<Contact> response = restTemplate.exchange(contact_base_url, HttpMethod.POST, entity, Contact.class);
return response.getBody();
}
I have tried a number of different approaches and nothing has made any difference. I tried this for example:
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
return builder.build();
}
I am logging all of the requests in each of my micro-services and it doesn't appear that the requests are actually hitting the other services. They just fail. According to the logs they fail in less than 50 ms so it isn't a timeout issue. For a couple of them I have implemented an exponential back-off retry but that isn't really a viable solution.

I also faced with same problem. Seems like the problem is with idle connections in pool.
This resolve problem in my case and no any error occasionally occurred
HttpClient httpClient = HttpClients
.custom()
.evictIdleConnections(60000, TimeUnit.MILLISECONDS)
.build();
HttpComponentsClientHttpRequestFactory clientHttpRequestFactory = new HttpComponentsClientHttpRequestFactory(httpClient);

Related

Connection is unable to read and will timeout after being idle

ServiceA uses a RestTemplate, created using a new instance of RestTemplateBuilder (not the default one created by Spring), to send requests to third-party services.
#Service
public class ServiceA {
private static final String URL_B = "...";
private RestTemplate restTemplate;
public ServiceA() {
restTemplate = new RestTemplateBuilder()
.uriTemplateHandler(new UriTemplateHandler() {
//...
})
.build();
}
public ResponseB getResponseAForServiceB() {
return restTemplate.getForObject(URL_B, ResponseB.class);
}
}
When the Spring Boot app starts up, requests made by ServiceA to ServiceB (third-party) will succeed without any issues. After about 1-1.5 hours of no requests, the first request made by ServiceA to ServiceB will cause a connection timeout for failing to read but a second request will succeed. What is causing the connection to not be able to read and timeout after being idle for a bit?

RestTemplate read timeout doesn't work

I have a springboot rest Service A calling rest service B using restTemplate.
Rest service A's restTemplate bean is created as follows with timeout settings as seen in the code snippet below.
#Bean
RestTemplate getRestTemplate()
{
CloseableHttpClient closeableHttpClient = HttpClientBuilder.create().build();
HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(closeableHttpClient);
RestTemplate restTemplate = new RestTemplate(requestFactory);
requestFactory.setConnectTimeout( 2000 );
requestFactory.setReadTimeout( 2000 );
return restTemplate;
}
A calls B as follows:
try{
restTemplate.postForEntity(urlSvcB, httpEntity, myObject.class);
}
catch (Exception ex){
.....some code here.....
}
When I put both A and B in bebug mode and wait at a breakpoint in B for more than 2 seconds, I except restTemplate call in A to detect a timeout of 2 seconds and straight away go in to the exception block BUT it doesn't. I also put a thread.sleep(5000) in B but still in vain.
Is there anything wrong I am doing because of which I don't see the expected ?
If you are using spring boot, then you could try:
#Bean
public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder)
{
return restTemplateBuilder
.setConnectTimeout(...)
.setReadTimeout(...)
.build();
}
If that is not okay, then in your current code, try setting all the props on requestFactory before creating a restTemplate OR test once by getting rid of CloseableHTTPClient like:
HttpComponentsClientHttpRequestFactory httpRequestFactory = new HttpComponentsClientHttpRequestFactory();
httpRequestFactory.setConnectionRequestTimeout(...);
httpRequestFactory.setConnectTimeout(...);
httpRequestFactory.setReadTimeout(...);
return new RestTemplate(httpRequestFactory);

Spring cloud multiple RestTemplate

I am using spring cloud: Spring Boot Application with Eureka + Ribbon default configuration.
I am using 2 RestTemplate configurations, both are #LoadBalanced currently and both of them have the same UriTemplateHandler.
I declared both the #SpringBootApplication and also the #RibbonClient(name="${service.name}") annotations.
My problem is:
When I am trying to access the first configured RestTemplate, the RestTemplate resolvs (by eureka and load balancing by ribbon) to a server , not as I requested as configured in the UriTemplateHandler.
For example: in the UriTemplateHandler I configured "A-Service" and in real time the restTemplate sends the httpRequest to "B-Service"
This behavior happens often, not just for a specific request, but it looks like it only happens when I'm accessing the first configured RestTemplate.
Is it a problem to use 2 RestTemplate with the same uri?
I have no idea why it happens, please advise.
When creating these rest templates as beans, name them uniquely, like e.g.
#LoadBalanced
#Bean("integrationRestTemplate")
public RestTemplate restTemplate() throws Exception {
// build and return your rest template
return ....
}
Then, the other one might be without any specific name e.g.
#Bean
public RestTemplate restTemplate() {
return new RestTemplate();
}
Now, if you have these two distinctive rest templates, you can inject the former one e.g. like that:
#Service
public class MyService {
private final RestTemplate restTemplate;
public ApplicantService(#Qualifier("integrationRestTemplate") RestTemplate restTemplate) {
this.restTemplate = restTemplate;
}
// service methods
...
}
Basically, the point is you can choose whatever rest template you want, by specifying a #Qualifier.

Spring Boot RestTemplate Basic Authentication using RestTemplateBuilder

In Spring Boot I'm trying to create a RestTemplate which will use basic authentication using
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
builder.basicAuthorization("username", "password");
RestTemplate template = builder.build();
return template;
}
I then inject the RestTemplate in my service class as
#Autowired
private RestTemplate restTemplate;
However, my requests fail with a 401 unauthorized exception:
Caused by: org.springframework.web.client.HttpClientErrorException: 401 Unauthorized
Using another REST Client (Postman) the requests to the same URL succeeds so I assume the basic authentication is not working correctly. From the debug output it looks as if the authentication header is not being set. What will make this work?
The problem is that you are using the RestTemplateBuilder in a wrong way. The RestTemplateBuilder is immutable. So when doing builder.basicAuthorization("username", "password") you actually get a new instance, with a BasicAuthorizationInterceptor added and configured, of the RestTemplateBuilder. (this applies to all configuration methods of the RestTemplateBuilder they all create a fresh copied instance).
However your code is discarding that specifically configured instance and you are basically using the non secured default RestTemplateBuilder.
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
builder.basicAuthorization("username", "password");
RestTemplate template = builder.build();
return template;
}
This code should be replaced with something like this.
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
return builder.basicAuthorization("username", "password").build();
}
Which will use the specifically configured instance.
One solution is to create the RestTemplate as follows:
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
RestTemplate template = builder.build();
template.setMessageConverters(
Arrays.asList(
new FormHttpMessageConverter(),
new StringHttpMessageConverter()
)
);
template.getInterceptors().add(new BasicAuthorizationInterceptor("username", "password"));
return template;
}

Spring RestTemplate timeout

I would like to set the connection timeouts for a rest service used by my web application. I'm using Spring's RestTemplate to talk to my service. I've done some research and I've found and used the xml below (in my application xml) which I believe is meant to set the timeout. I'm using Spring 3.0.
I've also seen the same problem here Timeout configuration for spring webservices with RestTemplate but the solutions don't seem that clean, I'd prefer to set the timeout values via Spring config
<bean id="RestOperations" class="org.springframework.web.client.RestTemplate">
<constructor-arg>
<bean class="org.springframework.http.client.CommonsClientHttpRequestFactory">
<property name="readTimeout" value="${restURL.connectionTimeout}" />
</bean>
</constructor-arg>
</bean>
It seems whatever I set the readTimeout to be I get the following:
Network cable disconnected:
Waits about 20 seconds and reports following exception:
org.springframework.web.client.ResourceAccessException: I/O error: No route to host: connect; nested exception is java.net.NoRouteToHostException: No route to host: connect
Url incorrect so 404 returned by rest service:
Waits about 10 seconds and reports following exception:
org.springframework.web.client.HttpClientErrorException: 404 Not Found
My requirements require shorter timeouts so I need to be able to change these. Any ideas as to what I'm doing wrong?
Many thanks.
For Spring Boot >= 1.4
#Configuration
public class AppConfig
{
#Bean
public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder)
{
return restTemplateBuilder
.setConnectTimeout(...)
.setReadTimeout(...)
.build();
}
}
For Spring Boot <= 1.3
#Configuration
public class AppConfig
{
#Bean
#ConfigurationProperties(prefix = "custom.rest.connection")
public HttpComponentsClientHttpRequestFactory customHttpRequestFactory()
{
return new HttpComponentsClientHttpRequestFactory();
}
#Bean
public RestTemplate customRestTemplate()
{
return new RestTemplate(customHttpRequestFactory());
}
}
then in your application.properties
custom.rest.connection.connection-request-timeout=...
custom.rest.connection.connect-timeout=...
custom.rest.connection.read-timeout=...
This works because HttpComponentsClientHttpRequestFactory has public setters connectionRequestTimeout, connectTimeout, and readTimeout and #ConfigurationProperties sets them for you.
For Spring 4.1 or Spring 5 without Spring Boot using #Configuration instead of XML
#Configuration
public class AppConfig
{
#Bean
public RestTemplate customRestTemplate()
{
HttpComponentsClientHttpRequestFactory httpRequestFactory = new HttpComponentsClientHttpRequestFactory();
httpRequestFactory.setConnectionRequestTimeout(...);
httpRequestFactory.setConnectTimeout(...);
httpRequestFactory.setReadTimeout(...);
return new RestTemplate(httpRequestFactory);
}
}
I finally got this working.
I think the fact that our project had two different versions of the commons-httpclient jar wasn't helping. Once I sorted that out I found you can do two things...
In code you can put the following:
HttpComponentsClientHttpRequestFactory rf =
(HttpComponentsClientHttpRequestFactory) restTemplate.getRequestFactory();
rf.setReadTimeout(1 * 1000);
rf.setConnectTimeout(1 * 1000);
The first time this code is called it will set the timeout for the HttpComponentsClientHttpRequestFactory class used by the RestTemplate. Therefore, all subsequent calls made by RestTemplate will use the timeout settings defined above.
Or the better option is to do this:
<bean id="RestOperations" class="org.springframework.web.client.RestTemplate">
<constructor-arg>
<bean class="org.springframework.http.client.HttpComponentsClientHttpRequestFactory">
<property name="readTimeout" value="${application.urlReadTimeout}" />
<property name="connectTimeout" value="${application.urlConnectionTimeout}" />
</bean>
</constructor-arg>
</bean>
Where I use the RestOperations interface in my code and get the timeout values from a properties file.
This question is the first link for a Spring Boot search, therefore, would be great to put here the solution recommended in the official documentation. Spring Boot has its own convenience bean RestTemplateBuilder:
#Bean
public RestTemplate restTemplate(
RestTemplateBuilder restTemplateBuilder) {
return restTemplateBuilder
.setConnectTimeout(Duration.ofSeconds(500))
.setReadTimeout(Duration.ofSeconds(500))
.build();
}
Manual creation of RestTemplate instances is a potentially troublesome approach because other auto-configured beans are not being injected in manually created instances.
Here are my 2 cents. Nothing new, but some explanations, improvements and newer code.
By default, RestTemplate has infinite timeout.
There are two kinds of timeouts: connection timeout and read time out. For instance, I could connect to the server but I could not read data. The application was hanging and you have no clue what's going on.
I am going to use annotations, which these days are preferred over XML.
#Configuration
public class AppConfig {
#Bean
public RestTemplate restTemplate() {
var factory = new SimpleClientHttpRequestFactory();
factory.setConnectTimeout(3000);
factory.setReadTimeout(3000);
return new RestTemplate(factory);
}
}
Here we use SimpleClientHttpRequestFactory to set the connection and read time outs.
It is then passed to the constructor of RestTemplate.
#Configuration
public class AppConfig {
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
return builder
.setConnectTimeout(Duration.ofMillis(3000))
.setReadTimeout(Duration.ofMillis(3000))
.build();
}
}
In the second solution, we use the RestTemplateBuilder. Also notice the parameters of the two methods: they take Duration. The overloaded methods that take directly milliseconds are now deprecated.
Edit
Tested with Spring Boot 2.1.0 and Java 11.
Here is a really simple way to set the timeout:
RestTemplate restTemplate = new RestTemplate(getClientHttpRequestFactory());
private ClientHttpRequestFactory getClientHttpRequestFactory() {
int timeout = 5000;
HttpComponentsClientHttpRequestFactory clientHttpRequestFactory =
new HttpComponentsClientHttpRequestFactory();
clientHttpRequestFactory.setConnectTimeout(timeout);
return clientHttpRequestFactory;
}
RestTemplate timeout with SimpleClientHttpRequestFactory
To programmatically override the timeout properties, we can customize the SimpleClientHttpRequestFactory class as below.
Override timeout with SimpleClientHttpRequestFactory
//Create resttemplate
RestTemplate restTemplate = new RestTemplate(getClientHttpRequestFactory());
//Override timeouts in request factory
private SimpleClientHttpRequestFactory getClientHttpRequestFactory()
{
SimpleClientHttpRequestFactory clientHttpRequestFactory
= new SimpleClientHttpRequestFactory();
//Connect timeout
clientHttpRequestFactory.setConnectTimeout(10_000);
//Read timeout
clientHttpRequestFactory.setReadTimeout(10_000);
return clientHttpRequestFactory;
}
RestTemplate timeout with HttpComponentsClientHttpRequestFactory
SimpleClientHttpRequestFactory helps in setting timeout but it is very limited in functionality and may not prove sufficient in realtime applications. In production code, we may want to use HttpComponentsClientHttpRequestFactory which support HTTP Client library along with resttemplate.
HTTPClient provides other useful features such as connection pool, idle connection management etc.
Read More : Spring RestTemplate + HttpClient configuration example
Override timeout with HttpComponentsClientHttpRequestFactory
//Create resttemplate
RestTemplate restTemplate = new RestTemplate(getClientHttpRequestFactory());
//Override timeouts in request factory
private SimpleClientHttpRequestFactory getClientHttpRequestFactory()
{
HttpComponentsClientHttpRequestFactory clientHttpRequestFactory
= new HttpComponentsClientHttpRequestFactory();
//Connect timeout
clientHttpRequestFactory.setConnectTimeout(10_000);
//Read timeout
clientHttpRequestFactory.setReadTimeout(10_000);
return clientHttpRequestFactory;
}
reference: Spring RestTemplate timeout configuration example
Simple timeout for restTemplate.I have set the read and write timeout for 3 seconds.
#Bean
public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder){
RestTemplate restTemplate= restTemplateBuilder.setConnectTimeout(Duration.ofMillis(3000)).setReadTimeout(Duration.ofMillis(3000)).build();
return restTemplate;
}
I had a similar scenario, but was also required to set a Proxy. The simplest way I could see to do this was to extend the SimpleClientHttpRequestFactory for the ease of setting the proxy (different proxies for non-prod vs prod). This should still work even if you don't require the proxy though. Then in my extended class I override the openConnection(URL url, Proxy proxy) method, using the same as the source, but just setting the timeouts before returning.
#Override
protected HttpURLConnection openConnection(URL url, Proxy proxy) throws IOException {
URLConnection urlConnection = proxy != null ? url.openConnection(proxy) : url.openConnection();
Assert.isInstanceOf(HttpURLConnection.class, urlConnection);
urlConnection.setConnectTimeout(5000);
urlConnection.setReadTimeout(5000);
return (HttpURLConnection) urlConnection;
}
To expand on benscabbia's answer:
private RestTemplate restCaller = new RestTemplate(getClientHttpRequestFactory());
private ClientHttpRequestFactory getClientHttpRequestFactory() {
int connectionTimeout = 5000; // milliseconds
int socketTimeout = 10000; // milliseconds
RequestConfig config = RequestConfig.custom()
.setConnectTimeout(connectionTimeout)
.setConnectionRequestTimeout(connectionTimeout)
.setSocketTimeout(socketTimeout)
.build();
CloseableHttpClient client = HttpClientBuilder
.create()
.setDefaultRequestConfig(config)
.build();
return new HttpComponentsClientHttpRequestFactory(client);
}
private static RestTemplate restTemplate;
static {
HttpComponentsClientHttpRequestFactory rf = new HttpComponentsClientHttpRequestFactory();
rf.setReadTimeout(3 * 1000);
rf.setConnectTimeout(2 * 1000);
restTemplate = new RestTemplate(rf);
restTemplate.getMessageConverters()
.add(0, new StringHttpMessageConverter(StandardCharsets.UTF_8));
}
Setting the timeout only in RestTemplateBuilder didn't work for me when i was using Apache's httpcomponents. I had to set the timeout in the RequestFactory as well.
Here's the entire code:
public RestTemplate templateBuilder() {
RestTemplate restTemplate = this.restTemplateBuilder
.setConnectTimeout(Duration.ofSeconds(connectTimeout))
.setReadTimeout(Duration.ofSeconds(readTimeout))
.build();
HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
requestFactory.setConnectTimeout((int) connectTimeout * 1000);
requestFactory.setReadTimeout((int) readTimeout * 1000);
requestFactory.setConnectionRequestTimeout((int) connectTimeout * 1000);
restTemplate.setRequestFactory(new BufferingClientHttpRequestFactory(requestFactory));
return restTemplate;
}

Resources