my resttemplate.exchange() failed on a POST request, with the server returning a 500 error.
I tried to set the root logging level to DEBUG, but nothing was logged before the 500 error was returned. to make sure that my logging config is right, I added a line before the resttemplate call
HttpClient client = new DefaultHttpClient();
client.execute(new HttpGet("http://google.com"));
in this case indeed a lot of logging messages appeared.
so how can I make RestTemplate export the debugging data?
Thanks
Yang
From your analysis it seems that you expect RestTemplate to use Apache HttpClient.
However, by default, Spring RestTemplate does not use Apache HttpClient but uses the JDK facilities (java.net.URL#openConnection() etc.) by means of SimpleClientHttpRequestFactory.
org.springframework.http.client.support.HttpAccessor declares:
private ClientHttpRequestFactory requestFactory = new
SimpleClientHttpRequestFactory();
As far as I know, this client does not support logging requests/responses.
To change RestTemplate to use HttpClient, try this:
new RestTemplate(new HttpComponentsClientHttpRequestFactory());
Logging configuration should then enable category org.apache.http.wire at level debug for complete requests/responses to be logged.
Related
I have one issue in PRD. we recently released a springboot application and it has REST API exposed. Mobile/web APP call a legacy spring application which is in spring [not sprintboot] and it is a web applicationwhich then routes and makes a call to the these failing apis in new springboot. We are seeing timeout exception for these apis only .
there are lots of other OUTBOUND api calls made from spring legacy web application to other applications eg : login API [which has apis heavy traffic but these legacy apis work well and call other legacy applications.
There are no exception/error in logs in springboot application which has these REST API exposed. Infact we only see timeout in spring web application -meaning connection is exhausted but that does not explain why other apis OUTBOUND call are not failing which use same wrapper HTTPClient. Those which fail with timeout dont have request logs in springboot [ obviously because they dont leave spring web application tomcat JVM and die there due to timeout ]
So if we say connection pool is exhausted, the other heavey traffic OUTBOUnd calls should also face same issue but we dont see that.
All API call OUTWARD use HTTPCLient [apache.]
Not clear what is causing issue. I also explicitly defined below in new springboot for server side [I just did it to see if that makes difference but in vain]:
server:
tomcat:
connection-timeout: 10s
max-connections: 20000
max-threads: 500
min-spare-threads: 10
tomcat Log at spring web applicaiton [caller]:
org.apache.http.conn.ConnectionPoolTimeoutException
org.apache.http.conn.ConnectionPoolTimeoutException.Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:313)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
at
Any inputs?
Code snippet of Wrapper HTTPClient :
SSLContext sslContext = SSLContexts.createDefault();
HostnameVerifier hostnameVerifier = new DefaultHostnameVerifier();
SSLConnectionSocketFactory secureSSLConnectionSocketFactory = new SSLConnectionSocketFactory(
sslContext,
sslProtocolsArray,
ciphersArray,
hostnameVerifier);
ConnectionSocketFactory nonSecureConnectionSocketFactory = PlainConnectionSocketFactory.getSocketFactory();
Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder
.<ConnectionSocketFactory>create()
.register("https", secureSSLConnectionSocketFactory)
.register("http", nonSecureConnectionSocketFactory)
.build();
securePoolingConnectionManager = new PoolingHttpClientConnectionManager(socketFactoryRegistry);
securePoolingConnectionManager.setMaxTotal(this.connectionMgrMaxTotalSecure);
securePoolingConnectionManager.setDefaultMaxPerRoute(this.connectionMgrMaxPerRouteSecure);
SocketConfig secureSocketConfig = SocketConfig
.custom()
.setSoKeepAlive(true)
.setTcpNoDelay(true)
.build();
secureHttpsClient = HttpClients
.custom()
.setSSLSocketFactory(secureSSLConnectionSocketFactory)
.setConnectionManager(securePoolingConnectionManager)
.setDefaultRequestConfig(secureRequestConfig)
.setDefaultSocketConfig(secureSocketConfig)
.disableAutomaticRetries()
.build();
Stacktrace after above is just failing at wrapper HTTPClient method where call is invoked :
protected String execute(HttpClient httpclient, HttpRequestBase http) throws IOException {
String result;
ResponseHandler<String> responseHandler = new BasicResponseHandler();
result = httpclient.execute(http, responseHandler);
return result;
}
So I have to dig in another wrapper which was also using this HTTP pool and was being used in our legacy which was leaking. Closing this. Fortunately there was pool statistics api exposed so that I can see leased connection count which confirmed leaking. Since this second wrapper was used rerely and we had used in this release this was suspect and removing it solved the issue. It is another matter to dig that wrapper and find out how the pool was handled but the cause was caught.
I have a restemplate which can make call to multiple external systems over https.
I configured the resttemplate like so :
SSLContext sslContext = SSLContextBuilder
.create()
.loadTrustMaterial(key, keyPassword)
.build();
HttpClient client = HttpClients
.custom()
.setSSLContext(sslContext)
.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE)
.build();
HttpComponentsClientHttpRequestFactory httpComponentsClientHttpRequestFactory = new HttpComponentsClientHttpRequestFactory(
client);
httpComponentsClientHttpRequestFactory.setConnectTimeout(5000);
httpComponentsClientHttpRequestFactory.setReadTimeout(30000);
return new RestTemplate(httpComponentsClientHttpRequestFactory);
I have setup 2 mock services on 2 separate VMs with ssl enabled and I am testing this restemplate by calling those services over https. And it works.
What I want to confirm is that configuring the restemplate as shown in the above code and importing the certificates of the different services to be called in by the client truststore should work without any additional configurations right?
I am confused because in the code in this example here : https://github.com/jonashackt/spring-boot-rest-clientcertificates-docker-compose the author has used 2 different factories i.e. serverTomClientHttpRequestFactory & serverAliceClientHttpRequestFactory, however mine works with a single restemplate. can someone please shed some light on this topic?
In order to configure RestTemplate I use the following configuration:
HttpComponentsClientHttpRequestFactory httpRequestFactory = new HttpComponentsClientHttpRequestFactory(HttpClients.createDefault());
httpRequestFactory.setConnectTimeout(connectionTimeoutMs);
httpRequestFactory.setConnectionRequestTimeout(readTimeoutMs);
httpRequestFactory.setReadTimeout(readTimeoutMs);
RestTemplate restTemplate = new RestTemplate(httpRequestFactory);
I understand the purpose of connection and read timeouts. But I don't understand the purpose of connection request timeout. It also not clear from Javadoc what does it mean. Could you please explain it ?
As the docs say :
Set the timeout in milliseconds used when requesting a connection from
the connection manager using the underlying HttpClient.
It means the maximum amount of time you will allow to the connection manager to give you an available connection from its pool (so it has nothing to do with the RESTservice itself you'll reach).
To define a custom connection manager, you can use this :
CloseableHttpClient httpClientBuilder = HttpClientBuilder.create().setConnectionManager(...).build();
HttpComponentsClientHttpRequestFactory httpRequestFactory = new HttpComponentsClientHttpRequestFactory(httpClientBuilder);
I have one question regarding Spring WebClient
In my application I need to do many similar API calls, sometimes I need change headers in the calls (Authentication token). So the question arises, what would be better of the two options:
To create one WebClient for all incoming requests to MyService.class, by making it private final field, like code below:
private final WebClient webClient = WebClient.builder()
.baseUrl("https://another_host.com/api/get_inf")
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.defaultHeader(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE)
.build();
Here arises another question: is WebClient thread-safe? (because service is used by many threads)
To create new WebClient for each new request incoming to service class.
I want to provide maximum performance, and to use it in right way, but I don't know how WebClient works inside it, and how it expects to be used.
Thank you.
Two key things here about WebClient:
Its HTTP resources (connections, caches, etc) are managed by the underlying library, referenced by the ClientHttpConnector that you can configure on the WebClient
WebClient is immutable
With that in mind, you should try to reuse the same ClientHttpConnector across your application, because this will share the connection pool - this is arguably the most important thing for performance. This means you should try to derive all WebClient instances from the same WebClient.create() call. Spring Boot helps you with that by creating and configuring for you a WebClient.Builder bean that you can inject anywhere in your app.
Because WebClient is immutable it is thread-safe. WebClient is meant to be used in a reactive environment, where nothing is tied to a particular thread (this doesn't mean you cannot use in a traditional Servlet application).
If you'd like to change the way requests are made, there are several ways to achieve that:
configure things in the builder phase
WebClient baseClient = WebClient.create().baseUrl("https://example.org");
configure things on a per-request basis
Mono<ClientResponse> response = baseClient.get().uri("/resource")
.header("token", "secret").exchange();
create a new client instance out of an existing one
// mutate() will *copy* the builder state and create a new one out of it
WebClient authClient = baseClient.mutate()
.defaultHeaders(headers -> {headers.add("token", "secret");})
.build();
From my experience, if you are calling an external API on a server you have no control over, don't use WebClient at all, or use it with the pooling mechanism turned off. Any performance gains from connection pooling are greatly overweighed by the assumptions built into the (default reactor-netty) library that will cause random errors on one API call when another was abruptly terminated by the remote host, etc. In some cases, you don't even know where the error occurred because the calls are all made from a shared worker thread.
I made the mistake of using WebClient because the doc for RestTemplate said it would be deprecated in the future. In hindsight, I would go with regular HttpClient or Apache Commons HttpClient, but if you are like me and already implemented with WebClient, you can turn off the pooling by creating your WebClient as follows:
private WebClient createWebClient(int timeout) {
TcpClient tcpClient = TcpClient.newConnection();
HttpClient httpClient = HttpClient.from(tcpClient)
.tcpConfiguration(client -> client.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, timeout * 1000)
.doOnConnected(conn -> conn.addHandlerLast(new ReadTimeoutHandler(timeout))));
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
}
*** Creating a separate WebClient does not mean that WebClient will have a separate connection pool. Just look at the code for HttpClient.create - it calls HttpResources.get() to get the global resources. You could provide the pool settings manually but considering the errors that occur even with the default setup, I don't consider it worth the risk.
I need to access a JAX-WS webservice protected by SSL using Jersey Client. I have successfully got this working with limited configuration and just letting the Client use the default HTTPURLConnection API.
This approach wont work however for my overall solution because it does not allow me the flexibility to change the credentials used when making a request to the WS. I'm trying to use DefaultHTTPClient instead and then passing it to the Client object on intialization.
NTCredentials credentials = new NTCredentials("username", "password",
computerName, domainName);
DefaultHttpClient httpClienttemp = new DefaultHttpClient();
DefaultHttpClient httpClient = wrapClient(httpClienttemp);
httpClient.getCredentialsProvider().setCredentials(AuthScope.ANY, credentials );
ClientConfig config = new DefaultClientConfig();
The wrapClient method creates an X509TrustManager and overrides the necessary methods so that all certificates are accepted. It also creates a SchemeRegistry entry for https access on port 443. This configuration results in a Connection refused exception.
The strange thing is, if i add an additional entry in the SchemeRegistry for http and give it a port of 443 then the request does get sent however a Connection Reset exception then gets thrown.
The Url i use to create the WebResource object is https however the SOAPAction i declare in the header uses http. Any ideas where im going wrong?
This is a limitation of the default HTTP Client (com.sun.jersey.api.client.Client) documented in the Jersey docs. You will have to use Apache HTTP Client to achieve this functionality.
Looks like someone already recommended doing this in the answer to your previous question: Jersey Client API - authentication.
EDIT: Corrected reference to the default Jersey HTTP Client to avoid confusion.