I am building out a web service which proxies and does some slight manipulation of HTTP requests. I'm handling requests going to multiple hosts of the same type but of which I don't know of until run time (I consume a web service that gives the host IPs). Each host that I interact with has different credentials (Basic-Auth, fetched from a non-local database, credentials change periodically). The way I handle things today is pretty naive. For every request, I am constructing a new RestTemplate like so:
public static RestOperations getRestOperations(int timeout, String username, String password)
{
RequestConfig requestConfig = RequestConfig.custom().setConnectionRequestTimeout(timeout).setConnectTimeout(timeout).setSocketTimeout(timeout).build();
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password));
CloseableHttpClient httpclient = HttpClients.custom()
.setDefaultCredentialsProvider(credsProvider)
.setDefaultRequestConfig(requestConfig)
.build();
ClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(
httpclient);
return new RestTemplate(requestFactory);
}
So each Controller method always starts out with:
UsernamePassword userPass = credentialService.getCredentials(request.getRemoteHost())
RestOperations restOps = getRestOperations(userPass.getUser(), userPass.getPass(), TIMEOUT_IN_MILLIS);
It seems to me that since I'm constructing a new RestTemplate with each request that any previous connections that have been made between my server and the host are not being reused.
Is this the case? If so, then it seems I will need some sort of RestTemplateFactory which can cache RestTemplate instances based on the host IP address so that connections can be reused. However if I do that, then I need some mechanism that makes sure that the credentials haven't changed and to check and update credentials if they do change. Is there a better solution?
Related
I have rest template config similar to the following. I am trying to release a connection from the pool if I get a status code that does not equal 2XX (long story but need this code). Is here a way I can get the connection Manager and release a specific connection?
#Bean
public RestTemplate restTemplate() {
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(100);
connectionManager.setDefaultMaxPerRoute(20);
RequestConfig requestConfig = RequestConfig
.custom()
.setConnectionRequestTimeout(5000) // timeout to get connection from pool
.setSocketTimeout(5000) // standard connection timeout
.setConnectTimeout(5000) // standard connection timeout
.build();
HttpClient httpClient = HttpClientBuilder.create()
.setConnectionManager(connectionManager)
.setDefaultRequestConfig(requestConfig).build();
ClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(httpClient);
return new RestTemplate(requestFactory);
}
Looking for a way to accomplish something similar to the following
if(!httpStatusCode.substr(1).equals("2")) {
restTemplate.getConnectionPool().relase().thisConnection();
}
enter code here
I am building an application that reads JSON response from certain endpoints and I am trying to authenticate in Apache HttpClient using NTLM authentication:
The class that is responsible for authentication HttpConnector tries to authentice right after its instantiation:
public static HttpConnector on(String username, String password) {
HttpConnector connector = new HttpConnector(username, password);
Credentials credentials = new NTCredentials(username, password, "host", "domain");
connector.getHttpClient().getState().setCredentials(AuthScope.ANY, credentials);
return connector;
}
but I always get response code 401 Unauthorized. As I read in internet including here in Stackoverflow I used NTCredentials that I am trying to set globally in the HttpClient. I have tested the endpoints in Postman, there I get the JSON response successfully but HttpClient cannot connect.
In the code I use GetMethod: httpMethod = new GetMethod(url);
Here I have also tried to configure authentication but it still does not work:
private void configureMethod(HttpMethod httpMethod) throws MalformedChallengeException {
httpMethod.getHostAuthState().setAuthRequested(true);
httpMethod.setRequestHeader("Content-Type", "application/json; charset=UTF-8");
httpMethod.setDoAuthentication(true);
httpMethod.getParams().setParameter(
HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
httpMethod.getHostAuthState().setAuthRequested(true);
httpMethod.getHostAuthState().setAuthScheme(
new NTLMScheme(
String.format("ntlm %s:%s", username, password)));
}
During debugging I see that I get: Connection reset by peer: socket write error. It happens in HttpMethodDirector::executeWithRetry(final HttpMethod method) method.
Can someone help me, what is the correctNTLM authentication setup in Apache HttpClient. Can I really use the global set of credentials or I have to setup credentials to every HttpMethod I create and how?
Thank you in advance!
I fixed this by formatting the client the following way:
HttpClient httpClient = new DefaultHttpClient();
httpClient.getCredentialsProvider().setCredentials(
new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT),
new NTCredentials(username, password, HOST, MY_DOMAIN));
And the used not GetMethod but HttpGet:
HttpGet = getRequest = new HttpGet(url);
and this time the connection owas successful.
I have a RestTemplate call to an API (get). This call, is the only we have of GET type, and go through a proxy. It seems that sometimes during a week, the call returns a 403 Forbidden with this exception: "sun.security.validator.ValidatorException"
We have a certificate between Spring and the API, but the certificate works fine (the application returns thousands of "200 ok" during a day).
But sometimes, only this call (not others that are POST) returns a "403 Forbidden".
We have done:
Launch Jmeter with curl through the proxy (everything seems ok)
Disable the TrustStore only to test (the result is ko)
This is the RestTemplate code:
SSLConnectionSocketFactory socketFactory;
socketFactory = new SSLConnectionSocketFactory(new SSLContextBuilder()
.loadTrustMaterial(ResourceUtils.getFile(this.trustStorePath), this.trustStorePassword.toCharArray())
.loadKeyMaterial(ResourceUtils.getFile(this.keyStorePath), this.keystorePassword.toCharArray(),
this.keystorePassword.toCharArray())
.build(), NoopHostnameVerifier.INSTANCE);
CloseableHttpClient client = HttpClients.custom().setSSLSocketFactory(socketFactory).setProxy(host)
.disableCookieManagement().disableRedirectHandling().build();
ClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(client);
RestTemplate restTemplateVar = new RestTemplate(requestFactory);
And this is the call:
response = this.restTemplate.getForEntity(this.host, String.class);
Could the number of concurrent connections be the cause?
Why only with GET and sometimes?
And the last one: If we change RestTemplate by Httpconnection, the result could be different?
Thank in advance
Setting this properties works fine (it depends on your metrics)
.setMaxConnTotal(1000)
.setMaxConnPerRoute(40)
CloseableHttpClient client = HttpClients.custom()
.setSSLSocketFactory(socketFactory)
.setProxy(host)
.disableCookieManagement()
.disableRedirectHandling()
.setMaxConnTotal(1000)
.setMaxConnPerRoute(40)
.build();
I'm using a Spring OAuth2RestTemplate with ClientCredentialsResourceDetails to acquire an API authorization token. The authorization server and the API endpoints are hidden behind the same load balancers (LB). We have an issues where the first connection to the API endpoint, after acquiring the token, fails with a 404 error message but subsequent calls to the same API endpoint with the same token are successful. I believe the LB is miss-configured in some way but we've been asked if we could try using separate TCP sessions for the acquisition of the token and then the REST call. Is there a way to get the Spring RestTemplate to do this?
UPDATE
Here's how I create and configure the template:
#Bean
public OAuth2RestTemplate oauth2RestTemplate(
#Value("${token.uri}") final String tokenUri,
#Value("${token.clientId:client}") final String clientId,
#Value("${token.secret:secret}") final String clientSecret,
#Value("${token.scope:platform}") final String scope,
final MappingJackson2HttpMessageConverter customJackson2HttpMessageConverter)
{
ClientCredentialsResourceDetails rd = new
ClientCredentialsResourceDetails();
rd.setAuthenticationScheme(AuthenticationScheme.header);
rd.setAccessTokenUri(tokenUri);
rd.setClientId(clientId);
rd.setClientSecret(clientSecret);
rd.setScope(Arrays.asList(scope));
OAuth2RestTemplate rt = new OAuth2RestTemplate(rd);
List<HttpMessageConverter<?>> converters = rt.getMessageConverters();
converters.add(customJackson2HttpMessageConverter);
rt.setMessageConverters(converters);
return rt;
}
and here's the call to the api:
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Arrays.asList(MediaType.APPLICATION_JSON));
headers.set("Connection", "close"); // hmm, gets replace by keep-alive on the token api request!
HttpEntity<String> entity = new HttpEntity<String>(headers);
ResponseEntity<MyObject[]> response = restTemplate.exchange(
"http://example.com/api/v1/rest/method",
HttpMethod.GET, entity, MyObject[].class);
Thanks.
Try adding the Connection request header with value as close while sending your request using resttemplate. This should force the TCP connection to be closed after each request. Not very performant though.
HttpHeaders headers = new HttpHeaders();
headers.set("Connection", "close");
This is only for the "but we've been asked if we could try using separate TCP sessions for the acquisition of the token and then the REST call." part of your question. It will not help resolve your 404 (that does seem to be an LB issue).
UPDATE: Since you're using OAuth2RestTemplate, create a ClientHttpRequestInterceptor which injects the header.
public class ConnectionCloseInterceptor implements ClientHttpRequestInterceptor {
public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution) throws IOException {
HttpHeaders headers = request.getHeaders();
headers.add("Connection", "close");
return execution.execute(request, body);
}
}
Use it in your rest template (OAuth2RestTemplate extends RestTemplate so below applies to both) like so (when you create the rest template bean):
List<ClientHttpRequestInterceptor> currentInterceptors = new ArrayList<>(restTemplate.getInterceptors()); //Don't want to lose the other interceptors!
currentInterceptors.add(new ConnectionCloseInterceptor()); //Add ours
restTemplate.setInterceptors(currentInterceptors);
I am writing a multi-threaded REST client that uses different APIs for different purposes, For making these requests I am using an HttpClient with its different methods (GET,PUT,POST)
Thread 1:
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(url);
httpclient.execute(httppost);
methodThatNeedsHttpClient(httpclient);
public void methodThatNeedsHttpClient(HttpClient client) {
//perform other GET/POST/PUT requests
}
Thread 2:
DefaultHttpClient httpclient2 = new DefaultHttpClient();
HttpPost httppost = new HttpPost(url);
httpclient2.execute(httppost);
// Other methods
I read that for managing httpConnections I should using a Connection Manager. I am on version 4.5 of the client, which connection manager should I be using? How does the connection manager ensure connections dont leak and are used efficiently?
I tried the following implementation :
PoolingClientConnectionManager connectionManager = new PoolingClientConnectionManager();
connectionManager.setMaxTotal(5);
// Perform REST operations
client.getConnectionManager().shutdown();
But I am not sure how connections are managed in the pool , for a multithreaded system, does the connection manager be initialized in every thread?
In most cases, connection pool should be a shared one, accessible by all httpClient instances.
When creating httpClient,
CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.setConnectionManagerShared(true)
.build();
And this will release the connection back to the pool,
EntityUtils.consume(httpResponse.getEntity());
While will close the connection,
httpResponse.close();
httpClient.close();
Since we have setConnectionManagerShared(true), httpClient.close() wouldn't shunt down the connection pool.