Configure Rest High Client with Elastic Search proxy - spring-boot

is there any way to configure my rest high client to connect with es using proxy. My configuration is
#Override
#Bean
public RestHighLevelClient elasticsearchClient() {
return new RestHighLevelClient(RestClient.builder(HttpHost.create(elasticSearchUrl)));}
My elastic search url is: aaa.bbbb.ccc.company.com/api/elastic-search-proxy
In that case I get No such host is known (aaa.bbbb.ccc.company.com/api/elastic-search-proxy) what is clear for me but is there any option to configure it ?

Its mentioned in the Elasticsearch documentation of JHLRC initialization , use below code:
RestClientBuilder builder = RestClient.builder(
new HttpHost("localhost", 9200, "http"));
builder.setHttpClientConfigCallback(new HttpClientConfigCallback() {
#Override
public HttpAsyncClientBuilder customizeHttpClient(
HttpAsyncClientBuilder httpClientBuilder) {
return httpClientBuilder.setProxy(
new HttpHost("proxy", 9000, "http"));
}
});
Set a callback that allows to modify the http client configuration (e.g. encrypted communication over ssl, or anything that the org.apache.http.impl.nio.client.HttpAsyncClientBuilder allows to set)
So in your case, you need to give your original host in below code
new HttpHost("localhost", 9200, "http"));
And then you need to define a callback to your proxy server in setHttpClientConfigCallback call back.
new HttpHost("proxy", 9000, "http"));

If someone using the latest Elastic Java client 8.x, you can use this way of configuring the proxy to your rest client. (note the proxy should already set in system properties). It might help someone.
val restClientBuilder = RestClient.builder(
HttpHost(randomHost, 443, https)
)*.setHttpClientConfigCallback {
HttpAsyncClientBuilder.create().useSystemProperties()
}*

Related

How to connect elastic server(with protected) from java springboot?

I can able to connect with elastic server without security by using below code.But i can connect elastic server with user credentials.
#Bean
public RestClient getRestClient() {
RestClient restClient = RestClient.builder(
new HttpHost("localhost", 9200)).build();
return restClient;
}
#Bean
public ElasticsearchTransport getElasticsearchTransport() {
return new RestClientTransport(
getRestClient(), new JacksonJsonpMapper());
}
#Bean
public ElasticsearchClient getElasticsearchClient(){
ElasticsearchClient client = new ElasticsearchClient(getElasticsearchTransport());
return client;
}`your text`
But my Elasticyour textSearch is protected with username and password. Httphost doesn't have username and password parameters,Could you please help me with that....
Thanks for the Advance.
I need to connect elastic server from springboot(java) while my elastic server has username and password .
I need to connect elastic server from springboot ,and i need to create index from springboot.

Unable to connect to SSL enabled Elastic Search from Google Dataflow

Unable to connect to SSL enabled Elastic Search using Google Cloud Dataflow. I have used google provide template from git I modified the source code to pass keystore, keystorepassword, truststore to the job along with the Elastic Search credentials
Elastic Search Class file
config = config.withKeystorePassword("XXXXXXXXXXXXXXXXXXXXXXXXXXX")
.withKeystorePath("PATH_TO_KEYSTORE")
Exception on running the job is as below
Caused by: java.lang.IllegalArgumentException: Cannot get
Elasticsearch version at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.getBackendVersion(ElasticsearchIO.java:1546)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.setup(ElasticsearchIO.java:1336)
Caused by: org.elasticsearch.client.ResponseException: method [GET],
host [https://elastic.irds.opus.dev.gcp.db.com], URI [/], status line
[HTTP/1.1 401 Unauthorized]
{"error":{"root_cause":[{"type":"security_exception","reason":"missing
authentication credentials for REST request
[/]","header":{"WWW-Authenticate":"Basic realm="security"
charset="UTF-8""}}],"type":"security_exception","reason":"missing
authentication credentials for REST request
[/]","header":{"WWW-Authenticate":"Basic realm="security"
charset="UTF-8""}},"status":401} at
org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.getBackendVersion(ElasticsearchIO.java:1530)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.setup(ElasticsearchIO.java:1336)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn$DoFnInvoker.invokeSetup(Unknown
Source)
We wrote a sample java program to ensure elastic search connectivity, which works fine as expected
SSLContextBuilder sslBuilder = SSLContexts.custom()
.loadKeyMaterial(keyStore,"*************".toCharArray())
.loadTrustMaterial(truststore,null);
final SSLContext sslContext = sslBuilder.build();
RestClientBuilder builder = RestClient.builder(
new HttpHost("https://X.X.X.X", 443, "https"))
.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
public HttpAsyncClientBuilder customizeHttpClient(
final HttpAsyncClientBuilder httpAsyncClientBuilder) {
httpAsyncClientBuilder.disableAuthCaching();
return httpAsyncClientBuilder.setSSLContext(sslContext);
}
});
RestHighLevelClient client = new RestHighLevelClient(builder);
GetRequest getRequest = new GetRequest("test_index1/_search", "1");
GetResponse response = client.get(getRequest, RequestOptions.DEFAULT);
System.out.println("\nSearch results:");
System.out.println(response.getSourceAsString());
Could you please provide some inputs on what changes do i need to make on dataflow template to connect to SSL enabled Elastic search

Elasticsearch RestHighLevelClient behind a corporate firewall via a proxy

I am trying to access cloud Elasticsearch installation from our network that requires using a proxy for external requests. This is the snippet of code I use to pass Elasticsearch credentials and our proxy settings:
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(elasticUser, elasticPassword));
RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost(hostName,port,"https")).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setProxy(new HttpHost(proxyURL", proxyPort, "http")));
RestHighLevelClient client = new RestHighLevelClient(restClientBuilder);
This results in this response from ES:
"Exception in thread "main" ElasticsearchStatusException[Elasticsearch exception
[type=security_exception, reason=action [indices:data/read/search] requires authentication]]"
It appears that Elasticsearch credentials are not passed for some reason.
should have been done like this:
RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost(hostName, port, "https"))
.setHttpClientConfigCallback(clientBuilder -> {
clientBuilder.setDefaultCredentialsProvider(credentialsProvider);
clientBuilder.setProxy(new HttpHost(proxyURL, proxyPort, "http"));
return clientBuilder;
});

Handling multiple RestTemplate instances for multiple hosts

I am building out a web service which proxies and does some slight manipulation of HTTP requests. I'm handling requests going to multiple hosts of the same type but of which I don't know of until run time (I consume a web service that gives the host IPs). Each host that I interact with has different credentials (Basic-Auth, fetched from a non-local database, credentials change periodically). The way I handle things today is pretty naive. For every request, I am constructing a new RestTemplate like so:
public static RestOperations getRestOperations(int timeout, String username, String password)
{
RequestConfig requestConfig = RequestConfig.custom().setConnectionRequestTimeout(timeout).setConnectTimeout(timeout).setSocketTimeout(timeout).build();
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password));
CloseableHttpClient httpclient = HttpClients.custom()
.setDefaultCredentialsProvider(credsProvider)
.setDefaultRequestConfig(requestConfig)
.build();
ClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(
httpclient);
return new RestTemplate(requestFactory);
}
So each Controller method always starts out with:
UsernamePassword userPass = credentialService.getCredentials(request.getRemoteHost())
RestOperations restOps = getRestOperations(userPass.getUser(), userPass.getPass(), TIMEOUT_IN_MILLIS);
It seems to me that since I'm constructing a new RestTemplate with each request that any previous connections that have been made between my server and the host are not being reused.
Is this the case? If so, then it seems I will need some sort of RestTemplateFactory which can cache RestTemplate instances based on the host IP address so that connections can be reused. However if I do that, then I need some mechanism that makes sure that the credentials haven't changed and to check and update credentials if they do change. Is there a better solution?

Proxy settings in Spring Boot

My application needs to fetch an XML file from the web, as follows:
#Bean
public HTTPMetadataProvider metadataProvider()
throws MetadataProviderException {
String metadataURL = "http://idp.ssocircle.com/idp-meta.xml";
final Timer backgroundTaskTimer = new Timer(true);
HTTPMetadataProvider provider =
new HTTPMetadataProvider(backgroundTaskTimer, httpClient(), metadataURL);
provider.setParserPool(parserPool());
return provider;
}
I'm working by using a filtered network, thus the app is unable to retrieve that file.
There is a way to setup an HTTP Proxy (e.g. myproxy.eu:8080) in Spring Boot?
Alternatively, I could retrieve the XML file by using the HTTPS protocol, but I should properly setup the metadata provider in order to support an encrypted connection... How?
This is not something you can configure in spring boot, HttpClient is not using java variables.
Therefor you need to set the proxy on the httpClient manually:
HostConfiguration hostConfig = new HostConfiguration();
hostConfig.setProxyHost(new ProxyHost("your.proxy.host", 8080));
httpClient.setHostConfiguration(hostConfig);

Resources