I am trying to access cloud Elasticsearch installation from our network that requires using a proxy for external requests. This is the snippet of code I use to pass Elasticsearch credentials and our proxy settings:
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(elasticUser, elasticPassword));
RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost(hostName,port,"https")).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setProxy(new HttpHost(proxyURL", proxyPort, "http")));
RestHighLevelClient client = new RestHighLevelClient(restClientBuilder);
This results in this response from ES:
"Exception in thread "main" ElasticsearchStatusException[Elasticsearch exception
[type=security_exception, reason=action [indices:data/read/search] requires authentication]]"
It appears that Elasticsearch credentials are not passed for some reason.
should have been done like this:
RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost(hostName, port, "https"))
.setHttpClientConfigCallback(clientBuilder -> {
clientBuilder.setDefaultCredentialsProvider(credentialsProvider);
clientBuilder.setProxy(new HttpHost(proxyURL, proxyPort, "http"));
return clientBuilder;
});
Related
I've been trying to connect to my ES instance on the Elastic cloud with no success using the Java client and following the Documentation
In the hostname I put https://myinstance-xx.europe-west1.gcp.cloud.es.io
private String hostName = "https://myinstance-xx.es.europe-west1.gcp.cloud.es.io";
private String username = "username";
private String password = "pass";
#Bean
public ElasticsearchClient client() {
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials(username, password));
RestClientBuilder builder = RestClient.builder(new HttpHost(hostName, 9200))
.setHttpClientConfigCallback(httpClientBuilder ->
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));
// Create the low-level client
RestClient restClient = builder.build();
// Create the transport with a Jackson mapper
ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
// And create the API client
return new ElasticsearchClient(transport);
}
However, I get an Connection Refused exception
java.net.ConnectException: Connection refused
What's the best way to connect to ES on Elastic Cloud through the Java Client?
Go to cloud.elastic.go/home after your are logged in, find the "Manage this deployment" link. It should be something like https://cloud.elastic.co/deployments/xxxxxxxx
Under "Applications" you should have an "Elasticsearch" enpoint (click on "Copy endpoint")
On the API console create an API Key:
POST /_security/api_key
{
"name": "my_key_name",
"expiration": "15d"
}
That's gonna give you something like:
{
"id": "API_KEY_ID",
"name": "my_key_name",
"expiration": 1670471984009,
"api_key": "API_KEY",
"encoded": "XxxxxxxxXXXXXxxxxxxXXXXxxxxxXXXX=="
}
Now in your Java code:
// endpoint you copied
String hostname = "youdepname.es.us-west1.gcp.cloud.es.io";
String apiKeyId = "API_KEY_ID";
String apiKeySecret = "API_KEY";
String apiKeyAuth =
Base64.getEncoder().encodeToString(
(apiKeyId + ":" + apiKeySecret)
.getBytes(StandardCharsets.UTF_8));
RestClientBuilder builder = RestClient.builder(
new HttpHost(hostname, 9243, "https"))
.setRequestConfigCallback(
new RestClientBuilder.RequestConfigCallback() {
#Override
public RequestConfig.Builder customizeRequestConfig(
RequestConfig.Builder requestConfigBuilder) {
return requestConfigBuilder
.setConnectTimeout(5000)
.setSocketTimeout(60000);
}
});
Header[] defaultHeaders =
new Header[]{new BasicHeader("Authorization",
"ApiKey " + apiKeyAuth)};
builder.setDefaultHeaders(defaultHeaders);
RestClient restClient = builder.build();
// Create the transport with a Jackson mapper
ElasticsearchTransport transport = new RestClientTransport(
restClient, new JacksonJsonpMapper());
// And create the API client
ElasticsearchClient client = new ElasticsearchClient(transport);
That should do it. There are other ways to connect, but this is my preferred way.
I'm building an application that need to call an endpoint using NTLM authentication. My approach is that I try to use the Apache HttpComponents for the NTLM authentication and integrate the Spring WebClient with it. However, the WebClient doesn't seem to send any request at all. There's no errors but the response won't be returned.
Below is my code:
BasicCredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(new AuthScope(null, -1), new NTCredentials(username, password, computername, domain));
HttpAsyncClientBuilder clientBuilder = HttpAsyncClients.custom();
clientBuilder.setDefaultRequestConfig(RequestConfig.DEFAULT);
ClientHttpConnector connector = new HttpComponentsClientHttpConnector(client);
WebClient.builder().clientConnector(connector).build();
ResponseDto response = webClient.post()
.uri("http://myhost:8080/api/notification/add")
.body(Mono.just(request), RequestDto.class)
.retrieve()
.bodyToMono(ResponseDto.class).block();
Unable to connect to SSL enabled Elastic Search using Google Cloud Dataflow. I have used google provide template from git I modified the source code to pass keystore, keystorepassword, truststore to the job along with the Elastic Search credentials
Elastic Search Class file
config = config.withKeystorePassword("XXXXXXXXXXXXXXXXXXXXXXXXXXX")
.withKeystorePath("PATH_TO_KEYSTORE")
Exception on running the job is as below
Caused by: java.lang.IllegalArgumentException: Cannot get
Elasticsearch version at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.getBackendVersion(ElasticsearchIO.java:1546)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.setup(ElasticsearchIO.java:1336)
Caused by: org.elasticsearch.client.ResponseException: method [GET],
host [https://elastic.irds.opus.dev.gcp.db.com], URI [/], status line
[HTTP/1.1 401 Unauthorized]
{"error":{"root_cause":[{"type":"security_exception","reason":"missing
authentication credentials for REST request
[/]","header":{"WWW-Authenticate":"Basic realm="security"
charset="UTF-8""}}],"type":"security_exception","reason":"missing
authentication credentials for REST request
[/]","header":{"WWW-Authenticate":"Basic realm="security"
charset="UTF-8""}},"status":401} at
org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.getBackendVersion(ElasticsearchIO.java:1530)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.setup(ElasticsearchIO.java:1336)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn$DoFnInvoker.invokeSetup(Unknown
Source)
We wrote a sample java program to ensure elastic search connectivity, which works fine as expected
SSLContextBuilder sslBuilder = SSLContexts.custom()
.loadKeyMaterial(keyStore,"*************".toCharArray())
.loadTrustMaterial(truststore,null);
final SSLContext sslContext = sslBuilder.build();
RestClientBuilder builder = RestClient.builder(
new HttpHost("https://X.X.X.X", 443, "https"))
.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
public HttpAsyncClientBuilder customizeHttpClient(
final HttpAsyncClientBuilder httpAsyncClientBuilder) {
httpAsyncClientBuilder.disableAuthCaching();
return httpAsyncClientBuilder.setSSLContext(sslContext);
}
});
RestHighLevelClient client = new RestHighLevelClient(builder);
GetRequest getRequest = new GetRequest("test_index1/_search", "1");
GetResponse response = client.get(getRequest, RequestOptions.DEFAULT);
System.out.println("\nSearch results:");
System.out.println(response.getSourceAsString());
Could you please provide some inputs on what changes do i need to make on dataflow template to connect to SSL enabled Elastic search
I'm trying to setup a WebClient connection in Spring Boot using a proxy. My implementation looks like the following:
final WebClient.Builder webclientBuilder = WebClient.builder();
final HttpClient httpClient = HttpClient.create();
httpClient.proxy(proxy -> proxy
.type(Proxy.HTTP)
.host(proxyName)
.port(Integer.parseInt(proxyPort)));
final ReactorClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);
webclientBuilder.clientConnector(connector);
final WebClient webClient = webclientBuilder
.baseUrl(baseUrl)
.build();
After running it and sending an API call, I receive a "Connection timed out: no further information". I should get back a Bad Request (in case my call is wrong), but I don't.
Is the implementation wrong?
the proxyName is written like this: "proxy.blabla.de"
After some trial and error and comparing I found a solution working for me:
String baseUrl = "https://mybaseurl";
String proxyName = "proxy.blabla.de";
int proxyPort = 1234;
public InitResponse addAccount() {
// for logging purposes, nothing to do with the proxy
LOGGER.info("LOGGER.info: addAccount()");
final InitRequest request = buildRequest();
HttpClient httpClient = HttpClient.create()
.proxy(proxy -> proxy.type(Proxy.HTTP)
.host(proxyName)
.port(proxyPort));
ReactorClientHttpConnector conn = new ReactorClientHttpConnector(httpClient);
WebClient webClient = WebClient.builder().clientConnector(conn).baseUrl(baseUrl).build();
is there any way to configure my rest high client to connect with es using proxy. My configuration is
#Override
#Bean
public RestHighLevelClient elasticsearchClient() {
return new RestHighLevelClient(RestClient.builder(HttpHost.create(elasticSearchUrl)));}
My elastic search url is: aaa.bbbb.ccc.company.com/api/elastic-search-proxy
In that case I get No such host is known (aaa.bbbb.ccc.company.com/api/elastic-search-proxy) what is clear for me but is there any option to configure it ?
Its mentioned in the Elasticsearch documentation of JHLRC initialization , use below code:
RestClientBuilder builder = RestClient.builder(
new HttpHost("localhost", 9200, "http"));
builder.setHttpClientConfigCallback(new HttpClientConfigCallback() {
#Override
public HttpAsyncClientBuilder customizeHttpClient(
HttpAsyncClientBuilder httpClientBuilder) {
return httpClientBuilder.setProxy(
new HttpHost("proxy", 9000, "http"));
}
});
Set a callback that allows to modify the http client configuration (e.g. encrypted communication over ssl, or anything that the org.apache.http.impl.nio.client.HttpAsyncClientBuilder allows to set)
So in your case, you need to give your original host in below code
new HttpHost("localhost", 9200, "http"));
And then you need to define a callback to your proxy server in setHttpClientConfigCallback call back.
new HttpHost("proxy", 9000, "http"));
If someone using the latest Elastic Java client 8.x, you can use this way of configuring the proxy to your rest client. (note the proxy should already set in system properties). It might help someone.
val restClientBuilder = RestClient.builder(
HttpHost(randomHost, 443, https)
)*.setHttpClientConfigCallback {
HttpAsyncClientBuilder.create().useSystemProperties()
}*