Unable to connect to SSL enabled Elastic Search using Google Cloud Dataflow. I have used google provide template from git I modified the source code to pass keystore, keystorepassword, truststore to the job along with the Elastic Search credentials
Elastic Search Class file
config = config.withKeystorePassword("XXXXXXXXXXXXXXXXXXXXXXXXXXX")
.withKeystorePath("PATH_TO_KEYSTORE")
Exception on running the job is as below
Caused by: java.lang.IllegalArgumentException: Cannot get
Elasticsearch version at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.getBackendVersion(ElasticsearchIO.java:1546)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.setup(ElasticsearchIO.java:1336)
Caused by: org.elasticsearch.client.ResponseException: method [GET],
host [https://elastic.irds.opus.dev.gcp.db.com], URI [/], status line
[HTTP/1.1 401 Unauthorized]
{"error":{"root_cause":[{"type":"security_exception","reason":"missing
authentication credentials for REST request
[/]","header":{"WWW-Authenticate":"Basic realm="security"
charset="UTF-8""}}],"type":"security_exception","reason":"missing
authentication credentials for REST request
[/]","header":{"WWW-Authenticate":"Basic realm="security"
charset="UTF-8""}},"status":401} at
org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.getBackendVersion(ElasticsearchIO.java:1530)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.setup(ElasticsearchIO.java:1336)
at
com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn$DoFnInvoker.invokeSetup(Unknown
Source)
We wrote a sample java program to ensure elastic search connectivity, which works fine as expected
SSLContextBuilder sslBuilder = SSLContexts.custom()
.loadKeyMaterial(keyStore,"*************".toCharArray())
.loadTrustMaterial(truststore,null);
final SSLContext sslContext = sslBuilder.build();
RestClientBuilder builder = RestClient.builder(
new HttpHost("https://X.X.X.X", 443, "https"))
.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
public HttpAsyncClientBuilder customizeHttpClient(
final HttpAsyncClientBuilder httpAsyncClientBuilder) {
httpAsyncClientBuilder.disableAuthCaching();
return httpAsyncClientBuilder.setSSLContext(sslContext);
}
});
RestHighLevelClient client = new RestHighLevelClient(builder);
GetRequest getRequest = new GetRequest("test_index1/_search", "1");
GetResponse response = client.get(getRequest, RequestOptions.DEFAULT);
System.out.println("\nSearch results:");
System.out.println(response.getSourceAsString());
Could you please provide some inputs on what changes do i need to make on dataflow template to connect to SSL enabled Elastic search
Related
I have setup WebClient with apache httpclient 5 connector. Here my code snippet
#Bean
public WebClient webClient() {
HttpAsyncClientBuilder clientBuilder = HttpAsyncClients.custom();
final PoolingAsyncClientConnectionManager connManager =
PoolingAsyncClientConnectionManagerBuilder
.create()
.build();
clientBuilder.setConnectionManager(connManager);
CloseableHttpAsyncClient client = clientBuilder.build();
ClientHttpConnector connector = new HttpComponentsClientHttpConnector(client);
return WebClient.builder().clientConnector(connector).build();
}
I did a loading test with my rest API which internally calls remote URL using WebClient configuration as mentioned above. I see many requests failed with error
Caused by: org.apache.hc.core5.util.DeadlineTimeoutException: Deadline: 2022-02-02T14:01:23.844+0530, -451 MILLISECONDS overdue
at org.apache.hc.core5.util.DeadlineTimeoutException.from(DeadlineTimeoutException.java:49)
at org.apache.hc.core5.pool.StrictConnPool.processPendingRequest(StrictConnPool.java:318)
at org.apache.hc.core5.pool.StrictConnPool.processNextPendingRequest(StrictConnPool.java:299)
I am not able to find what configuration should be added in order to fix the issue. Can anyone please help me to solve this issue?
I am building an application that reads JSON response from certain endpoints and I am trying to authenticate in Apache HttpClient using NTLM authentication:
The class that is responsible for authentication HttpConnector tries to authentice right after its instantiation:
public static HttpConnector on(String username, String password) {
HttpConnector connector = new HttpConnector(username, password);
Credentials credentials = new NTCredentials(username, password, "host", "domain");
connector.getHttpClient().getState().setCredentials(AuthScope.ANY, credentials);
return connector;
}
but I always get response code 401 Unauthorized. As I read in internet including here in Stackoverflow I used NTCredentials that I am trying to set globally in the HttpClient. I have tested the endpoints in Postman, there I get the JSON response successfully but HttpClient cannot connect.
In the code I use GetMethod: httpMethod = new GetMethod(url);
Here I have also tried to configure authentication but it still does not work:
private void configureMethod(HttpMethod httpMethod) throws MalformedChallengeException {
httpMethod.getHostAuthState().setAuthRequested(true);
httpMethod.setRequestHeader("Content-Type", "application/json; charset=UTF-8");
httpMethod.setDoAuthentication(true);
httpMethod.getParams().setParameter(
HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
httpMethod.getHostAuthState().setAuthRequested(true);
httpMethod.getHostAuthState().setAuthScheme(
new NTLMScheme(
String.format("ntlm %s:%s", username, password)));
}
During debugging I see that I get: Connection reset by peer: socket write error. It happens in HttpMethodDirector::executeWithRetry(final HttpMethod method) method.
Can someone help me, what is the correctNTLM authentication setup in Apache HttpClient. Can I really use the global set of credentials or I have to setup credentials to every HttpMethod I create and how?
Thank you in advance!
I fixed this by formatting the client the following way:
HttpClient httpClient = new DefaultHttpClient();
httpClient.getCredentialsProvider().setCredentials(
new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT),
new NTCredentials(username, password, HOST, MY_DOMAIN));
And the used not GetMethod but HttpGet:
HttpGet = getRequest = new HttpGet(url);
and this time the connection owas successful.
I am trying to access cloud Elasticsearch installation from our network that requires using a proxy for external requests. This is the snippet of code I use to pass Elasticsearch credentials and our proxy settings:
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(elasticUser, elasticPassword));
RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost(hostName,port,"https")).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setProxy(new HttpHost(proxyURL", proxyPort, "http")));
RestHighLevelClient client = new RestHighLevelClient(restClientBuilder);
This results in this response from ES:
"Exception in thread "main" ElasticsearchStatusException[Elasticsearch exception
[type=security_exception, reason=action [indices:data/read/search] requires authentication]]"
It appears that Elasticsearch credentials are not passed for some reason.
should have been done like this:
RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost(hostName, port, "https"))
.setHttpClientConfigCallback(clientBuilder -> {
clientBuilder.setDefaultCredentialsProvider(credentialsProvider);
clientBuilder.setProxy(new HttpHost(proxyURL, proxyPort, "http"));
return clientBuilder;
});
is there any way to configure my rest high client to connect with es using proxy. My configuration is
#Override
#Bean
public RestHighLevelClient elasticsearchClient() {
return new RestHighLevelClient(RestClient.builder(HttpHost.create(elasticSearchUrl)));}
My elastic search url is: aaa.bbbb.ccc.company.com/api/elastic-search-proxy
In that case I get No such host is known (aaa.bbbb.ccc.company.com/api/elastic-search-proxy) what is clear for me but is there any option to configure it ?
Its mentioned in the Elasticsearch documentation of JHLRC initialization , use below code:
RestClientBuilder builder = RestClient.builder(
new HttpHost("localhost", 9200, "http"));
builder.setHttpClientConfigCallback(new HttpClientConfigCallback() {
#Override
public HttpAsyncClientBuilder customizeHttpClient(
HttpAsyncClientBuilder httpClientBuilder) {
return httpClientBuilder.setProxy(
new HttpHost("proxy", 9000, "http"));
}
});
Set a callback that allows to modify the http client configuration (e.g. encrypted communication over ssl, or anything that the org.apache.http.impl.nio.client.HttpAsyncClientBuilder allows to set)
So in your case, you need to give your original host in below code
new HttpHost("localhost", 9200, "http"));
And then you need to define a callback to your proxy server in setHttpClientConfigCallback call back.
new HttpHost("proxy", 9000, "http"));
If someone using the latest Elastic Java client 8.x, you can use this way of configuring the proxy to your rest client. (note the proxy should already set in system properties). It might help someone.
val restClientBuilder = RestClient.builder(
HttpHost(randomHost, 443, https)
)*.setHttpClientConfigCallback {
HttpAsyncClientBuilder.create().useSystemProperties()
}*
I connect with my websocket client to non-SSL endpoint without any problem. But I cannot find any way how to connect to wss (SSL) endpoint. Where can I define the SSL factory etc. No object seem to have related set method.
WebSocketClient transport = new StandardWebSocketClient();
WebSocketStompClient stompClient = new WebSocketStompClient(transport);
stompClient.setMessageConverter(new MappingJackson2MessageConverter());
String url = cfg.getWebsocketEndpoint();
StompSessionHandler handler = new MySessionHandler();
WebSocketHttpHeaders headers = new WebSocketHttpHeaders();
stompClient.connect(url, handler);
I am using wss:// url and on the other side I have a server with self-signed certificate. However, this code does not throw any exception while connecting, but the session is not established.
EDIT: After enabling tracing for web.* I got a standard error, with
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
It occurs when connecting to server with self-signed certificate. However, for RestTemplate I already updated SSLContext with this code and REST calls are fine now, but I do not know why, StandardWebSocketClient is IGNORING the SSLContext. Why?
String keystoreType = "JKS";
InputStream keystoreLocation = new FileInputStream("src/main/resources/aaa.jks");
char [] keystorePassword = "zzz".toCharArray();
char [] keyPassword = "zzz".toCharArray();
KeyStore keystore = KeyStore.getInstance(keystoreType);
keystore.load(keystoreLocation, keystorePassword);
KeyManagerFactory kmfactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
kmfactory.init(keystore, keyPassword);
InputStream truststoreLocation = new FileInputStream("src/main/resources/aaa.jks");
char [] truststorePassword = "zzz".toCharArray();
String truststoreType = "JKS";
KeyStore truststore = KeyStore.getInstance(truststoreType);
truststore.load(truststoreLocation, truststorePassword);
TrustManagerFactory tmfactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
tmfactory.init(truststore);
KeyManager[] keymanagers = kmfactory.getKeyManagers();
TrustManager[] trustmanagers = tmfactory.getTrustManagers();
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keymanagers, trustmanagers, new SecureRandom());
SSLContext.setDefault(sslContext);
UPDATE: Unfortunately, I did not managed to do this with custom truststore. I installed the certificate with InstallCert.java.
I think that each websocket container implementation provides ways to do this.
You have to set this configuration using the StandardWebSocketClient .setUserProperties. All those properties are internally set in the ClientEndpointConfig used by the client.
Here's an example with Tomcat as a provider:
StandardWebSocketClient wsClient = //...;
SSLContext sslContext = //...;
wsClient.setUserProperties(WsWebSocketContainer.SSL_CONTEXT_PROPERTY, sslContext);
In any case you should refer to your provider reference documentation to know which configuration keys you should use.
I've faced a similar issue, where I need to use a different ssl context instead of the default one and I could not use the Tomcat provider version solution. I've met a lot of resources and examples about this solution which does not fit my case, and for sure I landed here too on this question.
When you need to use a specific/custom SSLContext to set and you don't need the 'tomcat version', you can go for the Jetty version because it allows you to set which trust store and/or key store you want to use. In this case my Spring application was on the 'client' side and not the server one, but Jetty SslContextFactory provides the same functionalities for the 'Server' case.
The short example code below is for the client side, but it shares methods and signatures with the server side (check the jetty documentation about)
final SslContextFactory.Client factory = new SslContextFactory.Client();
factory.setSslContext(sslContext); // a different loaded java SslContext instead of the default one
//Create the web socket client with jetty client factory
final JettyWebSocketClient client =
new JettyWebSocketClient(new WebSocketClient(new HttpClient(factory)));
client.start();
final WebSocketStompClient stomp = new WebSocketStompClient(client);
There're also method for setting the trust store or key store instead a fully loaded SslContext.