I am using the Java HighLevelRestClient to connect to my elasticsearch instance hosted on AWS. I can make requests against the URL on postman and from my browser just fine, but when I use the client library I receive
java.net.ConnectException: Connection Refused.
(I don't currently need any authentication as this is a small public test instance). This is my code:
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(restClientBuilder);
GetRequest getRequest = new GetRequest("some_index", "some_type","some_id");
final String[] elasticGetResponse = new String[1];
restHighLevelClient.getAsync(getRequest, new ActionListener() {
#Override
public void onResponse(GetResponse documentFields) {
try {
elasticGetResponse[0] = restHighLevelClient.get(getRequest).toString();
}
catch (IOException e) {
e.printStackTrace();
}
}
#Override
public void onFailure(Exception e) {
e.printStackTrace();
}
});
Please let me know how I can fix this... thanks!
Update: Here is my code for the restClientBuilder:
MySSLHelper sslHelper = new MySSLHelper(SSLConfig.builder()
.withKeyStoreProvider(myKeyStoreProvider)
.withTrustStoreProvider(InternalTrustStoreProvider.INSTANCE)
.build());
RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost("MY_ELASTICSEARCH_ENDPOINT")).setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
#Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpAsyncClientBuilder) {
return httpAsyncClientBuilder.setSSLContext(sslHelper.getContext());
}
});
I was the same problem and solved putting the port and protocol, like showed in this page:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.0/java-rest-high-getting-started-initialization.html
My code stayed like this:
RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(new HttpHost(elasticsearchHost, 9200, "http")));
Please try to do something like this:
RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost("MY_ELASTICSEARCH_ENDPOINT", "MY_ELASTICSEARCH_PORT", "MY_ELASTICSEARCH_PROTOCOL"))...
Hope this helps.
Good bye.
Related
I'm migrating from the HLRC to the new client, things were smooth but for some reason I cannot index a specific class/document. Here is my client implementation and index request:
#Configuration
public class ClientConfiguration{
#Autowired
private InternalProperties conf;
public ElasticsearchClient sslClient(){
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials(conf.getElasticsearchUser(), conf.getElasticsearchPassword()));
HttpHost httpHost = new HttpHost(conf.getElasticsearchAddress(), conf.getElasticsearchPort(), "https");
RestClientBuilder restClientBuilder = RestClient.builder(httpHost);
try {
SSLContext sslContext = SSLContexts.custom().loadTrustMaterial(null, (x509Certificates, s) -> true).build();
restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
#Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
return httpClientBuilder.setSSLContext(sslContext)
.setDefaultCredentialsProvider(credentialsProvider);
}
});
} catch (Exception e) {
e.printStackTrace();
}
RestClient restClient=restClientBuilder.build();
ElasticsearchTransport transport = new RestClientTransport(
restClient, new JacksonJsonpMapper());
ElasticsearchClient client = new ElasticsearchClient(transport);
return client;
}
}
#Service
public class ThisDtoIndexClass extends ConfigAndProperties{
public ThisDtoIndexClass() {
}
//client is declared in the class it's extending from
public ThisDtoIndexClass(#Autowired ClientConfiguration esClient) {
this.client = esClient.sslClient();
}
#KafkaListener(topics = "esTopic")
public void in(#Payload(required = false) customDto doc)
throws ThisDtoIndexClassException, ElasticsearchException, IOException {
if(doc!= null && doc.getId() != null) {
IndexRequest.Builder<customDto > indexReqBuilder = new IndexRequest.Builder<>();
indexReqBuilder.index("index-for-this-Dto");
indexReqBuilder.id(doc.getId());
indexReqBuilder.document(doc);
IndexResponse response = client.index(indexReqBuilder.build());
} else {
throw new ThisDtoIndexClassException("document is null");
}
}
}
This is all done in spring boot (v2.6.8) with ES 7.17.3. According to the debug, the payload is NOT null! It even fetches the id correctly while stepping through. For some reason, it throws me a org.springframework.kafka.listener.ListenerExecutionFailedException: in the last line (during the .build?). Nothing gets indexed, but the response comes back 200. I'm lost on where I should be looking. I have a different class that also writes to a different index, also getting a payload from kafka directly (all seperate consumers). That one functions just fine.
I suspect it has something to do with the way my client is set up and/or the kafka. Please point me in the right direction.
I solved it by deleting the default constructor. If I put it back it overwrites the extended constructor (or straight up doesn't acknowledge the extended constructor), so my client was always null. The error message it gave me was extremely misleading since it actually wasn't the Kafka's fault!
Removing the default constructor completely initializes the correct constructor and I was able to index again. I assume this was a spring boot loading related "issue".
I am using RestHighLevelClient version 7.2 to connect to the ElasticSearch cluster version 7.2. My cluster has 3 Master nodes and 2 data nodes. Data node memory config: 2 core and 8 GB. I have used to below code in my spring boot project to create RestHighLevelClient instance.
#Bean(destroyMethod = "close")
#Qualifier("readClient")
public RestHighLevelClient readClient(){
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials(elasticUser, elasticPass));
RestClientBuilder builder = RestClient.builder(new HttpHost(elasticHost, elasticPort))
.setHttpClientConfigCallback(httpClientBuilder ->httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider).setDefaultIOReactorConfig(IOReactorConfig.custom().setIoThreadCount(5).build()));
builder.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder.setConnectTimeout(30000).setSocketTimeout(60000)
);
RestHighLevelClient restClient = new RestHighLevelClient(builder);
return restClient;
}
RestHighLevelClient is a singleton bean. Intermittently I am getting SocketTimeoutException with both GET and PUT request. The index size is around 50 MB. I have tried increasing the socket timeout value, but still, I receive the same error. Am I missing some configuration? Any help would be appreciated.
I got the issue just wanted to share so that it can help others.
I was using Load Balancer to connect to the ElasticSerach Cluster.
As you can see from my RestClientBuilder code that I was using only the loadbalancer host and port. Although I have multiple master node, still RestClient was not retrying my request in case of connection timeout.
RestClientBuilder builder = RestClient.builder(new HttpHost(elasticHost, elasticPort))
.setHttpClientConfigCallback(httpClientBuilder ->httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider).setDefaultIOReactorConfig(IOReactorConfig.custom().setIoThreadCount(5).build()));
According to the RestClient code if we use a single host then it won't retry in case of any connection issue.
So I changed my code as below and it started working.
RestClientBuilder builder = RestClient.builder(new HttpHost(elasticHost, 9200),new HttpHost(elasticHost, 9201))).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));
For complete RestClient code please refer https://github.com/elastic/elasticsearch/blob/master/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
Retry code block in RestClient
private Response performRequest(final NodeTuple<Iterator<Node>> nodeTuple,
final InternalRequest request,
Exception previousException) throws IOException {
RequestContext context = request.createContextForNextAttempt(nodeTuple.nodes.next(), nodeTuple.authCache);
HttpResponse httpResponse;
try {
httpResponse = client.execute(context.requestProducer, context.asyncResponseConsumer, context.context, null).get();
} catch(Exception e) {
RequestLogger.logFailedRequest(logger, request.httpRequest, context.node, e);
onFailure(context.node);
Exception cause = extractAndWrapCause(e);
addSuppressedException(previousException, cause);
if (nodeTuple.nodes.hasNext()) {
return performRequest(nodeTuple, request, cause);
}
if (cause instanceof IOException) {
throw (IOException) cause;
}
if (cause instanceof RuntimeException) {
throw (RuntimeException) cause;
}
throw new IllegalStateException("unexpected exception type: must be either RuntimeException or IOException", cause);
}
ResponseOrResponseException responseOrResponseException = convertResponse(request, context.node, httpResponse);
if (responseOrResponseException.responseException == null) {
return responseOrResponseException.response;
}
addSuppressedException(previousException, responseOrResponseException.responseException);
if (nodeTuple.nodes.hasNext()) {
return performRequest(nodeTuple, request, responseOrResponseException.responseException);
}
throw responseOrResponseException.responseException;
}
I'm facing the same issue, and seeing this I realized that the retry is happening on my side too in each host (I have 3 host and the exception happens in 3 threads). I wanted to post it since you might face the same issue or someone else might come to this post because of the same SocketConnection Exception.
Searching the official docs, the HighLevelRestClient uses under the hood the RestClient, and the RestClient uses CloseableHttpAsyncClient which have a connection pool. ElasticSearch specifies that you should close the connection once that you are done, (which sounds ambiguous the definition of "done" in an application), but in general in internet I have found that you should close it when the application is closing or ending, rather than when you finished querying.
Now on the official documentation of apache they have an example to handle the connection pool, which i'm trying to follow, I'll try to replicate the scenario and will post if that fixes my issue, the code can be found here:
https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientEvictExpiredConnections.java
This is what i have so far:
#Bean(name = "RestHighLevelClientWithCredentials", destroyMethod = "close")
public RestHighLevelClient elasticsearchClient(ElasticSearchClientConfiguration elasticSearchClientConfiguration,
RestClientBuilder.HttpClientConfigCallback httpClientConfigCallback) {
return new RestHighLevelClient(
RestClient
.builder(getElasticSearchHosts(elasticSearchClientConfiguration))
.setHttpClientConfigCallback(httpClientConfigCallback)
);
}
#Bean
#RefreshScope
public RestClientBuilder.HttpClientConfigCallback getHttpClientConfigCallback(
PoolingNHttpClientConnectionManager poolingNHttpClientConnectionManager,
CredentialsProvider credentialsProvider
) {
return httpAsyncClientBuilder -> {
httpAsyncClientBuilder.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE);
httpAsyncClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
httpAsyncClientBuilder.setConnectionManager(poolingNHttpClientConnectionManager);
return httpAsyncClientBuilder;
};
}
public class ElasticSearchClientManager {
private ElasticSearchClientManager.IdleConnectionEvictor idleConnectionEvictor;
/**
* Custom client connection manager to create a connection watcher
*
* #param elasticSearchClientConfiguration elasticSearchClientConfiguration
* #return PoolingNHttpClientConnectionManager
*/
#Bean
#RefreshScope
public PoolingNHttpClientConnectionManager getPoolingNHttpClientConnectionManager(
ElasticSearchClientConfiguration elasticSearchClientConfiguration
) {
try {
SSLIOSessionStrategy sslSessionStrategy = new SSLIOSessionStrategy(getTrustAllSSLContext());
Registry<SchemeIOSessionStrategy> sessionStrategyRegistry = RegistryBuilder.<SchemeIOSessionStrategy>create()
.register("http", NoopIOSessionStrategy.INSTANCE)
.register("https", sslSessionStrategy)
.build();
ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
PoolingNHttpClientConnectionManager poolingNHttpClientConnectionManager =
new PoolingNHttpClientConnectionManager(ioReactor, sessionStrategyRegistry);
idleConnectionEvictor = new ElasticSearchClientManager.IdleConnectionEvictor(poolingNHttpClientConnectionManager,
elasticSearchClientConfiguration);
idleConnectionEvictor.start();
return poolingNHttpClientConnectionManager;
} catch (IOReactorException e) {
throw new RuntimeException("Failed to create a watcher for the connection pool");
}
}
private SSLContext getTrustAllSSLContext() {
try {
return new SSLContextBuilder()
.loadTrustMaterial(null, (x509Certificates, string) -> true)
.build();
} catch (Exception e) {
throw new RuntimeException("Failed to create SSL Context with open certificate", e);
}
}
public IdleConnectionEvictor.State state() {
return idleConnectionEvictor.evictorState;
}
#PreDestroy
private void finishManager() {
idleConnectionEvictor.shutdown();
}
public static class IdleConnectionEvictor extends Thread {
private final NHttpClientConnectionManager nhttpClientConnectionManager;
private final ElasticSearchClientConfiguration elasticSearchClientConfiguration;
#Getter
private State evictorState;
private volatile boolean shutdown;
public IdleConnectionEvictor(NHttpClientConnectionManager nhttpClientConnectionManager,
ElasticSearchClientConfiguration elasticSearchClientConfiguration) {
super();
this.nhttpClientConnectionManager = nhttpClientConnectionManager;
this.elasticSearchClientConfiguration = elasticSearchClientConfiguration;
}
#Override
public void run() {
try {
while (!shutdown) {
synchronized (this) {
wait(elasticSearchClientConfiguration.getExpiredConnectionsCheckTime());
// Close expired connections
nhttpClientConnectionManager.closeExpiredConnections();
// Optionally, close connections
// that have been idle longer than 5 sec
nhttpClientConnectionManager.closeIdleConnections(elasticSearchClientConfiguration.getMaxTimeIdleConnections(),
TimeUnit.SECONDS);
this.evictorState = State.RUNNING;
}
}
} catch (InterruptedException ex) {
this.evictorState = State.NOT_RUNNING;
}
}
private void shutdown() {
shutdown = true;
synchronized (this) {
notifyAll();
}
}
public enum State {
RUNNING,
NOT_RUNNING
}
}
}
I am using Elastic search, and it works well, but not when I try to use it with a webservice with jetty and jersey.
Here is an example of a function that I want to use :
public boolean insertUser(RestHighLevelClient client, User user) throws IOException
{
java.util.Map<String, Object> jsonMap = new HashMap<String, Object>();
jsonMap.put("username", user.username);
jsonMap.put("password", user.password);
jsonMap.put("mail", user.mail);
jsonMap.put("friends", user.friends);
jsonMap.put("maps", user.maps);
System.out.println("insertUser");
IndexRequest indexRequest = new IndexRequest("users", "doc",user.username)
.source(jsonMap);
try {
IndexResponse indexResponse = client.index(indexRequest);
System.out.println("insertUser 222");
if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {
System.out.println("user "+user.username+" créé");
}
else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {
System.out.println("user "+user.username+" update dans insertUser (pas normal)");
}
} catch (IOException e) {
e.printStackTrace();
return false;
}
return true;
}
This function works well when I try it inside a test class. But If i start my server like this :
Server server = new Server();
// Add a connector
ServerConnector connector = new ServerConnector(server);
connector.setHost("0.0.0.0");
connector.setPort(8081);
connector.setIdleTimeout(30000);
server.addConnector(connector);
DAO.ClientConnection("0.0.0.0",8081);
// Configure Jersey
ResourceConfig rc = new ResourceConfig();
rc.packages(true, "com.example.jetty_jersey.ws");
rc.register(JacksonFeature.class);
// Add a servlet handler for web services (/ws/*)
ServletHolder servletHolder = new ServletHolder(new ServletContainer(rc));
ServletContextHandler handlerWebServices = new ServletContextHandler(ServletContextHandler.SESSIONS);
handlerWebServices.setContextPath("/ws");
handlerWebServices.addServlet(servletHolder, "/*");
// Add a handler for resources (/*)
ResourceHandler handlerPortal = new ResourceHandler();
handlerPortal.setResourceBase("src/main/webapp/temporary-work");
handlerPortal.setDirectoriesListed(false);
handlerPortal.setWelcomeFiles(new String[] { "homepage.html" });
ContextHandler handlerPortalCtx = new ContextHandler();
handlerPortalCtx.setContextPath("/");
handlerPortalCtx.setHandler(handlerPortal);
// Activate handlers
ContextHandlerCollection contexts = new ContextHandlerCollection();
contexts.setHandlers(new Handler[] { handlerWebServices, handlerPortalCtx });
server.setHandler(contexts);
// Start server
server.start();
And when I enter a form, then call this webservice :
#POST
#Path("/signup")
#Produces(MediaType.APPLICATION_JSON)
// #Consumes(MediaType.APPLICATION_FORM_URLENCODED)
public SimpleResponse signup(#Context HttpServletRequest httpRequest,
#FormParam("username") String username,
#FormParam("email") String email,
#FormParam("password") String password,
#FormParam("passwordConfirm") String passwordConfirm) {
System.out.println("k");
//if (httpRequest.getSession().getAttribute("user") != null) { //httpRequest.getUserPrincipal() == null) {
try {
if (password.equals(passwordConfirm)) {
User user = new User("jeanOknewmail#gmail.com", "abc");
user.username = "jeanok";
user.maps = new ArrayList<String>();
user.friends = new ArrayList<String>();
System.out.println(user);
System.out.println("avant insert");
DAO.getActionUser().createIndexUser();
//System.out.println(DAO.getActionUser().getOneUser(DAO.client, "joe"));
System.out.println("rdctfygbhunji,k");
DAO.getActionUser().insertUser(DAO.client, user);
System.out.println("après insert");
return new SimpleResponse(true);
}
} catch (IOException e) {
e.printStackTrace();
}
//}
return new SimpleResponse(false);
}
I get lots of errors :
avax.servlet.ServletException: ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [PUT], host [http://0.0.0.0:8081], URI [/users/doc/jeanok?timeout=1m], status line [HTTP/1.1 404 Not Found]|];
...
Caused by:
ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [PUT], host [http://0.0.0.0:8081], URI [/users/doc/jeanok?timeout=1m], status line [HTTP/1.1 404 Not Found]|];
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:598)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:501)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:474)
at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:335)
at DAO.UserDAO.insertUser(UserDAO.java:160)
Do you have any idea why the behaviour of my function isn't the same when I launch my server? And why this error? Thanks for your help
I wasn't connected to elastic search. My client was connected to the wrong port. Now it works
I have an HTTP2C Embedded Jetty 9.x Server running ... note the server connector shows h2c ...
2016-03-21 09:25:44.082:INFO:oejs.ServerConnector:main: Started ServerConnector#66c7bd3f{HTTP/1.1,[http/1.1, h2c, h2c-17, h2c-16, h2c-15, h2c-14]}{0.0.0.0:8080}
I have an OkHttpClient 3 attempting to talk HTTP2C to this server , however it always gets downgraded to HTTP/1.1, what am I missing? Which Java client API supports HTTP2C? My client code is as below ...
package http2;
import java.util.Collections;
import okhttp3.ConnectionSpec;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
public class GetClear {
public static void main(String[] args) throws Exception {
ConnectionSpec spec = new ConnectionSpec.Builder(ConnectionSpec.CLEARTEXT).build();
OkHttpClient client = new OkHttpClient.Builder().connectionSpecs(Collections.singletonList(spec)).build();
Request request = new Request.Builder().url("http://localhost:8080/test").build();
Response response = client.newCall(request).execute();
System.out.println (response.body().string());
System.out.println("****");
response.body().close();
}
}
[The server prints the 'request.getProtocol' from a Jetty servlet and that shows HTTP/1.1 instead of HTTP/2].
HTTP/2 server and client on TLS works just fine using HTTP/2(client code and server code are different of course).
Any help will be truly appreciated.
Using a Jetty HTTP2C client, the same server code works. I guess OkHTTPClient does not support HTTP2C.
A complete h2c example using the HelloHandler example from the jetty doc:
public class HelloServer {
public static class HelloHandler extends AbstractHandler {
final String greeting;
final String body;
public HelloHandler() {
this("Hello World");
}
public HelloHandler(String greeting) {
this(greeting, null);
}
public HelloHandler(String greeting, String body) {
this.greeting = greeting;
this.body = body;
}
public void handle(String target,
Request baseRequest,
HttpServletRequest request,
HttpServletResponse response) throws IOException,
ServletException {
response.setContentType("text/html; charset=utf-8");
response.setStatus(HttpServletResponse.SC_OK);
PrintWriter out = response.getWriter();
out.println("<h1>" + greeting + "</h1>");
if (body != null) {
out.println(body);
}
baseRequest.setHandled(true);
}
}
public static void main(String[] args) throws Exception {
Server server = new Server();
server.setHandler(new HelloHandler());
HttpConfiguration httpConfig = new HttpConfiguration();
ConnectionFactory h1 = new HttpConnectionFactory(httpConfig);
ConnectionFactory h2c = new HTTP2CServerConnectionFactory(httpConfig);
ServerConnector serverConnector = new ServerConnector(server, h1, h2c);
serverConnector.setPort(8080);
server.setConnectors(new ServerConnector[] { serverConnector });
server.start();
server.join();
}
}
The Jetty log line shows that you have configured the server connector to have HTTP/1.1 to be the default protocol (that is the upper-case "HTTP/1.1" before the brackets containing the list of protocols supported).
You don't show your server-side code, but you have two choices:
Configure explicitly the default protocol for the server connector:
serverConnector.setDefaultProtocol("h2c");
Pass the ConnectionFactory objects in the right order to the server connector, since the first one will be the default protocol:
HttpConfiguration httpConfig = new HttpConfiguration();
ConnectionFactory h1 = new HttpConnectionFactory(httpConfig);
ConnectionFactory h2c = new HTTP2CServerConnectionFactory(httpConfig);
ServerConnector serverConnector = new ServerConnector(server, h2c, h1);
Is there a way to ignore SSL certificate verification while connecting elasticsearch 7.4 using high level rest client. I explored a couple of options but nothing worked in my case. I have a HTTPS ES cluster which I want to connect from my spring boot application by ignoring ssl certificate verification.
hope this will help you, I had the same problem and this is how I resolved.
#Bean
public RestHighLevelClient createSimpleElasticClient() throws Exception {
try {
SSLContextBuilder sslBuilder = SSLContexts.custom()
.loadTrustMaterial(null, (x509Certificates, s) -> true);
final SSLContext sslContext = sslBuilder.build();
RestHighLevelClient client = new RestHighLevelClient(RestClient
//port number is given as 443 since its https schema
.builder(new HttpHost(hostNameOrLoadbalancerURL, 443, "https"))
.setHttpClientConfigCallback(new HttpClientConfigCallback() {
#Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
return httpClientBuilder
.setSSLContext(sslContext)
.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE);
}
})
.setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
#Override
public RequestConfig.Builder customizeRequestConfig(
RequestConfig.Builder requestConfigBuilder) {
return requestConfigBuilder.setConnectTimeout(5000)
.setSocketTimeout(120000);
}
}));
System.out.println("elasticsearch client created");
return client;
} catch (Exception e) {
System.out.println(e);
throw new Exception("Could not create an elasticsearch client!!");
}
}