Elasticsearch client does not fetch result when a single client node goes down - elasticsearch

We have a very standard elasticsearch setup with 3 master nodes, 6 data nodes and 3 client nodes. Here is our connection code for connecting to Elasticsearch clients from our Java application.
Settings settings = Settings.settingsBuilder()
.put("cluster.name", configuration.getString("clusterName"))
.put("client.transport.sniff", false)
.put("client.transport.ping_timeout", "5s")
.build();
TransportClient client = TransportClient.builder().settings(settings).build();
for (String hostname : (Collection<String>)configuration.get("hostnames")){
try {
client = client.addTransportAddresses(
new InetSocketTransportAddress(InetAddress.getByName(hostname), 9300)
);
break;
} catch (UnknownHostException e) {
e.printStackTrace();
}
}
We have currently three different host in hostnames list. But any time a single client from this list of hostname goes down this Elasticsearch transport client stops responding. I have gone through transport client documentation on Elasticsearch site and have also tried looking at their Github issues, according to that whenever a node goes down only elasticsearch should remove it from list of nodes and continue working with other nodes, but in our case things just break down. Anyone has any idea what might be the problem?
We are using elasticsearch 2.4.3 right now.

It looks like you are breaking the loop after a single node has been added. Try removing the break statement:
for (String hostname : (Collection<String>)configuration.get("hostnames")){
try {
client = client.addTransportAddresses(
new InetSocketTransportAddress(InetAddress.getByName(hostname), 9300)
);
} catch (UnknownHostException e) {
e.printStackTrace();
}
}

Related

Redis java.net.SocketTimeoutException: Read timed out on setting value while High memory utilization on redis cluster

I am using Jedis client for redis in my spring service.
Below is the similar code which I use for setting value into a hashedkey.
Jedis jedis = null;
try {
1-- jedis = redisJedisPool.getResource();
2-- jedis.hset(key,"data", dataValue);
for (Entry<String, String> entry : hmap.entrySet()) {
if (some condition)
3-- jedis.hset(key, entry.getKey(), entry.getValue());
}
4-- jedis.expire(key, ttl);
}catch(Exception e) {
logger.error("Error for key {}, Reason: {}", key, e.getMessage());
}
finally {
RedisConnectionManager.closeJedisResource(jedis);
}
there was heavy load on this service and thus a high memory utilization of the redis cluster was noticed.
I faced a lot of SocketTimeoutException from the above function(all exception where catched in the catch block).
Exception:
Error for key: {key}, Reason: java.net.SocketTimeoutException: Read timed out
Main Issue: After this intermittent issue I see a lot of keys with ttl as -1(infinite expiry). Almost all these keys where logged in the above catch block.
Need thoughts on this from the community on what can be the possible issue here.
Action I took for verifying: I checked the data for few of these keys(with ttl -1) and saw that the data that was to be set at line 3(inside the for loop) didn't set all the data. But I came across couple of keys which had ttl as -1, not all data was set but that key was not logged in the above catch block. And this is the only func where the setting of data in cache takes place in my service.
After this I am not able to conclude the above hypothesis.

grpc custom load balancer not detecting new server addition in cluster

I am building a distributed workflow orchestrator, grpc is used to communicate with the server cluster by workers.If a new server is added to the server grpc client is not able to detect this change. However i have done a workaround by adding a max connection age to the server options
grpc.KeepaliveParams(keepalive.ServerParameters{
MaxConnectionAge: time.Minute * 1,
})
We have two implementation of workers, one in golang and other in java this workaround works perfectly in golang client. Every minute the client makes new connection and is able to detect new servers in cluster. But this is not working with java client.
public CustomNameResolverFactory(String host, int port) {
ManagedChannel managedChannel = NettyChannelBuilder
.forAddress(host, port)
.withOption( ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000 )
.usePlaintext().build();
GetServersRequest request = GetServersRequest.newBuilder().build();
GetServersResponse servers = TaskServiceGrpc.newBlockingStub(managedChannel).getServers(request);
List<Server> serversList = servers.getServersList();
System.out.println(servers);
LOGGER.info("found servers {}", servers);
for (Server server : serversList) {
String rpcAddr = server.getRpcAddr();
String[] split = rpcAddr.split(":");
String hostName = split[0];
int portN = Integer.parseInt(split[1]);
addresses.add(new EquivalentAddressGroup(new InetSocketAddress(hostName, portN)));
}
}
Java client code- https://github.com/Mohitkumar/orchy-worker-java/blob/master/src/main/java/com/orchy/client/CustomNameResolverFactory.java
Golang client code- https://github.com/Mohitkumar/orchy/blob/main/worker/lb/resolver.go

No server chosen by com.mongodb.async.client.ClientSessionHelpe from cluster description ClusterDescription

I am trying to connect to aws DocumentDB with async mongoClient.
I created a DocumentDB cluster in aws and success connect via ssh command line.
I went over here and created MongoClient and success connected and insert events.
But when I tried create com.mongodb.async.client.MongoClient, connection failed with folowing error:
No server chosen by WritableServerSelector from cluster description
ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE,
serverDescriptions=[ServerDescription{address=aws-cluster:27017,
type=UNKNOWN, state=CONNECTING,
exception={com.mongodb.MongoSocketReadTimeoutException: Timeout while
receiving message}, caused by
{io.netty.handler.timeout.ReadTimeoutException}}]}. Waiting for 30000
ms before timing out.
ClusterSettings clusterSettings = ClusterSettings.builder()
.applyConnectionString(new ConnectionString(connectionString)).build();
List<MongoCredential> credentials = new ArrayList<>();
credentials.add(
MongoCredential.createCredential(
mongoUserName,
mongoDBName,
mongoPassword));
MongoClientSettings settings = MongoClientSettings.builder()
.credentialList(credentials)
.clusterSettings(clusterSettings)
.streamFactoryFactory(new NettyStreamFactoryFactory())
.writeConcern(WriteConcern.ACKNOWLEDGED)
.build();
com.mongodb.async.client.MongoClient mongoClient = MongoClients.create(settings);
MongoDatabase testDB = mongoClient.getDatabase("myDB");
MongoCollection<Document> collection = testDB.getCollection("test");
Document doc = new Document("name", "MongoDB").append("type", "database");
//**trying insert document => here I got an error**
collection.insertOne(doc, new SingleResultCallback<Void>() {
#Override
public void onResult(final Void result, final Throwable t) {
System.out.println("Inserted!");
}
});
Do you have any ideas, why does it happen?
I solved it by using uri:
String uri = "mongodb://<username>:<Password>#<hostname>:27017/?ssl=true&ssl_ca_certs=cert";
MongoClientSettings settings = MongoClientSettings.builder()
.streamFactoryFactory(new NettyStreamFactoryFactory())
.applyConnectionString(new ConnectionString(uri))
.build();
com.mongodb.async.client.MongoClient mongoClient = MongoClients.create(settings);
I encountered a similar error , for me it was related to the TLS configs.
I disabled the TLS in documentDB https://docs.aws.amazon.com/documentdb/latest/developerguide/security.encryption.ssl.html
In my case I had to restart the cluster after disabling the TLS. (TLS was not needed for the use case). After the restart the connection was established successfully.

Create java RestHighLevelClient in elastic cluster mode

If elasticsearch runs on single mode, I can easily establish the RestHighLevel connection with this line of code:
RestHighLevelClient client = new RestHighLevelClient(
RestClient.builder(
new HttpHost("localhost", 9200, "http"),
new HttpHost("localhost", 9201, "http")));
But if my elastic cluster has 3 machines, e.g., "host1", "host2", "host3", how to create the rest high level client in cluster mode ?
Thanks
RestHighLevelClient client = new RestHighLevelClient(
RestClient.builder(
new HttpHost("host1", 9200, "http"),
new HttpHost("host2", 9200, "http"),
new HttpHost("host2", 9200, "http")
)
);
As the doc it looks like you were referencing states, RestClient.builder accepts an array of HttpHosts to connect to. The client (which under the hood is the ES low-level REST client) will round-robin requests to these hosts. See also the Javadoc.
As per the Elasticsearch docs you can pass multiple Elasticsearch hosts in RestClient.builder().
The better solution is to load the Elasticsearch hosts from configuration(application.conf in case of Scala-based application) instead of hardcoding it in the codebase.
Here is the Scala-based solution using Java Varargs(:_*).
application.conf
es_hosts = ["x.x.x.x","x.x.x.x","x.x.x.x"] // You can even use service-name/service-discovery
es_port = 9200
es_scheme = "http"
Code snippet
import collection.JavaConverters._
import com.typesafe.config.ConfigFactory
import org.apache.http.HttpHost
import org.elasticsearch.client.{RestClient, RestHighLevelClient}
val config = ConfigFactory.load()
val port = config.getInt(ES_PORT)
val scheme = config.getString(ES_SCHEME)
val es_hosts = config.getStringList(ES_HOSTS).asScala
val httpHosts = es_hosts.map(host => new HttpHost(host, port, scheme))
val low_level_client = RestClient.builder(httpHosts:_*)
val high_level_client: RestHighLevelClient = new RestHighLevelClient(low_level_client)
To create High level REST client using multiple hosts, you can do something like following:
String[] esHosts = new String[]{"node1-example.com:9200", "node2-example.com:9200",
"node3-example.com:9200"};
final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(esHosts)
.build();
RestHighLevelClient restClient = RestClients.create(clientConfiguration).rest();
// Hostnames used for building client can be verified as following
List<Node> nodes = restClient.getLowLevelClient().getNodes();
nodes.forEach(node -> System.out.println(node.toString()));
References:
Docs for High Level REST Client
Source code for ClientConfigurationBuilder
I am creating my elastic REST client with below steps:
RestHighLevelClient client=null;
List<HttpHost> hostList = new ArrayList<>();
for (String host : hosts) {
String[] hostDetails = host.split("\\:");hostList.add(new
HttpHost(hostDetails[0],Integer.parseInt(hostDetails[1]),https));
}
try(RestHighLevelClient client1 = new RestHighLevelClient(
RestClient.builder(hostList.toArray(new
HttpHost[hostList.size()]))
.setHttpClientConfigCallback(
httpClientBuilder ->
// to do this only if auth is enabled
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider))))
{
client = client1;
}
} catch (IOException e) {
log.error("exception occurred while setting elastic client");
}

Vert.x Hazelcast in Multidocker AWS

i try to run multiple Vert.x Instances on a EC2 Node via multiple Docker Containers.
Container A:
Port Forwardning: 5071 -> 5071
Local IP: 172.17.0.4
Container B:
Port Forwardning: 5072 -> 5072
Local IP: 172.17.0.5
Container C:
Port Forwardning: 5073 -> 5073
Local IP: 172.17.0.6
i use the Hazelcast Amazon EC2 Setup but this is not working, because the node himself has just one Public IP (set in the Hazelcastsetup) and no possibility to add ports.
How can i run multiple vertx via hazelcast in aws on different ports (maybe this different port solution is not the best one).
Thanks
Marcel
P.s.: i tried to add the nodes via tcp-ip setup, but it's not allowed to mixed AWS and tcp join.
P.p.s: i cannot and don't want use the "--net=host" in AWS ElasticBeanstalk
It looks like this one: https://github.com/hazelcast/hazelcast/issues/4537
Update:
my HC Config
JsonObject amazonConfig = clusterConfig.getJsonObject("aws");
String publicIp = null;
String privateIp = null;
String localIp = InetAddress.getLocalHost().getHostAddress();
logger.info("Found local IP: " + localIp);
try {
publicIp = doHttpUrlConnectionAction("http://169.254.169.254/latest/meta-data/public-ipv4");
logger.info("Found public IP: " + publicIp);
privateIp = doHttpUrlConnectionAction("http://169.254.169.254/latest/meta-data/local-ipv4");
logger.info("Found private IP: " + privateIp);
} catch (IOException | InterruptedException e) {
logger.fatal("Cannot detect public cloud ip");
throw e;
}
logger.info("AWS Cluster config loaded");
hazelcastConfig.getNetworkConfig().setPublicAddress(privateIp);
hazelcastConfig.getNetworkConfig().setPortAutoIncrement(false);
if (amazonConfig.containsKey("hazelcastPort")) {
logger.info("Use port " + amazonConfig.getString("hazelcastPort") + " for hazelcast");
hazelcastConfig.getNetworkConfig()
.setPublicAddress(privateIp + ":" + amazonConfig.getString("hazelcastPort"));
hazelcastConfig.getNetworkConfig().setPort(Integer.valueOf(amazonConfig.getString("hazelcastPort")));
}
// hazelcastConfig.setProperty("hazelcast.local.localAddress",
// localIp);
hazelcastConfig.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
hazelcastConfig.getNetworkConfig().getJoin().getAwsConfig().setEnabled(true);
// hazelcastConfig.getNetworkConfig().getInterfaces().setEnabled(true).addInterface(localIp);
if (amazonConfig.containsKey("region")) {
hazelcastConfig.getNetworkConfig().getJoin().getAwsConfig().setRegion(amazonConfig.getString("region"));
}
if (amazonConfig.containsKey("accessKey")) {
hazelcastConfig.getNetworkConfig().getJoin().getAwsConfig()
.setAccessKey(amazonConfig.getString("accessKey"));
}
if (amazonConfig.containsKey("secretKey")) {
hazelcastConfig.getNetworkConfig().getJoin().getAwsConfig()
.setSecretKey(amazonConfig.getString("secretKey"));
}
try {
String hazelcastGroup = System.getenv("HAZELCASTGROUP");
logger.info("Join Hazelcast Nodes with Tag HAZELCASTGROUP and Value " + hazelcastGroup);
hazelcastConfig.getNetworkConfig().getJoin().getAwsConfig().setTagKey("HAZELCASTGROUP");
hazelcastConfig.getNetworkConfig().getJoin().getAwsConfig().setTagValue(hazelcastGroup);
} catch (Exception e) {
logger.error("Cannot detect hazelcastgroup: " + e.getMessage(), e);
throw e;
}
mgr = new HazelcastClusterManager(hazelcastConfig);
vertxOptions = new VertxOptions().setClusterManager(mgr).setClustered(true);
Solution
// privateIp = doHttpUrlConnectionAction("http://169.254.169.254/latest/meta-data/local-ipv4");
hazelcastConfig.getNetworkConfig().setPublicAddress(privateIp);
Dont disable setPortAutoIncrement
for the 1st Docker Image, you should set the port to 5701 via
hazelcastConfig.getNetworkConfig().setPort(5701);
on the second Docker Image - 5702 and so on
You don't need to link the Docker Container. Just make a portmapping for each image.
Create a Security Group for this ports, so that other nodes can access the ports.
Here are my recommendations. If it doesn't solve the problem then please post the HZ log statements.
Uncomment the line which adds a property for localAddress. hazelcastConfig.setProperty("hazelcast.local.localAddress", localIp);
Disable tcp-ip configuration explicitly.
Remove the setting of Public address second time.
hazelcastConfig.getNetworkConfig().setPublicAddress(privateIp);
hazelcastConfig.getNetworkConfig().setPortAutoIncrement(false);
if (amazonConfig.containsKey("hazelcastPort")) {
logger.info("Use port " + amazonConfig.getString("hazelcastPort") + " for hazelcast");
hazelcastConfig.getNetworkConfig().setPort(Integer.valueOf(amazonConfig.getString("hazelcastPort")));
}
If you can, try to make use of default ports itself. As you pointed out in the comments, there was an issue with HZ not supporting custom ports. Also AWSClient specification doesn't allow specifying custom ports, they tend to use the default ports 5701,5702,5703. Here is the enhancement request that I had created few months back. https://github.com/hazelcast/hazelcast-aws/issues/3
Also make sure the docker containers are able to communicate with each other.

Resources