cross cluster elasticsearch on kuberntes - elasticsearch

Question

I have 2 elastics cluster on 2 different kubernetes VMS I tried to connect with cross cluster . but its not working, I add detailed below can someone assist and tell me what I did wrong or missed?
I tried to connect from one elastic to another as below:
GET _cluster/settings
{
"persistent" : {
"cluster" : {
"remote" : {
"cluster_three" : {
"mode" : "proxy",
"proxy_address" : "122.22.111.222:30005"
},
"cluster_two" : {
"mode" : "sniff",
"skip_unavailable" : "false",
"transport" : {
"compress" : "true"
},
"seeds" : [
"122.22.222.182:30005"
]
},
"cluster_one" : {
"seeds" : [
"127.0.0.1:9200"
],
"transport" : {
"ping_schedule" : "30s"
}
}
}
}
},
"transient" : { }
}
}
I tried to search on cluster two and I get the following error:
{"statusCode":502,"error":"Bad Gateway","message":"Client request timeout"}
but when I do curl on elastic to cluste_two I get this :
curl 122.22.222.182:30005
{
"name" : "elasticsearch-client-7dcc49ddsdsd4-ljwasdpl",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "bOkaIrcFTgetsadaaY114N4a1EQ",
"version" : {
"number" : "7.10.2",
"build_flavor" : "oss",
"build_type" : "docker",
"build_hash" : "747e1cc71def077253878a59143c1f785asdasafa92b9",
"build_date" : "2021-01-13T00:42:12.435326Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
this is my svc configured on kubernetes for cluste_two:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch-client NodePort 10.111.11.28 <none> 9200:30005/TCP 27m
elasticsearch-discovery ClusterIP 10.111.11.11 <none> 9300/TCP 27m

Answer 1

Elasticsearch discovery work on port 9300 instead of 9200 while you are running curl it's going as client request over port 30005.
Please check 9300 is open to connect cross cluster. as your elasticsearch-discovery service running as clusterIP you might have to change the type of it to expose out of K8s using NodePort of LoadBalancer as per requirement.
for example
# From cluster 1, we’ll define how the cluster-2 can be accessed
PUT /_cluster/settings
{
"persistent" : {
"cluster" : {
"remote" : {
"us-cluster" : {
"seeds" : [
"127.0.0.1:9300"
]
}
}
}
}
}
you can also look into : https://www.elastic.co/blog/cross-datacenter-replication-with-elasticsearch-cross-cluster-replication