Logstash not able to connect to Elasticsearch deployed on Kubernetes cluster - elasticsearch

Question

I have deployed Logstash and elasticsearch pod on EKS cluster. When I am checking the logs for logstash pod it is showing unreachable elasticserach server. Though my elasticsearch is up and running. Please find below yaml files and log error.
configMap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: "logstash-configmap-development"
namespace: "development"
labels:
app: "logstash-development"
data:
logstash.conf: |-
input {
http {
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["https://my-server.com/elasticsearch-development/"]
index => "%{[#metadata][beat]}-%{[#metadata][version]}-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}
deployment.yaml
---
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "logstash-development"
namespace: "development"
spec:
selector:
matchLabels:
app: "logstash-development"
replicas: 1
strategy:
type: "RollingUpdate"
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
minReadySeconds: 5
template:
metadata:
labels:
app: "logstash-development"
spec:
containers:
-
name: "logstash-development"
image: "logstash:7.10.2"
imagePullPolicy: "Always"
env:
-
name: "XPACK_MONITORING_ELASTICSEARCH_HOSTS"
value: "https://my-server.com/elasticsearch-development/"
-
name: "XPACK_MONITORING_ELASTICSEARCH_URL"
value: "https://my-server.com/elasticsearch-development/"
-
name: "SERVER_BASEPATH"
value: "logstash-development"
securityContext:
privileged: true
ports:
-
containerPort: 8080
protocol: TCP
volumeMounts:
-
name: "logstash-conf-volume"
mountPath: "/usr/share/logstash/pipeline/"
volumes:
-
name: "logstash-conf-volume"
configMap:
name: "logstash-configmap-development"
items:
- key: "logstash.conf"
path: "logstash.conf"
imagePullSecrets:
-
name: "logstash"
service.yaml
---
apiVersion: "v1"
kind: "Service"
metadata:
name: "logstash-development"
namespace: "development"
labels:
app: "logstash-development"
spec:
ports:
-
port: 55770
targetPort: 8080
selector:
app: "logstash-development"
Logstash pod log error
[2021-06-09T08:22:38,708][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://my-server.com/elasticsearch-development/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://my-server.com/elasticsearch-development/][Manticore::ConnectTimeout] connect timed out"}
Note:- Elasticsearch is up and running. And when I hit the logstash url it is giving as status ok.
I have checked with elasticsearch cluster-ip, their logstash is able to connect with Elasticsearch, but when I am giving ingress path url it is not able to connect to elasticsearch.
Also from the logs, I noticed it is taking incorrect url for elasticsearch.
My elasticsearch url is something like this:- https://my-server.com/elasticserach
but instead logstash is looking for https://my-server.com:9200/elasticsearch
With this url (https://my-server.com:9200/elasticsearch) elasticsearch is not accessible as a result it is giving connection timeout.
Can someone tell why it is taking (https://my-server.com:9200/elasticsearch) and not (https://my-server.com/elasticsearch)

Answer 1

I am now able to connect logstash with elasticsearch, if you are using elasticsearch with dns name, logstash by default will take the port of elasticsearch as 9200, so in my case it was taking the elasticsearch url as https://my-server.com:9200/elasticsearch-development/. But with that url, elasticsearch was not accessible, it was only accessible with (https://myserver.com/elasticsearch-development/). So I need to add the https port i.e 443 in my elasticsearch url, through which the logstash would be able to connect to elasticserach (https://my-server.com:443/elasticsearch-development/)
Long story short:-
In deployment.yaml file under env variable XPACK_MONITORING_ELASTICSEARCH_HOSTS and XPACK_MONITORING_ELASTICSEARCH_URL given the value as https://my-server.com:443/elasticsearch-development/
Same value was given in logstash.conf file.