Logstash not able to connect secured (ssl) Elastic search cluster - elasticsearch

Question

I have installed Logstash,elasticsearch and kibana in single instance and installed X-pack also for TLS communication. enabled ssl communication in elasticsearch and kibana working good but logstash unable to connect elasticsearch , but i can curl elasticsearch url https://localhost:9200 there is no firewall blocking also,
I have generated open ssl certificate and key file and kept in elasticsearch
input {
beats {
client_inactivity_timeout => 1000
port => 5044
}
}
filter {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601} %{LOGLEVEL:loglevel} zeppelin IDExtractionService transactionId %{WORD:transaction_id} operation %{WORD:otype} received request duration %{NUMBER:duration} exception %{WORD:error}" ]
}
}
filter {
if "beats_input_codec_plain_applied" in [tags] {
mutate {
remove_tag => ["beats_input_codec_plain_applied"]
}
}
}
filter {
if "_grokparsefailure" in [tags] {
mutate {
remove_tag => ["_grokparsefailure"]
}
}
}
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: https://localhost:9200
output {
elasticsearch {
hosts => ["http://localhost:9200"]
user => elastic
password => password
manage_template => false
# ssl_certificate_verification => false
ssl => true
cacert => '/etc/elasticsearch/ca/key.pem'
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
}
}
elastic search config file
cluster.name: my-application
network.host: 0.0.0.0
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /opt/elasticsearch/ca/ca.key
xpack.security.http.ssl.certificate: /opt/elasticsearch/ca/ca.crt
logstash log files
[2018-05-16T05:28:16,421][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx#localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx#localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-05-16T05:28:17,201][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>1, "stalling_thread_info"=>{"other"=>[{"thread_id"=>24, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2018-05-16T05:28:21,422][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx#localhost:9200/, :path=>"/"}
[2018-05-16T05:28:21,422][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx#localhost:9200/, :path=>"/"}
[2018-05-16T05:28:21,424][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx#localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx#localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-05-16T05:28:21,425][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx#localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx#localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-05-16T05:28:22,202][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>1, "stalling_thread_info"=>{"other"=>[{"thread_id"=>24, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2018-05-16T05:28:26,425][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx#localhost:9200/, :path=>"/"}
[2018-05-16T05:28:26,426][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx#localhost:9200/, :path=>"/"}
[2018-05-16T05:28:26,427][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx#localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx#localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-05-16T05:28:26,427][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx#localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx#localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-05-16T05:28:27,201][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>1, "stalling_thread_info"=>{"other"=>[{"thread_id"=>24, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
root#5c417caecc5f:/var/log/logstash#

Answer 1

You have to enable monitoring for elasticsearch in the logstash.yml configuration file.
/etc/logstash/logstash.yml
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://X.X.X.X:9200
See this post for more information :
https://discuss.elastic.co/t/elasticsearch-unreachable-error-in-logstash/75157/7
And the documentation (maybe needed for TLS/SSL monitoring settings) :
https://www.elastic.co/guide/en/logstash/6.2/configuring-logstash.html#monitoring-settings
xpack.monitoring.elasticsearch.ssl.ca
xpack.monitoring.elasticsearch.ssl.truststore.path
xpack.monitoring.elasticsearch.ssl.truststore.password
xpack.monitoring.elasticsearch.ssl.keystore.path
xpack.monitoring.elasticsearch.ssl.keystore.password
If this doesn't work, can I see your /etc/logstash/logstash.yml configuration file ?