find elasticsearch service endpoint - elasticsearch

Question

I'm on my trial to test elasticcloud. But now I got problem to create pipeline from logstash to elasticcloud. Here is my logstash.conf output
output {
stdout{codec=>rubydebug}
elasticsearch
{
hosts=>["https://<clusterid>.asia-southeast1.gcp.cloud.es.io:9243"]
index=>"testindex"
user=>elasticdeploymentcredentials
password=>elasticdeploymentcredentials
}
}
But it always returning error as:
[WARN ] 2021-03-29 12:24:50.148 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
[WARN ] 2021-03-29 12:24:55.158 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 12:25:00.163 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
[WARN ] 2021-03-29 12:25:05.170 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 12:25:10.175 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx#<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
It is possible for me to curl it with my credential as :
[root#localhost testconfig]# curl https://elasticdeploymentcredentials:elasticdeploymentcredentials#<clusterid>.asia-southeast1.gcp.elastic-cloud.com:9243
it returning
"name" : "name",
"cluster_name" : "<clusterid>",
"cluster_uuid" : "<clusteruuid>",
"version" : {
"number" : "7.12.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "build_hash",
"build_date" : "2021-03-18T06:17:15.410153305Z",
"build_snapshot" : false,
"lucene_version" : "8.8.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
am I missing something?

Answer 1

Instead of trying to connect to Elastic Cloud via the username/password from the deployment, try to use the Cloud_ID/Cloud_Auth combination:
output {
elasticsearch {
hosts => ["https://<clusterid>.asia-southeast1.gcp.cloud.es.io:9243"]
index => "%{[#metadata][beat]}-%{[#metadata][version]}-%{+YYYY.MM.dd}"
cloud_id => "your cloudid from the console"
cloud_auth => "elastic:password"
}
}
The cloud_auth parameter is where you are actually going to use the username/password from the deployment. More information here:
https://www.elastic.co/guide/en/logstash/7.12/connecting-to-cloud.html