Kibana is not responding but cluster is working fine - elasticsearch

We are using the Kibana hosted endpoint that comes with Elasticsearch on AWS accessible with a URL like https://*******.us-east-1.es.amazonaws.com/_plugin/kibana/
As of a few days ago, Kibana became unaccessible and Chrome/cURL returns a ERR_TIMED_OUT error.
Telnet shows that the port 443 is timing out:
telnet ******.us-east-1.es.amazonaws.com 443
Trying 10.0.1.238...
telnet: connect to address 10.0.1.238: Operation timed out
Trying 10.0.0.28...
telnet: connect to address 10.0.0.28: Operation timed out
telnet: Unable to connect to remote host
Everything else on the cluster seems to be working fine. Accessing Elastic thru code works and the cluster status is "Status: Green". The "Cluster Health" page on AWS does not show any visible issues.
The only other issue I am aware of is our inability to migrate Elastic from 6.x to 7.x due to this error:
Checking upgrade eligibility - Failed
Cluster has 3082.0 shards per node which exceeds the setting cluster.max_shards_per_node value 1000
According to AWS documentation "The 7.x versions of Elasticsearch have a default setting of no more than 1,000 shards per node" so I am unsure if this is related to our inability to access Kibana or not.
Would anyone know of a good next step to re-gain access to Kibana?

Related

How to Access ElasticSearch From Server?

I am using elastic search in my ubuntu16.04 server. When i am trying to access elasticsearch from browser by using url ip:port/_cat/indices?v. I am getting site can't be reached. After that i am change the network.host value to network.host: 0.0.0.0. After change the network.host ip the search engine not started. How can i access the elasticsearch in my browser.I changed the port also.
Thank you..
There can be many reasons for ES not being reachable. I would start with the obvious and make sure that:
ES is listening on the port: on the ES instance when you run 'curl
ip:port' you should get an answer. if not the service didn't start
well.
make sure there are no firewall rules/security groups that prevent
access from remote network.
make sure network.publish_host is configure correctly:
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#advanced-network-settings
more info here: ElasticSearch instance not reachable from outside the server - Azure Windows 2012

Kibana unable to connect to Elasticsearch

I am having trouble getting Kibana to connect to elasticsearch. The error I get is below;
plugin:elasticsearch#5.6.1 Request Timeout after 3000ms
Elasticsearch has been deployed via kubernetes and runs as a service called monitoring-es with a cluster IP of 100.63.124.194.
In my Kibana.yml file I have set the elasticsearch url as;
elasticsearch.url: "http://100.63.124.194:9200"
However I still get the same error. Is this the correct value or is there something I am missing?
Thanks

ElasticSearch on EC2 - search queries are blocked

I have installed ElasticSearch 5.5 on EC2 (not using managed service). the EC2 has elastic ip and all incoming and outgoing rules are open.
I am able to add data and query the mapping, cluster status, templates and so on, however when i try to run a query a data of a certain index, I am getting a timeout. when sending the request within the VPC (using private IP) all is ok. I run tcpdump to see what is going on and for my surprise I didnt see any incoming request when i did a search query, all the other requests were seen in the dump. Then I have installed on a new EC2 elasticsearch cluster, and again the same behavior
what is going on?
it was a firewall issue that blocked the request

ElasticSearch-Kibana Connection Error: Unable to Connect

I have an elasticsearch server with host h1 running at port 9200. And i have a separate kibana server with host h2 running on port 5601. I have configured both of them properly and then tried to see the GUI of kibana.
It says Status:Red and in the error i see "Unable to connect to Elasticsearch at h1:9200"
I ran the basic troubleshoot mentioned in the elk documentation i.e. to check whether kibana.yml's elasticsearch.url field is properly referring to h1:9200 or not and vice versa for elasticsearch.yml's fields. Everything seems to be fine. What could be any other reason for this error? How to fix the same?
Could be for a number of reasons, connectivity/network configuration usually the prime suspect - check out this troubleshooting article for possible solutions

Kibana web interface not loading

Despite ElasticSearch and Kibana both running on my production server, I'm unable to visit the GUI over the public IP: http://52.4.153.19:5601/
Localhost curls return 200 but console errors on the browser report timeouts after a few images are retrieved.
I've successfully installed, run, and accessed Kibana on my local (Windows 10) and on my staging AWS EC2 Ubuntu 14.04 environment. I'm able to access both over port 5601 on localhost and the staging environment is accessible over the public IP address and all domains addressed accordingly. The reverse proxy also works and all status indicators are green on the dashboard.
I'm running Kibana 4.5, ElasticSearch 2.3.1, Apache 2.4.12
I've used the same exact volume from the working environment to attach to the production instance, so everything is identical on the two volumes, except that the staging environment's apache vhost uses a subdomain while the production environment's servername is the base domain. Both are configured for SSL wildcards. Both are in separate availability zones at Amazon. I've tried altering the server block to use a subdomain on the production server, just to see if the domain was impactful but the error remains.
I also tried running one instance individually, in case EC2 had some kind of networking error with 0.0.0.0 but I'm unable to come to a resolution. All logs and configurations are identical between the two servers for ElasticSearch and Kibana.
I've tried deleting and re-creating the kibana index, tried alternate settings inclusive of the host, elasticsearch url, extending the max ping and timeout, max retries, extended the apache limits, http.cors to allow different origins. I've tried other ports but both servers are indicating that 5601 is listening in the same way.
I also had the same problem on a completely different volume that was previously attached to this instance.
The only difference I can see is that the working version pings fine while the non-working version has a 100% packet loss when pinging the IP, although I can't imagine why that would be, as I'm able to reach the website on 80, just fine. I can also access various other tools running on other ports. I assume there might be some kind of networking conflict. Any ideas?
May be port 5601 is blocked by firewall
Allow incoming connections to port 5601 by:
sudo iptables -I INPUT -p tcp --dport 5601 -j ACCESS
For security:
Modify above mentioned command and accept connection only from specific address. (See man iptables)
or use Shield plugin for elasticseach
Sorry, forgot to update this question. The answer turned out being that I simply needed to deploy a new instance. Simply by creating a clone of the instance, I was able to resolve the issue. I've had networking problems at AWS, before, with their internal dns/ip conflicts, so I've had to do so, in the past and this turned out to be the quickest and cleanest solution, albeit not providing any definitive insight into the cause.

Resources