Why won't Kibana Node server start up? - elasticsearch

Question

I just upgraded to Elasticsearch and Kibana 6.0.1 from 5.6.4 and I'm having trouble getting the Kibana server running. The service appears to be running, but nothing is binding to the port and I don't see any errors in the logs.
Verifying the version I have running:
root#my-server:/var/log# /usr/share/kibana/bin/kibana --version
6.0.1
Checking the service status:
root#my-server:/var/log# sudo service kibana start
root#my-server:/var/log# sudo service kibana status
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2017-12-08 21:17:53 UTC; 1s ago
Main PID: 17766 (node)
Tasks: 6 (limit: 4915)
Memory: 86.6M
CPU: 1.981s
CGroup: /system.slice/kibana.service
└─17766 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
The contents of my /etc/kibana/kibana.yml config file:
elasticsearch.password: mypassword
elasticsearch.url: http://my-server:9200
elasticsearch.username: elastic
logging.dest: /var/log/kibana.log
logging.verbose: true
server.basePath: /kibana
server.host: localhost
server.port: 5601
The contents of my log file:
root#my-server:/var/log# tail /var/log/kibana.log -n1000
{"type":"log","#timestamp":"2017-12-08T21:17:04Z","tags":["plugins","debug"],"pid":17712,"dir":"/usr/share/kibana/plugins","message":"Scanning `/usr/share/kibana/plugins` for plugins"}
{"type":"log","#timestamp":"2017-12-08T21:17:04Z","tags":["plugins","debug"],"pid":17712,"dir":"/usr/share/kibana/src/core_plugins","message":"Scanning `/usr/share/kibana/src/core_plugins` for plugins"}
{"type":"log","#timestamp":"2017-12-08T21:17:16Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/plugins/x-pack/index.js","message":"Found plugin at /usr/share/kibana/plugins/x-pack/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:16Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/console/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/console/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:16Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/elasticsearch/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/elasticsearch/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:16Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/kbn_doc_views/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/kbn_doc_views/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:16Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/kbn_vislib_vis_types/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/kbn_vislib_vis_types/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:17Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/kibana/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/kibana/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:17Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/markdown_vis/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/markdown_vis/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:17Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/metrics/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/metrics/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:17Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/region_map/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/region_map/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:17Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/spy_modes/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/spy_modes/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:17Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/state_session_storage_redirect/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/state_session_storage_redirect/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:17Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/status_page/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/status_page/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:17Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/table_vis/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/table_vis/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:17Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/tagcloud/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/tagcloud/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:17Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/tile_map/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/tile_map/index.js"}
{"type":"log","#timestamp":"2017-12-08T21:17:17Z","tags":["plugins","debug"],"pid":17712,"path":"/usr/share/kibana/src/core_plugins/timelion/index.js","message":"Found plugin at /usr/share/kibana/src/core_plugins/timelion/index.js"}
{"type":"ops","#timestamp":"2017-12-08T21:17:18Z","tags":[],"pid":17712,"os":{"load":[1.03271484375,1.29541015625,2.1494140625],"mem":{"total":2094931968,"free":763858944},"uptime":10018},"proc":{"uptime":16.017,"mem":{"rss":269451264,"heapTotal":239005696,"heapUsed":200227592,"external":489126},"delay":3.2269310001283884},"load":{"requests":{},"concurrents":{"5601":0},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 191.0MB uptime: 0:00:16 load: [1.03 1.30 2.15] delay: 3.227"}
{"type":"log","#timestamp":"2017-12-08T21:17:18Z","tags":["info","optimize"],"pid":17712,"message":"Optimizing and caching bundles for graph, monitoring, ml, kibana, stateSessionStorageRedirect, timelion, login, logout, dashboardViewer and status_page. This may take a few minutes"}
Confirming that ES is up and running:
root#my-server:/var/log# curl -u elastic:"mypassword" http://my-
server:9200/
{
"name" : "vf9xM-O",
"cluster_name" : "my-server",
"cluster_uuid" : "pdwwLfCOTgehc_5B8oB-8g",
"version" : {
"number" : "6.0.1",
"build_hash" : "601be4a",
"build_date" : "2017-12-04T09:29:09.525Z",
"build_snapshot" : false,
"lucene_version" : "7.0.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
When I try to CURL the Kibana Node server (which should be running on 5601 as per the config):
root#my-server:/var/log# curl 'localhost:5601'
curl: (7) Failed to connect to localhost port 5601: Connection refused
Indeed when I list the open ports, I see lots of things, but nothing on 5601:
root#my-server:/var/log# netstat -ntlp | grep LISTEN
tcp 0 0 0.0.0.0:9191 0.0.0.0:* LISTEN 1412/uwsgi
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 1413/systemd-resolv
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 15924/nginx: master
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1519/sshd
tcp 0 0 0.0.0.0:3031 0.0.0.0:* LISTEN 1412/uwsgi
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 1591/postgres
tcp6 0 0 :::5355 :::* LISTEN 1413/systemd-resolv
tcp6 0 0 :::9200 :::* LISTEN 16108/java
tcp6 0 0 :::9300 :::* LISTEN 16108/java
tcp6 0 0 :::22 :::* LISTEN 1519/sshd
tcp6 0 0 :::5432 :::* LISTEN 1591/postgres
I'm not sure what else to try to troubleshoot Kibana, any ideas are really really appreciated!

Answer 1

Well I'm not really sure why, but this seemed to work after I rebooted the machine:
root#my-server:~# sudo reboot
after a minute I SSHed back in and voila:
root#my-server:~# netstat -ntlp | grep LISTEN
tcp 0 0 127.0.0.1:5601 0.0.0.0:* LISTEN 1428/node
tcp 0 0 0.0.0.0:9191 0.0.0.0:* LISTEN 1414/uwsgi
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 1427/systemd-resolv
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1467/nginx: master
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1510/sshd
tcp 0 0 0.0.0.0:3031 0.0.0.0:* LISTEN 1414/uwsgi
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 1578/postgres
tcp6 0 0 :::5355 :::* LISTEN 1427/systemd-resolv
tcp6 0 0 :::9200 :::* LISTEN 2011/java
tcp6 0 0 :::9300 :::* LISTEN 2011/java
tcp6 0 0 :::22 :::* LISTEN 1510/sshd
tcp6 0 0 :::5432 :::* LISTEN 1578/postgres
¯_(ツ)_/¯