Adding requestTimeout causes Kibana to fail at startup - elasticsearch

Question

I am experimenting with Elasticsearch-Kibana-Logstash for processing web log files.
Some of the queries take a bit of time and Kibana times out quickly, so I wanted to increase the amount of time that Kibana will wait for a response from elasticsearch. After a bit of searching, I found some suggestions to set elasticsearch.requestTimeout. I tried to increase the timeout by adding this to my kibana.yml file:
elasticsearch.requestTimeout: 5000
This causes Kibana to fail immediately on startup with this error:
kibserver_1 | FATAL { Error: Payload timeout must be shorter than socket timeout: POST /elasticsearch/{index}/_search
kibserver_1 | at Object.exports.assert (/usr/share/kibana/node_modules/hoek/lib/index.js:736:11)
kibserver_1 | at new module.exports.internals.Route (/usr/share/kibana/node_modules/hapi/lib/route.js:69:10)
kibserver_1 | at internals.Connection._addRoute (/usr/share/kibana/node_modules/hapi/lib/connection.js:387:19)
kibserver_1 | at internals.Connection._route (/usr/share/kibana/node_modules/hapi/lib/connection.js:379:18)
kibserver_1 | at internals.Plugin._apply (/usr/share/kibana/node_modules/hapi/lib/plugin.js:572:14)
kibserver_1 | at internals.Plugin.route (/usr/share/kibana/node_modules/hapi/lib/plugin.js:542:10)
kibserver_1 | at createProxy (/usr/share/kibana/src/core_plugins/elasticsearch/lib/create_proxy.js:85:14)
kibserver_1 | at ScopedPlugin.init [as externalInit] (/usr/share/kibana/src/core_plugins/elasticsearch/index.js:110:37)
kibserver_1 | at ScopedPlugin.tryCatcher (/usr/share/kibana/node_modules/bluebird/js/main/util.js:26:23)
kibserver_1 | at Promise.attempt.Promise.try (/usr/share/kibana/node_modules/bluebird/js/main/method.js:30:24)
kibserver_1 | at /usr/share/kibana/src/server/plugins/plugin.js:196:46
kibserver_1 | at next (native)
kibserver_1 | at step (/usr/share/kibana/src/server/plugins/plugin.js:25:191)
kibserver_1 | at /usr/share/kibana/src/server/plugins/plugin.js:25:361
kibserver_1 | cause:
kibserver_1 | Error: Payload timeout must be shorter than socket timeout: POST /elasticsearch/{index}/_search
kibserver_1 | at Object.exports.assert (/usr/share/kibana/node_modules/hoek/lib/index.js:736:11)
kibserver_1 | at new module.exports.in how to resolternals.Route (/usr/share/kibana/node_modules/hapi/lib/route.js:69:10)
kibserver_1 | at internals.Connection._addRoute (/usr/share/kibana/node_modules/hapi/lib/connection.js:387:19)
kibserver_1 | at internals.Connection._route (/usr/share/kibana/node_modules/hapi/lib/connection.js:379:18)
kibserver_1 | at internals.Plugin._apply (/usr/share/kibana/node_modules/hapi/lib/plugin.js:572:14)
kibserver_1 | at internals.Plugin.route (/usr/share/kibana/node_modules/hapi/lib/plugin.js:542:10)
kibserver_1 | at createProxy (/usr/share/kibana/src/core_plugins/elasticsearch/lib/create_proxy.js:85:14)
kibserver_1 | at ScopedPlugin.init [as externalInit] (/usr/share/kibana/src/core_plugins/elasticsearch/index.js:110:37)
kibserver_1 | at ScopedPlugin.tryCatcher (/usr/share/kibana/node_modules/bluebird/js/main/util.js:26:23)
kibserver_1 | at Promise.attempt.Promise.try (/usr/share/kibana/node_modules/bluebird/js/main/method.js:30:24)
kibserver_1 | at /usr/share/kibana/src/server/plugins/plugin.js:196:46
kibserver_1 | at next (native)
kibserver_1 | at step (/usr/share/kibana/src/server/plugins/plugin.js:25:191)
kibserver_1 | at /usr/share/kibana/src/server/plugins/plugin.js:25:361,
kibserver_1 | isOperational: true }
This one baffles me. I can't seem to find any reference to a "payload timeout" in the ElasticSearch documentation. My web searches suggest this might be coming from hapijs, but I'm not sure how to resolve this. Does anyone out there know?
(Kibana, ElasticSearch, and Logstash are all v 6.1.0)

Answer 1

I think the problem is that you're setting the timeout to a value that's too small, it's in ms and the default is 30000 (see https://www.elastic.co/guide/en/kibana/6.1/settings.html):
elasticsearch.requestTimeout:
Default: 30000 Time in milliseconds to wait for responses from the back end or Elasticsearch. This value must be a positive integer.
What might be possible is that the elasticsearch.requestTimeout is used to set the socket timeout in the hapijs and since the default for the payload time out seems to be 10 s (from here):
route.options.payload.timeout
Default value: to 10000 (10 seconds).
It will fail when checking if the payload timeout is shorter than the socket timeout. But that's just an hypotheses and I have failed find any proof in Kibana's code.