Kibana configuration foe es with production and monitoring clusters - elasticsearch

Question

I referred this and this
Why do we need two fields to let Kibana know where is the monitoring data.
elasticsearch.hosts
monitoring.ui.elasticsearch.hosts
But when I give my monitoring cluster at either of these properties, it works. I assumed badly that elasticsearch.hosts is my actual production cluster than monitoring cluster.
Apart from why part, is my understanding correct about this integration attributes?
Any thoughts? Thanks.
Kibana.yml:
server.host: "ip.ad.re.ss"
#elasticsearch.hosts: ["http://host1:9200","http://host2:9200","FewMoreHosts"]
monitoring.ui.elasticsearch.hosts: ["http://MonitoringNode:9200"]
I haven't changed any part in elasticsearch.yml of monitoring node.
metricbeat.yml:
output.elasticsearch:
host: ["http://MonitoringNode:9200"]
setup.kibana:
host: kibanaHost
In modules.d/elasticsearch-xpack.yml, I left the default configurations.
elasticsearch.yml:
cluster.name: es_cluster
node.name: master-1
node.data: false
node.master: true
node.ingest: true
node.max_local_storage_nodes: 3
transport.tcp.port: 9300
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["master1.ip", "master2.ip","master3.ip"]
cluster.initial_master_nodes: ["master-1","master-2"]
Monitoring cluster yml:
network.host: 0.0.0.0
discovery.type: single-node
When I enable both properties in Kibana.yml, I get the below error in the log.
{
"type": "log",
"#timestamp": "2021-04-21T14:48:34-04:00",
"tags": [
"error",
"plugins",
"data",
"data",
"indexPatterns"
],
"pid": 29959,
"message": "Error: No indices match pattern \"metricbeat-*\"\n at createNoMatchingIndicesError (/usr/share/kibana/src/plugins/data/server/index_patterns/fetcher/lib/errors.js:45:29)\n at convertEsError (/usr/share/kibana/src/plugins/data/server/index_patterns/fetcher/lib/errors.js:71:12)\n at callFieldCapsApi (/usr/share/kibana/src/plugins/data/server/index_patterns/fetcher/lib/es_api.js:69:38)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:93:5)\n at getFieldCapabilities (/usr/share/kibana/src/plugins/data/server/index_patterns/fetcher/lib/field_capabilities/field_capabilities.js:35:23)\n at IndexPatternsFetcher.getFieldsForWildcard (/usr/share/kibana/src/plugins/data/server/index_patterns/fetcher/index_patterns_fetcher.js:49:31)\n at IndexPatternsApiServer.getFieldsForWildcard (/usr/share/kibana/src/plugins/data/server/index_patterns/index_patterns_api_client.js:27:12)\n at IndexPatternsService.refreshFieldSpecMap (/usr/share/kibana/src/plugins/data/common/index_patterns/index_patterns/index_patterns.js:216:27)\n at IndexPatternsService.getSavedObjectAndInit (/usr/share/kibana/src/plugins/data/common/index_patterns/index_patterns/index_patterns.js:320:23) {\n data: null,\n isBoom: true,\n isServer: false,\n output: {\n statusCode: 404,\n payload: {\n statusCode: 404,\n error: 'Not Found',\n message: 'No indices match pattern \"metricbeat-*\"',\n code: 'no_matching_indices'\n },\n headers: {}\n }\n}"
}
But if I set only monitoring.ui.elasticsearch.hosts, Kibana shows the data.

Answer 1

The elasticsearch.hosts is the place where you will set the hosts where your data is stored, the data you want to query, this should be your production cluster.
The monitoring.ui.elasticsearch.hosts is the place where you will set the hosts of your monitoring cluster if you have a separated monitoring cluster.
Depending on the size of your cluster it is recommended to have a separated cluster just for monitoring, this could be a single-node cluster using the basic license, for example.

Answer 2

The security tokens that are used in these contexts are cluster-specific, therefore you cannot use a single Kibana instance to connect to both production and monitoring clusters
As mentioned in the documentation, I think that we need to have separate Kibana instance to collect data from both clusters.
I am not sure much about the security tokens.