I recently noted that Elasticsearch was not starting after a reboot. Eventualy I deleted it and re-installed from scratch. However I'm still having the same problem. When I try to start it, systemctl shows the following status:
$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: failed (Result: timeout) since Wed 2020-09-16 16:22:32 EDT; 3min 23s ago
Docs: https://www.elastic.co
Process: 1716 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143)
Main PID: 1716 (code=exited, status=143)
Sep 16 16:20:59 enk-network-sniffer systemd[1]: Starting Elasticsearch...
Sep 16 16:22:29 enk-network-sniffer systemd[1]: elasticsearch.service: start operation timed out. Terminating.
Sep 16 16:22:32 enk-network-sniffer systemd[1]: elasticsearch.service: Failed with result 'timeout'.
Sep 16 16:22:32 enk-network-sniffer systemd[1]: Failed to start Elasticsearch.
Looking at
/var/log/elasticsearch/elasticsearch.log
shows the following logs:
[2020-09-16T16:42:33,268][INFO ][o.e.e.NodeEnvironment ] [enk-network-sniffer] using [1] data paths, mounts [[/ (/dev/sda5)]], net usable_space [23.4gb], net total_space [38.6gb], types [ext4]
[2020-09-16T16:42:33,301][INFO ][o.e.e.NodeEnvironment ] [enk-network-sniffer] heap size [1gb], compressed ordinary object pointers [true]
[2020-09-16T16:42:33,717][INFO ][o.e.n.Node ] [enk-network-sniffer] node name [enk-network-sniffer], node ID [RljBYx0ZQx6E87E__MV3gQ], cluster name [elasticsearch]
[2020-09-16T16:42:33,720][INFO ][o.e.n.Node ] [enk-network-sniffer] version[7.7.1], pid[1690], build[default/deb/ad56dce891c901a492bb1ee393f12dfff473a423/2020-05-28T16:30:01.040088Z], OS[Linux/5.4.0-47-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[2020-09-16T16:42:33,721][INFO ][o.e.n.Node ] [enk-network-sniffer] JVM home [/usr/share/elasticsearch/jdk]
[2020-09-16T16:42:33,722][INFO ][o.e.n.Node ] [enk-network-sniffer] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-8868144046098870714, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2020-09-16T16:42:54,521][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [aggs-matrix-stats]
[2020-09-16T16:42:54,522][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [analysis-common]
[2020-09-16T16:42:54,531][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [constant-keyword]
[2020-09-16T16:42:54,531][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [flattened]
[2020-09-16T16:42:54,532][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [frozen-indices]
[2020-09-16T16:42:54,535][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [ingest-common]
[2020-09-16T16:42:54,535][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [ingest-geoip]
[2020-09-16T16:42:54,536][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [ingest-user-agent]
[2020-09-16T16:42:54,538][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [lang-expression]
[2020-09-16T16:42:54,539][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [lang-mustache]
[2020-09-16T16:42:54,543][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [lang-painless]
[2020-09-16T16:42:54,543][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [mapper-extras]
[2020-09-16T16:42:54,547][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [parent-join]
[2020-09-16T16:42:54,548][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [percolator]
[2020-09-16T16:42:54,551][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [rank-eval]
[2020-09-16T16:42:54,552][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [reindex]
[2020-09-16T16:42:54,552][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [repository-url]
[2020-09-16T16:42:54,555][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [search-business-rules]
[2020-09-16T16:42:54,556][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [spatial]
[2020-09-16T16:42:54,556][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [systemd]
[2020-09-16T16:42:54,561][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [tasks]
[2020-09-16T16:42:54,562][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [transform]
[2020-09-16T16:42:54,562][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [transport-netty4]
[2020-09-16T16:42:54,565][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [vectors]
[2020-09-16T16:42:54,567][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-analytics]
[2020-09-16T16:42:54,571][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-async-search]
[2020-09-16T16:42:54,574][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-autoscaling]
[2020-09-16T16:42:54,574][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-ccr]
[2020-09-16T16:42:54,577][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-core]
[2020-09-16T16:42:54,580][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-deprecation]
[2020-09-16T16:42:54,581][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-enrich]
[2020-09-16T16:42:54,582][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-eql]
[2020-09-16T16:42:54,583][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-graph]
[2020-09-16T16:42:54,586][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-identity-provider]
[2020-09-16T16:42:54,588][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-ilm]
[2020-09-16T16:42:54,592][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-logstash]
[2020-09-16T16:42:54,593][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-ml]
[2020-09-16T16:42:54,594][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-monitoring]
[2020-09-16T16:42:54,602][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-ql]
[2020-09-16T16:42:54,602][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-rollup]
[2020-09-16T16:42:54,602][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-security]
[2020-09-16T16:42:54,612][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-sql]
[2020-09-16T16:42:54,612][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-voting-only-node]
[2020-09-16T16:42:54,633][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-watcher]
[2020-09-16T16:42:54,634][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] no plugins loaded
I'm not being able to find any error or warning in these files. Any idea on what might be wrong or where else I can look at to find more clues?
UPDATE:
Here is the /etc/elasticsearch/elasticsearch.yml file:
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
discovery.type: single-node
cluster.initial_master_nodes: []
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
After increasing the heap size, I obtained the following logs:
[2020-09-17T11:47:21,107][INFO ][o.e.e.NodeEnvironment ] [enk-network-sniffer] using [1] data paths, mounts [[/ (/dev/sda5)]], net usable_space [174.4gb], net total_space [196.1gb], types [ext4]
[2020-09-17T11:47:21,129][INFO ][o.e.e.NodeEnvironment ] [enk-network-sniffer] heap size [4gb], compressed ordinary object pointers [true]
[2020-09-17T11:47:23,118][INFO ][o.e.n.Node ] [enk-network-sniffer] node name [enk-network-sniffer], node ID [RljBYx0ZQx6E87E__MV3gQ], cluster name [elasticsearch]
[2020-09-17T11:47:23,128][INFO ][o.e.n.Node ] [enk-network-sniffer] version[7.7.1], pid[1966], build[default/deb/ad56dce891c901a492bb1ee393f12dfff473a423/2020-05-28T16:30:01.040088Z], OS[Linux/5.4.0-47-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[2020-09-17T11:47:23,129][INFO ][o.e.n.Node ] [enk-network-sniffer] JVM home [/usr/share/elasticsearch/jdk]
[2020-09-17T11:47:23,129][INFO ][o.e.n.Node ] [enk-network-sniffer] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms4g, -Xmx4g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-11788675126322072724, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=2147483648, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2020-09-17T11:47:54,845][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [aggs-matrix-stats]
[2020-09-17T11:47:54,845][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [analysis-common]
[2020-09-17T11:47:54,849][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [constant-keyword]
[2020-09-17T11:47:54,850][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [flattened]
[2020-09-17T11:47:54,850][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [frozen-indices]
[2020-09-17T11:47:54,850][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [ingest-common]
[2020-09-17T11:47:54,850][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [ingest-geoip]
[2020-09-17T11:47:54,851][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [ingest-user-agent]
[2020-09-17T11:47:54,852][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [lang-expression]
[2020-09-17T11:47:54,852][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [lang-mustache]
[2020-09-17T11:47:54,853][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [lang-painless]
[2020-09-17T11:47:54,853][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [mapper-extras]
[2020-09-17T11:47:54,854][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [parent-join]
[2020-09-17T11:47:54,854][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [percolator]
[2020-09-17T11:47:54,855][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [rank-eval]
[2020-09-17T11:47:54,855][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [reindex]
[2020-09-17T11:47:54,856][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [repository-url]
[2020-09-17T11:47:54,856][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [search-business-rules]
[2020-09-17T11:47:54,856][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [spatial]
[2020-09-17T11:47:54,856][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [systemd]
[2020-09-17T11:47:54,857][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [tasks]
[2020-09-17T11:47:54,857][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [transform]
[2020-09-17T11:47:54,858][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [transport-netty4]
[2020-09-17T11:47:54,858][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [vectors]
[2020-09-17T11:47:54,859][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-analytics]
[2020-09-17T11:47:54,860][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-async-search]
[2020-09-17T11:47:54,860][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-autoscaling]
[2020-09-17T11:47:54,860][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-ccr]
[2020-09-17T11:47:54,861][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-core]
[2020-09-17T11:47:54,861][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-deprecation]
[2020-09-17T11:47:54,862][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-enrich]
[2020-09-17T11:47:54,862][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-eql]
[2020-09-17T11:47:54,862][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-graph]
[2020-09-17T11:47:54,863][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-identity-provider]
[2020-09-17T11:47:54,863][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-ilm]
[2020-09-17T11:47:54,864][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-logstash]
[2020-09-17T11:47:54,864][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-ml]
[2020-09-17T11:47:54,865][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-monitoring]
[2020-09-17T11:47:54,867][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-ql]
[2020-09-17T11:47:54,868][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-rollup]
[2020-09-17T11:47:54,868][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-security]
[2020-09-17T11:47:54,868][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-sql]
[2020-09-17T11:47:54,869][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-voting-only-node]
[2020-09-17T11:47:54,869][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] loaded module [x-pack-watcher]
[2020-09-17T11:47:54,870][INFO ][o.e.p.PluginsService ] [enk-network-sniffer] no plugins loaded
ES is still failing to start.
Thanks.
As there is no error message, its difficult to pinpoint to the exact cause, but could you provide your elasticsearch.yml and I also see you have defined 1gb of Heap space, could you try increasing it?
Only interesting this in the logs are:
Active: failed (Result: timeout) since Wed 2020-09-16 16:22:32 EDT;
3min 23s ago
Docs: https://www.elastic.co
Process: 1716 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p
${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143) Main
PID: 1716 (code=exited, status=143)
You can notice the timeout and 143 status and you can read more about it in
https://serverfault.com/questions/695849/services-remain-in-failed-state-after-stopped-with-systemctl and see if you can figure out the error logs or more info to debug the issue.
Related
I'm running Elasticsearch version 7.10.2 with 8 nodes of cluster (3 master, 3 data, and 2 client node).
It was running smoothly till now , now the nodes began to disconnect from the cluster one node at a day , when I checked the node Elasticsearch was stopped. I have noticed that node get disconnected randomly
last node that was disconnected was data-node-3 at 12:10 , I want to know the main reason why Elasticsearch getting stopped also please suggest on how to resolve this problem.
Note:-we only restart master-1(Monday) , master-2(Tuesday) and master-3(Wednesday) using cronjob we have not set any cronjob for data and client nodes.
cluster information:-
Name RAM Heap Cpu
client 1 4gb 2gb 2
client 2 4gb 2gb 2
data 1 8gb 4gb 2
data 2 8gb 4gb 2
data 3 8gb 4gb 2
master 1 2gb 1gb 2
master 2 2gb 1gb 2
master 3 2gb 1gb 2
I am attaching log here of datanode-3
[2022-08-21T12:10:00,193][INFO ][o.e.n.Node ] [data-node-3] version[7.10.2], pid[534], build[default/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.15.0-1016-gcp/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-08-21T12:10:00,381][INFO ][o.e.n.Node ] [data-node-3] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-08-21T12:10:00,382][INFO ][o.e.n.Node ] [data-node-3] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms4g, -Xmx4g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-5474094125273943758, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=2147483648, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2022-08-21T12:10:18,654][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [aggs-matrix-stats]
[2022-08-21T12:10:18,656][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [analysis-common]
[2022-08-21T12:10:18,656][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [constant-keyword]
[2022-08-21T12:10:18,660][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [flattened]
[2022-08-21T12:10:18,661][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [frozen-indices]
[2022-08-21T12:10:18,663][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [ingest-common]
[2022-08-21T12:10:18,663][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [ingest-geoip]
[2022-08-21T12:10:18,672][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [ingest-user-agent]
[2022-08-21T12:10:18,673][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [kibana]
[2022-08-21T12:10:18,674][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [lang-expression]
[2022-08-21T12:10:18,674][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [lang-mustache]
[2022-08-21T12:10:18,675][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [lang-painless]
[2022-08-21T12:10:18,675][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [mapper-extras]
[2022-08-21T12:10:18,677][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [mapper-version]
[2022-08-21T12:10:18,679][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [parent-join]
[2022-08-21T12:10:18,680][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [percolator]
[2022-08-21T12:10:18,681][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [rank-eval]
[2022-08-21T12:10:18,682][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [reindex]
[2022-08-21T12:10:18,682][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [repositories-metering-api]
[2022-08-21T12:10:18,684][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [repository-url]
[2022-08-21T12:10:18,685][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [search-business-rules]
[2022-08-21T12:10:18,685][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [searchable-snapshots]
[2022-08-21T12:10:18,697][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [spatial]
[2022-08-21T12:10:18,698][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [systemd]
[2022-08-21T12:10:18,698][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [transform]
[2022-08-21T12:10:18,699][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [transport-netty4]
[2022-08-21T12:10:18,701][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [unsigned-long]
[2022-08-21T12:10:18,702][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [vectors]
[2022-08-21T12:10:18,702][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [wildcard]
[2022-08-21T12:10:18,704][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-analytics]
[2022-08-21T12:10:18,709][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-async]
[2022-08-21T12:10:18,710][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-async-search]
[2022-08-21T12:10:18,710][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-autoscaling]
[2022-08-21T12:10:18,711][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-ccr]
[2022-08-21T12:10:18,712][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-core]
[2022-08-21T12:10:18,713][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-data-streams]
[2022-08-21T12:10:18,714][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-deprecation]
[2022-08-21T12:10:18,714][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-enrich]
[2022-08-21T12:10:18,717][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-eql]
[2022-08-21T12:10:18,719][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-graph]
[2022-08-21T12:10:18,722][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-identity-provider]
[2022-08-21T12:10:18,723][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-ilm]
[2022-08-21T12:10:18,724][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-logstash]
[2022-08-21T12:10:18,724][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-ml]
[2022-08-21T12:10:18,725][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-monitoring]
[2022-08-21T12:10:18,732][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-ql]
[2022-08-21T12:10:18,742][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-rollup]
[2022-08-21T12:10:18,743][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-security]
[2022-08-21T12:10:18,743][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-sql]
[2022-08-21T12:10:18,744][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-stack]
[2022-08-21T12:10:18,744][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-voting-only-node]
[2022-08-21T12:10:18,745][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-watcher]
[2022-08-21T12:10:18,746][INFO ][o.e.p.PluginsService ] [data-node-3] no plugins loaded
[2022-08-21T12:10:19,097][INFO ][o.e.e.NodeEnvironment ] [data-node-3] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [77.9gb], net total_space [96.7gb], types [ext4]
[2022-08-21T12:10:19,098][INFO ][o.e.e.NodeEnvironment ] [data-node-3] heap size [4gb], compressed ordinary object pointers [true]
[2022-08-21T12:10:22,654][INFO ][o.e.n.Node ] [data-node-3] node name [data-node-3], node ID [KG1DQKuASBq2GpwYCz4pkQ], cluster name [la-prod7-cluster], roles [transform, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold]
[2022-08-21T12:10:37,839][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [data-node-3] [controller/974] [Main.cc#114] controller (64 bit): Version 7.10.2 (Build 40a3af639d4698) Copyright (c) 2021 Elasticsearch BV
[2022-08-21T12:10:38,910][INFO ][o.e.x.s.a.s.FileRolesStore] [data-node-3] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2022-08-21T12:10:40,701][INFO ][o.e.t.NettyAllocator ] [data-node-3] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=512kb, suggested_max_allocation_size=512kb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=2mb}]
[2022-08-21T12:10:40,845][INFO ][o.e.d.DiscoveryModule ] [data-node-3] using discovery type [zen] and seed hosts providers [settings]
[2022-08-21T12:10:41,685][WARN ][o.e.g.DanglingIndicesState] [data-node-3] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-08-21T12:10:42,542][INFO ][o.e.n.Node ] [data-node-3] initialized
[2022-08-21T12:10:42,543][INFO ][o.e.n.Node ] [data-node-3] starting ...
[2022-08-21T12:10:43,019][INFO ][o.e.t.TransportService ] [data-node-3] publish_address {10.115.0.141:9300}, bound_addresses {10.115.0.141:9300}
[2022-08-21T12:10:44,459][INFO ][o.e.b.BootstrapChecks ] [data-node-3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-08-21T12:10:44,464][INFO ][o.e.c.c.Coordinator ] [data-node-3] cluster UUID [uSrx3heuSVypgStFinWJrw]
[2022-08-21T12:10:48,038][INFO ][o.e.c.s.ClusterApplierService] [data-node-3] master node changed {previous [], current [{master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false}]}, added {{master-node-2}{t4Pbwly1S0q084vosUjmNw}{m6Rwe3xLSquVPA2Oj0bFxQ}{10.115.0.49}{10.115.0.49:9300}{m}{xpack.installed=true, transform.node=false},{data-node-1}{aesOUIaRTA-4xHPDSZ-ZyA}{x1xAw9K8RGuGCzahKeE3EA}{10.115.0.95}{10.115.0.95:9300}{cdhlrstw}{ml.machine_memory=8340889600, ml.max_open_jobs=20, xpack.installed=true, transform.node=true},{master-node-3}{ka0XuT6zQMOLL_CTqa3ysA}{AmKed0FWR8CnZoNE5JsUAw}{10.115.0.114}{10.115.0.114:9300}{m}{xpack.installed=true, transform.node=false},{master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false},{client-node-2}{UOSU11VmS_GByWchh0jV8w}{X2uXIsMsQPu_34SaOKk5uw}{10.115.0.81}{10.115.0.81:9300}{ilr}{ml.machine_memory=4126511104, ml.max_open_jobs=20, xpack.installed=true, transform.node=false},{data-node-2}{qEU0XgvoSCmgqfBLZIEOLg}{zfVDj9HWT-OlVnhfyv6VSg}{10.115.0.112}{10.115.0.112:9300}{cdhlrstw}{ml.machine_memory=8332742656, ml.max_open_jobs=20, xpack.installed=true, transform.node=true},{client-node-1}{FvYReNu1SF-SX7aT5d3GLQ}{iKJIdSP_RAey-Q9N-OkH2Q}{10.115.0.80}{10.115.0.80:9300}{ilr}{ml.machine_memory=4126502912, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}}, term: 168, version: 15421, reason: ApplyCommitRequest{term=168, version=15421, sourceNode={master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false}}
[2022-08-21T12:10:48,072][INFO ][o.e.n.Node ] [data-node-3] stopping ...
[2022-08-21T12:10:48,077][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [data-node-3] [controller/974] [Main.cc#154] ML controller exiting
[2022-08-21T12:10:48,078][INFO ][o.e.x.m.p.NativeController] [data-node-3] Native controller process has stopped - no new native processes can be started
[2022-08-21T12:10:48,080][INFO ][o.e.x.w.WatcherService ] [data-node-3] stopping watch service, reason [shutdown initiated]
[2022-08-21T12:10:48,081][INFO ][o.e.x.w.WatcherLifeCycleService] [data-node-3] watcher has stopped and shutdown
[2022-08-21T12:10:48,511][INFO ][o.e.h.AbstractHttpServerTransport] [data-node-3] publish_address {10.115.0.141:9200}, bound_addresses {10.115.0.141:9200}
[2022-08-21T12:10:48,511][INFO ][o.e.n.Node ] [data-node-3] started
[2022-08-21T12:10:49,584][INFO ][o.e.c.s.ClusterSettings ] [data-node-3] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2022-08-21T12:10:50,534][INFO ][o.e.x.s.a.TokenService ] [data-node-3] refresh keys
[2022-08-21T12:10:50,880][INFO ][o.e.x.s.a.TokenService ] [data-node-3] refreshed keys
[2022-08-21T12:10:51,977][INFO ][o.e.l.LicenseService ] [data-node-3] license [4d78bbbf-fb01-4d13-8178-86cc235014f1] mode [basic] - valid
[2022-08-21T12:10:51,980][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [data-node-3] Active license is now [BASIC]; Security is enabled
[2022-08-21T12:10:51,995][INFO ][o.e.c.c.Coordinator ] [data-node-3] master node [{master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [master-node-1][10.115.0.21:9300][disconnected] disconnected
[2022-08-21T12:10:52,001][INFO ][o.e.c.c.JoinHelper ] [data-node-3] failed to join {master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false} with JoinRequest{sourceNode={data-node-3}{KG1DQKuASBq2GpwYCz4pkQ}{KmQunMToQcS5Stxb5IT7MA}{10.115.0.141}{10.115.0.141:9300}{cdhlrstw}{ml.machine_memory=8332750848, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, minimumTerm=168, optionalJoin=Optional.empty}
org.elasticsearch.transport.NodeDisconnectedException: [master-node-1][10.115.0.21:9300][internal:cluster/coordination/join] disconnected
[2022-08-21T12:10:52,002][INFO ][o.e.c.c.JoinHelper ] [data-node-3] failed to join {master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false} with JoinRequest{sourceNode={data-node-3}{KG1DQKuASBq2GpwYCz4pkQ}{KmQunMToQcS5Stxb5IT7MA}{10.115.0.141}{10.115.0.141:9300}{cdhlrstw}{ml.machine_memory=8332750848, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, minimumTerm=168, optionalJoin=Optional.empty}
org.elasticsearch.transport.NodeDisconnectedException: [master-node-1][10.115.0.21:9300][internal:cluster/coordination/join] disconnected
[2022-08-21T12:10:52,073][INFO ][o.e.c.c.Coordinator ] [data-node-3] master node [{master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false}] failed, restarting discovery
org.elasticsearch.transport.ConnectTransportException: [master-node-1][10.115.0.21:9300] disconnected during check
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:279) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:605) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:248) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.LeaderChecker.updateLeader(LeaderChecker.java:158) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:635) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.Coordinator.handlePublishRequest(Coordinator.java:342) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.acceptState(PublicationTransportHandler.java:230) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:210) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$new$0(PublicationTransportHandler.java:102) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:224) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:401) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:205) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:129) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:120) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:305) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.10.2.jar:7.10.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: org.elasticsearch.transport.NodeNotConnectedException: [master-node-1][10.115.0.21:9300] Node not connected
at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:189) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:682) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:602) ~[elasticsearch-7.10.2.jar:7.10.2]
... 31 more
[2022-08-21T12:10:52,100][INFO ][o.e.n.Node ] [data-node-3] stopped
[2022-08-21T12:10:52,101][INFO ][o.e.n.Node ] [data-node-3] closing ...
[2022-08-21T12:10:52,161][INFO ][o.e.n.Node ] [data-node-3] closed
[2022-08-21T13:25:16,191][INFO ][o.e.n.Node ] [data-node-3] version[7.10.2], pid[1587], build[default/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.15.0-1016-gcp/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
I got error like below in both servers. I setup 2 server for running elasticsearch. Config file also attached as below. I am using ubuntu 18.04.
I got error saying failed to bind transport port 9093. I changed default value. I there any other i need to change ?
I am using open jdk version 8 - 181 version and elasticsearch version 6.4.3
[2018-11-11T12:38:21,155][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-11-11T12:38:21,157][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
[2018-11-11T12:38:21,158][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 16777216, hard limit: 16777216
[2018-11-11T12:38:21,158][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
[2018-11-11T12:38:21,158][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2018-11-11T12:38:21,304][INFO ][o.e.n.Node ] [linux-1] initializing ...
[2018-11-11T12:38:21,401][INFO ][o.e.e.NodeEnvironment ] [linux-1] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [24.6gb], net total_space [28.9gb], types [ext4]
[2018-11-11T12:38:21,401][INFO ][o.e.e.NodeEnvironment ] [linux-1] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-11-11T12:38:21,402][INFO ][o.e.n.Node ] [linux-1] node name [linux-1], node ID [h03oGLGESzqeHmeNJLl0LQ]
[2018-11-11T12:38:21,402][INFO ][o.e.n.Node ] [linux-1] version[6.4.3], pid[11401], build[oss/deb/fe40335/2018-10-30T23:17:19.084789Z], OS[Linux/4.15.0-1030-azure/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-11-11T12:38:21,403][INFO ][o.e.n.Node ] [linux-1] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb]
[2018-11-11T12:38:22,242][INFO ][o.e.p.PluginsService ] [linux-1] loaded module [aggs-matrix-stats]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService ] [linux-1] loaded module [analysis-common]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService ] [linux-1] loaded module [lang-expression]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService ] [linux-1] loaded module [rank-eval]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService ] [linux-1] loaded module [reindex]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService ] [linux-1] loaded module [repository-url]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService ] [linux-1] loaded module [transport-netty4]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService ] [linux-1] loaded module [tribe]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService ] [linux-1] no plugins loaded
[2018-11-11T12:38:25,636][INFO ][o.e.d.DiscoveryModule ] [linux-1] using discovery type [zen]
[2018-11-11T12:38:26,235][INFO ][o.e.n.Node ] [linux-1] initialized
[2018-11-11T12:38:26,235][INFO ][o.e.n.Node ] [linux-1] starting ...
[2018-11-11T12:38:26,561][ERROR][o.e.b.Bootstrap ] [linux-1] Exception
org.elasticsearch.transport.BindTransportException: Failed to bind to [9093]
at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:821) ~[elasticsearch-6.4.3.jar:6.4.3]
at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:786) ~[elasticsearch-6.4.3.jar:6.4.3]
at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:134) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:66)
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
[2018-11-11T12:38:26,581][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [linux-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: BindTransportException[Failed to bind to [9093]]; nested: BindException[Cannot assign requested address];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.3.jar:6.4.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.3.jar:6.4.3]
Caused by: org.elasticsearch.transport.BindTransportException: Failed to bind to [9093]
at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:821) ~[elasticsearch-6.4.3.jar:6.4.3]
at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:786) ~[elasticsearch-6.4.3.jar:6.4.3]
... 6 more
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2018-11-11T12:38:27,284][INFO ][o.e.n.Node ] [linux-1] stopping ...
[2018-11-11T12:38:27,288][INFO ][o.e.n.Node ] [linux-1] stopped
[2018-11-11T12:38:27,288][INFO ][o.e.n.Node ] [linux-1] closing ...
[2018-11-11T12:38:27,321][INFO ][o.e.n.Node ] [linux-1] closed
My config file for servers as below
Linux-1 server
cluster.name: linux-elk
node.name: linux-1
node.data: true
node.ingest: true
node.master: false
node.max_local_storage_nodes: 2
indices.query.bool.max_clause_count: 10000
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 24.95.245.313
http.port: 9092
transport.tcp.port: 9093
discovery.zen.ping.unicast.hosts:
- 178.51.190.47
discovery.zen.minimum_master_nodes: 1
Linux-2 server
cluster.name: linux-elk
node.name: linux-2
node.data: true
node.ingest: false
node.master: true
node.max_local_storage_nodes: 2
indices.query.bool.max_clause_count: 10000
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 178.51.190.47
http.port: 9092
transport.tcp.port: 9093
discovery.zen.ping.unicast.hosts:
- 24.95.245.313
discovery.zen.minimum_master_nodes: 1
plese help ? what i need to do now ?
Another application may be bind to port 9093. need to check it first.
I've installed ElasticSearch 5.3 on my Windows 10 machine recently and after following the installation process, ES does not respond when I access http://localhost:9200 and from Chrome's Sense.
However, when I use curl 'http://localhost:9200' on the command line, it sends this:
{
"name" : "test",
"cluster_name" : "my-application",
"cluster_uuid" : "9-wvh6UXTHSKmkdpblqyyA",
"version" : {
"number" : "5.3.0",
"build_hash" : "3adb13b",
"build_date" : "2017-03-23T03:31:50.652Z",
"build_snapshot" : false,
"lucene_version" : "6.4.1"
},
"tagline" : "You Know, for Search"
}
which indicates that the ES is running/configured correctly(?).
I also configured network.host: to 0.0.0.0/127.0.0.1/:: of elasticsearch.yml still no effect.
Here is the latest log info:
[2017-04-07T18:21:06,481][INFO ][o.e.n.Node ] [test] initializing ...
[2017-04-07T18:21:06,633][INFO ][o.e.e.NodeEnvironment ] [test] using [1] data paths, mounts [[OSDisk (C:)]], net usable_space [129.2gb], net total_space [232.3gb], spins? [unknown], types [NTFS]
[2017-04-07T18:21:06,634][INFO ][o.e.e.NodeEnvironment ] [test] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-04-07T18:21:06,681][INFO ][o.e.n.Node ] [test] node name [test], node ID [Ab5g3zN0S7qNOuP_asF-iQ]
[2017-04-07T18:21:06,682][INFO ][o.e.n.Node ] [test] version[5.3.0], pid[312], build[3adb13b/2017-03-23T03:31:50.652Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_121/25.121-b13]
[2017-04-07T18:21:08,288][INFO ][o.e.p.PluginsService ] [test] loaded module [aggs-matrix-stats]
[2017-04-07T18:21:08,289][INFO ][o.e.p.PluginsService ] [test] loaded module [ingest-common]
[2017-04-07T18:21:08,291][INFO ][o.e.p.PluginsService ] [test] loaded module [lang-expression]
[2017-04-07T18:21:08,291][INFO ][o.e.p.PluginsService ] [test] loaded module [lang-groovy]
[2017-04-07T18:21:08,292][INFO ][o.e.p.PluginsService ] [test] loaded module [lang-mustache]
[2017-04-07T18:21:08,294][INFO ][o.e.p.PluginsService ] [test] loaded module [lang-painless]
[2017-04-07T18:21:08,295][INFO ][o.e.p.PluginsService ] [test] loaded module [percolator]
[2017-04-07T18:21:08,296][INFO ][o.e.p.PluginsService ] [test] loaded module [reindex]
[2017-04-07T18:21:08,296][INFO ][o.e.p.PluginsService ] [test] loaded module [transport-netty3]
[2017-04-07T18:21:08,298][INFO ][o.e.p.PluginsService ] [test] loaded module [transport-netty4]
[2017-04-07T18:21:08,299][INFO ][o.e.p.PluginsService ] [test] loaded plugin [ltr-query]
[2017-04-07T18:21:11,330][INFO ][o.e.n.Node ] [test] initialized
[2017-04-07T18:21:11,330][INFO ][o.e.n.Node ] [test] starting ...
[2017-04-07T18:21:11,612][INFO ][o.e.t.TransportService ] [test] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2017-04-07T18:21:14,657][INFO ][o.e.c.s.ClusterService ] [test] new_master {test}{Ab5g3zN0S7qNOuP_asF-iQ}{XYHTUZ8AQN6kYrEPLChFNg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-04-07T18:21:14,826][INFO ][o.e.h.n.Netty4HttpServerTransport] [test] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2017-04-07T18:21:14,826][INFO ][o.e.n.Node ] [test] started
[2017-04-07T18:21:14,982][INFO ][o.e.g.GatewayService ] [test] recovered [1] indices into cluster_state
[2017-04-07T18:21:15,510][INFO ][o.e.c.r.a.AllocationService] [test] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[tmdb][0]] ...]).
Got it! Referring from Network Settings Documentation, I tried to run it using this script: elasticsearch.bat -E network.host=_local_, and it worked!
I also changed the configuration file(./config/elasticsearch.yml). Just uncomment the line and change it to network.host: _local_. After that, I simply ran elasticsearch.bat directly.
OR
to put it simply just comment the network.host line.
i want to run Elasticsearch on two servers in the same cluster.
The Problem is : I can't connect the 2 servers on elasticsearch. I mean i can't get the 2 nodes on the same cluster.
can somebody tell me if my Configuration in the elasticsearch.yml is right and
Sever 1:
cluster.name: MyData
node.name: Node_1
network.host: '192.160.122.4'
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.160.122.4","192.160.122.3"]
Server2:
cluster.name: MyData
node.name: Node_2
network.host: '192.160.122.3'
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.160.122.4","192.160.122.3"]
what i need to change it ?
Thanks
to the Logs:
Log of server 1:
[2017-03-21T10:38:18,859][INFO ][o.e.n.Node ] [Node_1]
initialized
[2017-03-21T10:38:18,906][INFO ][o.e.n.Node ] [Node_1]
starting ...
[2017-03-21T10:38:19,764][INFO ][o.e.t.TransportService ] [Node_1]
publish_address {192.160.122.4:9300}, bound_addresses {192.160.122.4:9300}
[2017-03-21T10:38:19,764][INFO ][o.e.b.BootstrapChecks ] [Node_1] bound
or publishing to a non-loopback or non-link-local address, enforcing
bootstrap checks
[2017-03-21T10:38:22,899][INFO ][o.e.c.s.ClusterService ] [Node_1]
new_master {Master-Node_1}{4ZEftg6TRCOJqE0kEv-Mrg}{MYrWUFABQ5OOT0US73j13w}
{192.160.122.4}{192.160.122.4:9300}, reason: zen-disco-elected-as-master
([0]
nodes joined)
[2017-03-21T10:38:22,950][INFO ][o.e.h.HttpServer ] [Node_1]
publish_address {192.160.122.4:9200}, bound_addresses {192.160.122.4:9200}
[2017-03-21T10:38:22,997][INFO ][o.e.n.Node ] [Node_1] started
[2017-03-21T10:38:23,948][INFO ][o.e.g.GatewayService ] [Node_1]
recovered [5] indices into cluster_state
[2017-03-21T10:38:32,357][INFO ][o.e.c.r.a.AllocationService] [Node_1]
Cluster health status changed from [RED] to [YELLOW] (reason: [shards
started [[node_1][4]] ...]).
Log of Server2:
[2017-03-21T11:55:57,277][INFO ][o.e.n.Node ] [Node_2] initialized
[2017-03-21T11:55:57,293][INFO ][o.e.n.Node ] [Node_2] starting ...
[2017-03-21T11:56:01,099][INFO ][o.e.t.TransportService ] [Node_2] publish_address {192.160.122.3:9300}, bound_addresses {192.160.122.3:9300}
[2017-03-21T11:56:01,115][INFO ][o.e.b.BootstrapChecks ] [Node_2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-03-21T11:56:01,115][ERROR][o.e.b.Bootstrap ] [Node_2] node validation exception
bootstrap checks failed
JVM is using the client VM [Java HotSpot(TM) Client VM] but should be using a server VM for the best performance
[2017-03-21T11:56:01,146][INFO ][o.e.n.Node ] [Node_2] stopping ...
[2017-03-21T11:56:01,193][INFO ][o.e.n.Node ] [Node_2] stopped
[2017-03-21T11:56:01,193][INFO ][o.e.n.Node ] [Node_2] closing ...
[2017-03-21T11:56:01,240][INFO ][o.e.n.Node ] [Node_2] closed
Before x-pack plugin installed，My ElasticSearch is health；
After x-pack plugin installed，some security error occured，
so,I'm setting
xpack.security.enabled: false
in my elasticsearch.yml and kibana.yml, and then restarting elasticsearch successed.
But sometime the following errors occured.
deMacBook-Pro:bin yyq$ ./elasticsearch
[2017-02-16T11:44:15,349][INFO ][o.e.n.Node ] [yml-es-node-1] initializing ...
[2017-02-16T11:44:15,542][INFO ][o.e.e.NodeEnvironment ] [yml-es-node-1] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [9.1gb], net total_space [232.6gb], spins? [unknown], types [hfs]
[2017-02-16T11:44:15,543][INFO ][o.e.e.NodeEnvironment ] [yml-es-node-1] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-02-16T11:44:15,665][INFO ][o.e.n.Node ] [yml-es-node-1] node name [yml-es-node-1], node ID [hqVAcDOdRu-LXuAgUEX9oQ]
[2017-02-16T11:44:15,670][INFO ][o.e.n.Node ] [yml-es-node-1] version[5.2.0], pid[18949], build[24e05b9/2017-01-24T19:52:35.800Z], OS[Mac OS X/10.11.6/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_25/25.25-b02]
[2017-02-16T11:44:21,007][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [aggs-matrix-stats]
[2017-02-16T11:44:21,008][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [ingest-common]
[2017-02-16T11:44:21,008][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [lang-expression]
[2017-02-16T11:44:21,008][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [lang-groovy]
[2017-02-16T11:44:21,008][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [lang-mustache]
[2017-02-16T11:44:21,008][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [lang-painless]
[2017-02-16T11:44:21,008][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [percolator]
[2017-02-16T11:44:21,009][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [reindex]
[2017-02-16T11:44:21,009][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [transport-netty3]
[2017-02-16T11:44:21,009][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [transport-netty4]
[2017-02-16T11:44:21,010][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded plugin [x-pack]
[2017-02-16T11:44:28,308][INFO ][o.e.n.Node ] [yml-es-node-1] initialized
[2017-02-16T11:44:28,309][INFO ][o.e.n.Node ] [yml-es-node-1] starting ...
[2017-02-16T11:44:28,688][INFO ][o.e.t.TransportService ] [yml-es-node-1] publish_address {127.0.0.1:9301}, bound_addresses {[fe80::1]:9301}, {[::1]:9301}, {127.0.0.1:9301}
[2017-02-16T11:44:31,856][INFO ][o.e.c.s.ClusterService ] [yml-es-node-1] detected_master {yml-es-node-2}{5flQkMYSR-ab1dgNA6coog}{fdX0iBHhSzG3Mrri_5ooNA}{127.0.0.1}{127.0.0.1:9300}, added {{yml-es-node-2}{5flQkMYSR-ab1dgNA6coog}{fdX0iBHhSzG3Mrri_5ooNA}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [master {yml-es-node-2}{5flQkMYSR-ab1dgNA6coog}{fdX0iBHhSzG3Mrri_5ooNA}{127.0.0.1}{127.0.0.1:9300} committed version [1]])
[2017-02-16T11:44:31,881][INFO ][o.e.h.HttpServer ] [yml-es-node-1] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200}
[2017-02-16T11:44:31,882][INFO ][o.e.n.Node ] [yml-es-node-1] started
[2017-02-16T11:44:32,853][INFO ][o.e.l.LicenseService ] [yml-es-node-1] license [854880ad-c73b-4bdc-8813-918c125d442b] mode [trial] - valid
[2017-02-16T11:47:32,980][INFO ][o.e.m.j.JvmGcMonitorService] [yml-es-node-1] [gc][184] overhead, spent [261ms] collecting in the last [1s]
[2017-02-16T11:48:37,797][DEBUG][o.e.a.s.TransportSearchAction] [yml-es-node-1] All shards failed for phase: [query]
org.elasticsearch.action.NoShardAvailableActionException: null
at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:115) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:154) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:53) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:145) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:87) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.search.TransportMultiSearchAction.executeSearch(TransportMultiSearchAction.java:113) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.search.TransportMultiSearchAction.doExecute(TransportMultiSearchAction.java:88) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.search.TransportMultiSearchAction.doExecute(TransportMultiSearchAction.java:41) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:145) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:87) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:75) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:64) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.support.AbstractClient.multiSearch(AbstractClient.java:560) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.rest.action.search.RestMultiSearchAction.lambda$prepareRequest$0(RestMultiSearchAction.java:78) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.rest.action.search.RestMultiSearchAction$$Lambda$1847/1830071645.accept(Unknown Source) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:82) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:162) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.http.HttpServer.dispatchRequest(HttpServer.java:115) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:515) [transport-netty4-5.2.0.jar:5.2.0]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:70) [transport-netty4-5.2.0.jar:5.2.0]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:66) [transport-netty4-5.2.0.jar:5.2.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at org.elasticsearch.http.netty4.cors.Netty4CorsHandler.channelRead(Netty4CorsHandler.java:76) [transport-netty4-5.2.0.jar:5.2.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
anyone can help me?
Delete data directory and restarting elasticsearch,kibana,elasticsearch-head,show some error messages.
deMacBook-Pro:bin yyq$ ./elasticsearch
[2017-02-17T10:56:37,391][INFO ][o.e.n.Node ] [yml-es-node-1] initializing ...
[2017-02-17T10:56:37,875][INFO ][o.e.e.NodeEnvironment ] [yml-es-node-1] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [4.6gb], net total_space [232.6gb], spins? [unknown], types [hfs]
[2017-02-17T10:56:37,875][INFO ][o.e.e.NodeEnvironment ] [yml-es-node-1] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-02-17T10:56:37,879][INFO ][o.e.n.Node ] [yml-es-node-1] node name [yml-es-node-1], node ID [nIE4QoyTQ9yh8Nz_vidq5A]
[2017-02-17T10:56:37,888][INFO ][o.e.n.Node ] [yml-es-node-1] version[5.2.0], pid[45986], build[24e05b9/2017-01-24T19:52:35.800Z], OS[Mac OS X/10.11.6/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_25/25.25-b02]
[2017-02-17T10:56:42,151][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [aggs-matrix-stats]
[2017-02-17T10:56:42,160][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [ingest-common]
[2017-02-17T10:56:42,161][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [lang-expression]
[2017-02-17T10:56:42,161][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [lang-groovy]
[2017-02-17T10:56:42,162][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [lang-mustache]
[2017-02-17T10:56:42,162][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [lang-painless]
[2017-02-17T10:56:42,162][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [percolator]
[2017-02-17T10:56:42,164][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [reindex]
[2017-02-17T10:56:42,165][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [transport-netty3]
[2017-02-17T10:56:42,166][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded module [transport-netty4]
[2017-02-17T10:56:42,167][INFO ][o.e.p.PluginsService ] [yml-es-node-1] loaded plugin [x-pack]
[2017-02-17T10:56:49,011][INFO ][o.e.n.Node ] [yml-es-node-1] initialized
[2017-02-17T10:56:49,012][INFO ][o.e.n.Node ] [yml-es-node-1] starting ...
[2017-02-17T10:56:49,531][INFO ][o.e.t.TransportService ] [yml-es-node-1] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}
[2017-02-17T10:56:55,760][INFO ][o.e.c.s.ClusterService ] [yml-es-node-1] detected_master {yml-es-node-2}{0got1rmqQX-HcD9Wp3uEUA}{2C-_m1pkQv-YKzAPs3v5JA}{127.0.0.1}{127.0.0.1:9301}, added {{yml-es-node-2}{0got1rmqQX-HcD9Wp3uEUA}{2C-_m1pkQv-YKzAPs3v5JA}{127.0.0.1}{127.0.0.1:9301},}, reason: zen-disco-receive(from master [master {yml-es-node-2}{0got1rmqQX-HcD9Wp3uEUA}{2C-_m1pkQv-YKzAPs3v5JA}{127.0.0.1}{127.0.0.1:9301} committed version [1]])
[2017-02-17T10:56:55,797][INFO ][o.e.h.HttpServer ] [yml-es-node-1] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200}
[2017-02-17T10:56:55,797][INFO ][o.e.n.Node ] [yml-es-node-1] started
[2017-02-17T10:56:56,701][INFO ][o.e.l.LicenseService ] [yml-es-node-1] license [1e32f145-313f-4129-b3f2-666bbf279cc5] mode [trial] - valid
------------------------------------------
deMacBook-Pro:bin yyq$ ./kibana
log [02:57:14.219] [info][status][plugin:kibana#5.2.0] Status changed from uninitialized to green - Ready
log [02:57:14.302] [info][status][plugin:elasticsearch#5.2.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [02:57:14.327] [info][status][plugin:xpack_main#5.2.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [02:57:14.545] [info][status][plugin:graph#5.2.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [02:57:14.554] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml
log [02:57:14.558] [info][status][plugin:reporting#5.2.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [02:57:14.594] [info][status][plugin:searchprofiler#5.2.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [02:57:14.603] [info][status][plugin:tilemap#5.2.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [02:57:14.613] [info][status][plugin:console#5.2.0] Status changed from uninitialized to green - Ready
log [02:57:14.853] [info][status][plugin:timelion#5.2.0] Status changed from uninitialized to green - Ready
log [02:57:14.859] [info][listening] Server running at http://localhost:5601
log [02:57:14.860] [info][status][ui settings] Status changed from uninitialized to yellow - Elasticsearch plugin is yellow
log [02:57:19.563] [info][status][plugin:xpack_main#5.2.0] Status changed from yellow to yellow - No existing Kibana index found
log [02:57:19.565] [info][status][plugin:graph#5.2.0] Status changed from yellow to yellow - No existing Kibana index found
log [02:57:19.565] [info][status][plugin:reporting#5.2.0] Status changed from yellow to yellow - No existing Kibana index found
log [02:57:19.567] [info][status][plugin:searchprofiler#5.2.0] Status changed from yellow to yellow - No existing Kibana index found
log [02:57:19.568] [info][status][plugin:tilemap#5.2.0] Status changed from yellow to yellow - No existing Kibana index found
log [02:57:19.568] [info][status][plugin:elasticsearch#5.2.0] Status changed from yellow to yellow - No existing Kibana index found
log [02:57:49.583] [error][status][plugin:xpack_main#5.2.0] Status changed from yellow to red - Unable to create Kibana index ".kibana"
log [02:57:49.584] [error][status][plugin:graph#5.2.0] Status changed from yellow to red - Unable to create Kibana index ".kibana"
log [02:57:49.584] [error][status][plugin:reporting#5.2.0] Status changed from yellow to red - Unable to create Kibana index ".kibana"
log [02:57:49.587] [error][status][plugin:searchprofiler#5.2.0] Status changed from yellow to red - Unable to create Kibana index ".kibana"
log [02:57:49.590] [error][status][plugin:tilemap#5.2.0] Status changed from yellow to red - Unable to create Kibana index ".kibana"
log [02:57:49.593] [error][status][plugin:elasticsearch#5.2.0] Status changed from yellow to red - Unable to create Kibana index ".kibana"
log [02:57:49.594] [error][status][ui settings] Status changed from yellow to red - Elasticsearch plugin is red
log [02:57:52.120] [error][status][plugin:xpack_main#5.2.0] Status changed from red to red - Elasticsearch is still initializing the kibana index.
log [02:57:52.120] [error][status][plugin:graph#5.2.0] Status changed from red to red - Elasticsearch is still initializing the kibana index.
log [02:57:52.121] [error][status][plugin:reporting#5.2.0] Status changed from red to red - Elasticsearch is still initializing the kibana index.
log [02:57:52.122] [error][status][plugin:searchprofiler#5.2.0] Status changed from red to red - Elasticsearch is still initializing the kibana index.
log [02:57:52.123] [error][status][plugin:tilemap#5.2.0] Status changed from red to red - Elasticsearch is still initializing the kibana index.
log [02:57:52.124] [error][status][plugin:elasticsearch#5.2.0] Status changed from red to red - Elasticsearch is still initializing the kibana index.
log [03:00:45.436] [warning][process] UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 3): Error: reply interface called twice
at emitPendingUnhandledRejections (internal/process/promises.js:57:27)
at runMicrotasksCallback (internal/process/next_tick.js:61:9)
at _combinedTickCallback (internal/process/next_tick.js:67:7)
at process._tickDomainCallback (internal/process/next_tick.js:122:9)
error [03:00:45.465] TypeError: Cannot read property 'toJSON' of undefined
at withXpackInfo (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/plugins/x-pack/plugins/xpack_main/server/lib/replace_injected_vars.js:5:32)
at replaceInjectedVars$ (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/plugins/x-pack/plugins/xpack_main/server/lib/replace_injected_vars.js:10:12)
at tryCatch (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/regenerator/runtime.js:61:40)
at GeneratorFunctionPrototype.invoke [as _invoke] (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/regenerator/runtime.js:328:22)
at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/regenerator/runtime.js:94:21)
at invoke (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/regenerator/runtime.js:136:37)
at callInvokeWithMethodAndArg (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/regenerator/runtime.js:172:16)
at previousPromise (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/regenerator/runtime.js:194:19)
at AsyncIterator.enqueue (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/regenerator/runtime.js:193:13)
at AsyncIterator.prototype.(anonymous function) [as next] (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/regenerator/runtime.js:94:21)
at Object.runtime.async (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/regenerator/runtime.js:215:14)
at replaceInjectedVars (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/plugins/x-pack/plugins/xpack_main/server/lib/replace_injected_vars.js:3:22)
at /Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/src/ui/index.js:67:22
at undefined.next (native)
at step (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/src/ui/index.js:9:273)
at /Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/src/ui/index.js:9:443
at /Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/src/ui/index.js:9:99
at tryCatcher (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/bluebird/js/main/util.js:26:23)
at ReductionPromiseArray._promiseFulfilled (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/bluebird/js/main/reduce.js:109:18)
at ReductionPromiseArray.init (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/bluebird/js/main/promise_array.js:92:18)
at ReductionPromiseArray.init (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/bluebird/js/main/reduce.js:42:10)
at Async._drainQueue (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/bluebird/js/main/async.js:128:12)
at Async._drainQueues (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/bluebird/js/main/async.js:133:10)
at Immediate.Async.drainQueues (/Users/yyq/Documents/es/kibana-5.2.0-darwin-x86_64/node_modules/bluebird/js/main/async.js:15:14)
-------------------------------------
deMacBook-Pro:elasticsearch-head-master yyq$ grunt server
Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100