Elasticsearch node gets disconnected from cluster - elasticsearch

Question

I'm running Elasticsearch version 7.10.2 with 8 nodes of cluster (3 master, 3 data, and 2 client node).
It was running smoothly till now , now the nodes began to disconnect from the cluster one node at a day , when I checked the node Elasticsearch was stopped. I have noticed that node get disconnected randomly
last node that was disconnected was data-node-3 at 12:10 , I want to know the main reason why Elasticsearch getting stopped also please suggest on how to resolve this problem.
Note:-we only restart master-1(Monday) , master-2(Tuesday) and master-3(Wednesday) using cronjob we have not set any cronjob for data and client nodes.
cluster information:-
Name RAM Heap Cpu
client 1 4gb 2gb 2
client 2 4gb 2gb 2
data 1 8gb 4gb 2
data 2 8gb 4gb 2
data 3 8gb 4gb 2
master 1 2gb 1gb 2
master 2 2gb 1gb 2
master 3 2gb 1gb 2
I am attaching log here of datanode-3
[2022-08-21T12:10:00,193][INFO ][o.e.n.Node ] [data-node-3] version[7.10.2], pid[534], build[default/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.15.0-1016-gcp/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-08-21T12:10:00,381][INFO ][o.e.n.Node ] [data-node-3] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-08-21T12:10:00,382][INFO ][o.e.n.Node ] [data-node-3] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms4g, -Xmx4g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-5474094125273943758, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=2147483648, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2022-08-21T12:10:18,654][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [aggs-matrix-stats]
[2022-08-21T12:10:18,656][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [analysis-common]
[2022-08-21T12:10:18,656][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [constant-keyword]
[2022-08-21T12:10:18,660][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [flattened]
[2022-08-21T12:10:18,661][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [frozen-indices]
[2022-08-21T12:10:18,663][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [ingest-common]
[2022-08-21T12:10:18,663][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [ingest-geoip]
[2022-08-21T12:10:18,672][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [ingest-user-agent]
[2022-08-21T12:10:18,673][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [kibana]
[2022-08-21T12:10:18,674][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [lang-expression]
[2022-08-21T12:10:18,674][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [lang-mustache]
[2022-08-21T12:10:18,675][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [lang-painless]
[2022-08-21T12:10:18,675][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [mapper-extras]
[2022-08-21T12:10:18,677][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [mapper-version]
[2022-08-21T12:10:18,679][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [parent-join]
[2022-08-21T12:10:18,680][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [percolator]
[2022-08-21T12:10:18,681][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [rank-eval]
[2022-08-21T12:10:18,682][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [reindex]
[2022-08-21T12:10:18,682][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [repositories-metering-api]
[2022-08-21T12:10:18,684][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [repository-url]
[2022-08-21T12:10:18,685][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [search-business-rules]
[2022-08-21T12:10:18,685][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [searchable-snapshots]
[2022-08-21T12:10:18,697][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [spatial]
[2022-08-21T12:10:18,698][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [systemd]
[2022-08-21T12:10:18,698][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [transform]
[2022-08-21T12:10:18,699][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [transport-netty4]
[2022-08-21T12:10:18,701][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [unsigned-long]
[2022-08-21T12:10:18,702][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [vectors]
[2022-08-21T12:10:18,702][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [wildcard]
[2022-08-21T12:10:18,704][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-analytics]
[2022-08-21T12:10:18,709][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-async]
[2022-08-21T12:10:18,710][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-async-search]
[2022-08-21T12:10:18,710][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-autoscaling]
[2022-08-21T12:10:18,711][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-ccr]
[2022-08-21T12:10:18,712][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-core]
[2022-08-21T12:10:18,713][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-data-streams]
[2022-08-21T12:10:18,714][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-deprecation]
[2022-08-21T12:10:18,714][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-enrich]
[2022-08-21T12:10:18,717][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-eql]
[2022-08-21T12:10:18,719][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-graph]
[2022-08-21T12:10:18,722][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-identity-provider]
[2022-08-21T12:10:18,723][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-ilm]
[2022-08-21T12:10:18,724][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-logstash]
[2022-08-21T12:10:18,724][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-ml]
[2022-08-21T12:10:18,725][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-monitoring]
[2022-08-21T12:10:18,732][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-ql]
[2022-08-21T12:10:18,742][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-rollup]
[2022-08-21T12:10:18,743][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-security]
[2022-08-21T12:10:18,743][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-sql]
[2022-08-21T12:10:18,744][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-stack]
[2022-08-21T12:10:18,744][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-voting-only-node]
[2022-08-21T12:10:18,745][INFO ][o.e.p.PluginsService ] [data-node-3] loaded module [x-pack-watcher]
[2022-08-21T12:10:18,746][INFO ][o.e.p.PluginsService ] [data-node-3] no plugins loaded
[2022-08-21T12:10:19,097][INFO ][o.e.e.NodeEnvironment ] [data-node-3] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [77.9gb], net total_space [96.7gb], types [ext4]
[2022-08-21T12:10:19,098][INFO ][o.e.e.NodeEnvironment ] [data-node-3] heap size [4gb], compressed ordinary object pointers [true]
[2022-08-21T12:10:22,654][INFO ][o.e.n.Node ] [data-node-3] node name [data-node-3], node ID [KG1DQKuASBq2GpwYCz4pkQ], cluster name [la-prod7-cluster], roles [transform, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold]
[2022-08-21T12:10:37,839][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [data-node-3] [controller/974] [Main.cc#114] controller (64 bit): Version 7.10.2 (Build 40a3af639d4698) Copyright (c) 2021 Elasticsearch BV
[2022-08-21T12:10:38,910][INFO ][o.e.x.s.a.s.FileRolesStore] [data-node-3] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2022-08-21T12:10:40,701][INFO ][o.e.t.NettyAllocator ] [data-node-3] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=512kb, suggested_max_allocation_size=512kb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=2mb}]
[2022-08-21T12:10:40,845][INFO ][o.e.d.DiscoveryModule ] [data-node-3] using discovery type [zen] and seed hosts providers [settings]
[2022-08-21T12:10:41,685][WARN ][o.e.g.DanglingIndicesState] [data-node-3] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-08-21T12:10:42,542][INFO ][o.e.n.Node ] [data-node-3] initialized
[2022-08-21T12:10:42,543][INFO ][o.e.n.Node ] [data-node-3] starting ...
[2022-08-21T12:10:43,019][INFO ][o.e.t.TransportService ] [data-node-3] publish_address {10.115.0.141:9300}, bound_addresses {10.115.0.141:9300}
[2022-08-21T12:10:44,459][INFO ][o.e.b.BootstrapChecks ] [data-node-3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-08-21T12:10:44,464][INFO ][o.e.c.c.Coordinator ] [data-node-3] cluster UUID [uSrx3heuSVypgStFinWJrw]
[2022-08-21T12:10:48,038][INFO ][o.e.c.s.ClusterApplierService] [data-node-3] master node changed {previous [], current [{master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false}]}, added {{master-node-2}{t4Pbwly1S0q084vosUjmNw}{m6Rwe3xLSquVPA2Oj0bFxQ}{10.115.0.49}{10.115.0.49:9300}{m}{xpack.installed=true, transform.node=false},{data-node-1}{aesOUIaRTA-4xHPDSZ-ZyA}{x1xAw9K8RGuGCzahKeE3EA}{10.115.0.95}{10.115.0.95:9300}{cdhlrstw}{ml.machine_memory=8340889600, ml.max_open_jobs=20, xpack.installed=true, transform.node=true},{master-node-3}{ka0XuT6zQMOLL_CTqa3ysA}{AmKed0FWR8CnZoNE5JsUAw}{10.115.0.114}{10.115.0.114:9300}{m}{xpack.installed=true, transform.node=false},{master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false},{client-node-2}{UOSU11VmS_GByWchh0jV8w}{X2uXIsMsQPu_34SaOKk5uw}{10.115.0.81}{10.115.0.81:9300}{ilr}{ml.machine_memory=4126511104, ml.max_open_jobs=20, xpack.installed=true, transform.node=false},{data-node-2}{qEU0XgvoSCmgqfBLZIEOLg}{zfVDj9HWT-OlVnhfyv6VSg}{10.115.0.112}{10.115.0.112:9300}{cdhlrstw}{ml.machine_memory=8332742656, ml.max_open_jobs=20, xpack.installed=true, transform.node=true},{client-node-1}{FvYReNu1SF-SX7aT5d3GLQ}{iKJIdSP_RAey-Q9N-OkH2Q}{10.115.0.80}{10.115.0.80:9300}{ilr}{ml.machine_memory=4126502912, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}}, term: 168, version: 15421, reason: ApplyCommitRequest{term=168, version=15421, sourceNode={master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false}}
[2022-08-21T12:10:48,072][INFO ][o.e.n.Node ] [data-node-3] stopping ...
[2022-08-21T12:10:48,077][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [data-node-3] [controller/974] [Main.cc#154] ML controller exiting
[2022-08-21T12:10:48,078][INFO ][o.e.x.m.p.NativeController] [data-node-3] Native controller process has stopped - no new native processes can be started
[2022-08-21T12:10:48,080][INFO ][o.e.x.w.WatcherService ] [data-node-3] stopping watch service, reason [shutdown initiated]
[2022-08-21T12:10:48,081][INFO ][o.e.x.w.WatcherLifeCycleService] [data-node-3] watcher has stopped and shutdown
[2022-08-21T12:10:48,511][INFO ][o.e.h.AbstractHttpServerTransport] [data-node-3] publish_address {10.115.0.141:9200}, bound_addresses {10.115.0.141:9200}
[2022-08-21T12:10:48,511][INFO ][o.e.n.Node ] [data-node-3] started
[2022-08-21T12:10:49,584][INFO ][o.e.c.s.ClusterSettings ] [data-node-3] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2022-08-21T12:10:50,534][INFO ][o.e.x.s.a.TokenService ] [data-node-3] refresh keys
[2022-08-21T12:10:50,880][INFO ][o.e.x.s.a.TokenService ] [data-node-3] refreshed keys
[2022-08-21T12:10:51,977][INFO ][o.e.l.LicenseService ] [data-node-3] license [4d78bbbf-fb01-4d13-8178-86cc235014f1] mode [basic] - valid
[2022-08-21T12:10:51,980][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [data-node-3] Active license is now [BASIC]; Security is enabled
[2022-08-21T12:10:51,995][INFO ][o.e.c.c.Coordinator ] [data-node-3] master node [{master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [master-node-1][10.115.0.21:9300][disconnected] disconnected
[2022-08-21T12:10:52,001][INFO ][o.e.c.c.JoinHelper ] [data-node-3] failed to join {master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false} with JoinRequest{sourceNode={data-node-3}{KG1DQKuASBq2GpwYCz4pkQ}{KmQunMToQcS5Stxb5IT7MA}{10.115.0.141}{10.115.0.141:9300}{cdhlrstw}{ml.machine_memory=8332750848, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, minimumTerm=168, optionalJoin=Optional.empty}
org.elasticsearch.transport.NodeDisconnectedException: [master-node-1][10.115.0.21:9300][internal:cluster/coordination/join] disconnected
[2022-08-21T12:10:52,002][INFO ][o.e.c.c.JoinHelper ] [data-node-3] failed to join {master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false} with JoinRequest{sourceNode={data-node-3}{KG1DQKuASBq2GpwYCz4pkQ}{KmQunMToQcS5Stxb5IT7MA}{10.115.0.141}{10.115.0.141:9300}{cdhlrstw}{ml.machine_memory=8332750848, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, minimumTerm=168, optionalJoin=Optional.empty}
org.elasticsearch.transport.NodeDisconnectedException: [master-node-1][10.115.0.21:9300][internal:cluster/coordination/join] disconnected
[2022-08-21T12:10:52,073][INFO ][o.e.c.c.Coordinator ] [data-node-3] master node [{master-node-1}{Z3sr97BDRFmgi2Mu_6NxXA}{Ge6zQsnLShmFzLCiSE2i2g}{10.115.0.21}{10.115.0.21:9300}{m}{xpack.installed=true, transform.node=false}] failed, restarting discovery
org.elasticsearch.transport.ConnectTransportException: [master-node-1][10.115.0.21:9300] disconnected during check
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:279) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:605) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:248) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.LeaderChecker.updateLeader(LeaderChecker.java:158) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:635) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.Coordinator.handlePublishRequest(Coordinator.java:342) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.acceptState(PublicationTransportHandler.java:230) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:210) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$new$0(PublicationTransportHandler.java:102) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:224) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:401) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:205) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:129) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:120) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:305) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.10.2.jar:7.10.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: org.elasticsearch.transport.NodeNotConnectedException: [master-node-1][10.115.0.21:9300] Node not connected
at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:189) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:682) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:602) ~[elasticsearch-7.10.2.jar:7.10.2]
... 31 more
[2022-08-21T12:10:52,100][INFO ][o.e.n.Node ] [data-node-3] stopped
[2022-08-21T12:10:52,101][INFO ][o.e.n.Node ] [data-node-3] closing ...
[2022-08-21T12:10:52,161][INFO ][o.e.n.Node ] [data-node-3] closed
[2022-08-21T13:25:16,191][INFO ][o.e.n.Node ] [data-node-3] version[7.10.2], pid[1587], build[default/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.15.0-1016-gcp/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]