Unable to Start ElasticSearch with Public IP instead of localhost - elasticsearch

I am trying to start elastic search with private ip address but it does not get started it shows some errors in error log which i have shared below.
elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#Also Tried with Private IP Address network.host: 52.50.122.93
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["52.50.122.93", "127.0.0.1", "[::1]"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
elasticsearch.log
[2019-05-21T17:22:28,068][ERROR][o.e.b.Bootstrap ] [WIN-CQKBIA6F350] Exception
java.lang.IllegalStateException: failed to obtain node locks, tried [[C:\ELKStack\elasticsearch-7.1.0\data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:297) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.node.Node.<init>(Node.java:272) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.1.0.jar:7.1.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.1.0.jar:7.1.0]
[2019-05-21T17:22:28,085][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [WIN-CQKBIA6F350] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[C:\ELKStack\elasticsearch-7.1.0\data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.0.jar:7.1.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.0.jar:7.1.0]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[C:\ELKStack\elasticsearch-7.1.0\data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:297) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.node.Node.<init>(Node.java:272) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]
... 6 more

you need to set one of these values
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
ex discovery.seed_hosts:
- 192.168.1.10:9300
- 192.168.1.11
- seeds.mydomain.com

The error is clear in the log file
the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
You have to set the cluster.initial_master_node or discovery.seed_hosts setting
Also don't forget to set the node.name and the cluster.name, you can also start ES and set the master node with this command-line:
bin/elasticsearch -Ecluster.initial_master_nodes=master-a,master-b,master-c
https://www.elastic.co/guide/en/elasticsearch/reference/master/important-settings.html
https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-discovery-bootstrap-cluster.html
https://www.elastic.co/guide/en/elasticsearch/reference/master/discovery-settings.html

Related

Elastic search 7.14 with Centos 7 in Linode Server

I have installed Elastic search7.14 with the help of this article
The elasticsearch.yml file is below. What I believe the problem is in the configuration. I want to use elastic search with magento 2.4.3 and the recomended versioun for magento 2.4.3 is Elasticsearch 7.10
My server is linode with centos 7 and 4core cpu and 8gb of ram. I have also changed the /jvm.options with Xms and Xmx with 1024mb
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
# cluster.name: myCluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
# node.name: My First Node
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#node.data : true
#node.roles: [ master ]
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
bootstrap.system_call_filter: false
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: localhost
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
discover.seed_hosts: []
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#cluster.initial_master_nodes : []
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
But I'm unable to start my service it says
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2022-06-02 21:58:16 IST; 6s ago
Docs: https://www.elastic.co
Process: 20876 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 20876 (code=exited, status=1/FAILURE)
Further debugging with journalctl -xe says
-- Unit user-1001.slice has finished shutting down.
Jun 02 21:59:01 li2147-225.members.linode.com dovecot[13248]: lmtp(21106): Connect from local
Jun 02 21:59:01 li2147-225.members.linode.com dovecot[13248]: lmtp(21106): Disconnect from local: Logged out (state=GREETING)
Jun 02 21:59:01 li2147-225.members.linode.com dovecot[13248]: imap-login: Login: user=<__cpanel__service__auth__imap__q6pfahcr7hjsl
Jun 02 21:59:01 li2147-225.members.linode.com dovecot[13248]: imap(__cpanel__service__auth__imap__q6pfahcr7hjslimd)<21120><uNTkgHng
Jun 02 21:59:01 li2147-225.members.linode.com pure-ftpd[21128]: (?#127.0.0.1) [INFO] New connection from 127.0.0.1
Jun 02 21:59:01 li2147-225.members.linode.com pure-ftpd[21128]: (?#127.0.0.1) [INFO] __cpanel__service__auth__ftpd__tUDT5E9wzfQWUKh
Jun 02 21:59:01 li2147-225.members.linode.com pure-ftpd[21128]: (__cpanel__service__auth__ftpd__tUDT5E9wzfQWUKhS#127.0.0.1) [INFO]
Jun 02 21:59:11 li2147-225.members.linode.com sshd[21168]: Invalid user mama from 165.232.141.0 port 60618
Jun 02 21:59:11 li2147-225.members.linode.com sshd[21168]: input_userauth_request: invalid user mama [preauth]
Jun 02 21:59:12 li2147-225.members.linode.com sshd[21168]: Received disconnect from 165.232.141.0 port 60618:11: Bye Bye [preauth]
Jun 02 21:59:12 li2147-225.members.linode.com sshd[21168]: Disconnected from 165.232.141.0 port 60618 [preauth]
If you want to check the logs then, the last a few lines of this is like
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:23) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:22) ~[?:?]
at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:115) ~[?:?]
at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:126) ~[?:?]
at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
at org.elasticsearch.node.Node.start(Node.java:971) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:313) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:408) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.14.1.jar:7.14.1]
Please give some advice, on how to fix this.
Thanks.

Run filebeat on windows 10

I 'm trying to run filebeat on windows 10 and send to data to elasticsearch and kibana all on localhost. This is my config file filebeat.yml
###################### Filebeat Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log
- D:\AppData\Elastic\filebeat\logs
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: ['.gz$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
### Multiline options
# Multiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
#multiline.pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
#multiline.negate: false
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
#multiline.match: after
# filestream is an input for collecting log messages from files. It is going to replace log input in the future.
- type: filestream
# Change to true to enable this input configuration.
enabled: false
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log
- D:\AppData\Elastic\filebeat\logs
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#prospector.scanner.exclude_files: ['.gz$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
host: "localhost:5601"
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# =============================== Elastic Cloud ================================
# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]
# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#monitoring.enabled: false
# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
# ============================== Instrumentation ===============================
# Instrumentation support for the filebeat.
#instrumentation:
# Set to true to enable instrumentation of filebeat.
#enabled: false
# Environment in which filebeat is running on (eg: staging, production, etc.)
#environment: ""
# APM Server hosts to report instrumentation results to.
#hosts:
# - http://localhost:8200
# API Key for the APM Server(s).
# If api_key is set then secret_token will be ignored.
#api_key:
# Secret token for the APM Server(s).
#secret_token:
# ================================= Migration ==================================
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
i 've run
./filebeat -c filebeat.yml -e
This is the result
2021-09-20T09:55:05.324+0700 INFO instance/beat.go:665 Home path: [D:\AppData\Elastic\filebeat] Config path: [D:\AppData\Elastic\filebeat] Data path: [D:\AppData\Elastic\filebeat\data] Logs path: [D:\AppData\Elastic\filebeat\logs]
2021-09-20T09:55:05.327+0700 INFO instance/beat.go:673 Beat ID: 3957662b-f353-4de0-a6a7-3260cb6481ba
2021-09-20T09:55:05.375+0700 INFO [beat] instance/beat.go:1014 Beat info {"system_info": {"beat": {"path": {"config": "D:\\AppData\\Elastic\\filebeat", "data": "D:\\AppData\\Elastic\\filebeat\\data", "home": "D:\\AppData\\Elastic\\filebeat", "logs": "D:\\AppData\\Elastic\\filebeat\\logs"}, "type": "filebeat", "uuid": "3957662b-f353-4de0-a6a7-3260cb6481ba"}}}
2021-09-20T09:55:05.376+0700 INFO [beat] instance/beat.go:1023 Build info {"system_info": {"build": {"commit": "703d589a09cfdbfd7f84c1d990b50b6b7f62ac29", "libbeat": "7.14.1", "time": "2021-08-26T09:12:57.000Z", "version": "7.14.1"}}}
2021-09-20T09:55:05.376+0700 INFO [beat] instance/beat.go:1026 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":12,"version":"go1.16.6"}}}
2021-09-20T09:55:05.403+0700 INFO [beat] instance/beat.go:1030 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-09-20T08:27:03.12+07:00","name":"nnhai2","ip":["fe80::6036:7939:ebe1:1d3e/64","192.168.82.42/23","fe80::984a:b076:b82f:bedf/64","169.254.190.223/16","fe80::d148:87f2:9bc8:8452/64","169.254.132.82/16","fe80::c4c:5978:a65:9c2a/64","169.254.156.42/16","fe80::488c:c4a3:51de:f987/64","169.254.249.135/16","fe80::e1fb:7ed2:192d:c665/64","169.254.198.101/16","::1/128","127.0.0.1/8","fe80::a42f:ed21:3139:b3a7/64","172.28.96.1/20"],"kernel_version":"10.0.19041.1237 (WinBuild.160101.0800)","mac":["70:b5:e8:5a:d1:0a","ac:82:47:8d:80:2e","ac:82:47:8d:80:2f","ae:82:47:8d:80:2e","00:09:0f:fe:00:01","ac:82:47:8d:80:32","00:15:5d:d4:9f:62"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"19042.1237"},"timezone":"+07","timezone_offset_sec":25200,"id":"85952915-f150-4943-835a-55ae79b7bcb0"}}}
2021-09-20T09:55:05.404+0700 INFO [beat] instance/beat.go:1059 Process info {"system_info": {"process": {"cwd": "D:\\AppData\\Elastic\\filebeat", "exe": "D:\\AppData\\Elastic\\filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 15268, "ppid": 21388, "start_time": "2021-09-20T09:55:04.931+0700"}}}
2021-09-20T09:55:05.405+0700 INFO instance/beat.go:309 Setup Beat: filebeat; Version: 7.14.1
2021-09-20T09:55:05.405+0700 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.14.1' as ILM is enabled.
2021-09-20T09:55:05.405+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-09-20T09:55:05.406+0700 INFO [publisher] pipeline/module.go:113 Beat name: nnhai2
2021-09-20T09:55:05.410+0700 INFO instance/beat.go:473 filebeat start running.
2021-09-20T09:55:05.410+0700 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2021-09-20T09:55:05.418+0700 INFO memlog/store.go:119 Loading data file of 'D:\AppData\Elastic\filebeat\data\registry\filebeat' succeeded. Active transaction id=0
2021-09-20T09:55:05.418+0700 INFO memlog/store.go:124 Finished loading transaction log file for 'D:\AppData\Elastic\filebeat\data\registry\filebeat'. Active transaction id=0
2021-09-20T09:55:05.420+0700 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2021-09-20T09:55:05.420+0700 INFO [crawler] beater/crawler.go:71 Loading Inputs: 2
2021-09-20T09:55:05.420+0700 INFO [input] log/input.go:164 Configured paths: [D:\AppData\Elastic\filebeat\logs] {"input_id": "444806ec-503a-4a80-812f-a8c78e3f69a4"}
2021-09-20T09:55:05.421+0700 INFO [crawler] beater/crawler.go:141 Starting input (ID: 1263043090716372778)
2021-09-20T09:55:05.456+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_deprecation.log c:\ProgramData\Elastic\Elasticsearch\logs\*_deprecation.json] {"input_id": "aa771bd1-e31c-4061-bfe6-2897ff20dde4"}
2021-09-20T09:55:05.456+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\gc.log.* c:\ProgramData\Elastic\Elasticsearch\logs\gc.log] {"input_id": "5c90cdc0-ca6b-4d51-a33c-3ce661ff324b"}
2021-09-20T09:55:05.457+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*.log c:\ProgramData\Elastic\Elasticsearch\logs\*_server.json] {"input_id": "c7ba61e1-c4cb-42e6-8d9e-7acaa5c0d982"}
2021-09-20T09:55:05.457+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_index_search_slowlog.log c:\ProgramData\Elastic\Elasticsearch\logs\*_index_indexing_slowlog.log c:\ProgramData\Elastic\Elasticsearch\logs\*_index_search_slowlog.json c:\ProgramData\Elastic\Elasticsearch\logs\*_index_indexing_slowlog.json] {"input_id": "a83c170f-f55c-4de2-b6e3-969a3686c403"}
2021-09-20T09:55:05.458+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_access.log c:\ProgramData\Elastic\Elasticsearch\logs\*_audit.log c:\ProgramData\Elastic\Elasticsearch\logs\*_audit.json] {"input_id": "0696e6da-a676-4e5b-a550-0165db7b89af"}
2021-09-20T09:55:05.472+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\MySQL\MySQL Server*\error.log*] {"input_id": "cc0b720d-c796-47a2-87e7-7a3244fe8174"}
2021-09-20T09:55:05.472+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\MySQL\MySQL Server*\mysql-slow.log*] {"input_id": "2129affd-14ec-4468-8710-39eccfffb356"}
2021-09-20T09:55:05.487+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\nginx\logs\*access.log*] {"input_id": "44009138-8a88-4b07-8bd2-d04ab9d520bd"}
2021-09-20T09:55:05.487+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\nginx\logs\error.log*] {"input_id": "daa74eb0-538d-45fc-b657-13431186e186"}
2021-09-20T09:55:05.488+0700 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2021-09-20T09:55:05.488+0700 INFO cfgfile/reload.go:164 Config reloader started
2021-09-20T09:55:05.498+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*.log c:\ProgramData\Elastic\Elasticsearch\logs\*_server.json] {"input_id": "77a9d5c1-6f98-414d-b368-dec4f7163ed2"}
2021-09-20T09:55:05.499+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_index_search_slowlog.log c:\ProgramData\Elastic\Elasticsearch\logs\*_index_indexing_slowlog.log c:\ProgramData\Elastic\Elasticsearch\logs\*_index_search_slowlog.json c:\ProgramData\Elastic\Elasticsearch\logs\*_index_indexing_slowlog.json] {"input_id": "beaa87b1-8ce8-4374-bee5-7372846a1968"}
2021-09-20T09:55:05.500+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_access.log c:\ProgramData\Elastic\Elasticsearch\logs\*_audit.log c:\ProgramData\Elastic\Elasticsearch\logs\*_audit.json] {"input_id": "b6c46042-3acf-440e-b788-3dfdaf789c10"}
2021-09-20T09:55:05.500+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_deprecation.log c:\ProgramData\Elastic\Elasticsearch\logs\*_deprecation.json] {"input_id": "d673b872-e9b2-496d-96b5-1e9792169b76"}
2021-09-20T09:55:05.501+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\gc.log.* c:\ProgramData\Elastic\Elasticsearch\logs\gc.log] {"input_id": "5e4465a8-9ffb-42c6-ab6f-c9d269419ed4"}
2021-09-20T09:55:05.501+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-09-20T09:55:08.389+0700 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2021-09-20T09:55:15.302+0700 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.14.1
2021-09-20T09:55:15.454+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-gc-pipeline"}
2021-09-20T09:55:15.874+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-server-pipeline"}
2021-09-20T09:55:15.986+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-server-pipeline-plaintext"}
2021-09-20T09:55:16.108+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-server-pipeline-json"}
2021-09-20T09:55:16.227+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-slowlog-pipeline"}
2021-09-20T09:55:16.335+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-slowlog-pipeline-plaintext"}
2021-09-20T09:55:16.450+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-slowlog-pipeline-json"}
2021-09-20T09:55:16.558+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-audit-pipeline"}
2021-09-20T09:55:16.674+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-audit-pipeline-json"}
2021-09-20T09:55:16.789+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-audit-pipeline-plaintext"}
2021-09-20T09:55:16.898+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-deprecation-pipeline"}
2021-09-20T09:55:17.003+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-deprecation-pipeline-plaintext"}
2021-09-20T09:55:17.111+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-deprecation-pipeline-json"}
2021-09-20T09:55:17.116+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\MySQL\MySQL Server*\error.log*] {"input_id": "0043e24c-4f0f-487a-ab3e-2d2254e613ac"}
2021-09-20T09:55:17.117+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\MySQL\MySQL Server*\mysql-slow.log*] {"input_id": "eddd4b6e-3b7e-4cca-a0b0-e5422c6b7ccf"}
2021-09-20T09:55:17.117+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-09-20T09:55:17.120+0700 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.14.1
2021-09-20T09:55:17.137+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\nginx\logs\*access.log*] {"input_id": "4af5fcde-53a7-4e67-9819-c153919b5f05"}
2021-09-20T09:55:17.137+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\nginx\logs\error.log*] {"input_id": "f4dafaa1-89cf-48d8-927e-eb292c5b186f"}
2021-09-20T09:55:17.137+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-09-20T09:55:17.141+0700 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.14.1
2021-09-20T09:55:17.153+0700 INFO cfgfile/reload.go:224 Loading of config files completed.
2021-09-20T09:55:35.468+0700 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":78,"time":{"ms":93}},"total":{"ticks":390,"time":{"ms":405},"value":390},"user":{"ticks":312,"time":{"ms":312}}},"handles":{"open":306},"info":{"ephemeral_id":"f9fdf685-5d19-459c-a646-fceea8d53c4e","uptime":{"ms":30152},"version":"7.14.1"},"memstats":{"gc_next":19839152,"memory_alloc":12307216,"memory_sys":34381016,"memory_total":62482208,"rss":62062592},"runtime":{"goroutines":78}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":3,"starts":3},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":10,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":12}}}}}
2021-09-20T09:56:05.478+0700 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":109,"time":{"ms":32}},"total":{"ticks":421,"time":{"ms":32},"value":421},"user":{"ticks":312}},"handles":{"open":307},"info":{"ephemeral_id":"f9fdf685-5d19-459c-a646-fceea8d53c4e","uptime":{"ms":60164},"version":"7.14.1"},"memstats":{"gc_next":19839152,"memory_alloc":12714808,"memory_total":62889800,"rss":62160896},"runtime":{"goroutines":78}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":10,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2021-09-20T09:56:35.470+0700 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":46}},"total":{"ticks":452,"time":{"ms":46},"value":452},"user":{"ticks":312}},"handles":{"open":304},"info":{"ephemeral_id":"f9fdf685-5d19-459c-a646-fceea8d53c4e","uptime":{"ms":90157},"version":"7.14.1"},"memstats":{"gc_next":19839152,"memory_alloc":13110648,"memory_total":63285640,"rss":62164992},"runtime":{"goroutines":72}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":10,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
but i have n't seen any data in kibana.
This is the result of GET _cat/indices?v
#! Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.14/security-minimal-setup.html to enable security.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .geoip_databases qeZJyRoKRlW7Xu7EE2ytcw 1 0 42 0 40.8mb 40.8mb
green open .apm-custom-link 5aba-WkIS5yl-Eui7vpfcQ 1 0 0 0 208b 208b
green open .kibana_task_manager_7.14.1_001 1zkRA6c6SXiMcqCtSyrSqQ 1 0 14 344 1.4mb 1.4mb
green open .apm-agent-configuration dppb9LlKQLWTzYBiLZjViA 1 0 0 0 208b 208b
yellow open filebeat-7.14.1-2021.09.19-000001 TpsVGELhRwC-_dxGH7nGKQ 1 1 0 0 208b 208b
green open .async-search KZgq-leNT_qt_dM8TOUQ6A 1 0 0 0 231b 231b
green open .kibana_7.14.1_001 D4UISLPMQlGmCjNgGIrOTw 1 0 2251 11 2.7mb 2.7mb
green open .kibana-event-log-7.14.1-000001 cspR3zh9T1emwvNA131noQ 1 0 3 0 16.4kb 16.4kb
green open .tasks 5dHd_BZpSVilDmVQy6kE7w 1 0 4 0 27.3kb 27.3kb
is it required specific structure log file or i can put any thing in there or where can i get sample log file to test the connection to put in my folder at D:\AppData\Elastic\filebeat\logs ?
Also, where can i find some best practice to config filebeat, i 've read the document at https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html. But it is too simple, many things were not explained like how to config and test modules (we have dozens modules pensando, postgresql, proofpoint, rabbitmq,....)

Add node to running cluster elasticsearch causes master not discovered exception

Problem
I have a running cluster and I would like to add a data node into it. The running cluster is
x.x.x.246
and the data node is
x.x.x.99
each server can see each other by ping.
Machine OS: CentOS7
Elasticsearch: 7.61
configs:
here is elasticsearch.yml of x.x.x.246:
cluster.name: elasticsearch
node.master: true
node.name: Node_master
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: x.x.x.246
http.port: 9200
discovery.seed_hosts: ["x.x.x.99:9300"]
cluster.initial_master_nodes: ["x.x.x.246:9300"]
here is elasticsearch.yml of x.x.x.99
cluster.name: elasticsearch
node.name: Node_master
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: x.x.x.99
http.port: 9200
discovery.seed_hosts: ["x.x.x.245:9300"]
cluster.initial_master_nodes: ["x.x.x.246:9300"]
Testing running elasticsearch on machine
When I run systemctl start elasticsearch on each machine, it works well.
test run on x.x.x.246
curl -X GET "X.X.X.246:9200/_cluster/health?pretty"
show:number of the node not changing
curl -X GET "X.X.X.99:9200/_cluster/health?pretty
show:
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
edited
here is elasticsearch.yml of x.x.x.246:
cluster.name: elasticsearch
node.name: master
node.master: true
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["x.x.x.99","x.x.x.246]
cluster.initial_master_nodes: ["x.x.x.246"]
logger.org.elasticsearch.discovery: TRACE
here is elasticsearch.yml of x.x.x.99
cluster.name: elasticsearch
node.name: node
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["x.x.x.246","x.x.x.99"]
cluster.initial_master_nodes: ["x.x.x.246"]
logger.org.elasticsearch.discovery: TRACE
log on x.x.x.99:
[root#dev ~]# tail -30 /var/log/elasticsearch/elasticsearch.log
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:830) ~[?:?]
[2020-03-19T12:12:04,462][INFO ][o.e.c.c.JoinHelper ] [node-1] failed to join {master}{0UHYehfNQ2-WCadTC_VVkA}{1FNy5AJrTpKOCAejBLKR2w}{10.64.2.246}{10.64.2.246:9300}{dilm}{ml.machine_memory=1907810304, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{a4TYDhG7SWqL3CSG4tusEg}{10.64.2.99}{10.64.2.99:9300}{d}{xpack.installed=true}, optionalJoin=Optional[Join{term=178, lastAcceptedTerm=8, lastAcceptedVersion=100, sourceNode={node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{a4TYDhG7SWqL3CSG4tusEg}{10.64.2.99}{10.64.2.99:9300}{d}{xpack.installed=true}, targetNode={master}{0UHYehfNQ2-WCadTC_VVkA}{1FNy5AJrTpKOCAejBLKR2w}{10.64.2.246}{10.64.2.246:9300}{dilm}{ml.machine_memory=1907810304, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [master][10.64.2.246:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:514) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:244) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.6.1.jar:7.6.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-1][10.64.2.99:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid P4QlwvuRRGSmlT77RroSjA than local cluster uuid oUoIe2-bSbS2UPg722ud9Q, rejecting
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:830) ~[?:?]
For node x.x.x.99 the entry for seed host is wrong. It should be as below:
discovery.seed_hosts: ["x.x.x.246:9300"]
The discovery.seed_hosts list is used to detect the master node, since this list contains the address to the nodes which are master eligible nodes and hold the information of the current master node as well, Since it is pointed to x.x.x.245 instead of x.x.x.246 in the configuration of x.x.x.99, the node x.x.x.99 is unable to detect the master.
Post discussion in comment correct configuration should be:
Master node:
cluster.name: elasticsearch
node.name: master
node.master: true
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["x.x.x.246]
cluster.initial_master_nodes: ["master"]
Note that if you want the above node to be master only and not hold data then set
node.data: false
Data node:
cluster.name: elasticsearch
node.name: data-node-1
node.data: true
node.master: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["x.x.x.246"]
Also since node x.x.x.99 could not join cluster it has stale cluster state. So delete data folder on x.x.x.99 and restart this node.
The reason, why it wasn't able to elect a master, is mention of discovery.seed_hosts: ["x.x.x.245:9300"] which is not part of the current master node config and is not part of master node config as well. as mentioned in this official ES docs it's used to elect a master node.
You should read in details the 2 important configs related to master selection:
discovery.seed_hosts
initial_master_nodes
You can turn DEBUG logging on Discovery module to better understand it, by adding below the line in your elasticsearch.yml
logger.org.elasticsearch.discovery: DEBUG
You can do a few modifications in both the elasticsearch.yml.
node.name has same name in both nodes elasticsearch.yml.
It's better to just mention ip without port 9200.
Better to give network.host: 0.0.0.0 value, instead of node ip in both elasticsearch.yml.
node.data: true is the default, so no need to mention it.
So better and concise version looks like below:
Master node elasticsearch.yml
cluster.name: elasticsearch
node.name: master
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.seed_hosts: ["x.x.x.99", "x.x.x.246"] -->note this
cluster.initial_master_nodes: ["x.x.x.246"] :- note this
Another data node elasticsearch.yml
cluster.name: elasticsearch
node.name: data
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["x.x.x.99", "x.x.x.246"] --> you need to change this and include both nodes
cluster.initial_master_nodes: ["x.x.x.246"]
Verify the master node
You can hit <your-any-node-ip>:9200/_cat/master and this should return the elected master node which would be in your case node with name master. more info on this.
I had also same issue, when I was trying to access elastic search from outside AWS windows server, I was not able to access it, after that I have added
network.host : aws_private_ip
and after that we need to restart elastic service, but it was throwing an error in restarting, and finally when had added below line, it works for me,
cluster.initial_master_nodes: node-1

ElasticSearch xpack.security.enabled: true Error on start

I want to set password to my elasticsearch. I have not paid or started my free-trial so I guess I am using basic plan as default.
I have followed official guide to install elasticsearch on ubuntu EC2.
I don't think I have installed OSS version but when i run:
/usr/share/elasticsearch$ sudo bin/elasticsearch-plugin list --verbose
Plugins directory: /usr/share/elasticsearch/plugins
it does not print xpack.
I tried Removing and Installing ElasticSearch clean just in case i have set something wrong.
Only thing I did to my elasticsearch.yml is adding: xpack.security.enabled:true
However starting elastic search by systemctl start elasticsearch.service outputs this error message:
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2019-11-13 02:53:44 UTC; 9min ago
Docs: http://www.elastic.co
Process: 20330 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 20330 (code=exited, status=1/FAILURE)
Nov 13 02:53:44 ip-172-31-47-37 elasticsearch[20330]: at org.yaml.snakeyaml.scanner.ScannerImpl.needMoreTokens(ScannerImpl.java:280)
Nov 13 02:53:44 ip-172-31-47-37 elasticsearch[20330]: at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:225)
Nov 13 02:53:44 ip-172-31-47-37 elasticsearch[20330]: at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:557)
Nov 13 02:53:44 ip-172-31-47-37 elasticsearch[20330]: at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:157)
Nov 13 02:53:44 ip-172-31-47-37 elasticsearch[20330]: at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:167)
Nov 13 02:53:44 ip-172-31-47-37 elasticsearch[20330]: at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:340)
Nov 13 02:53:44 ip-172-31-47-37 elasticsearch[20330]: ... 13 more
Nov 13 02:53:44 ip-172-31-47-37 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 02:53:44 ip-172-31-47-37 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Nov 13 02:53:44 ip-172-31-47-37 systemd[1]: Failed to start Elasticsearch
Also after I added xpack.security.enabled:true, listing plugin shows this error message:
/usr/share/elasticsearch$ sudo bin/elasticsearch-plugin list --verbose
Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: MarkedYAMLException[while scanning a simple key
in 'reader', line 90, column 1:
xpack.security.enabled:true
^
could not find expected ':'
in 'reader', line 91, column 1:
^
at [Source: sun.nio.ch.ChannelInputStream#6155d082; line: 37, column: 34]]; nested: ScannerException[while scanning a simple key
in 'reader', line 90, column 1:
xpack.security.enabled:true
^
could not find expected ':'
in 'reader', line 91, column 1:
^
];
at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1097)
at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1070)
at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:83)
at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:95)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: while scanning a simple key
in 'reader', line 90, column 1:
xpack.security.enabled:true
^
could not find expected ':'
in 'reader', line 91, column 1:
^
at [Source: sun.nio.ch.ChannelInputStream#6155d082; line: 37, column: 34]
at com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException.from(MarkedYAMLException.java:27)
at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:343)
at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52)
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1093)
... 9 more
Caused by: while scanning a simple key
in 'reader', line 90, column 1:
xpack.security.enabled:true
^
could not find expected ':'
in 'reader', line 91, column 1:
^
at org.yaml.snakeyaml.scanner.ScannerImpl.stalePossibleSimpleKeys(ScannerImpl.java:465)
at org.yaml.snakeyaml.scanner.ScannerImpl.needMoreTokens(ScannerImpl.java:280)
at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:225)
at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:557)
at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:157)
at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:167)
at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:340)
... 14 more
Here's my elasticsearch.yml:
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.enabled:true
What do i need to do to successfully launch elastic search?
Thank you in advance
Try to add a space after the colon for your xpack setting.
xpack.security.enabled: true
Syntax on yaml files can be pretty specific.
https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html
"A dictionary is represented in a simple key: value form (the colon must be followed by a space)"

How to configure multi-node Apache Storm cluster

I'm following http://jayatiatblogs.blogspot.com/2011/11/storm-installation.html & http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup to set up Apache Storm cluster in Ubuntu 14.04 LTS at AWS EC2.
My master node is 10.0.0.185.
My slave nodes are 10.0.0.79, 10.0.0.124 & 10.0.0.84 with myid of 1, 2 and 3 in their zookeeper-data respectively. I set up an ensemble of Apache Zookeeper consists of all the 3 slave nodes.
Below are my zoo.cfg for my slave nodes:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/ubuntu/zookeeper-data
clientPort=2181
server.1=10.0.0.79:2888:3888
server.2=10.0.0.124:2888:3888
server.3=10.0.0.84:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
Below are my storm.yaml for my slave nodes:
########### These MUST be filled in for a storm configuration
storm.zookeeper.server:
- "10.0.0.79"
- "10.0.0.124"
- "10.0.0.84"
# - "localhost"
storm.zookeeper.port: 2181
# nimbus.host: "localhost"
nimbus.host: "10.0.0.185"
storm.local.dir: "/home/ubuntu/storm/data"
java.library.path: "/usr/lib/jvm/java-7-oracle"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704
#
# worker.childopts: "-Xmx768m"
# nimbus.childopts: "-Xmx512m"
# supervisor.childopts: "-Xmx256m"
#
# ##### These may optionally be filled in:
#
## List of custom serializations
# topology.kryo.register:
# - org.mycompany.MyType
# - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
# - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
# - "server1"
# - "server2"
## Metrics Consumers
# topology.metrics.consumer.register:
# - class: "backtype.storm.metric.LoggingMetricsConsumer"
# parallelism.hint: 1
# - class: "org.mycompany.MyMetricsConsumer"
# parallelism.hint: 1
# argument:
# - endpoint: "metrics-collector.mycompany.org"
Below are the storm.yaml for my master node:
########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
- "10.0.0.79"
- "10.0.0.124"
- "10.0.0.84"
# - "localhost"
#
storm.zookeeper.port: 2181
nimbus.host: "10.0.0.185"
# nimbus.thrift.port: 6627
# nimbus.task.launch.secs: 240
# supervisor.worker.start.timeout.secs: 240
# supervisor.worker.timeout.secs: 240
ui.port: 8772
# nimbus.childopts: "‐Xmx1024m ‐Djava.net.preferIPv4Stack=true"
# ui.childopts: "‐Xmx768m ‐Djava.net.preferIPv4Stack=true"
# supervisor.childopts: "‐Djava.net.preferIPv4Stack=true"
# worker.childopts: "‐Xmx768m ‐Djava.net.preferIPv4Stack=true"
storm.local.dir: "/home/ubuntu/storm/data"
java.library.path: "/usr/lib/jvm/java-7-oracle"
# supervisor.slots.ports:
# - 6700
# - 6701
# - 6702
# - 6703
# - 6704
# worker.childopts: "-Xmx768m"
# nimbus.childopts: "-Xmx512m"
# supervisor.childopts: "-Xmx256m"
# ##### These may optionally be filled in:
#
## List of custom serializations
# topology.kryo.register:
# - org.mycompany.MyType
# - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
# - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
# - "server1"
# - "server2"
## Metrics Consumers
# topology.metrics.consumer.register:
# - class: "backtype.storm.metric.LoggingMetricsConsumer"
# parallelism.hint: 1
# - class: "org.mycompany.MyMetricsConsumer"
# parallelism.hint: 1
# argument:
# - endpoint: "metrics-collector.mycompany.org"
I start my zookeeper in all my slave nodes, then start my storm nimbus in my master node, then start storm supervisor in all my slave nodes. However, when I view in my Storm UI, there is only 1 supervisor with total 5 slots in the cluster summary & only 1 supervisor information in the supervisor summary, why so?
How many slave nodes is actually working if I submit a topology in this case?
Why it is not 3 supervisors with total 15 slots?
What should I do in order to have 3 supervisors?
When I check in the supervisor.log in the slave nodes, the causes is as below:
2015-05-29T09:21:24.185+0000 b.s.d.supervisor [INFO] 5019754f-cae1-4000-beb4-fa0
16bd1a43d still hasn't started
What you are doing perfect and its works too.
The only thing you should change is your storm.dir. It is same in the slave and the master nodes just change the path in the storm.dir path in nimbus & supervisor nodes (don't use same local path). When you use same local path the nimbus and supervisor share same id. They come into play but you don’t see 8 slots they just show you 4 slots as workers.
Change the (storm.local.dir:/home/ubuntu/storm/data) and don`t use same path in supervisor and nimbus.
Are you referring to Nimbus as the master node?
Generally, Zookeeper cluster should be started first and then nimbus and then the supervisors. Zookeeper and Nimbus should be always available for the Storm cluster to function correctly.
You should check the supervisor logs to check for the failures. The Nimbus host and the Zookeeper machines should be accessible from to Supervisor machines.

Resources