Exiting: Error in initing prospector: No paths were defined for prospector accessing config - elasticsearch

I am trying to visualize sample data on Kibana using Windows. I followed the link to Security Analytics section to setup Elasticsearch, Kibana and Filebeats.
Link to installation
I have installed Elastic search and Kibana, and have been able to successfully launch both. The description in the link states to configure the filebeat.yml file.
I have configured the filebeat.yml file as follows
###################### Filebeat Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- type: log
# Change to true to enable this prospector configuration.
enabled: false
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: ['.gz$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
### Multiline options
# Mutiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
#multiline.pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
#multiline.negate: false
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
#multiline.match: after
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false
#================================ General =====================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "localhost:5601"
#============================= Elastic Cloud ==================================
# These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
username: "elastic"
password: "n2yHQc8Cp1K2iRrOrNcV"
#----------------------------- Logstash output --------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
#================================ Logging =====================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
#============================== Xpack Monitoring ===============================
# filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:
After running the command ".\filebeat -e -modules=system --setup", filebeat begins, successfully connecting to Elasticsearch and loading Kibana dashboards.
But when I click on the dashboard section on Kibana, the Filebeat process exits with an error message saying "Exiting: Error in initing prospector: No paths were defined for prospector accessing config".
Am I doing something wrong? How can this issue be rectified?
Here is the filtered version of the filebeat config file:
filebeat.prospectors:
- type: log
enabled: false
paths:
- /var/log/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
output.elasticsearch:
hosts: ["localhost:9200"]
username: "elastic"
password: "n2yHQc8Cp1K2iRrOrNcV"

Related

How to configure Name/IndexPattern and amount of DataStreams in filebeat.yml

For my application i would like to have a two Data-Views inside Kibana. I'm using filebeat as data shiper and configure kibana with my filebeat.yml (see below). I've got one Data view with the correct Index-Pattern and name but it's configured with the setup.dashboard.index. If i comment that out or delete it i will get the default filebeat-* name and pattern which doesn't match anything. And espacially since i need to use two different data views inside a dashboard the setup.dashboard.index can't be uses because this is overwriting my default settings with the two different index-pattern for data-views
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
setup.template.name: pac-filebeat-%{[agent.version]}
setup.template.pattern: pac-filebeat-%{[agent.version]}
setup.template.fields: ${path.config}/fields.yml
setup.dashboards.enabled: false
setup.dashboards.directory: ${path.config}\kibana\custom
setup.dashboards.index: pac-filebeat*
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
protocol: "http"
index: pac-filebeat-%{[agent.version]}
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
So i would like to configure two Data-Views in my Project with different names and different Index-Pattern for their corresponding data-Streams.

Elastic index name not getting set correctly

###################### Filebeat Configuration #########################
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/myindex-app/*.log
# matching on this type 2022-07-20 10:56:29,393
multiline:
pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}'
negate: true
match: after
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template:
#name: "myindex-%{[agent.version]}"
#pattern: "myindex-%{[agent.version]}-*"
overwrite: true
settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
#========================== Modules configuration =============================
filebeat.modules:
#-------------------------------- Nginx Module --------------------------------
- module: nginx
# Access logs
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/nginx/access.log"]
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/nginx/error.log"]
# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
# Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
#ingress_controller:
# enabled: false
#
# # Set custom paths for the log files. If left empty,
# # Filebeat will choose the paths depending on your OS.
# #var.paths:
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["${ELASTIC_URL}"]
# Protocol - either `http` (default) or `https`.
protocol: "https"
# Certificate for SSL client authentication
# Client Certificate Key
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
username: ${ELASTIC_USERNAME}
password: ${ELASTIC_PASSWORD}
# %{[fileset.module]}-%{[fileset.name]} to be added as an option - TBC
index: "myindex-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.ilm:
enabled: true
policy_name: "myindex"
overwrite: true
rollover_alias: "myindex-%{[agent.version]}"
#pattern: "{now/d}-0000001"
policy_file: "/usr/share/filebeat/config/myindex.policy.json"
I am trying to get my logs written to an index "myindex" which itself has ilm configured. The above file doesn't error, but neither do the logs go to myindex, they go to filebeat...
I have spent a couple of weeks now trying to get this working, so anything that can be suggested would be really appreciated. I have tried many many combinations with and without many of the commented entries.
What am i doing wrong?

Filebeat 7.5.1 Missing a step for custom index names config?

I should be able to see a new custom index name dev-7.5.1-yyyy.MM.dd within Kibana, but instead it is still called filebeat.
I have the following set it my filebeat.yml config file, which should allow me to see dev-* index in Kibana. However these are still indexing as filebeat-7.5.1-yyyy-MM.dd. Filebeat is running on a ec2 instance that can communicate with the elastic-search server on port 9200.
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.elasticsearch:
hosts: ["my_es_host_ip:9200"]
index: "dev-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.enabled: true
setup.template.name: "dev-%{[agent.version]}"
setup.template.pattern: "dev-%{[agent.version]}-*"
Is there step I'm missing? From reading over the the docs and the reference file, I believe I have everything set correctly.
Appreciate any advice.
Looks like the following setting is needed in his case:
setup.ilm.enabled: false
After adding that to my config, it worked as expected. There is a open issue to improve how to find this out here: https://github.com/elastic/beats/issues/11866
Use these configuration to have customer index template and name.
setup.ilm.enabled: false
setup.template.name: "dev"
setup.template.pattern: "dev-*"
index: "dev-%{[agent.version]}-%{+yyyy.MM.dd}"
When index lifecycle management (ILM) is enabled, the default index is
"filebeat-%{[agent.version]}-%{+yyyy.MM.dd}-%{index_num}", for
example, "filebeat-7.13.0-2021-05-25-000001". Custom index settings
are ignored when ILM is enabled. If you’re sending events to a cluster
that supports index lifecycle management, see Index lifecycle
management (ILM) to learn how to change the index name.
If index lifecycle management is enabled (which is typically the
default), setup.template.name and setup.template.pattern are ignored.
Ref:
https://www.elastic.co/guide/en/beats/filebeat/current/change-index-name.html
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html
https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html

Content repeat collecting problems while use filebeat

Recently we will use filebeat to collect our system logs to elasticsearch vias:
${local_log_file} -> filebeat -> kafka -> logstash -> elasticsearch -> kibana
While testing our system, we found a scenario that filebeat will repeatly collect logs which means that it will collect logs from the start of file once there is a change.
here is my configuration for filebeat:
filebeat.prospectors:
- input_type: log
paths:
- /home/XXX/exp/*.log
scan_frequency: 1s
#tail_files: true
#================================ Outputs =====================================
#----------------------------- Logstash output --------------------------------
# output.logstash:
# hosts: ["localhost:5044"]
#----------------------------- Kafka output -----------------------------------
output.kafka:
enabled: true
hosts: ["10.10.1.103:9092"]
topic: egou
#----------------------------- console output --------------------------------
output.console:
enabled: true
pretty: true
Notice:
we construct the log files manually, and we are sure that there is a blank line at the end of file
to make a console, we open the output.console
once there is content appended to the end of log file, filebeat will collect from the beginning of the file.But we hope just fetching the change of file.
filebeat version is 5.6.X
Hope any useful hint can be offered by u all
I think this is because of the editor, you are using, creates a new file on save with new meta-data. Filebeat identifies the state of the file using its meta-data, not the content.
try,
echo "something" >> /path/to/file.log
ref: https://discuss.elastic.co/t/filebeat-repeatedly-sending-old-entries-in-log-file/55796

How can I parse custom Metricbeat dictionary for Kibana?

I have logstash set up sending to Kibana, and it tags each log file with two custom fields -- Cluster and Node:
I would like to add the same two fields to my Metricbeat configuration using the "fields" option, but when I do this it comes through as a dictionary in Kibana:
Here is the Metricbeat config file I'm using:
metricbeat.modules:
- module: system
metricsets:
# CPU stats
- cpu
# System Load stats
- load
# Per CPU core stats
#- core
# IO stats
#- diskio
# Per filesystem stats
- filesystem
# File system summary stats
- fsstat
# Memory stats
- memory
# Network stats
- network
# Per process stats
- process
# Sockets (linux only)
#- socket
enabled: true
period: 10s
processes: ['.*']
#Add Supervisor Cluster and Node ID
fields:
cluster: "Upswell ELK Testing Cluster"
node: "Nina's Macbook"
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["http://localhost:9200"]
I can't find any documentation about how to format these fields; is this an adjustment I should be making in Logstash or in the Metricbeat configuration?
From : https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-general.html
fields : By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true.

Resources