i have the following config for metricbeat:
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
metricbeat_modules:
- module: system
metricsets:
- cpu
- load
- memory
- network
- diskio
enabled: true
period: 10s
tags: ['os']
cpu.metrics: ['percentages']
core.metrics: ['percentages']
setup.template:
name: {{ metricbeat_index }}
pattern: {{ metricbeat_index }}-*
settings:
index:
number_of_shards: 1
codec: best_compression
tags: [{{ metricbeat_tags | join(', ') }}]
fields:
env: {{ metricbeat_env }}
output.elasticsearch:
hosts: {{ metricbeat_output_es_hosts | to_json }}
index: "{{ metricbeat_index }}-%{+yyyy-MM-dd}"
setup.dashboards.directory: /usr/share/metricbeat/kibana
setup.kibana:
host: {{ metricbeat_kibana_url }}
processors:
- drop_fields:
fields: ["beat.name","beat.hostname"]
processors:
- add_host_metadata:
netinfo.enabled: false
processors:
- add_cloud_metadata: ~
it worked as expected as i had the metricsets process and process_summary enabled. since i removed them it seems still to harvest those metrics. i restarted, stopped/started metricbeat again but it still has no effect.
Thanks for ideas as i cannot see any reason why this should happen this way :/
I digged a bit more into you issue.
You sepcify a module config folder with this part of your config:
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
If you run look info that folder I'm sure you'll find this:
aerospike.yml.disabled
apache.yml.disabled
ceph.yml.disabled
couchbase.yml.disabled
docker.yml.disabled
dropwizard.yml.disabled
elasticsearch.yml.disabled
envoyproxy.yml.disabled
etcd.yml.disabled
golang.yml.disabled
graphite.yml.disabled
haproxy.yml.disabled
http.yml.disabled
jolokia.yml.disabled
kafka.yml.disabled
kibana.yml.disabled
kubernetes.yml.disabled
kvm.yml.disabled
logstash.yml.disabled
memcached.yml.disabled
mongodb.yml.disabled
munin.yml.disabled
mysql.yml.disabled
nginx.yml.disabled
php_fpm.yml.disabled
postgresql.yml.disabled
prometheus.yml.disabled
rabbitmq.yml.disabled
redis.yml.disabled
system.yml
traefik.yml.disabled
uwsgi.yml.disabled
vsphere.yml.disabled
windows.yml.disabled
zookeeper.yml.disabled
See that system.yml file ?
This is the configuration that is loaded.
So you can remove process from this configuration file or not use metricbeat.config.modules.path
Hope it helped.
Shoudln't you have metricbeat.modules instead of metricbeat_modules ?
Related
in our cluster some apps are sending logs as multiline, and the problem is that the log structure is different from app to app.
How can we set up an 'if' condition that will include the
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
In it?
Our code:
filebeatConfig:
filebeat.yml: |
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- drop_event:
when:
contains:
container.image.name: "kibana"
output.logstash:
hosts: ["logstash-listener:5044"]
You need to use auto-discovery (either Docker or Kubernetes) with template conditions.
You will probably have at least two templates, one for capturing your containers that emit multiline messages and another for other containers.
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition: <--- your multiline condition goes here
contains:
kubernetes.namespace: xyz-namespace
config:
- type: container
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
multiline:
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- drop_event:
when:
contains:
container.image.name: "kibana"
I am trying to merge a dictionary with a yaml file content and pass them to some salt state.
metricbeat.yml content:
metricbeat:
config:
modules:
path: /etc/metricbeat/modules.d/*.yml
reload.enabled: true
reload.period: 10s
output.logstash:
hosts:
worker: 1
compression_level: 3
loadbalance: true
ssl:
certificate: /usr/share/metricbeat/metricbeat.crt
key: /usr/share/metricbeat/metricbeat.key
verification_mode: none
logging:
level: debug
to_files: true
files:
path: /var/tellme/log/metricbeat
name: metricbeat.log
rotateeverybytes: 10485760
keepfiles: 7
config.yml content:
metricbeat:
config:
modules:
reload.period: 100s
Statefile:
{% import_yaml "config.yml" as config %}
manage_file:
file.managed:
- name: /etc/metricbeat/metricbeat.yml
- source: salt://metricbeat.yml
- template: jinja
conf_file:
file.serialize:
- name: /etc/metricbeat/metricbeat.yml
- dataset:
output.logstash:
hosts: ['exacmple.domain.com:5158']
{{ config | yaml }}
- serializer: yaml
- merge_if_exists: true
But I am getting the below error:
example-1.domain.com:
Data failed to compile:
----------
Rendering SLS 'base:test' failed: could not find expected ':'
What am I doing wrong ?
Fixed the issue as below
{% import_yaml "config.yml" as config %}
manage_file:
file.managed:
- name: /etc/metricbeat/metricbeat.yml
- source: salt://metricbeat.yml
- template: jinja
conf_file:
file.serialize:
- name: /etc/metricbeat/metricbeat.yml
- dataset:
output.logstash:
hosts: ['exacmple.domain.com:5158:5158']
{{ config | yaml(false) | indent(8) }}
- serializer: yaml
- merge_if_exists: true
"yaml(false)" is for multiline yaml and proper indentation with "indent".
I have tried every combination I can conceive of to specify to deploy a google compute instance into a particular subnet (subnetX) in network (networkY).
- name: create a instance
google.cloud.gcp_compute_instance:
name: test_object
machine_type: n1-standard-1
disks:
- auto_delete: 'true'
boot: 'true'
source: "{{ disk }}"
- auto_delete: 'true'
interface: NVME
type: SCRATCH
initialize_params:
disk_type: local-ssd
labels:
environment: production
network_interfaces: # <<< does not work. API request is made without a network_interface
- network:
selfLink: "https://blah/blah/blah/networkY"
subnetwork:
selfLink: "https://blah/blah/blah/subnetworkX"
zone: us-central1-a
project: test_project
auth_kind: serviceaccount
service_account_file: "/tmp/auth.pem"
state: present
When using subnetwork you should not specify a network.
To be more precise this is the workaround for this problem.
---
- name: create a network
gcp_compute_network:
name: ansible-network
auto_create_subnetworks: yes
project: "{{ lookup('env','GCP_PROJECT') }}"
state: present
register: network
- name: Get Network URL
set_fact:
network_url: "{{ network | json_query(jmesquery) }}"
vars:
jmesquery: "{selfLink: selfLink}"
- name: create a firewall
gcp_compute_firewall:
name: ansible-firewall
network: "{{ network_url }}"
allowed:
- ip_protocol: tcp
ports: ['80','22']
target_tags:
- apache-http-server
source_ranges: ['0.0.0.0/0']
project: "{{ lookup('env','GCP_PROJECT') }}"
state: present
register: firewall
We are using journalbeat to push logs of kubernetes cluster to elastic search. It working fine and pushing the logs. However its also pushing event like "200 OK" and "INFO" which we do not want. The journalbeat.yaml is as follows
journalbeat.yaml
journalbeat.yml: |
name: "${NODENAME}"
journalbeat.inputs:
- paths: []
seek: cursor
cursor_seek_fallback: tail
processors:
- add_kubernetes_metadata:
host: "${NODENAME}"
in_cluster: true
default_indexers.enabled: false
default_matchers.enabled: false
indexers:
- container:
matchers:
- fields:
lookup_fields: ["container.id"]
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 1
target: ""
overwrite_keys: true
- drop_event.when:
or:
- regexp.kubernetes.pod.name: "filebeat-.*"
- regexp.kubernetes.pod.name: "journalbeat-.*"
- regexp.kubernetes.pod.name: "nginx-ingress-controller-.*"
- regexp.kubernetes.pod.name: "prometheus-operator-.*"
setup.template.enabled: false
setup.template.name: "journal-${ENVIRONMENT}-%{[agent.version]}"
setup.template.pattern: "journal-${ENVIRONMENT}-%{[agent.version]}-*"
setup.template.settings:
index.number_of_shards: 10
index.refresh_interval: 10s
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
index: "journal-${ENVIRONMENT}-system-%{[agent.version]}-%{+YYYY.MM.dd}"
indices:
- index: "journal-${ENVIRONMENT}-k8s-%{[agent.version]}-%{+YYYY.MM.dd}"
when.has_fields:
- 'kubernetes.namespace'
How can i exclude logs like "INFO" and "200 OK" events?
As far as I'm aware there is no way to exclude logs in Journalbeat. It's working other way around, meaning you tell it what input to look for.
You should read about Configuration input:
By default, Journalbeat reads log events from the default systemd journals. To specify other journal files, set the paths option in the journalbeat.inputs section of the journalbeat.yml file. Each path can be a directory path (to collect events from all journals in a directory), or a file path.
journalbeat.inputs:
- paths:
- "/dev/log"
- "/var/log/messages/my-journal-file.journal"
Within the configuration file, you can also specify options that control how Journalbeat reads the journal files and which fields are sent to the configured output. See Configuration options for a list of available options.
Get familiar with the Configuration options and using the translated fields to target the exact input you want to.
{beatname_lc}.inputs:
- id: consul.service
paths: []
include_matches:
- _SYSTEMD_UNIT=consul.service
- id: vault.service
paths: []
include_matches:
- _SYSTEMD_UNIT=vault.service
You should use it to target the inputs you want to have pushed to elastic.
As an alternative to Journalbeat you could use Filebeat and the exclude might look like this:
type: log
paths:
{{ range $i, $path := .paths }}
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
exclude_lines: ['.*INFO.*']
Hope this helps you a bit.
To apply filter use:
logging.level: warning
Use this instruction to drop event journalbeat.service:
processors:
- drop_event:
when:
equals:
systemd.unit: "journalbeat.service"
I'm creating instance in Google Cloud with Ansible, but when I want to attach existing disk to new compute engine, I can't attach it or add it to instance.
- name: Launch instances
gce:
instance_names: mongo
machine_type: "n1-standard-1"
image: "debian-9"
service_account_email: "xxxx#xxxx.iam.gserviceaccount.com"
credentials_file: "gcp-credentials.json"
project_id: "learning"
disk_size: 10
disk_auto_delete: false
preemptible: true
tags: "mongo-server"
register: gce
- name: Wait for SSH for instances
wait_for:
delay: 1
host: "{{ item.public_ip }}"
port: 22
state: started
timeout: 30
with_items: "{{ gce.instance_data }}"
The error I have is:
The error was: libcloud.common.google.ResourceExistsError: {'domain': 'global', 'message': "The resource 'projects/xxx-xxx/zones/us-central1-a/disks/mongo' already exists", 'reason': 'alreadyExists'}
There are any form to configure this option with Ansible? To do that now I'm using external scripts.
Existing disks can be provided as a list under 'disks' attribute, first entry needs to be Boot dik
https://docs.ansible.com/ansible/2.6/modules/gce_module.html
- gce:
instance_names: my-test-instance
zone: us-central1-a
machine_type: n1-standard-1
state: present
metadata: '{"db":"postgres", "group":"qa", "id":500}'
tags:
- http-server
- my-other-tag
disks:
- name: disk-2
mode: READ_WRITE
- name: disk-3
mode: READ_ONLY