I have filebeat rpm installed onto a unix server and I am attempting to read 3 files with multiline logs and I know a bit about multiline matching using filebeat but I am wondering if its possible to have matching for 3 separate logs.
Thanks
you might basically need multiple prospectors,
Example, (not tested)
filebeat.prospectors:
- input_type: log
paths:
- /var/log/app1/file1.log
multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: false
multiline.match: after
- input_type: log
paths:
- "/var/log/app2/file2.log"
- input_type: log
paths:
- "/var/log/app3/file3.log"
negate: true and match: after => specify that any line that does not match the specified pattern belongs to the previous line.
References
https://www.elastic.co/guide/en/beats/filebeat/current/multiple-prospectors.html
https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
Understanding Multiline
Related
I've been trying to finish up a playbook for deploying a new server. I'm struggling with changing data within brackets containing quotes via lineinfile and a regex:
- name: "Configuring: filebeat agent - configuring output to logstash"
lineinfile:
dest: "/etc/filebeat/filebeat.yml"
regexp: '#hosts: ["localhost:5044"]'
line: 'hosts: ["elk.home:5044"]'
tags: application
After the playbook is executed, the desired line:
#hosts: ["localhost:5044"]
is not updated to reflect:
hosts: ["elk.home:5044"]
What I'm trying to achieve is:
#hosts: ["localhost:5044"] is replaced with hosts: ["elk.home:5044"]
There are no errors generated. I've tried varying " and ' along with escapes \, but I can't get the expression correct. Any suggestions would be greatly appreciated!
Thanks seshadri_c and β.εηοιτ.βε!
I was able to reach a resolution with the following lines:
- name: "Configuring: filebeat agent - enabling logstash output hosts"
lineinfile:
dest: "/etc/filebeat/filebeat.yml"
regexp: '#hosts: \["localhost:5044"\]'
line: 'hosts: ["elk.home:5044"]'
tags:
- configuration
- application
- filebeat
After completing the playbook, I had an issue with whitespace. I added two spaces that correctly modified the line
- name: "Configuring: filebeat agent - enabling logstash output hosts"
lineinfile:
dest: "/etc/filebeat/filebeat.yml"
regexp: '#hosts: \["localhost:5044"\]'
line: ' hosts: ["elk.home:5044"]'
tags:
- configuration
- application
- filebeat
I have a simple problem:
My logfiles have timestamps in their name, i.e.:
/var/log/html/access-2021-11-27.log
/var/log/html/access-2021-11-28.log
/var/log/html/access-2021-11-29.log
Promtail is scraping this but does not "see" that access-2021-11-28.log is a continuation of access-2021-11-27.log. So it will "detect" a log file access-2021-11-28.log on the 28th and not show the access-2021-11-27.log anymore. I would want to see just "access.log" with data for several days.
I would assume this should be a well-known scenario, but I cannot find anything on this on the Internet.
The only way is to change log configuration of the application which is generating the logs, to use a unique access.log instead of the schema of the access-xxxx-xx-xx.log files. Unfortunately, this is not always possible.
But...
The old files can still be shown, it only depends on the time range used. Here is an example:
You can use regular expressions to perform the query, like in this example:
{filename=~".*JIRA_INSTALL/logs/access_log\\..*"}
If you want to statically override the filename field you can so something as simple as this:
scrape_configs:
- job_name: system
static_configs:
- labels:
job: remotevarlogs
__path__: /var/log/html/access-*.log
pipeline_stages:
- match:
selector: '{job="remotevarlogs"}'
stages:
- static_labels:
filename: '/var/log/html/access.log'
For those of you searching how to dynamically change the filepath prefix. For example, I'm using FreeBSD jails to nullfs mount my logs from other jails into a promtail jail. I don't want the local mount location (/mnt/logs/<hostname>) to show up as part of the path. Mounting shared folder could similarly be done with NFS or Docker.
scrape_configs:
- job_name: system
static_configs:
- labels:
job: remotevarlogs
__path__: /mnt/logs/*/**/*.log
pipeline_stages:
- match:
selector: '{job="remotevarlogs"}'
stages:
- regex:
source: filename
expression: "/mnt/logs/(?P<host>\\S+?)/(?P<relativepath>\\S+)"
- template:
source: host
template: '{{ .Value }}.mylocaldomain.com'
- template:
source: relativepath
template: '/var/log/{{ .Value }}'
- labels:
host:
filename: relativepath
- labeldrop:
- job
- relativepath
/etc/fstab for loki jail to pass-in /var/log/ directory from the grafana jail:
# Device Mountpoint FStype Options Dump Pass#
...
/jails/grafana/root/var/log/ /jails/loki/root/mnt/logs/grafana nullfs ro,nosuid,noexec 0 0
...
Now when I browse the logs, instead of seeing /mnt/logs/grafana/nginx/access.log, I see /var/log/nginx/access.log from grafana.mylocaldomain.com.
A YAML file as follows:
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node-exporter'
relabel_configs:
- source_labels: [__address__]
target_label: instance
regex: '([^:]+)(:[0-9]+)?'
replacement: '${1}'
static_configs:
- labels:
type: 'YYY'
group: 'Bar'
targets:
- '192.168.1.134:80'
- '192.168.1.146:80'
- labels:
type: 'YYY'
group: 'Foo'
targets:
- '192.168.2.136:80'
- labels:
type: 'ZZZ'
group: 'Foo'
targets:
- '192.168.2.100:80'
I'd like to use a bash script to add the "targets" to the YAML based on "type" and "group". For example:
./add_host.sh 192.168.1.17:80 YYY Bar
Added bonus for deleting entries :)
Things I've tried (and failed):
Using jq;
Tried using sed to insert values, but I have trouble in finding the right section to insert it to (Block parsing).
You don't mean that you want to parse the YAML file with bash, do you? That would be tedious and error-prone, not to mention slow to execute. You must mean you want a command-line utility to manipulate the file.
What you want is something that parses the YAML into a data structure. If I were doing it, I'd try the Python PyYAML module. It represents the whole file as a nested data structure. Read it in, change the parts you want to change, and write it out.
I have directory structure as below, wants to fetch the specific sub directory files using ansible rather than all.
/mnt/server1 ->
----> yyy.deb
----> /mnt/server1/All/tttsss.deb
----> /mnt/server1/HS-CLONE/gggg.deb
----> /mnt/server1/HS-TEST/kkkk.deb
I need to find only files present under /mnt/server1/All/tttsss.deb and /mnt/server1/HS-CLONE/gggg.deb directories. I don't require all other files.
When i trying using below logic, the parent directory file yyy.deb is also coming as output.
- name: Ansible find files in subdirectory examples
find:
paths: /mnt/server1
file_type: file
recurse: yes
use_regex: yes
patterns:
- 'All'
- "HS-CLONE"
- '.*deb$'
register: files_matched_subdirectory
With the above logic output as:
Output:
yyy.deb
/mnt/server1/All/tttsss.deb
/mnt/server1/HS-CLONE/gggg.deb
Expected output should be:
/mnt/server1/All/tttsss.deb
/mnt/server1/HS-CLONE/gggg.deb
This is happening because you have below specific pattern in your find command which will match yyy.deb in the directory /mnt/server1
patterns:
- '.*deb$'
you can use excludes parameter to explicitly exclude this particular file :
- name: Ansible find files in subdirectory examples
find:
paths: /mnt/server1
file_type: file
recurse: yes
use_regex: yes
patterns:
- 'All'
- "HS-CLONE"
- '.*deb$'
excludes:
- 'yyy.deb'
register: files_matched_subdirectory
Or may be you can try something like below in your pattern parameter :
patterns:
- 'All'
- "HS-CLONE"
- '*/.*deb$'
I need create some configuration file in yml wtith ansible role. This example what i want:
filebeat.inputs:
- type: log
paths:
- /var/log/system.log
- /var/log/wifi.log
In ansible dirctory structue in Templates dir i have file with such text:
filebeat.inputs:
- type: {{filebeat_input.type}}
{{ filebeat_input.paths | to_yaml}}
In Directory defaults i have such main.yml:
filebeat_create_config: true
filebeat_input:
type: log
paths:
- "/var/log/*.log"
- "/var/somelog/*.log"
When im run ansible-playbook with this role im got this:
filebeat.inputs:
- type: log
[/var/log/*.log, /var/sdfsadf]
Where i`m wrong ? what and where i need change to get exactly what i whant (see example above).
Thanks for any help !
filebeat_input.paths is a list, so when you pass it to to_yaml you get a list. You just need to structure your YAML template so that you're putting the list in the right place, e.g:
filebeat.inputs:
- type: {{filebeat_input.type}}
paths: {{ filebeat_input.paths | to_yaml}}
Note that:
paths: [/var/log/*.log, /var/sdfsadf]
Is exactly equivalent to:
paths:
- /var/log/*.log
- /var/sdfsadf