Our fluentd configuration looks like this:
Input:
a fluentd/configs.d folder which contains
input
filter
output
Inside input we have two .conf files which are defining our sources:
This is one of the .confs
<source>
#type tail
#label #INGRESS
path /var/log/containers/*.log
pos_file /var/log/es-containers.log.pos
time_format %Y-%m-%dT%H:%M:%S
tag kubernetes.*
format json
keep_time_key true
read_from_head true
</source>
The other config has the same label #INGRESS
In our filter section we have some config's to filter on all the sources from #INGRESS. The output section defines what to do whith our filtered logs.
our global fluentd.conf looks like this:
## sources
#include configs.d/input/docker/*.conf
#include configs.d/input/syslog/*.conf
##
<label #INGRESS>
## filters
#include configs.d/filter/k8s_meta.conf
#include configs.d/filter/kibana_transform.conf
#include configs.d/filter/k8s_flatten_hash.conf
#include configs.d/filter/k8s_record_transform.conf
#include configs.d/filter/syslog_record_transform.conf
##
## matches
#include configs.d/output/operations.conf
#include configs.d/output/applications.conf
##
</label>
So it's all inside the #INGRESS label and this is working fine.
But now we want another filter on our the same 2 sources. This filter will also have an outputfile (to tell what to do with the filtered logs). But we only want THAT filter, not the other filters which are inside the #INGRESS section (in fluent.conf).
It's not possible to add 2 labels to a source and add a label in our fluent.conf file.
So what is our solution to add an independent filter and output file on the same source as the source for everyting with the #INGRESS label?
Thanks.
This is the github repo with the sourcecode we are using for our fluentd.
Related
I work on page drafts in my sphinx doc that I don't want to be available when I run sphinx-build.
for example I have files named _bob.md , _mistbuddy.md etc.
in my conf.py file I put:
exclude_patterns = [ '**/_*']
to ignore these file.
This works on load - files ignored :
However, if I then click on one of the topics, and ALL pages are available.
I have tried .. only:: <tag> however, a directive doesn't work on all contents in a file.
Thank you.
I would like to write an application config in go using flags and then have it writeout into an .ini file.
I've done this with JSON files, but can't figure out the ini.
Any suggestions?
Ini file will look like so:
Name=*flag input
[Output]
Mode=*flag input
[Input]
BaseCX=*flag input
BaseCY=*flag input
Common=*flag input
Can't figure it out.
I am trying to process log files with .gz extension in fluentd using cat_sweep plugin, and failed in my attempt. As shown in the below config, I am trying to process all files under /opt/logfiles/* location. However when the file format is .gz, cat_sweep is unable to process the file, and starts deleting the file, but if I unzip the file manually inside the /opt/logfiles/ location, cat_sweep is able to process, the file.
<source>
#type cat_sweep
file_path_with_glob /opt/logfiles/*
format none
tag raw.log
waiting_seconds 0
remove_after_processing true
processing_file_suffix .processing
error_file_suffix .error
run_interval 5
</source>
So now I need some plugin that can unzip a given file. I tried searching for plugins that can unzip a zipped file. I came close when I found about the plugin, which acts like a terminal, where I can use something like gzip -d file_path
Link to the plugin:
http://docs.fluentd.org/v0.12/articles/in_exec
But the problem I see here, is that I cannot send the path of the file to be unzipped at run-time.
Can someone help me with some pointers?
Looking at your requirement, you can still achieve it by using in_exec module,
What you have to do is, to simply create a shell script which accepts path to look for .gz files and the wildcard pattern to match file names. And inside the shell script you can unzip files inside the folder_path that was passed with the given wildcard pattern. Basically your shell execution should look like:
sh unzip.sh <folder_path_to_monitor> <wildcard_to_files>
And use the above command in in_exec tag in your config. And your config will look like:
<source>
#type exec
format json
tag unzip.sh
command sh unzip.sh <folder_path_to_monitor> <wildcard_to_files>
run_interval 10s
</source>
I wrote some liblognormalize rules to parse postgresql logs.
My rule file contains two rules and each rule has some tags like:
rule=POSTGRESQL,CHECKPOINT: ....
rule=POSTGRESQL,SLOWQUERY: ....
After running mmnormalize in my rsyslog configuration, I would like to know which rule actually matched the log line being processed. The simplest solution would be to get the tags. I know that mmnormalize export some variables like $parsesuccess. Is there any variable containing the tags of the rule used ?
I do not know where I found it in the docs but in a rsyslog config file that I wrote some time ago I found that I can access the list of tags that liblognorm assigns to a message via event.tags in rsyslog.
I have for example
template( # ...
property(name="$!event.tags")
)
or
if $!event.tags != "" then { # ...
The solution seems to be the annotate feature:
rule=POSTGRESQL,CHECKPOINT: ....
annotate=CHECKPOINT:+checkpoint="complete"
Basically the annotate line will add a field checkpoint containing the value complete to all the log lines matching the rules with a tag CHECKPOINT
Found it here
I'm fairly new to fluentd and I am not sure if it can do what I am trying right now. I am using it to collect log-data from a whole bunch of Docker-containers running on the same host, so the "hostname"-variable that is often discussed is not helping me. Instead, the log-data I receive includes a field "container_name".
Now what I would like is to have Fluentd write log-files with a name of "container_name-id_timestamp" but none of the ways I've tried so far have worked. I do not know how to evaluate fields from within the data packet.
I went ahead and matched everything like this:
<match *.**>
type file
path /var/log/fluent/
time_slice_format %Y-%m-%d
time_slice_wait 10m
time_format %Y-%m-%dT-%H-%M-%S-%z
</match>
And then tried all kinds of variables like
path /var/log/fluent/${container_name}_%Y-%m-%d
or
path /var/log/fluent/${tag_parts[2]}_%Y-%m-%d
But instead of interpreting it takes it literal. What am I missing?
I'd also be fine with subfolders for each container, which I am having the same problem with.
Thank you.
Anyone coming here looking for a solution, I've found out how to do it:
<match docker.*>
type rewrite_tag_filter
rewriterule1 container_name ^\/(.*)$ tagged.$1
</match>
<match tagged.*>
type forest
subtype file
remove_prefix service
<template>
time_slice_format %Y-%m-%d
path /var/log/fluentd/${tag}.*.log
</template>
</match>
What is happening?
Look for a match to i.e. the docker.325435abcd-tag
Use fluent_plugin_rewrite_tag_filter to get the container_name from within the data
Rewrite the tag with the container_name
Match the tagged data
Use the forest_plugin to template the log-file-name to use the tag
Done