Reference Conf File within Conf File or Apply Rule to All Listening RSyslog Ports - rsyslog

We have a number of individual conf files with their own ruleset that's bound to its unique port. We want to create a single conf file that will filter/drop specific things such as, if msg from IP drop it or if msg contains x drop it. And have the drop filtering apply to all listening ports. Is this possible to do? Should we avoid using rulesets?
We're trying to avoid updating the drop/filter in each conf file for each port every time the filter has a new update.
Would anyone happen to know if one of the following things is possible with RSyslog?
Have 1 conf file that will listen on all rsyslog ports and be processed first? Without specifying each open port.
Have a conf file that calls another file with a rule in it?
Appreciate any help with this.

Typically, the default configuration file, say /etc/rsyslog.conf, will contain a line near the start saying something like
$IncludeConfig /etc/rsyslog.d/*.conf
or the equivalent RainerScript syntax
include(file="/etc/rsyslog.d/*.conf")
If not you can add it.
This will include all files matching the glob pattern, in alphabetical order. So you can optionally put any configuration in that directory, for example in arbitrarily named files 00-some.conf, and 10-somemore.conf and so on.
One file could have lots of input() statements like:
module(load="imtcp" MaxSessions="500")
input(type="imtcp" port="514")
input(type="imtcp" port="10514")
input(type="imtcp" port="20514")
assuming you are expecting to receive incoming tcp connections from remote
clients. See imtcp.
All the data from those remotes will be affected by any following rules.
For example, the last included file in the directory could hold lines like:
if ($msg contains "Password: ") then stop
if ($msg startswith "Debug") then stop
if ($hostname startswith "test") then stop
These will stop further processing of any matching input messages, effectively
deleting them.
The above inputs are all collected into a single global input queue.
All the if rules are applied to all the messages from that queue.
If you want to, you can partition some of the inputs into a new queue,
and write rules that will only apply to that new independent queue. The rest of the
configuration will know nothing about this new queue and rules.
This is called a ruleset. See
here and
here.
For example, you can have a ruleset called "myrules". Move one or more
inputs into the ruleset by adding the extra option:
input(type="imtcp" port="514" ruleset="myrules")
input(type="imtcp" port="10514" ruleset="myrules")
Move the rules to apply to that queue into a ruleset definition:
ruleset(name="myrules"){
if ($msg contains "Password: ") then stop
if ($msg startswith "Debug") then stop
*.* /var/log/mylogfile
}

Related

Weird messages "rsyslogd: msg: ruleset ' &è·Æ ' could not be found and could not be assgined to message object" in rsyslog logs

We have an rsyslog configured to receive messages from multiple sources on different ports.
Messages are then assigned to different action rulesets depending on the incoming port.
We have noticed that sometimes (but not systematically), after an rsyslog restart, there are error logged in /var/log/messages with content like
"2022-08-16T16:46:26.841640+02:00 mysyslogserver rsyslogd: msg: ruleset ' 6È B ' could not be found and could not be assgined to message object. This possibly leads to the message being processed incorrectly. We cannot do anything against this, but wanted to let you know. [v8.32.0 try http://www.rsyslog.com/e/3003 ]"
The name of ruleset is changing every time and seems to be a random binary string. Such message is logged several thousands of time (with same ruleset name), at a rate which often exceeds ratelimit for internal messages.
(And of course we don't have rulesets with such names in our config file... )
Would you know what could be the cause of such issue ? Is it a bug ?
Note that in some rulesets we use "call" statement to call sub-rulesets, but we don't use "call_indirectly".
Thanks in advance for any help.
S.Hemelaer

How to send two files into a single MQ message with different MsgSeqNumber

Good Afternoon,
I'm working from a unix machine.
I've two XML files and I need to load them into a queue. The two files should be 'considered' a single MQ message.
First, I tried to concatenate the files together and to upload them on a queue with
ma01_q -o QueueName -m ManagerQueue -F Filename;
However, the receiver of the message didn't succeed to make the distinction between the two files.
He provided me some part of what was expected in the hexadecimal message:
...
A GRP 000...
A MSQ 1
...
A MSF 8
...
The content of the first file in hexadecimal
...
A GRP 000...
A MSQ 2
...
A MSF 24
...
The content of the second file in hexadecimal
So according to him, I should use the parameters GroupId and MsgSeqNumber.
For the first file:
Put 'MQMF_MSG_IN_GROUP' in field MsgFlags of the message descriptor.
Use 'MQPMO_LOGICAL_ORDER' on the MQPUT.
Perform the MQPUT.
For the second file:
Put 'MQMF_LAST_MSG_IN_GROUP' in field MsgFlags of the message descriptor.
Use option 'MQPMO_LOGICAL_ORDER' on the MQPUT.
Perform the MQPUT.
This will automatically generate a 'GroupID' and 'MsgSeqNumber' for each file. Using the flag 'MQMF_LAST_MSG_IN_GROUP' will ensure that the message group is 'closed'.
The two files are one message each but grouped together using MQ "Message Grouping".
Is it possible to do that on unix with ma01_q or qload?
Neither ma01 nor the qload supports creating message groups.
You could use the mqput2 program from support pack ih03. The mqput2 program can use the MQMD header provided in the file. So you would need to add the MQMD to your files, with the desired message group and sequence number details, then use mqput2 to put your files on the queue.
But I think it would be better to write your own program to do this.

Logstash Duplicate Events

I have two configuration files for Logstash: test1.conf and test2.conf.
Each one of them has it's own flow of input -> filter -> ouput.
Both of them have the same filter and elasticsearch output writing to the same index.
My problem is that Logstash is writing duplicate events to the ElasticSearch index, no matter which input I choose to test (every event becomes two identical events instead of one).
How can I fix this?
By default, Logstash has one pipeline named main which automatically detects all .conf files in conf.d folder; this configuration is set at pipelines.yml file:
- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
If you have multiple .conf files under one pipeline, Logstash will merge them together, causing all filters and outputs to be performed on all of the inputs, so in this case, no matter which input is receiving events, it will go through two paths of filter/output, causing duplicate writing to ElasticSearch (identical events if the filters/outputs are the same for both .conf files).
Solutions
1. Move filter/output into a separate file
If your filters/outputs are the same across the config files, move filter/output into a separate file. So now you have two .conf files, one for each input, and a third .conf file for the filter/output. With this setup every input will go through only one processing path.
For example:
input1.conf
input {
# input 1
}
input2.conf
input {
# input 2
}
filter_output.conf
filter {
# common filter
}
output {
# common output
}
You can check out this answer for another example when this solution should be chosen.
Note that If the filters/output are the same but you still want to refer them as complete different processing paths, please keep reading.
2. Split the .conf files to different pipelines
If you need every .conf file to be independent, split the .conf files to different pipelines.
In order to do that, just edit pipelines.yml file.
For example:
pipelines.yml
- pipeline.id: test1
path.config: "/etc/logstash/conf.d/test1.conf"
- pipeline.id: test2
path.config: "/etc/logstash/conf.d/test2.conf"
Read more about Multiple Pipelines
3. Separate by types
Tag each input with different type and check it later on the filters/outputs with if statement.
You can read more about it in this answer.

logstash won’t read single line files

I'm trying to make a pipeline that sends xml documents to elasticsearch. Problem is that each document is in its own separate file as a single line without \n in the end.
Any way to tell logstash not to wait for \n but read whole file till EOF and send it?
Can you specify which logstash version you are using, and can you share your configuration?
It may depends on the mode you set: it may be tail or read. it defaults to tail, which means it listens on your file and it waits for default 1 hour before closing it and stopping waiting for new lines.
You may have to change this parameter fro 1 hour to 1 second if you know you have reached the EOF yet:
file {
close_older=> "1 second"
}
Let me know if that works!
Docs here: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-close_older

Wait for File Processing to be finished

I am using Spring Integration to process/load data from csv files.
My Configuration is -
1) Poll For incoming File
2) Split the file using splitter - this gives me individual lines(records) of the file
3) Tokenize the line - this gives me the values or columns
4) Use aggregator to aggregate/collect lines(records) and write it to database in a batch
Poller -> Splitter -> Tokenizer -> Aggregator
Now I want to wait till all the content of the file has been written to the database and then move the file to a different folder.
But how to identify when the file processing is finished ?
Problem here is, if the file has 1 million records and my aggregator has batch size of 500, how would i know when every record of my file has been aggregated and written out to the database.
The FileSplitter can optionally add markers (BOF, EOF) to the output - you would have to filter and/or route them before your secondary splitter.
See FileSplitter.
(markers) Set to true to emit start/end of file marker messages before and after the file data. Markers are messages with FileSplitter.FileMarker payloads (with START and END values in the mark property). Markers might be used when sequentially processing files in a downstream flow where some lines are filtered. They enable the downstream processing to know when a file has been completely processed. In addition, a header file_marker containing START or END are added to these messages. The END marker includes a line count. If the file is empty, only START and END markers are emitted with 0 as the lineCount. Default: false. When true, apply-sequence is false by default. Also see markers-json.

Resources