Is there a way parse log messages using rsyslog config and transform them to structured messages? - rsyslog

Question

I am trying to parse log messages and transform them to structured messages using rsyslog. Is there a way support such operation with rsyslog config? I have not yet explored the option to write custom parser or message modification plugin for this.
I found template list properties which can do some of it. Is there a way to do the following?
Map 2 fields to single output name. Ex: "__ts": "2018-09-20 10:18:56.363" (first 2 fields in example below). Would not use regex here as I am looking for a solution that does not depend on value of the fields. Ex: the two fields could be two strings or some other values not just dates.
Extract what is left in msg after extracting all known fields based on position. Ex: "msg": "Unregistering application nameOfAnApiHere with someOtherName with status DOWN".
Is there a way to use local variables to hold the values of fields from msg and use the variables in templates?
Example Log message:
2018-09-20 10:18:56.363 INFO --- [Thread-68] x.y.z.key1Value Unregistering application nameOfAnApiHere with someOtherName with status DOWN
1. rsyslog config template definition
template(name="structure-log-format" type="list") {
constant(value="{")
# This only extracts the first field with value 2018-09-20.
# TODO: What is a way to map first 2 fields to map to __ts field?
property(outname="__ts" name="msg" field.number="1" field.delimiter="32" format="jsonf") constant(value=", ")
constant(value="\"event\":[{")
constant(value="\"payload\":{")
property(outname="_log_" name="syslogtag" format="jsonf") constant(value=", ")
property(outname="__loglvl" name="msg" field.number="4" field.delimiter="32" format="jsonf") constant(value=", ")
property(outname="__thread" name="msg" field.number="7" field.delimiter="32" format="jsonf") constant(value=", ")
property(outname="__key1" name="msg" field.number="8" field.delimiter="32" format="jsonf") constant(value=", ")
# The following setting will include full message value starting from "2018-09-20 ... DOWN"
# TODO: What is a way to only include message starting from "Unregistering ... DOWN"?
property(name="msg" format="jsonf" droplastlf="on" )
constant(value="}")
constant(value="}]} \n")
}
2. Expected result:
{
"__ts": "2018-09-20 10:18:56.363",
"event": [
{
"payload": {
"_log_": "catalina",
"__loglvl": "INFO",
"__thread": "Thread-68",
"__key1": "x.y.z.key1Value",
"msg": "Unregistering application nameOfAnApiHere with someOtherName with status DOWN"
}
}
]
}
3. Actual result:
{
"__ts": "2018-09-20",
"event": [
{
"payload": {
"_log_": "catalina",
"__loglvl": "INFO",
"__thread": "Thread-68",
"__key1": "x.y.z.key1Value",
"msg": "2018-09-20 10:18:56.363 INFO 2144 --- [Thread-68] x.y.z.key1Value Unregistering application nameOfAnApiHere with someOtherName with status DOWN"
}
}
]
}
Thank you.

Answer 1

You can also use regular expressions to match parts of a message. For example, replace your outname="__ts" property with:
property(outname="__ts" name="msg"
regex.expression="([^ ]+ +[^ ]+)"
regex.type="ERE"
regex.submatch="1" format="jsonf")
Here the extended regular expression (ERE) looks for not-a-space ([^ ]) one or more of them (+), followed by a space or more, and another not-a-space. These 2 words are captured as a submatch by the () and you select this one, counting from 1. The result should be as you want.
You can similarly use a regex for the second requirement, either by counting "words" and spaces again, or some more precise other match. Here the regex skips 6 words by putting a repeat count {6} after the word-and-spaces pattern, then captures the rest (.*). Since there are 2 sets of (), the submatch to keep is now 2, not 1:
property(name="msg"
regex.expression="([^ ]+ +){6}(.*)"
regex.type="ERE"
regex.submatch="2" format="jsonf" droplastlf="on" )