logstash array of key value pairs - filter

Question

I am using logstash and I would like to know if there is a way to handle the following:
Using the xml filter I am able to extract a properties field
<?xml version="1.0"?>
<event logger="RemoteEventReceiver1" timestamp="2016-07-21T12:39:04.0607421-05:00" level="DEBUG" thread="26" domain="/LM/W3SVC/2/ROOT-1-131135962764935573" username="TOOTHLESS\dvdp4">
<message>Test nessage</message>
<properties>
<data name="log4net:HostName" value="Toothless"/>
<data name="log4net:Customer" value="Bob"/>
</properties>
</event>
that looks like this
"properties" => [
[0] {
"data" => [
[0] {
"name" => "HostName",
"value" => "Toothless"
},
[1] {
"name" => "Customer",
"value" => "Bob"
}
]
}
]
how can I convert it to this?
“propertiesParsed” => {
“HostName” => “Toothless”,
“Customer” => “Bob”
}
* UPDATE ADDING CONFIG AND DATA FILE *
input {
file {
type => "log4net"
path => ["D:/temp/MR4SPO.log"]
start_position => "beginning"
sincedb_path => "nul"
}
}
filter
{
mutate {
# remove xml prefices in the message field
gsub => [ "message", "log4net:", "" ]
}
xml {
source => "message"
target => "log4net"
add_field => {
log4net_message => "%{[log4net][message]}"
# "[log4net][messagetest]" => [log4net][message]
# xxx => "%{[log4net][properties][0][data]}"
}
remove_field => "message"
}
# get json message from log4net
if [log4net_message] =~ "^LS:\s{" {
ruby { code => "event['log4net_message'] = event['log4net_message'][3..-1]" }
json {
source => "log4net_message"
# target => "log4net_json"
}
mutate {
add_field => { forMQ => true }
}
}
mutate {
remove_field => "log4net_message"
}
}
# output logs to console and to elasticsearch
output {
if [forMQ] {
stdout { codec => rubydebug }
}
# elasticsearch { hosts => ["localhost:9200"] }
}
* DATA FILE *
<log4net:event logger="SPMRDLAdd_InWeb.Services.RemoteEventReceiver1" timestamp="2016-07-21T12:39:03.0607421-05:00" level="DEBUG" thread="26" domain="/LM/W3SVC/2/ROOT-1-131135962764935573" username="TOOTHLESS\dvdp4"><log4net:message>My test one</log4net:message><log4net:properties><log4net:data name="log4net:HostName" value="Toothless" /></log4net:properties></log4net:event>
<log4net:event logger="SPMRDLAdd_InWeb.Services.RemoteEventReceiver1" timestamp="2016-07-21T12:39:04.0607421-05:00" level="DEBUG" thread="26" domain="/LM/W3SVC/2/ROOT-1-131135962764935573" username="TOOTHLESS\dvdp4"><log4net:message>LS: { "name" : "file123.jpg", "size" : 50 }</log4net:message><log4net:properties><log4net:data name="log4net:HostName" value="Toothless" /></log4net:properties></log4net:event>

Answer 1

You can add that ruby filter:
...
ruby {
code => "
event['propertiesParsed'] = {}
for value in event['log4net']['properties']
for data in value['data']
event['propertiesParsed'][data['name']] = data['value']
end
end
"
}
...