How to aggregate metrics from netdata in logstash? - ruby

Question

And Thank you in advance for any help!
I'm using Netdata to collect metrics from servers and then send them to Logstash and to Elastic.
My need is to aggregate metrics with same fields and create a single event but in nested format.
This a example of input from Netdata:
{"host":"centosdns","#version":"1","port":52212,"#timestamp":"2019-01-19T16:16:22.117Z","message":"netdata.centosdns.disk_await.centos_swap.reads 0.0000000 1547914548"}
{"host":"centosdns","#version":"1","port":52212,"#timestamp":"2019-01-19T16:16:22.117Z","message":"netdata.centosdns.disk_await.centos_swap.writes 0.0000000 1547914548"}
{"host":"centosdns","#version":"1","port":52212,"#timestamp":"2019-01-19T16:16:22.117Z","message":"netdata.centosdns.disk_await.centos_root.reads 0.0000000 1547914548"}
{"host":"centosdns","#version":"1","port":52212,"#timestamp":"2019-01-19T16:16:22.117Z","message":"netdata.centosdns.disk_await.centos_root.writes 0.0000000 1547914548"}
My config file of logstash looks like:
input {
tcp {
port => 1234
}
}
filter {
# I take 'message' field and separate in different fields
grok {
named_captures_only => "true"
pattern_definitions => {
"CHART" => "[a-z]\w+"
"FAMILY" => "[_a-z0-9]+"
}
match => {
"message" => "%{WORD:prefix}\.%{WORD:hostname}\.%{CHART:chart}\.%{FAMILY:family}\.%{NOTSPACE:dimension} %{NUMBER:val} %{NUMBER:timestamp}"
}
}
if "_grokparsefailure" not in [tags] {
mutate {
remove_field => [ "#version", "host", "port", "prefix" ]
}
# Attempt to create a nested field and then aggregate
mutate {
id => "chart_field"
add_field => { "[%{chart}][%{family}][%{dimension}][value]" => "%{val}"
}
}
aggregate {
task_id => "[%{chart}][%{family}]"
code => "
# I tried many codes to aggregate but without success
event.cancel()
"
push_previous_map_as_event => true
timeout => 5
}
mutate {
# Remove unnecessary fields
id => "netdata_mutate_remove"
remove_field => [ "timestamp", "message"]
}
} else {
drop{}
}
output {
# TESTING PURPOSES
if "_aggregateexception" in [tags] {
file {
path => "/var/log/logstash/netdata/aggregatefailures-%{+MM-dd}.log"
}
} else {
file {
path => "/var/log/logstash/netdata/netdata-%{+MM-dd}-aggregate.log"
}
}
stdout { codec => rubydebug }
}
Take the input above:
"netdata.centosdns.disk_await.centos_swap.reads 0.0000000"
"netdata.centosdns.disk_await.centos_swap.writes 0.0000000"
My objective is make a nested field like:
disk_await: { # Chart
centos_swap: { # Family
[
reads => 0.0000000, # Dimension => Value
writes => 0.0000000 # Dimension => Value
]
}
}
I pretend to aggregate all 'Dimension\'Value'' in the same 'Chart'\'Family', this is only four lines of metrics but in reality we talk about 1000 per second or even more in some cases, all metrics are dynamic, it's virtual impossible to know all the names.
At this moment I'm using:
Logstash v.6.5.4 on a Virtualbox CentOS 7 minimal
All plugins (inputs/filters/outputs) updated