Logstash split root message - elasticsearch

Question

I am collecting some metrics about my application and periodically export them over REST one by one. The output json looks like:
{
"name": "decoder.example.type-3",
"value": 2000,
"from": 1517847790049
"to": 1517847840004
}
This is my logstash configuration that is working well. It will remove all http headers, the original counter name, and add example as interface and type-3 as transaction.
input {
http {
port => 31311
}
}
filter {
json {
source => "message"
}
grok {
match => [ "name", "decoder.%{WORD:interface}.%{NOTSPACE:transaction}" ]
}
mutate {
remove_field => [ "name", "headers", "message" ]
}
}
output {
elasticsearch {
hosts => [ "http://localhost:9200" ]
index => "metric.decoder-%{+YYYY.MM.dd}"
}
}
What I am trying to do now is send all my metrics at once as json array and split all these messages and apply the same logic that was applied to them one by one. An example of the input message would look like:
[
{
"name": "decoder.example.type-3",
"value": 2000,
"from": 1517847790049,
"to": 1517847840004
},
{
"name": "decoder.another.type-0",
"value": 3500,
"from": 1517847790049,
"to": 1517847840004
}
]
I am pretty certain I am supposed to use split filter, but I can't figure out how to use it. I have tried putting split before and after my json plugin, using different field settings, targets, but nothing seems to work as expected.
Could someone point me in the right direction?

Answer 1

In my config I used split first, then I did the logic. Yours should look based on that something like this:
input {
http {
port => 31311
}
}
filter {
json {
source => "message"
}
split{
field => "message"
}
grok {
match => [ "name", "decoder.%{WORD:interface}.%{NOTSPACE:transaction}" ]
}
mutate {
remove_field => [ "name", "headers", "message" ]
}
}
output {
elasticsearch {
hosts => [ "http://localhost:9200" ]
index => "metric.decoder-%{+YYYY.MM.dd}"
}
}
But this presumes that you always have a message field that is an array.
Oh yeah, and I think you should check whether you have the new message field contain the object that you posted. Because if so, your grok won't find anything under name, you need to match message.name. (I usually create a temp field from [message][name] and remove temp later because I didn't care to look up how to call nested fields. There must be a smarter way.)

Answer 2

This is the configuration I ended up with. Perhaps it can be done in fewer steps, but this works well. I had to move some fields around to keep the same structure so it is a bit bigger than my initial one which worked one by one.
The basic idea is to put the parsed json into a specific field, not in the root, and then split that new field.
input {
http {
port => 31311
}
}
filter {
json {
source => "message"
target => "stats"
}
split {
field => "stats"
}
grok {
match => [ "[stats][name]", "decoder.%{WORD:interface}.%{NOTSPACE:transaction}" ]
}
mutate {
add_field => {
"value" => "%{[stats][value]}"
"from" => "%{[stats][from]}"
"to" => "%{[stats][to]}"
}
remove_field => [ "headers", "message", "stats" ]
}
mutate {
convert => {
"value" => "integer"
"from" => "integer"
"to" => "integer"
}
}
}
output {
elasticsearch {
hosts => [ "http://localhost:9200" ]
index => "metric.decoder-%{+YYYY.MM.dd}"
}
}