Hierarchical data matching and displaying - elasticsearch

Question

In my log files, I have data that represents a the hierarchy of items, much like an http log file might show the hierarchy of a website.
I may have data such as this
41 2016-01-01 01:41:32-500 show:category:all
41 2016-01-01 04:11:20-500 show:category:animals
42 2016-01-02 01:41:32-500 show:item:wallaby
42 2016-01-02 01:41:32-500 show:home
and I would have 3 items in here... %{NUMBER:terminal} %{TIMESTAMP_ISO8601:ts} and (?<info>([^\r])*)
I parse the info data into an array using mutate and split to convert lvl1:lvl2:lvl3 into ['lvl1','lvl2','lvl3'].
I'm interested in aggregating the data to get counts at various levels easily, such as counting all records where info[0] is the same or where info[0] and info[1] are the same. (and be able to select time range and terminal)
Is there a way to set up kibana to visualize this kind of information?
Or should I change the way the filter is matching the data to make the data easier to access?
the depth of levels varies but I can be pretty certain that the max levels are 5, so I could parse the text into various fields lvl1 lvl2 lvl3 lvl4 lvl5 instead of putting them in an array.

Answer 1

As per your question, I agree with your way of parsing data. But I would like to add on more to make it directly aggregatable & visualize using Kibana.
The approach should be :-
Filter the data using %{NUMBER:terminal} %{TIMESTAMP_ISO8601:ts} and (?([^\r])*) {As per information given by you}
Mutate
Filter
Then after using mutate & filter you will get data in terms of array {as you have mentioned}
Now you can add a field as level 1 by mentioning add_field => [ "fieldname", "%{[arrayname][0]}" ]
Now you can add a field as level 2 by mentioning add_field => [ "fieldname", "%{[arrayname][1]}" ]
Now you can add a field as level 3 by mentioning add_field => [ "fieldname", "%{[arrayname][2]}" ]
Then you can directly use Kibana to visualize such information.

Answer 2

my solution
input {
file {
path => "C:/Temp/zipped/*.txt"
start_position => beginning
ignore_older => 0
sincedb_path => "C:/temp/logstash_temp2.sincedb"
}
}
filter {
grok {
match => ["message","^%{NOTSPACE}\[%{NUMBER:terminal_id}\] %{NUMBER:log_level} %{NUMBER} %{TIMESTAMP_ISO8601:ts} \[(?<facility>([^\]]*))\] (?<lvl>([^$|\r])*)"]
}
mutate {
split => ["lvl", ":"]
add_field => {"lvl_1" => "%{lvl[0]}"}
add_field => {"lvl_2" => "%{lvl[1]}"}
add_field => {"lvl_3" => "%{lvl[2]}"}
add_field => {"lvl_4" => "%{lvl[3]}"}
add_field => {"lvl_5" => "%{lvl[4]}"}
add_field => {"lvl_6" => "%{lvl[5]}"}
add_field => {"lvl_7" => "%{lvl[6]}"}
add_field => {"lvl_8" => "%{lvl[7]}"}
lowercase => [ "terminal_id" ] # set to lowercase so that it can be used for index - additional filtering may be required
}
date {
match => ["ts", "YYYY-MM-DD HH:mm:ssZZ"]
}
}
filter {
if [lvl_1] =~ /%\{lvl\[0\]\}/ {mutate {remove_field => [ "lvl_1" ]}}
if [lvl_2] =~ /%\{lvl\[1\]\}/ {mutate {remove_field => [ "lvl_2" ]}}
if [lvl_3] =~ /%\{lvl\[2\]\}/ {mutate {remove_field => [ "lvl_3" ]}}
if [lvl_4] =~ /%\{lvl\[3\]\}/ {mutate {remove_field => [ "lvl_4" ]}}
if [lvl_5] =~ /%\{lvl\[4\]\}/ {mutate {remove_field => [ "lvl_5" ]}}
if [lvl_6] =~ /%\{lvl\[5\]\}/ {mutate {remove_field => [ "lvl_6" ]}}
if [lvl_7] =~ /%\{lvl\[6\]\}/ {mutate {remove_field => [ "lvl_7" ]}}
if [lvl_8] =~ /%\{lvl\[7\]\}/ {mutate {remove_field => [ "lvl_8" ]}}
mutate{
remove_field => [ "lvl","host","ts" ] # do not keep this data
}
}
output {
if [facility] == "mydata" {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-mydata-%{terminal_id}-%{+YYYY.MM.DD}"
}
} else {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-other-%{terminal_id}-%{+YYYY.MM.DD}"
}
}
# stdout { codec => rubydebug }
}