Split a message using grok - elasticsearch

Question

I have logs in the format:
2018-09-17 15:24:34;Count of files in error folder in;C:\Scripts\FOLDER\SUBFOLDER\error;1
I want to put in a separate field the path to the folder and the number after.
Like
dirTEST=C:\Scripts\FOLDER\SUBFOLDER\
count.of.error.filesTEST=1
or
dir=C:\Scripts\FOLDER\SUBFOLDER\
count.of.error.files=1
I use for this grok pattern in logstash config:
if "TestLogs" in [tags] {
grok{
match => { "message" => "%{DATE:date_in_log}%{SPACE}%{TIME:time.in.log};%{DATA:message.text.log};%{WINPATH:dir};%{INT:count.of.error.files}" }
add_field => { "dirTEST" => "%{dir}" }
add_field => { "count.of.error.filesTEST" => "%{count.of.error.files}" }
}
}
No errors in logstash logs.
But in the Kibana I get the usual log without new fields.

Answer 1

A couple of notes here. First of all, it must be said that the solution seems to be doing what you expect, so probably the problem is that your Index Pattern has not been updated with the new fields. To do so in Kibana you can go to Management -> Kibana -> Index Patterns and refresh the field list in the upper right corner (Next to the delete Index Pattern button).
Second is that you must take into account that using points to separate the terms makes the structured data look like this:
{
"date_in_log": "18-09-17",
"count": {
"of": {
"error": {
"files": "1"
}
}
},
"time": {
"in": {
"log": "15:24:34"
}
},
"message": {
"text": {
"log": "Count of files in error folder in"
}
},
"dir": "C:\\Scripts\\FOLDER\\SUBFOLDER\\error"
}
I don't know if this is how you want your data to be represented, but maybe you should consider other solution changing the naming of the fields in the grok pattern.