How to remove first few lines of CSV in Logstash - elasticsearch

Question

This is the input I am using for logstash.
ItemId,AssetId,ItemName,Comment
11111,07,ABCDa,XYZa
11112,07,ABCDb,XYZb
11113,07,ABCDc,XYZc
11114,07,ABCDd,XYZd
11115,07,ABCDe,XYZe
11116,07,ABCDf,XYZf
11117,07,ABCDg,XYZg
Date,Time,Mill Sec,rows,columns
19-05-2020,13:03:46,534,2,2
19-05-2020,13:03:46,539,2,2
19-05-2020,13:03:46,544,2,2
19-05-2020,13:03:46,549,2,2
19-05-2020,13:03:46,554,2,2
I need to remove first 8 lines from the csv and make the next line as column header and parse rest of lines as usual. Is there a way to do that in logstash?

Answer 1

You could do this using the file input and then read it line by line using grok to make sure it has the right amount of fields comma separated and ignore the header one
Your input will look like this:
input {
file {
path => "/path/to/my.csv"
start_position => beginning
}
}
This will read each line into an event with the data in the field named message and then send it to your filters.
In your filter you'll use grok with a pattern like this:
filter {
grok {
match => { "message" => [
"^%{DATE:Date},%{TIME:Time},%{NUMBER:Mill_Sec},%{NUMBER:rows},%{NUMBER:colums}$"
]
}
}
}
This will present each line as an event looking like this:
{
"colums": "2",
"Time": "13:03:46",
"Mill_Sec": "554",
"rows": "2",
"Date": "19-05-2020"
}
You can use mutate to remove unwanted fields (like message) prior to going to your output part. If there is no match with the pattern defined you'll get a tag with the value _grokparsefailure in your tags, you can use that to decide to send it to your output or not. As you defined that it has to be numbers, it will also fail on the header one and thus leave you with only 'real' events.
This can be done by having your output defined like this:
output {
if "_grokparsefailure" not in [tags] {
elasticsearch {
...
}
}
}

Answer 2

You should do this before the file gets to Logstash. There are ways to do it within Logstash, for example by using a mutliline code then doing exotic grok matches to remove the first N lines (or removing lines until a particular regex), then doing a split followed by a plain ol' csv filter. You need to be even more careful than usual with header rows. It's a big mess.
Much better to put something in front of Logstash to handle this issue.
If the files are local to your logstash instance, you could use the Exec input plugin to deal with the irregularities.
input {
exec {
command => "/path/to/command_or_script" # sh or py or js etc
interval => 60
}
}
On Linux, this command will print a file from the 8th line on...
command => "tail +8 /path/to/file"
This one (again for Linux) will drop everything until a line that starts with date, and print everything after that
command => "sed -n -e '/^date/,$p' /path/to/file"
You can avoid read the same file over and over again by deleting or archiving it in a script (rather than a one-liner as used in these examples)
After trimming the unwanted leading lines, you should be able to use the csv filter in a normal way.
Note that if you want to autodetect_column_names that pipeline workers must be set to 1.

Answer 3

Your content is not CSV format. Your task is convert it to true CSV format.