Remove nested field with NULL value before insert to Elasticsearch - elasticsearch

Question

I'm using Metricbeat to query my database and ship the result to Logstash. One of the queries returns a date column that might contain a NULL value , lets call it "records_last_failure".
I'm getting the following error in logstash :
[2020-12-29T15:45:59,077][WARN ][logstash.outputs.elasticsearch][my_pipeline] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"my-index", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x4f7d2f63>], :response=>{"index"=>{"_index"=>"my-index", "_type"=>"_doc", "_id"=>"wRItr3YBUsMlz2Mnbnlz", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [sql.metrics.string.records_last_failure] of type [date] in document with id 'wRItr3YBUsMlz2Mnbnlz'. Preview of field's value: 'NULL'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [NULL] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}
My goal is to remove this field before trying to save the document into Elasticsearch. I tried to use the following two filters in Logstash but both of them didn't help and I still keep getting the error mentioned above. The filters :
filter {
......
......
if "[sql.metrics.string.records_last_failure]" == "NULL" {
mutate {
remove_field => [ "[sql.metrics.string.records_last_failure]" ]
}
}
if "[sql][metrics][string][records_last_failure]" == "NULL" {
mutate {
remove_field => [ "[sql][metrics][string][records_last_failure]" ]
}
}
}

Answer 1

Not sure why or how, but after a few restarts logstash started working well.
My json looked like the following example :
"a" : 1,
"b" : 2,
"sql" : {
"metrics" : {
"string" : {
"record_last_failure" : "NULL",
"other_field" : 5
}
}
}
I wanted to remove the field sql.metrics.string.record_last_failure if it was set to null. The syntax that I used in my filter :
if "[sql][metrics][string][records_last_failure]" == "NULL" {
mutate {
remove_field => [ "[sql][metrics][string][records_last_failure]" ]
}