How to check if specific index exists in EKL using logstash pipeline? - elasticsearch

Question

I want to write a logstash pipeline to check if specific index exists or not in ES env; if yes then mark incoming event as "valid", else "invalid".
To check index validity using cURL :
curl -u elastic:elastic -I http://localhost:9200/sampletest1
valid output - HTTP/1.1 200 OK
invalid output - HTTP/1.1 400 Not Found
My logstash script:
input {
beats {
port => "5044"
}
}
filter {
#execute curl to check for index http://localhost:9200/%{process-code}
#if response has 200 then mutate with add_tags "valid". else add tag "invalid"
if "valid" in [tags] {
} else {
#delete event; prevent it from going to output section
}
}
output {
#print only valid events
stdout {
codec => rubydebug
}
}
I am stuck at 2 # lines mentioned in filter section. We cant use exec plugin in filter section !

Answer 1

Solved it using "http filter plugin" as suggested by Badger in comments.
filter {
json { source => "message"}
http {
url => "http://localhost:9200/%{process-code}"
verb => "HEAD"
body_format => "json"
user => "elastic"
password => "elastic"
}
if "_httprequestfailure" in [tags] {
#index not present. drop will prevent it from going to output section
drop {}
} else {
#index present
mutate { add_tag => [ "found" ]}
}
}
Note: http plugin above adds _jsonparsefailure tag in output event; to avoid it we can use tag_on_json_failure=>[].
Ref: https://discuss.elastic.co/t/http-filter-plugin-adds-jsonparsefailure-in-tag/277744