logstash remove_field not working in order to upload csv to elasticsearch - elasticsearch

Question

I'm using elasticsearch, kibana and logstash 6.0.1.
I wish to upload csv data to elasticsearch by logstash and removing fields (path, #timestamp, #version, host and message). I'm showing logstash.conf and emp.csv files below. The upload will work if I don't use the remove_field instruction but I need to. Furthermore, the index was not created.
logstash.conf:
input {
file {
path => "e:\emp.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["code","color"]
remove_field => ["path", "#timestamp", "#version", "host", "message"]
}
mutate {convert => ["code", "string"]}
mutate {convert => ["color", "string"]}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "emp5"
user => "elastic"
password => "password"
}
stdout {}
}
emp.csv:
1,blue
2,red
What is missing in this case?

Answer 1

In your csv file the data is not available that you are trying to delete.
Instead try this to delete for example path and host field:
(...)
filter {
csv {
separator => ","
columns => ["code","color"]
}
mutate {
remove_field => ["path", "host"]
}
(...)
And for information, if field path and/or host doesn't exist, there's no problem. The plugin will remove field if field exists, and just do nothing if field does not exist.
Edit:
I have tested it on fresh elastic stack:
You can delete index with:
curl -X DELETE "localhost:9200/emp5"
Also note that in your current config logstash will read the file only once.
You can change that behaviour by adding sincedb_path => "/dev/null"
or in Windows case: sincedb_path => "NUL" inside:
input {
file {
(...) # here
}
section.
Then after logstash work verify result with:
curl -X GET "localhost:9200/emp5?pretty"
{
"emp5" : {
"aliases" : { },
"mappings" : {
"doc" : {
"properties" : {
"#timestamp" : {
"type" : "date"
},
"#version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"code" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"color" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
},
"settings" : {
"index" : {
"number_of_shards" : "5",
"blocks" : {
"read_only_allow_delete" : "true"
},
"provided_name" : "emp5",
"creation_date" : "1576099826712",
"number_of_replicas" : "1",
"uuid" : "reXYzqPgQryYcASoov9l5A",
"version" : {
"created" : "6080599"
}
}
}
}
}
As you can see there is no host and path field.