Logstash - reading only appended data in file - elasticsearch

Question

I'm learning how to use logstash and I'm facing some problems in reading a file with logstash which is constantly updated. Here is my test:
logstash.conf
input {
file {
path => ["/usr/share/logs_data/first_log_test.log"]
start_position => "beginning"
}
}
filter {
grok {
match => ["message", "(?<execution_date>\d{4}-\d{2}-\d{2}) (?<execution_time>\d{2}:\d{2}:\d{2})%{GREEDYDATA}ParaBrutos/configs/pipelines/(?<crawler_category>([^/])+)/(?<crawler_subcategory>([^-])+)-(?<crawler_name>([^.])+).json"]
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => "elastic"
password => "changeme"
ecs_compatibility => disabled
index => "logs_second"
}
}
First, I use this repo's code, for installing ELK stack in a dockerized way.
I started with this test log file empty, and then added few lines that maches the pattern, one by one, in a text editor, simultaneously seeing the index patterns in kibana updating. However, each new row I add in this test log is not added alone, and I see hits of old entries in kibana index patterns.
What happens?
Could it be anything related to not selecting time filters in index pattern creation?
Or something related to since_db settings? But what? Because by default, isn't it supposed to save the last read position of the file?
Or something related to start_position? Even though, by the plugin documentation, it is supposed to make effect only when the first read of the file is done?
I'm a bit lost, tried lots of things and still am not understanding well what's happening. Could you help me?

Answer 1

If you are using a text editor then you are probably creating a new file each time you exit it.
That could be an inode reuse issue. There are links to various issues in the META issue 211. Especially see 251.
Tracking which files have been read when those files can get rotated is an extremely hard problem. Way harder than most folks would initially think. A good option to get it right is to checksum the file contents (although this is not foolproof). The file input does not do that, because it can get ridiculously expensive. Instead it implements a very cheap technique that almost always gets it right (but in a few cases it decides it has already read a file that it has not read).
There are other cases where it gets it wrong by duplicating data (which is what you are hitting). As I said, it is a really hard problem.