Content repeat collecting problems while use filebeat - elasticsearch

Question

Recently we will use filebeat to collect our system logs to elasticsearch vias:
${local_log_file} -> filebeat -> kafka -> logstash -> elasticsearch -> kibana
While testing our system, we found a scenario that filebeat will repeatly collect logs which means that it will collect logs from the start of file once there is a change.
here is my configuration for filebeat:
filebeat.prospectors:
- input_type: log
paths:
- /home/XXX/exp/*.log
scan_frequency: 1s
#tail_files: true
#================================ Outputs =====================================
#----------------------------- Logstash output --------------------------------
# output.logstash:
# hosts: ["localhost:5044"]
#----------------------------- Kafka output -----------------------------------
output.kafka:
enabled: true
hosts: ["10.10.1.103:9092"]
topic: egou
#----------------------------- console output --------------------------------
output.console:
enabled: true
pretty: true
Notice:
we construct the log files manually, and we are sure that there is a blank line at the end of file
to make a console, we open the output.console
once there is content appended to the end of log file, filebeat will collect from the beginning of the file.But we hope just fetching the change of file.
filebeat version is 5.6.X
Hope any useful hint can be offered by u all

Answer 1

I think this is because of the editor, you are using, creates a new file on save with new meta-data. Filebeat identifies the state of the file using its meta-data, not the content.
try,
echo "something" >> /path/to/file.log
ref: https://discuss.elastic.co/t/filebeat-repeatedly-sending-old-entries-in-log-file/55796