Logstash stuck problem on pipeline running - elasticsearch

Question

I am trying to read logs to load them into elasticsearch using logstash im running it in a RHEL 7.9 integrated with elasticsearch and Kibana, but when I run it it stops at:
[INFO ] 2021-10-22 13:40:00.704 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
My file config is:
file {
path => [
"/home/logstash/connectors.log"
]
start_position => "beginning"
}
}
filter {
grok {
break_on_match => false
match => {
"message" => "%{TIMESTAMP_ISO8601:fecha} \[(?<threadname>[^\]]+)\] %{LOGLEVEL:loglevel}\s*\(%{JAVAFILE:file}:%{INT:line}\)\s*-\s*Dato\s*a\s*enviar:\s*\[%{GREEDYDATA:xml}\]"
}
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:8087"]
document_id => "%{[#metadata][fingerprint]}"
index => "wilobank-%{+YYYY.MM.dd}"
}
}
And the log of execution is:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-10-22 13:48:29.490 [main] runner - Starting Logstash {"logstash.version"=>"7.15.1", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.12+7 on 11.0.12+7 +indy +jit [linux-x86_64]"}
[WARN ] 2021-10-22 13:48:29.818 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-10-22 13:48:30.990 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2021-10-22 13:48:31.589 [Converge PipelineAction::Create<main>] Reflections - Reflections took 73 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2021-10-22 13:48:32.589 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-10-22 13:48:32.648 [Converge PipelineAction::Create<main>] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-10-22 13:48:32.970 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-10-22 13:48:33.021 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2021-10-22 13:48:33.085 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:8087"]}
[INFO ] 2021-10-22 13:48:33.418 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:8087/]}}
[WARN ] 2021-10-22 13:48:33.566 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:8087/"}
[INFO ] 2021-10-22 13:48:33.616 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.15.1) {:es_version=>7}
[WARN ] 2021-10-22 13:48:33.618 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[WARN ] 2021-10-22 13:48:33.705 [[main]-pipeline-manager] grok - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2021-10-22 13:48:33.741 [Ruby-0-Thread-10: :1] elasticsearch - Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[INFO ] 2021-10-22 13:48:33.880 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/home/logstash/test.conf"], :thread=>"#<Thread:0x7a96f84a run>"}
[INFO ] 2021-10-22 13:48:34.785 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.9}
[INFO ] 2021-10-22 13:48:34.840 [[main]-pipeline-manager] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_a171bd20c3269483fada27f50b68caf2", :path=>["/home/logstash/itecban-connectors.log"]}
[INFO ] 2021-10-22 13:48:34.860 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-10-22 13:48:34.897 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2021-10-22 13:48:34.916 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Can someone help me?
Thanks in advance

Answer 1

If you want to parse again a complete file, you need to :
delete sindedb files
OR only delete the corresponding line in sincedb file
Then, restart Logstash. Logstash will reparse the file.
For more info: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#sincedb_path