Data ingestion got stuck though the Logstash pipelines are running - elasticsearch

Question

We have deployed 10 logstash config files last year. We started all the config files at once by using the folder where we have kept all the config files(as service). On 24th May, we found that, few of them are not pushing data in the elasticsearch, checked from Kibana Discover. Though rest of them were working fine. We checked the status of all 3 ELK components first (by systemctl status), found that all 3 services are running. We have then checked logs of logstash by using journalctl but did not find anything that caused the issue. We then stopped all the PIDs that were running for logstash and then started the logstash pipelines and that fixed the issue.
Please find below the portion of the logs from journalctl:
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,503][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>http://localhost:9200/}
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,619][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,623][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
Aug 26 12:24:22 xyz.com systemd[1]: Stopping logstash...
Aug 26 12:24:22 xyz.com logstash[827]: [2020-08-26T12:24:22,665][WARN ][logstash.runner ] SIGTERM received. Shutting down.
Aug 26 12:24:27 xyz.com logstash[827]: [2020-08-26T12:24:27,864][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>33, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-5.1.9-java/lib/logstash/inputs/beats.rb:212:in `run'"}, {"thread_id"=>25, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>26, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>27, "name"=>"[main]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>28, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>29, "name"=>"[main]>worker4", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>30, "name"=>"[main]>worker5", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>31, "name"=>"[main]>worker6", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>32, "name"=>"[main]>worker7", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}]}}
Aug 26 12:24:27 xyz.com logstash[827]: [2020-08-26T12:24:27,866][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
Aug 26 12:24:29 xyz.com logstash[827]: [2020-08-26T12:24:29,852][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x2c879c55 run>"}
Aug 26 12:24:29 xyz.com logstash[827]: [2020-08-26T12:24:29,855][INFO ][logstash.runner ] Logstash shut down.
Aug 26 12:24:29 xyz.com systemd[1]: Stopped logstash.
May 24 08:11:46 xyz.com systemd[1]: Started logstash.
May 24 08:12:11 xyz.com logstash[19174]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
May 24 08:12:11 xyz.com logstash[19174]: 2021-05-24 08:12:11,794 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.open0(Native Method)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
May 24 08:12:11 xyz.com logstash[19174]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:640)
May 24 08:12:25 xyz.com logstash[19174]: [2021-05-24T08:12:25,120][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>
May 24 08:12:25 xyz.com logstash[19174]: [2021-05-24T08:12:25,363][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Main process exited, code=killed, status=9/KILL
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Failed with result 'signal'.
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1.
May 24 15:38:50 xyz.com systemd[1]: Stopped logstash.
May 24 15:38:50 xyz.com systemd[1]: Started logstash.
May 24 15:39:06 xyz.com logstash[25666]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
May 24 15:39:07 xyz.com logstash[25666]: 2021-05-24 15:39:07,101 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFo
May 24 15:39:07 xyz.com logstash[25666]: at java.io.FileOutputStream.open0(Native Method)
May 24 15:39:07 xyz.com logstash[25666]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
Server OS: Ubuntu 18.04
ELK Version: 7.11.1
Please help us to find the cause of the issue as we are struggling to locate the issue.

Answer 1

Focus at
java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
Check your file/folder permission (chmod: read, write, execute).
Use root account, check permission
ls -l /path/to/file
ls -l /var/log/logstash/logstash-plain.log
See more at https://askubuntu.com/a/528433/299516