Filebeat does not collecting logs? - elasticsearch

Question

Problem
Ubuntu 18.04. The logs for some files not sending. For example I have has 16 log files on 2020-06-23. But only #5 & #8 got collected into data.json. Others are not found in data.json
Here's a script I use to found files on disk but not in data.json.
sudo python -c '
import json; import os;
raw = os.listdir("/path/to/my/logdir")
f = open("/var/lib/filebeat/registry/filebeat/data.json", "r")
data=json.load(f)
harvested=[d["source"].split("/")[-1] for d in data]
substraction=[x for x in raw if not x in harvested]
print("\n".join(substraction))
'
The script result, there's just a lot:
app-2020-06-21.20.log
app-2020-06-21.25.log
app-2020-06-23.11.log
app-2020-06-22.1.log
app-2020-06-22.48.log
app-2020-06-21.41.log
app-2020-06-23.15.log
...
And there are only 2 types of logs End of file reached and Non-zero metrics in the last 30s :
Jun 23 12:23:12 filebeat[32738]: 2020-06-23T12:23:12.223Z DEBUG [harvester] log/log.go:107 End of file reached: /path/to/my/logdir/app-2020-06-21.43.log; Backoff now.
Jun 23 12:23:12 filebeat[32738]: 2020-06-23T12:23:12.344Z DEBUG [harvester] log/log.go:107 End of file reached: /path/to/my/logdir/app-2020-06-22.9.log; Backoff now.
Jun 23 12:23:12 filebeat[32738]: 2020-06-23T12:23:12.364Z DEBUG [harvester] log/log.go:107 End of file reached: /path/to/my/logdir/app-2020-06-21.34.log; Backoff now.
Jun 23 12:23:12 filebeat[32738]: 2020-06-23T12:23:12.444Z DEBUG [harvester] log/log.go:107 End of file reached: /path/to/my/logdir/app-2020-06-23.5.log; Backoff now.
Jun 23 12:23:15 filebeat[32738]: 2020-06-23T12:23:15.144Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":350,"time":{"ms":10}},"total":{"ticks":1710,"ti
Jun 23 12:23:45 filebeat[32738]: 2020-06-23T12:23:45.144Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":350,"time":{"ms":9}},"total":{"ticks":1720,"tim
Jun 23 12:24:15 filebeat[32738]: 2020-06-23T12:24:15.144Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":360,"time":{"ms":6}},"total":{"ticks":1740,"tim
Jun 23 12:24:45 filebeat[32738]: 2020-06-23T12:24:45.144Z
Details
Filebeat config
output:
logstash:
enabled: true
hosts:
- x.x.x.x:5044
filebeat:
inputs:
-
paths:
- "/path/to/log/dir/*"
document_type: myapp
multiline.pattern: '^[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
multiline.negate: true
multiline.match: after
clean_removed: true
close_removed: true
logging.level: debug
name: "myapp"
tags: ["tag1", "tag2"]
Disk
Here's is the usage. I guess disk is fine
Filesystem Size Used Avail Use% Mounted on
... 2.0G 0 2.0G 0% /dev
... 395M 820K 394M 1% /run
... 30G 12G 18G 41% /
Here's the inode checking result. You can see that there's no duplicated inode in 06-23 logs.
ls -il * | grep 06-23
768289 -rw-r--r-- 1 root root 10485996 Jun 23 00:33 app-2020-06-23.0.log
768372 -rw-r--r-- 1 root root 10486447 Jun 23 01:02 app-2020-06-23.1.log
768292 -rw-r--r-- 1 root root 10485819 Jun 23 05:36 app-2020-06-23.10.log
800654 -rw-r--r-- 1 root root 10499153 Jun 23 05:59 app-2020-06-23.11.log
794052 -rw-r--r-- 1 root root 10486575 Jun 23 06:32 app-2020-06-23.12.log
768487 -rw-r--r-- 1 root root 10492683 Jun 23 06:59 app-2020-06-23.13.log
800633 -rw-r--r-- 1 root root 10490445 Jun 23 07:27 app-2020-06-23.14.log
794067 -rw-r--r-- 1 root root 10500849 Jun 23 07:55 app-2020-06-23.15.log
788191 -rw-r--r-- 1 root root 10489159 Jun 23 08:28 app-2020-06-23.16.log
788410 -rw-r--r-- 1 root root 10486744 Jun 23 09:30 app-2020-06-23.17.log
800624 -rw-r--r-- 1 root root 10486794 Jun 23 10:00 app-2020-06-23.18.log
794048 -rw-r--r-- 1 root root 10490002 Jun 23 10:39 app-2020-06-23.19.log
768461 -rw-r--r-- 1 root root 10486161 Jun 23 01:36 app-2020-06-23.2.log
794051 -rw-r--r-- 1 root root 10488204 Jun 23 11:12 app-2020-06-23.20.log
794081 -rw-r--r-- 1 root root 10487146 Jun 23 11:46 app-2020-06-23.21.log
794071 -rw-r--r-- 1 root root 10492866 Jun 23 12:16 app-2020-06-23.22.log
787673 -rw-r--r-- 1 root root 10490849 Jun 23 12:51 app-2020-06-23.23.log
787698 -rw-r--r-- 1 root root 3491076 Jun 23 13:00 app-2020-06-23.24.log
768478 -rw-r--r-- 1 root root 10486306 Jun 23 02:08 app-2020-06-23.3.log
768507 -rw-r--r-- 1 root root 10486690 Jun 23 02:34 app-2020-06-23.4.log
800620 -rw-r--r-- 1 root root 10496353 Jun 23 03:00 app-2020-06-23.5.log
800623 -rw-r--r-- 1 root root 10503668 Jun 23 03:36 app-2020-06-23.6.log
768521 -rw-r--r-- 1 root root 10520722 Jun 23 04:05 app-2020-06-23.7.log
774652 -rw-r--r-- 1 root root 10487379 Jun 23 04:38 app-2020-06-23.8.log
784704 -rw-r--r-- 1 root root 10553972 Jun 23 05:05 app-2020-06-23.9.log