ccze (Log colorizer) shows no output when filtering certain strings. Why? - bash

Question

I am using a (bash) alias on an Ubuntu 22.04 system to quickly tail the system log without all the garbage that usually is of no importance. (I do want all this logged, but not displayed all the time.) The current alias looks something like this:
$ tail -f --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: (REFUSED unexpected RCODE|timed out) resolving)|\[INFO\] (New connection from ::1|Logout.)"
This works fine and give me some output:
Dec 7 13:32:15 root named[1044]: clients-per-query decreased to 20
Dec 7 13:32:29 root mariadbd[1188]: 2022-12-07 13:32:29 83187 [Warning] Aborted connection 83187 to db: 'XXX' user: 'XXX' host: 'localhost' (Got an error reading communication packets)
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.168.192.bl.blocklist.de/A/IN': 46.252.24.212#53
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.215.85.bl.blocklist.de/A/IN': 46.252.24.212#53
Dec 7 13:39:00 root systemd[1]: Starting Clean php session files...
Dec 7 13:39:03 root systemd[1]: phpsessionclean.service: Deactivated successfully.
Dec 7 13:39:03 root systemd[1]: Finished Clean php session files.
Dec 7 13:39:03 root systemd[1]: phpsessionclean.service: Consumed 3.081s CPU time.
Dec 7 13:52:15 root named[1044]: clients-per-query decreased to 19
Dec 7 13:53:06 root named[1044]: connection refused resolving '_.54.142.bl.blocklist.de/A/IN': 46.252.24.212#53
Dec 7 13:55:58 root mariadbd[1188]: 2022-12-07 13:55:58 83536 [Warning] Access denied for user 'xxx'#'localhost' (using password: YES)
Dec 7 13:55:58 root mariadbd[1188]: 2022-12-07 13:55:58 83537 [Warning] Access denied for user ''#'localhost' (using password: NO)
Dec 7 13:59:01 root kernel: [241023.546785] sh (1569030): drop_caches: 3
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> Received signal: wake up
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> ClamAV update process started at Wed Dec 7 14:04:19 2022
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> ^Your ClamAV installation is OUTDATED!
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> ^Local version: 0.103.6 Recommended version: 0.103.7
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> DON'T PANIC! Read https://docs.clamav.net/manual/Installing.html
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> daily.cld database is up-to-date (version: 26743, sigs: 2013590, f-level: 90, builder: raynman)
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
Dec 7 14:04:52 root named[1044]: validating _spf.eu.mailgun.org/TXT: no valid signature found
Dec 7 14:09:00 root systemd[1]: Starting Clean php session files...
Dec 7 14:09:03 root systemd[1]: phpsessionclean.service: Deactivated successfully.
Dec 7 14:09:03 root systemd[1]: Finished Clean php session files.
Dec 7 14:09:03 root systemd[1]: phpsessionclean.service: Consumed 3.038s CPU time.
Dec 7 14:09:38 root clamd[489]: Wed Dec 7 14:09:38 2022 -> SelfCheck: Database status OK.
However, when I pipe this output through ccze, I get no output at all:
$ tail -f --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: (REFUSED unexpected RCODE|timed out) resolving)|\[INFO\] (New connection from ::1|Logout.)" | \
ccze -A
It gets weirder: after removing the RCODE line from my exclusion list it works:
$ tail -f --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: timed out resolving)|\[INFO\] (New connection from ::1|Logout.)" | \
ccze -A
Dec 7 13:32:15 root named[1044]: clients-per-query decreased to 20
Dec 7 13:32:29 root mariadbd[1188]: 2022-12-07 13:32:29 83187 [Warning] Aborted connection 83187 to db: 'xxx' user: 'xxx' host: 'localhost' (Got an error reading communication packets)
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.168.192.bl.blocklist.de/A/IN': 46.252.24.212#53
(...)
And if I first cat the syslog to a file and then pipe it through ccze, it also works:
$ tail --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: (REFUSED unexpected RCODE|timed out) resolving)|\[INFO\] (New connection from ::1|Logout.)" \
> syslog-ccze.bug
$ cat syslog-ccze.bug | ccze -A
Dec 7 13:32:15 root named[1044]: clients-per-query decreased to 20
Dec 7 13:32:29 root mariadbd[1188]: 2022-12-07 13:32:29 83187 [Warning] Aborted connection 83187 to db: 'xxx' user: 'xxx' host: 'localhost' (Got an error reading communication packets)
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.168.192.bl.blocklist.de/A/IN': 46.252.24.212#53
(...)
For some reason, ccze doesn't like me filtering out the "unexpected RCODE" named messages. Am I missing something obvious, or am I hitting some weird race condition in ccze, or what is happening?

Answer 1

Bah. Once you take the time to actually write down your question, you'll find answers by yourself. :-(
ccze apparently has an internal buffer. It needs a minimum amount of lines before outputting ANYTHING. Increasing the --lines= parameter for tail in my alias helped.
This seems to be undocumented behaviour, so I'll leave the question here to be searchable. Hope it helps and saves you some time. :-)