I am using a (bash) alias on an Ubuntu 22.04 system to quickly tail the system log without all the garbage that usually is of no importance. (I do want all this logged, but not displayed all the time.) The current alias looks something like this:
$ tail -f --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: (REFUSED unexpected RCODE|timed out) resolving)|\[INFO\] (New connection from ::1|Logout.)"
This works fine and give me some output:
Dec 7 13:32:15 root named[1044]: clients-per-query decreased to 20
Dec 7 13:32:29 root mariadbd[1188]: 2022-12-07 13:32:29 83187 [Warning] Aborted connection 83187 to db: 'XXX' user: 'XXX' host: 'localhost' (Got an error reading communication packets)
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.168.192.bl.blocklist.de/A/IN': 46.252.24.212#53
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.215.85.bl.blocklist.de/A/IN': 46.252.24.212#53
Dec 7 13:39:00 root systemd[1]: Starting Clean php session files...
Dec 7 13:39:03 root systemd[1]: phpsessionclean.service: Deactivated successfully.
Dec 7 13:39:03 root systemd[1]: Finished Clean php session files.
Dec 7 13:39:03 root systemd[1]: phpsessionclean.service: Consumed 3.081s CPU time.
Dec 7 13:52:15 root named[1044]: clients-per-query decreased to 19
Dec 7 13:53:06 root named[1044]: connection refused resolving '_.54.142.bl.blocklist.de/A/IN': 46.252.24.212#53
Dec 7 13:55:58 root mariadbd[1188]: 2022-12-07 13:55:58 83536 [Warning] Access denied for user 'xxx'#'localhost' (using password: YES)
Dec 7 13:55:58 root mariadbd[1188]: 2022-12-07 13:55:58 83537 [Warning] Access denied for user ''#'localhost' (using password: NO)
Dec 7 13:59:01 root kernel: [241023.546785] sh (1569030): drop_caches: 3
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> Received signal: wake up
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> ClamAV update process started at Wed Dec 7 14:04:19 2022
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> ^Your ClamAV installation is OUTDATED!
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> ^Local version: 0.103.6 Recommended version: 0.103.7
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> DON'T PANIC! Read https://docs.clamav.net/manual/Installing.html
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> daily.cld database is up-to-date (version: 26743, sigs: 2013590, f-level: 90, builder: raynman)
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
Dec 7 14:04:19 root freshclam[1929]: Wed Dec 7 14:04:19 2022 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
Dec 7 14:04:52 root named[1044]: validating _spf.eu.mailgun.org/TXT: no valid signature found
Dec 7 14:09:00 root systemd[1]: Starting Clean php session files...
Dec 7 14:09:03 root systemd[1]: phpsessionclean.service: Deactivated successfully.
Dec 7 14:09:03 root systemd[1]: Finished Clean php session files.
Dec 7 14:09:03 root systemd[1]: phpsessionclean.service: Consumed 3.038s CPU time.
Dec 7 14:09:38 root clamd[489]: Wed Dec 7 14:09:38 2022 -> SelfCheck: Database status OK.
However, when I pipe this output through ccze, I get no output at all:
$ tail -f --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: (REFUSED unexpected RCODE|timed out) resolving)|\[INFO\] (New connection from ::1|Logout.)" | \
ccze -A
It gets weirder: after removing the RCODE line from my exclusion list it works:
$ tail -f --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: timed out resolving)|\[INFO\] (New connection from ::1|Logout.)" | \
ccze -A
Dec 7 13:32:15 root named[1044]: clients-per-query decreased to 20
Dec 7 13:32:29 root mariadbd[1188]: 2022-12-07 13:32:29 83187 [Warning] Aborted connection 83187 to db: 'xxx' user: 'xxx' host: 'localhost' (Got an error reading communication packets)
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.168.192.bl.blocklist.de/A/IN': 46.252.24.212#53
(...)
And if I first cat the syslog to a file and then pipe it through ccze, it also works:
$ tail --lines=200 /var/log/syslog | \
egrep -v "(named\[.*\]: (REFUSED unexpected RCODE|timed out) resolving)|\[INFO\] (New connection from ::1|Logout.)" \
> syslog-ccze.bug
$ cat syslog-ccze.bug | ccze -A
Dec 7 13:32:15 root named[1044]: clients-per-query decreased to 20
Dec 7 13:32:29 root mariadbd[1188]: 2022-12-07 13:32:29 83187 [Warning] Aborted connection 83187 to db: 'xxx' user: 'xxx' host: 'localhost' (Got an error reading communication packets)
Dec 7 13:32:39 root named[1044]: connection refused resolving '_.168.192.bl.blocklist.de/A/IN': 46.252.24.212#53
(...)
For some reason, ccze doesn't like me filtering out the "unexpected RCODE" named messages. Am I missing something obvious, or am I hitting some weird race condition in ccze, or what is happening?
Bah. Once you take the time to actually write down your question, you'll find answers by yourself. :-(
ccze apparently has an internal buffer. It needs a minimum amount of lines before outputting ANYTHING. Increasing the --lines= parameter for tail in my alias helped.
This seems to be undocumented behaviour, so I'll leave the question here to be searchable. Hope it helps and saves you some time. :-)
after upgrading via mc command i get this error when i try to login to the (kind of new) minio console:
Post "https://fqdn.org/": dial tcp 127.0.1.1:443: connect: connection refused
I have a signed and valid SSL Certificate.
Downgrading minio (aka restore Snapshot of VM) solves the problem.
Any ideas?
This is my config:
MINIO_SERVER_URL="https://fqdn.org"
MINIO_ACCESS_KEY="key"
MINIO_VOLUMES="/mnt/hdd2/minio/"
MINIO_OPTS="-C /etc/minio --address :9000 --console-address :9001"
MINIO_SECRET_KEY="minio"
This is my minio startup log:
● minio.service - MinIO
Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-11-11 08:41:14 CET; 4min 50s ago
Docs: https://docs.min.io
Process: 3567 ExecStartPre=/bin/bash -c if [ -z "${MINIO_VOLUMES}" ]; then echo "Variable MINIO_VOLUMES not set in /etc/default/minio"; exit 1; fi (code=exited, status=0/SUCCESS)
Main PID: 3568 (minio)
Tasks: 9 (limit: 2351)
Memory: 101.9M
CGroup: /system.slice/minio.service
└─3568 /home/minio/minio server -C /etc/minio --address :9000 --console-address :9001 /mnt/hdd2/minio/
Nov 11 08:41:14 pmit-minio-test systemd[1]: Starting MinIO...
Nov 11 08:41:14 pmit-minio-test systemd[1]: Started MinIO.
Nov 11 08:41:17 pmit-minio-test minio[3568]: WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
Nov 11 08:41:17 pmit-minio-test minio[3568]: Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
Nov 11 08:41:17 pmit-minio-test minio[3568]: API: https://fqdn.org
Nov 11 08:41:17 pmit-minio-test minio[3568]: Console: https://191.164.213.7:9001 https://127.0.0.1:9001
Nov 11 08:41:17 pmit-minio-test minio[3568]: Documentation: https://docs.min.io
Please see the answer here:
https://github.com/minio/minio/issues/13639#issuecomment-966244704
I had to change this line:
MINIO_SERVER_URL="https://fqdn.org:9000"
Hi after downloading the latest rpm for CentIS and and installing for the first time I am getting this error in the logs:
Jun 22 09:47:31 ssd316r.simpleservers.co.uk systemd[1]: Starting Elasticsearch...
Jun 22 09:47:32 ssd316r.simpleservers.co.uk systemd-entrypoint[2501]: ERROR: Temporary file directory [/usr/share/elasticsearch/tmp] does not exist or is not accessible
Jun 22 09:47:32 ssd316r.simpleservers.co.uk systemd[1]: elasticsearch.service: main process exited, code=exited, status=78/n/a
Jun 22 09:47:32 ssd316r.simpleservers.co.uk systemd[1]: Failed to start Elasticsearch.
Jun 22 09:47:32 ssd316r.simpleservers.co.uk systemd[1]: Unit elasticsearch.service entered failed state.
Jun 22 09:47:32 ssd316r.simpleservers.co.uk systemd[1]: elasticsearch.service failed.
Error is due to below log:
Jun 22 09:47:32 ssd316r.simpleservers.co.uk systemd-entrypoint[2501]:
ERROR: Temporary file directory [/usr/share/elasticsearch/tmp] does
not exist or is not accessible
Can you check /usr/share/elasticsearch/tmp is present on your server or not, if not please create this folder at the same location and make sure your elasticsearch process has write access to it.
Dear StackOverflow community,
I was running Kibana/Elasticsearch without a problem until installing a Kibana plugin. Then the service failed and I noticed that the problem is that Elasticsearch stopped. I tried several ways to fix it, and then even reinstalled all. But the problem still avoiding to launch Elasticsearch, even with a fresh installation.
Installation on Debian 9 using apt install.
systemctl start elasticsearch.service
results on:
Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
[0.000s][error][logging] Error opening log file '/gc.log': Permission denied
Full log with journalctl -xe
-- Unit elasticsearch.service has begun starting up.
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"Unable to revive connection: http://localhost:9200/"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"No living connections"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"Unable to revive connection: http://localhost:9200/"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"No living connections"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: output:
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: [0.000s][error][logging] Error opening log file '/gc.log': Permission denied
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: [0.000s][error][logging] Initialization of output 'file=/var/log/elasticsearch/gc.log' using options 'filecount=32,filesize=64m' failed.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: error:
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Error: Could not create the Java Virtual Machine.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Error: A fatal exception has occurred. Program will exit.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:118)
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:86)
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:92)
Feb 07 14:09:06 Debian-911-stretch-64-minimal systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Feb 07 14:09:06 Debian-911-stretch-64-minimal systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit elasticsearch.service has failed.
The mentioned gc.log file was not in that folder. And the permissions were:
drwxr-s--- 2 elasticsearch elasticsearch 4096 Jan 15 13:20 elasticsearch
I created the file and also played with permissions until having these:
-rwxrwxrwx 1 root elasticsearch 0 Feb 7 15:19 gc.log
...and even changed the ownership:
-rwxrwxrwx 1 root root 0 Feb 7 15:19 gc.log
But no success, I still having the same issue.
Thanks
Make sure you are running CMD as Administrator.
This error also happens if you are using docker & running the container as a different user. You have to add --group_add flag to docker command or set TAKE_FILE_OWNERSHIP environment variable as mentioned here
Using docker-compose:
user: 1007:1007
group_add:
- 0
Using docker:
--group-add 0
Firstly, I didn't know why gc.log file was not present. Have you changed the logs folder path or something? The gc.log path can be set in jvm.options file. By default ES logs and java garbage collection logs are fed into the logs folder inside $ES_HOME directory.
About user perspective, elastic search can't be run as root user. So from the ES directory details its showing you have an elasticsearch user created, and trying to run the cluster by that user.
The problem here can be solved by changing the permissions of files insdie the ES directory where all it belongs. Now the gc.log file is owned by root user and it cannot be accessed by the elasticsearch user.
Try this: sudo chown <user> <path/to/es/directory> -R
Here it becomes : sudo chown elasticsearch elasticsearch/ -R
If the issue still persists, check the jvm.options file whether its all configured correctly. Unless you change the -Xloggc:logs/gc.log option, the gc.log won't be pushing to /var/log.
Feb 09 17:09:02 server elasticsearch[2199]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Your log says, the option is given as file=/var/log/elasticsearch/gc.log. Correct any wrong configurations as per documentation : https://www.elastic.co/guide/en/elasticsearch/reference/master/jvm-options.html
sudo systemctl -l status elasticsearch.service
Returns this log:
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─override.conf
Active: failed (Result: exit-code) since Sun 2020-02-09 17:09:02 CET; 2min 48s ago
Docs: http://www.elastic.co
Process: 2199 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 2199 (code=exited, status=1/FAILURE)
Feb 09 17:09:02 server elasticsearch[2199]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Feb 09 17:09:02 server elasticsearch[2199]: Error: Could not create the Java Virtual Machine.
Feb 09 17:09:02 server elasticsearch[2199]: Error: A fatal exception has occurred. Program will exit.
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:118)
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:86)
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:92)
Feb 09 17:09:02 server systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Feb 09 17:09:02 server systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Feb 09 17:09:02 server systemd[1]: Failed to start Elasticsearch.
At this point I'm doing a fresh install. Not able to find the solution I need to continue working...
When I try to start Freeswitch using bellow command :
/etc/init.d/freeswitch
It gives errors:-
[....] Starting freeswitch (via systemctl): freeswitch.serviceJob for
freeswitch.service failed. See 'systemctl status
freeswitch.service' and 'journalctl -xn' for details. failed!
Then I run this systemctl status freeswitch.service command that time it gives me this error:
● freeswitch.service - freeswitch Loaded: loaded
(/lib/systemd/system/freeswitch.service; enabled) Active: failed
(Result: start-limit) since Tue 2016-03-29 00:52:08 EDT; 8s ago
Process: 25997 ExecStart=/usr/bin/freeswitch -u freeswitch -g
freeswitch -ncwait $DAEMON_OPTS (code=exited, status=1/FAILURE)
Mar 29 00:52:07 systemd[1]: Failed to start freeswitch. Mar 29
00:52:07 systemd[1]: Unit freeswitch.service entered failed state.
Mar 29 00:52:08 systemd[1]: freeswitch.service start request repeated
too quickly, refusing to start. Mar 29 00:52:08 systemd[1]: Failed to
start freeswitch. Mar 29 00:52:08 media03 systemd[1]: Unit
freeswitch.service entered failed state.
Then I run journalctl -xn this command it gives me this error:
-- Logs begin at Tue 2016-03-22 15:12:38 EDT, end at Tue 2016-03-29 01:17:01 EDT. -- Mar 29 01:05:54 freeswitch[26066]: FreeSWITCH[26066]
Error starting system! pid:26067 Mar 29 01:05:54 systemd[1]:
freeswitch.service: control process exited, code=exited status=1 Mar
29 01:05:54 systemd[1]: Failed to start freeswitch.
-- Subject: Unit freeswitch.service has failed
-- Defined-By: systemd
-- Unit freeswitch.service has failed.
-- The result is failed. Mar 29 01:05:54 systemd[1]: Unit freeswitch.service entered failed state. Mar 29 01:17:01 CRON[26073]:
pam_unix(cron:session): session opened for user root by (uid=0) Mar 29
01:17:01 CRON[26074]: (root) CMD ( cd / && run-parts --report
/etc/cron.hourly) Mar 29 01:17:01 CRON[26073]:
pam_unix(cron:session): session closed for user root
Freeswitch.log
2016-03-18 13:13:16.027486 [NOTICE] switch_loadable_module.c:1183 Deleting File Format 'ul'
2016-03-18 13:13:16.027486 [NOTICE] switch_loadable_module.c:1183 Deleting File Format 'ulaw'
2016-03-18 13:13:16.027486 [NOTICE] switch_loadable_module.c:1183 Deleting File Format 'al'
2016-03-18 13:13:16.027486 [NOTICE] switch_loadable_module.c:1183 Deleting File Format 'alaw'
2016-03-18 13:13:16.027486 [NOTICE] switch_loadable_module.c:1183 Deleting File Format 'adpcm'
2016-03-18 13:13:16.027486 [NOTICE] switch_loadable_module.c:1183 Deleting File Format 'vox'
2016-03-18 13:13:16.027486 [CONSOLE] switch_loadable_module.c:2008 Stopping: mod_sndfile
2016-03-18 13:13:16.027486 [DEBUG] switch_loadable_module.c:1330 Write lock interface 'en' to wait for existing references.
2016-03-18 13:13:16.027486 [NOTICE] switch_loadable_module.c:1338 Deleting Say interface 'en'
2016-03-18 13:13:16.027486 [CONSOLE] switch_loadable_module.c:2011 mod_say_en has no shutdown routine
This problem is related to your init script is not working properly.
so try below
/usr/local/freeswitch/bin/freeswitch
If still problem exist just try to killall freeswitch processes.
Then again start the freeswitch