I'm using syslog to capture logging from an application. I've configured syslog to write my applications logs to an application-specific file in asl.conf:
# Redirect foo to /var/log/foo.log
? [= Sender foo] file /var/log/foo.log
This works, but repeated messages are culled. For example:
Jun 21 17:22:03 hostname.domain foo[346] <Debug>: This is a message!
Jun 21 17:22:03 --- last message repeated 3 times ---
How can I disable the squashing of repeated messages?
If you restart the syslogd daemon with the option -dup_delay and set the value to 0, it will stop this happening, but it's there to prevent the log getting filled with unnecessary messages.
More detail of syslogd is described here.
Related
Problem description:
I have relatively big /var/log/messages file which is rotated.
The file list looks like this:
ls -l /var/log/messages*
-rw-------. 1 root 928873050 Mar 5 10:37 /var/log/messages
-rw-------. 1 root 889843643 Mar 5 07:49 /var/log/messages.1
-rw-------. 1 root 890148183 Mar 5 07:50 /var/log/messages.2
-rw-------. 1 root 587333632 Mar 5 07:51 /var/log/messages.3
My filebeat configuration snippet:
filebeat.prospectors:
- input_type: log
paths:
- /var/log/messages
- /var/lib/ntp/drift
- /var/log/syslog
- /var/log/secure
tail_files: True
With multiple /var/log/messages* files as shown above each time filebeat is restarted it starts to harvest and ingest the old log files.
When I have just one /var/log/messages file, this issue is not observed.
On Linux systems, Filebeat keeps track of files not by filename but with inode number which doesn't change when renamed. This is from Filebeat documentation.
The harvester is responsible for opening and closing the file, which
means that the file descriptor remains open while the harvester is
running. If a file is removed or renamed while it’s being harvested,
Filebeat continues to read the file. This has the side effect that the
space on your disk is reserved until the harvester closes. By default,
Filebeat keeps the file open until close_inactive is reached.
Which means this is what happens in your case
Reads current messages file (inode#1) and keeps track of its inode number in the registry.
Filebeat Stops, but messages file rotated to messages.1 (inode#1) and new messages (inode#2) file got created.
When Filebeat restarts then it will start reading
messages.1 (inode#1) file from where it left off
messages (inode#2) since it matches the path you configured (/var/log/messages)
If your plan is to harvest all messages file even the rotated ones, then it would be better to configure the path as
/var/log/messages*
It seems like the syslog and security plugins were ON in the configuration. That triggered the loading of the rotated syslog files.
How can I save the output of the NetworkManager command that listens for and prints the current activity?
The command is nmcli monitor (https://developer.gnome.org/NetworkManager/stable/nmcli.html).
Logging Messages
I will show how to modify the logging levels by NetworkManager.
NetworkManager supports on the fly changing of the logging levels and allows for a fine control over what is logged.
First check what is the current configuration by below command:
root#hostname ~: nmcli general logging
As a result you will be presented with the information about current conf:
LEVEL DOMAINS INFO PLATFORM,RFKILL,ETHER,WIFI,BT,MB,DHCP4,DHCP6,PPP,IP4,IP6,AUTOIP4,DNS,VPN,SHARING,SUPPLICANT,AGENTS,SETTINGS,SUSPEND,CORE,DEVICE,OLPC,INFINIBAND,FIREWALL,ADSL,BOND,VLAN,BRIDGE,TEAM,CONCHECK,DCB,DISPATCH,AUDIT,SYSTEMD
It is possible to change the level either globally or for each domain separately. The command to achieve this is:
nmcli general logging [level <level> [domain <domain>]]
The < level > is the desired log level, here are some examples:
ERR: will log only critical errors
WARN: will log warnin messages
INFO: will log various informational messages
DEBUG: enables verbose logging for debugging purposes
< domain > is the category of messages that shall be logged with given severity.
WIFI: will include only WiFi related messages, IP4 will include only IPv4 related messages
And so on, i cant provide info for every setting but this is how it works in general.
Other things you may wanna try:
Systemd:
journalctl -u NetworkManager > tomyfile.txt
Use debug mode in general:
sudo /usr/lib/NetworkManager/debug-helper.py --nm debug > tomyfile.txt
I was able to log the output of nmcli monitor to a file using:
nmcli monitor >> log.txt
Are there any GUCs or commands which I can get debug messages in HAWQ log? Now, I can only get ERROR or FATAL message but can not get any DEBUG messages. How to print these DEBUG messages in Log file?
You can set log_min_messages level in postgres.conf of hawq master data directory. Log level can be the following values in order of decreasing detail:
# debug5
# debug4
# debug3
# debug2
# debug1
# info
# notice
# warning
# error
# log
# fatal
# panic
It needs to restart cluster if you change the postgres.conf. But you can set GUC log_min_messages in the PSQL session if you just want to log the debug info within this session.
Different component of apache hawq support different level of debugging message.
The overall supported levels are as below. You may refer to https://github.com/apache/incubator-hawq/blob/master/src/include/utils/elog.h for details.
/* Error level codes */
Level Value
------------------
DEBUG5 10
DEBUG4 11
DEBUG3 12
DEBUG2 13
DEBUG1 14
LOG 15
COMMERROR 16
INFO 17
NOTICE 18
WARNING 19
ERROR 20
FATAL 21
PANIC 22
To get the DEBUG message you want, you need to check the component you care about regarding the supported level of debugging. Then before run your query, use below setting to get debug information:
either persistent level of GUC ("hawq config -c log_min_messages -v DEBUG_LEVEL" and then "hawq restart cluster -a")
or use session level debugging ("set log_min_messages = DEBUG_LEVEL")
If you don't find enough log information even with highest level debugging level, you can try to add it in apache hawq source code yourself.
DEBUG you refer to may have two meaning, One is DEBUG log level in hawq code, which is answered by ztao1987, and the other is when you debug using gdb/lldb, where is the output of your print function.
The answer is in the master/segment log too. stdout has been redirected to log file by HAWQ, For example, when you want to print a tupletableslot in lldb, just type"expr print_slot(yourslot)", and tail -f your.log, the slot info will be printed on the screen.
I've setup a remote rsyslog server for testing but I can't seem to get it to log from a remote system. I have an app on my desktop (windows) called "Syslog Test Message Utility 1.0" which sends test syslog messages on UDP 514. I see the message appear on my Centos box on port 514 (using Wireshark interface) but no corresponding line appears in /var/log/messages as I would expect.
I've verified that the logging does work (e.g.. logger test) but just not from the remote system. Here is my etc/rsyslog.conf file..
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
$ModLoad imklog # reads kernel messages (the same are read from journald)
$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
#$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
# ### end of the forwarding rule ###
and I've verified that the host is listening on port 514 as expected
tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 15273/rsyslogd
tcp6 0 0 :::514 :::* LISTEN 15273/rsyslogd
udp 0 0 0.0.0.0:514 0.0.0.0:* 15273/rsyslogd
udp6 0 0 :::514 :::* 15273/rsyslogd
I'm just not even sure what to look for next.. I can't seem to figure out why my message coming from my Syslog app isn't creating a log entry in my messages file.
It turns out that CentOS 7 (and I'm assuming RHEL 7) have a firewall other than iptables called firewalld. After disabling this firewall in my dev environment I was able to successfully syslog to 514.
systemctl disable firewalld
systemctl stop firewalld
systemctl status firewalld
Disabling the firewall is admittedly overkill, as I'm sure you can create a rule for 514, but since my server is in a lab...it was acceptable in my case.
I'm trying to get pf working under Mountain Lion. Since it appears that Apple neglected to include pflogd, we have to make our own like:
/sbin/ifconfig pflog0 create
/usr/sbin/tcpdump -lnettti pflog0 | /usr/bin/logger -t pf
This dumps messages into system.log, but I'm trying to find a way to sort them out into their own. I was able to figure this out with ipfw, but ipfw left nice logs like:
Mar 5 11:34:44 flamingo kernel[0] <Debug>: ipfw: 65534 Deny ICMP:3.10 192.168.4.233 192.168.92.60 in via en0
For pf, I'm getting logs like:
Mar 5 11:57:50 flamingo.mydomain.com pf[51938]: 00:00:00.000000 rule 1/8(ip-option): pass in on en0: 172.24.32.41 > 224.0.0.1: igmp query v2
With OSXes 'logger', I can use -p to set a facility, but then the logs quietly vanish. I found references to using 'syslog -s -k facility whatever', but when I pipe my tcpdump into syslog, the process dies after a few seconds or minutes.
How can I A) find out how to keep syslog running, or B) either set or predict what I could filter the results from logger in asl.conf?
You're missing one ingredient to get logging.
This worked for me:
root# touch /var/log/pffirewall.log` to create the log file.
Then, add the following line to /etc/syslog.conf:
local2.* /var/log/pffirewall.log
Now your logger process output will show up in /var/log/pffirewall.log. Apparently logger is still tied to the old syslog facility and /etc/syslog.conf.