rsyslog to send data to the same server showing error on modules - rsyslog

My app is producing logs under
/var/log/myapp/app.log
I need to send all the logs under app.log to my syslog file /var/log/syslog.
I created a file with following content /etc/rsyslog.d
$WorkDirectory /var/spool/rsyslog
$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%"
# Log shipment rsyslog target servers
$ActionQueueFileName appfile
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
$ActionResumeRetryCount 250
*.* #10.x.x.1;RFC3164fmt
# Log files
$InputFileName /var/log/myapp/app.log
$InputFileTag app:
$InputFileStateFile state-app
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
10.x.x.1 is the same node where rsyslog is installed and my app is running.
once i restart rsyslog am getting following error in syslog file
invalid or yet-unknown config file command 'InputFileName' - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ]
invalid or yet-unknown config file command 'InputFileTag' - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ]
invalid or yet-unknown config file command 'InputFileStateFile' - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ]
invalid or yet-unknown config file command 'InputFileFacility' - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ]
invalid or yet-unknown config file command 'InputFilePollInterval' - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ]
invalid or yet-unknown config file command 'InputFilePersistStateInterval' - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ]
invalid or yet-unknown config file command 'InputRunFileMonitor' - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ]
I also checked couple of SO links and changed my /etc/rsyslog.conf file entry to change the user . but I
#$PrivDropToUser syslog
$PrivDropToUser appuser
#$PrivDropToGroup syslog
$PrivDropToGroup appuser

I missed to load the module .
Adding the following line in the beginning resolved the issue
$ModLoad imfile

Related

Unable to parse logstash.yml file within /etc/logstash [ELK Stack]

Good Evening,
I receive the error: unable to parse logstash.yml
My OS: lubuntu 20.04.3
Here is the uncommented line of code within the etc/logstash/logstash.yml
path.conf: "/etc/logstash/conf.d"
Within the conf.d directory, I have my logstash.conf file
I run the command using: /usr/share/logstash/bin/logstash -f, --path.settings /etc/logstash`
Here is the whole logstash.yml file: https://pastebin.com/gTk1Cuhc
I've tried:
path.conf: /etc/logstash/conf.d
and
path.conf: '/etc/logstash/conf.d'

Homebrew Logstash failed to open a file under ~/Documents

I installed the Logstash via Homebrew and configured the Logstash to tail a log file under my ~/Documents folder. However, the Logstash log indicate it failed to open the file with the following error.
[2020-09-14T13:58:27,892][WARN ][filewatch.tailmode.handlers.createinitial][others][7b0d387b8f4792cb946034cfce3f23950334ccb21b9d8b1792d718ecbf4d5c3a] failed to open /Users/xxx/Documents/project/logs/all: #<IOError: Operation not permitted>, ["org/jruby/RubyIO.java:1234:in `sysopen'", "org/jruby/RubyFile.java:365:in `initialize'", "org/jruby/RubyIO.java:1156:in `open'"]
I tried to add following file into the list of Security & Privary -> Full Disk Access but still failed.
/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java
/usr/local/Cellar/logstash/7.9.1/bin/logstash
/usr/local/Cellar/logstash/7.9.1/bin/logstash-plugin
/usr/local/Cellar/logstash/7.9.1/libexec/bin/logstash
I confirmed the Logstash can access other files (e.g. /var/log/system.log)
Here is my Logstash's input configuration.
input {
file {
path => [
"/Users/xxx/Documents/project/logs/all"
]
mode => "tail"
}
}
Does anyone know why may be the cause of this failure?

logstash not runs config

I'm using filebeat on client side > logstash on serverside > elasticsearch on server side
filebeat on clientside works properly by sending file, but the configuration i've made on logstash returning
Fail
[WARN ] 2019-12-18 14:53:30.987 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2019-12-18 14:53:31.341 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[ERROR] 2019-12-18 14:53:31.364 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Here is my configfile
input {
beats {
port =>5044
}
}
filter {
grok {
match => { "message" =>"%{TIMESTAMP_ISO8601:timestamp}] %{WORD:test}\[%{NUMBER:nom}]\[%{DATA:tes}\] %{DATA:module_name}\: %{WORD:method}%{GREEDYDATA:log_message}" }
}
}
output {
elasticsearch
{
hosts => "127.0.0.1:9200"
index=>"test_log_pbx"
}
}
code to run my logstash config
/usr/share/logstash/bin/logstash -f logstash.conf
when i run configtest it returns
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-12-18 14:59:53.300 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-12-18 14:59:56.566 [LogStash::Runner] Reflections - Reflections took 139 ms to scan 1 urls, producing 20 keys and 40 values
Configuration OK
[INFO ] 2019-12-18 14:59:57.923 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
please help me i dont know whats wrong
A logstash instance already running, so you can not run another instance.If you made your logstash as service, you should stop the service. If you want to run multiple instances, you should modify pipelines.yml
If you want to learn more about pipelines.yml, I put link the below.
https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

rsyslog 8.34.0: could not load module '/usr/lib/rsyslog/omuxsock.so'

My project requires forwarding log using rsyslog to a socket. rsyslog provides omuxsock output module for the same. When I try to use it using standard example, I see below error.
rsyslogd: could not load module '/usr/lib/rsyslog/omuxsock.so', dlopen: Error loading shared library /usr/lib/rsyslog/omuxsock.so: No such file or directory [v8.34.0 try http://www.rsyslog.com/e/2066 ]
Could someone please help me in solving this issue?
System Info
Alpine container = v3.8
rsyslog = 8.34.0-r0
Full log :-
/ # rsyslogd -N6 | head -10
rsyslogd: version 8.34.0, config validation run (level 6), master config /etc/rsyslog.conf
rsyslogd: could not load module '/usr/lib/rsyslog/omuxsock.so', dlopen: Error loading shared library /usr/lib/rsyslog/omuxsock.so: No such file or directory [v8.34.0 try http://www.rsyslog.com/e/2066 ]
rsyslogd: invalid or yet-unknown config file command 'OMUxSockSocket' - have you forgotten to load a module? [v8.34.0 try http://www.rsyslog.com/e/3003 ]
rsyslogd: error during parsing file /etc/rsyslog.conf, on or before line 30: errors occured in file '/etc/rsyslog.conf' around line 30 [v8.34.0 try http://www.rsyslog.com/e/2207 ]
rsyslogd: error during parsing file /etc/rsyslog.d/rsyslog_stats.conf, on or before line 15: invalid character '\' in expression - is there an invalid escape sequence somewhere? [v8.34.0 try http://www.rsyslog.com/e/2207 ]

When rsyslog (v8.39.0) mmnormalize is not recognized. How can this be fixed?

I am trying to use mmnormalize to structure text logs. Both with rsyslog 8.16.x and with 8.39.0 when trying to use mmnormalize as an action logs show that the module was not recognized. Below are details.
How can I set up to use mmnormalize with rsyslog?
remediation already tried
Installed separately liblognorm, libstr, json-c,libfastjson
Upgraded rsyslog from 8.16.x to 8.39.0
rsyslog .conf
module(load="mmnormalize") # text parsing
syslog log messages
Dec 3 11:33:55 sys1 systemd[1]: Starting System Logging Service...
Dec 3 11:33:55 sys1 systemd[1]: Started System Logging Service.
Dec 3 11:33:55 sys1 rsyslogd: could not load module 'mmnormalize',
errors: trying to load module /usr/lib/rsyslog/mmnormalize.so:
/usr/lib/rsyslog/mmnormalize.so: cannot open shared object file: No
such file or directory [v8.39.0 try http://www.rsyslog.com/e/2066 ]
Dec 3 11:33:55 sys1 rsyslogd: module name 'mmnormalize' is unknown
[v8.39.0 try http://www.rsyslog.com/e/2209 ]
Dec 3 11:33:55 sys1 rsyslogd: error during parsing file
/etc/rsyslog.d/52-tomcat.conf, on or before line 52: errors occured in
file '/etc/rsyslog.d/52-tomcat.conf' around line 52 [v8.39.0 try
http://www.rsyslog.com/e/2207 ]
The following way to install mmnormalize worked for me. I was running this on Ububutu (Xenial)
sudo apt-get install rsyslog-mmnormalize
Rich Megginson (thank you) answered as below to the same question I posted to rsyslog mailing list (rsyslog#lists.adiscon.com). As mentioned above it worked for me.
"On RHEL/CentOS/Fedora and similar platforms, the rsyslog-mmnormalize is a separate RPM that must be installed separately e.g.
yum install rsyslog rsyslog-mmnormalize ....
"

Resources