I'm using Websphere Liberty 8.5.5 on Linux and I'm trying to redirect its logging into Unix syslog. The logging configuration in the server.xml has an entry for copySystemStreams="true", which
according to the documentation for the server.xml logging should have the following behaviour:
"If true, write System.out to the system output stream and System.err to the system error stream. If false, System.out and System.err will write to configured logs like messages.log or trace.log, but not to the system streams. The default value is true."
However, I do not see any Liberty entries in syslog if I set copySystemStreams to true. Am I misunderstanding the documentation or is this a missing piece of functionality?
Peter
Check this page from the Liberty documentation Liberty profile: Logging and Trace, it has newer, more accurate description of this field. So it will not log to the syslog, no matter what is set in copySystemStreams.
copySystemStreams
If true, messages written to the System.out and System.err streams
are copied to console.log. If false, those messages are written to
configured logs such as messages.log or trace.log, but they are not
copied to console.log. The default value is true.
Related
I have a graylog instance that's running a UDP-Syslog-Input on Port 1514.
It's working wonderfully well for all the system logs of the linux servers.
When I try to ingest payara logs though [1], the "source" of the message is set to "localhost" in graylog, while it's normally the hostname of the sending server.
This is suboptimal, because in the best case I want the application logs with correct source in graylog also.
I googled around and found:
https://github.com/payara/Payara/blob/payara-server-5.2021.5/nucleus/core/logging/src/main/java/com/sun/enterprise/server/logging/SyslogHandler.java#L122
It seems like the syslog "source" is hard-coded into payara (localhost).
Is there a way to accomplish sending payara-logs with the correct "source" set?
I have nothing to do with the application server itself, I just want to receive the logs with the correct source (the hostname of the sending server).
example log entry in /var/log/syslog for payara
Mar 10 10:00:20 localhost [ INFO glassfish ] Bootstrapping Monitoring Console Runtime
I suspect I want the "localhost" in above example set to fqdn of the host.
Any ideas?
Best regards
[1]
logging.properties:com.sun.enterprise.server.logging.SyslogHandler.useSystemLogging=true
Try enabling "store full message" in the syslog input settings.
That will add the full_message field to your log messages and will contain the header, in addition to what you see in the message field. Then you can see if the source IP is in the UDP packet. If so, collect those messages via a raw/plaintext UDP input and the source should show correctly.
You may have to parse the rest of the message via an extractor or pipeline rule, but at least you'll have the source....
Well,
this might not exactly be a good solution but I tweaked the rsyslog template for graylog.
I deploy the rsyslog-config via Puppet, so I can generate "$YOURHOSTNAME-PAYARA" dynamically using the facts.
This way, I at least have the correct source set.
$template GRAYLOGRFC5424,"<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% YOURHOSTNAME-PAYARA %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
if $msg contains 'glassfish' then {
*.* #loghost.domain:1514;GRAYLOGRFC5424
& ~
} else {
*.* #loghost.domain:1514;RSYSLOG_SyslogProtocol23Format
}
The other thing we did is actually activating application logging through log4j and it's syslog appender:
<Syslog name="syslog_app" appName="DEMO" host="loghost" port="1514" protocol="UDP" format="RFC5424" facility="LOCAL0" enterpriseId="">
<LoggerFields>
<KeyValuePair key="thread" value="%t"/>
<KeyValuePair key="priority" value="%p"/>
<KeyValuePair key="category" value="%c"/>
<KeyValuePair key="exception" value="%ex"/>
</LoggerFields>
</Syslog>
This way, we can ingest the glassfish server logs and the independent application logs into graylog.
The "LoggerFields" in log4j.xml appear to be key-value pairs for the "StructuredDataElements" according to RFC5424.
https://logging.apache.org/log4j/2.x/manual/appenders.html
https://datatracker.ietf.org/doc/html/rfc5424
That's the problem with UDP Syslog. The sender gets to set the source in the header. There is no "best answer" to this question. When the information isn't present, it's hard for Graylog to pass it along.
It sounds like you may have found an answer that works for you. Go with it. Using log4j solves two problems and lets you define the source yourself.
For those who face a similar issue, a simpler way to solve the source problem might be to use a static field. If you send the payara syslog messages to their own input, you can create a static field that could substitute for the source to identify traffic from that source. Call it "app_name" or "app_source" or something and use that field for whatever sorting you need to do.
Alternatively, if you have just one source for application messages, you could use a pipeline to set the value of the source field to the IP or FQDN of the payara server. Then it displays like all the rest.
The documentation at https://www.ibm.com/support/knowledgecenter/en/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/rwlp_logging.html lists the following config snippet for configuring logs to go to the console in JSON format (a useful configuration when Liberty is running inside a Linux container like in a kubernetes pod):
com.ibm.ws.logging.console.format=json
com.ibm.ws.logging.console.log.level=info
com.ibm.ws.logging.console.source=message,trace,accessLog,ffdc,audit
However, I noticed that when I increase the log level inside my server.xml (say to FINER to enable tracing), the trace messages are also emitted to the console. I assume its because the source is configured to pick up messages from the trace, but what I don't understand is whether/how that interacts with the consoleLogLevel setting.
Why specify a com.ibm.ws.logging.console.log.level if its just going to include everything from message and trace log files anyway?
The com.ibm.ws.logging.console.source bootstrap property controls which sources the console handler will subscribe to. This means, if there is any trace (typically levels FINE / FINER / FINEST) enabled by the traceSpecification, that content will be consumed by the console handler and output as JSON to console.
So, setting...
com.ibm.ws.logging.console.source=message,trace,accessLog,ffdc,audit
means that the console handler will receive any log, trace, access log, ffdc, and audit events that are generated by the system.
If com.ibm.ws.logging.console.log.level is specified, it controls which log events are filtered out by the console handler. By setting com.ibm.ws.logging.console.log.level=WARNING you won't see INFO level messages in the console output unless you also explicitly set consoleLogLevel=info in the server.xml.
Note that logging settings explicitly set in the <logging> element in the server.xml (which is processed when the config manager is ready) override any configuration set using environment variables or the bootstrap.properties file (which are processed very early in the server startup process). In practice this means you should use bootstrap.properties or environment variables for logging configuration, and just use the logging element in the server.xml file to specify things you want to override while the server is running.
I deployed a go application in google cloud using kubernetes which automatically logs to google stackdriver. Oddly, all log statements are being tagged with severity "ERROR"
For example:
log.Println("This should have log level info")
will be tagged as an error.
Their docs say "Severities: By default, logs written to the standard output are on the INFO level and logs written to the standard error are on the ERROR level."
Anyone know what could be wrong with my setup?
Take a look at this logging package: github.com/teltech/logger, with an accompanying blog post. It will output your logs in a JSON format, including the severity, that is readable by the Stackdriver Fluentd agent.
I am trying to configure log shipping/ consolidation using logstash. My tomcat servers run on Windows. I am running into a few problems with my configuration - Tomcat on windows, logging using log4j, redis consolidator/ elasticsearch/ logstash/ kibana running on a single linux server.
Fewer log shippers available on Windows. It looks like nxlog does not work with redis out of the box. So, I have reverted to using logstash to ship. I would like to learn what others prefer to use
Rather use custom appenders I would rather have tomcat use log4j to log to file and then feed the file as input to be shipped to Redis. I don't want to the log formats.
No json-event format for me - http://spredzy.wordpress.com/2013/03/02/monitor-your-cluster-of-tomcat-applications-with-logstash-and-kibana/. I can't seem to get the right file config in the shipper.conf
Any sample config for log4j files - fed to logstash via redis would help.
Thanks
I'm currently writing a Java library to send logs to Logstash using ZeroMQ (no central redis broker required).
Disclaimer: it's not quite perfect yet, but may be worth keeping an eye on.
https://github.com/stuart-warren/logit
You can setup the standard juli log configuration (or log4j if you are using that), plus with the tomcat-valve jar you can send access logs as well by configuring the server.xml.
It does however send it in json-event format by default.
I'm confused as to why you wouldn't want to save all the processing on the Logstash server? You can (and currently probably should) log to file in standard format as well.
logging.properties file.
# "handlers" specifies a comma separated list of log Handler
# classes. These handlers will be installed during VM startup.
# Note that these classes must be on the system classpath.
# By default we only configure a ConsoleHandler, which will only
# show messages at the INFO and above levels.
handlers= com.stuartwarren.logit.jul.ZmqAppender
# handlers= com.stuartwarren.logit.jul.ZmqAppender, java.util.logging.ConsoleHandler
# Default global logging level.
# This specifies which kinds of events are logged across
# all loggers. For any given facility this global level
# can be overriden by a facility-specific level.
# Note that the ConsoleHandler also has a separate level
# setting to limit messages printed to the console.
.level=INFO
# Limit the messages that are printed on the console to INFO and above.
com.stuartwarren.logit.jul.ZmqAppender.level=INFO
com.stuartwarren.logit.jul.ZmqAppender.socketType=PUSHPULL
com.stuartwarren.logit.jul.ZmqAppender.endpoints=tcp://localhost:2120
com.stuartwarren.logit.jul.ZmqAppender.bindConnect=CONNECT
com.stuartwarren.logit.jul.ZmqAppender.linger=1000
com.stuartwarren.logit.jul.ZmqAppender.sendHWM=1000
com.stuartwarren.logit.jul.ZmqAppender.layout=com.stuartwarren.logit.jul.Layout
com.stuartwarren.logit.jul.Layout.layoutType=logstashv1
com.stuartwarren.logit.jul.Layout.detailThreshold=WARNING
com.stuartwarren.logit.jul.Layout.tags=tag1,tag2,tag3
com.stuartwarren.logit.jul.Layout.fields=field1:value1,field2:value2,field3:value3
java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
server.xml
<Valve className="com.stuartwarren.logit.tomcatvalve.ZmqAppender"
layout="com.stuartwarren.logit.tomcatvalve.Layout"
socketType="PUSHPULL"
endpoints="tcp://localhost:2120"
bindConnect="CONNECT"
linger="1000"
sendHWM="1000"
layoutType="logstashv1"
iHeaders="Referer,User-Agent"
oHeaders=""
cookies=""
tags="tag1,tag2,tag3"
fields="field1:value1,field2:value2,field3:value3" />
By default, log, error, and trace information for all processes and applications on a process server is written to the SystemOut.log, But our requirement is to only log Request and Responses . Is there any setup in admin console to do this?
Thanks in advance.....
When WebSphere starts, it designates the SystemOut.log file as the file into which all System.out prints will go to. Therefore, whenever any code is issuing, for example, System.out.println, the output will end up going into SystemOut.log - and that is true for both your application code and WebSphere's internal code.
To achieve the effect that you're looking for, consider using some logging framework, such as Log4J, SLF4J or Java's standard logging API.