I'm using the logstashCollector-1.0 feature on Liberty 16.0.0.3 to collect logs on a ELK stack.
I followed instructions here for the setup:
http://www.ibm.com/support/knowledgecenter/SSAW57_9.0.0/com.ibm.websphere.wlp.nd.doc/ae/cwlp_analytics_collectors.html
All is working well, but I do not have any of the stacktraces I'm logging in the code.
I get the exception message in logstash, but not the stack.
The stack appears in message.log file on the server itself, but are not forwarded by the logstashCollector feature (neither in liberty_trace nor in liberty_message items).
Is there some configuration I can tweak to get the stacktraces, or is it impossible? Not having the stacks is a big showstopper for my devs...
Stack trace was not included to the liberty_message events prior to Liberty 17.0.0.1. There was no server configuration users can set to add this.
In Liberty 17.0.0.1, the stack trace has been appended to the message field of the liberty_message events. No configuration change is required.
See Liberty APAR PI76200 for more details.
HTH
Related
What makes Kibana to not show docker container logs in APM "Transactions" page under "Logs" tab.
I verified the logs are successfully being generated with the "trace.id" associated for proper linking.
I have the exact same environment and configs (7.16.2) up via docker-compose and it works perfectly.
Could not figure out why this feature works locally but does not show in Elastic Cloud deploy.
UPDATE with Solution:
I just solved the problem.
It's related to the Filebeat version.
From 7.16.0 and ON, the transaction/logs linking stops working.
Reverted Filebeat back to version 7.15.2 and it started working again.
If you are not using file beats, for example - We rolled our own logging implementation to send logs from a queue in batches using the Bulk API.
We have our own "ElasticLog" class and then use Attributes to match the logs-* Schema for the Log Stream.
In particular we had to make sure that trace.id was the same as the the actual Traces, trace.id property. Then the logs started to show up here (It does take a few minutes sometimes)
Some more info on how to get the ID's
We use OpenTelemetry exporter for Traces and ILoggerProvider for Logs. The fire off batches independently of each other.
We populate the Trace Id's at the time of instantiation of the class as a default value. This way you in the context of the Activity. Also helps set the timestamp exactly when the log was created.
This LogEntry then gets passed into the ElasticLogger processor and mapped as displayed above to the ElasticLog entry with the Attributes needed for ES
The documentation at https://www.ibm.com/support/knowledgecenter/en/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/rwlp_logging.html lists the following config snippet for configuring logs to go to the console in JSON format (a useful configuration when Liberty is running inside a Linux container like in a kubernetes pod):
com.ibm.ws.logging.console.format=json
com.ibm.ws.logging.console.log.level=info
com.ibm.ws.logging.console.source=message,trace,accessLog,ffdc,audit
However, I noticed that when I increase the log level inside my server.xml (say to FINER to enable tracing), the trace messages are also emitted to the console. I assume its because the source is configured to pick up messages from the trace, but what I don't understand is whether/how that interacts with the consoleLogLevel setting.
Why specify a com.ibm.ws.logging.console.log.level if its just going to include everything from message and trace log files anyway?
The com.ibm.ws.logging.console.source bootstrap property controls which sources the console handler will subscribe to. This means, if there is any trace (typically levels FINE / FINER / FINEST) enabled by the traceSpecification, that content will be consumed by the console handler and output as JSON to console.
So, setting...
com.ibm.ws.logging.console.source=message,trace,accessLog,ffdc,audit
means that the console handler will receive any log, trace, access log, ffdc, and audit events that are generated by the system.
If com.ibm.ws.logging.console.log.level is specified, it controls which log events are filtered out by the console handler. By setting com.ibm.ws.logging.console.log.level=WARNING you won't see INFO level messages in the console output unless you also explicitly set consoleLogLevel=info in the server.xml.
Note that logging settings explicitly set in the <logging> element in the server.xml (which is processed when the config manager is ready) override any configuration set using environment variables or the bootstrap.properties file (which are processed very early in the server startup process). In practice this means you should use bootstrap.properties or environment variables for logging configuration, and just use the logging element in the server.xml file to specify things you want to override while the server is running.
I am trying to configure log shipping/ consolidation using logstash. My tomcat servers run on Windows. I am running into a few problems with my configuration - Tomcat on windows, logging using log4j, redis consolidator/ elasticsearch/ logstash/ kibana running on a single linux server.
Fewer log shippers available on Windows. It looks like nxlog does not work with redis out of the box. So, I have reverted to using logstash to ship. I would like to learn what others prefer to use
Rather use custom appenders I would rather have tomcat use log4j to log to file and then feed the file as input to be shipped to Redis. I don't want to the log formats.
No json-event format for me - http://spredzy.wordpress.com/2013/03/02/monitor-your-cluster-of-tomcat-applications-with-logstash-and-kibana/. I can't seem to get the right file config in the shipper.conf
Any sample config for log4j files - fed to logstash via redis would help.
Thanks
I'm currently writing a Java library to send logs to Logstash using ZeroMQ (no central redis broker required).
Disclaimer: it's not quite perfect yet, but may be worth keeping an eye on.
https://github.com/stuart-warren/logit
You can setup the standard juli log configuration (or log4j if you are using that), plus with the tomcat-valve jar you can send access logs as well by configuring the server.xml.
It does however send it in json-event format by default.
I'm confused as to why you wouldn't want to save all the processing on the Logstash server? You can (and currently probably should) log to file in standard format as well.
logging.properties file.
# "handlers" specifies a comma separated list of log Handler
# classes. These handlers will be installed during VM startup.
# Note that these classes must be on the system classpath.
# By default we only configure a ConsoleHandler, which will only
# show messages at the INFO and above levels.
handlers= com.stuartwarren.logit.jul.ZmqAppender
# handlers= com.stuartwarren.logit.jul.ZmqAppender, java.util.logging.ConsoleHandler
# Default global logging level.
# This specifies which kinds of events are logged across
# all loggers. For any given facility this global level
# can be overriden by a facility-specific level.
# Note that the ConsoleHandler also has a separate level
# setting to limit messages printed to the console.
.level=INFO
# Limit the messages that are printed on the console to INFO and above.
com.stuartwarren.logit.jul.ZmqAppender.level=INFO
com.stuartwarren.logit.jul.ZmqAppender.socketType=PUSHPULL
com.stuartwarren.logit.jul.ZmqAppender.endpoints=tcp://localhost:2120
com.stuartwarren.logit.jul.ZmqAppender.bindConnect=CONNECT
com.stuartwarren.logit.jul.ZmqAppender.linger=1000
com.stuartwarren.logit.jul.ZmqAppender.sendHWM=1000
com.stuartwarren.logit.jul.ZmqAppender.layout=com.stuartwarren.logit.jul.Layout
com.stuartwarren.logit.jul.Layout.layoutType=logstashv1
com.stuartwarren.logit.jul.Layout.detailThreshold=WARNING
com.stuartwarren.logit.jul.Layout.tags=tag1,tag2,tag3
com.stuartwarren.logit.jul.Layout.fields=field1:value1,field2:value2,field3:value3
java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
server.xml
<Valve className="com.stuartwarren.logit.tomcatvalve.ZmqAppender"
layout="com.stuartwarren.logit.tomcatvalve.Layout"
socketType="PUSHPULL"
endpoints="tcp://localhost:2120"
bindConnect="CONNECT"
linger="1000"
sendHWM="1000"
layoutType="logstashv1"
iHeaders="Referer,User-Agent"
oHeaders=""
cookies=""
tags="tag1,tag2,tag3"
fields="field1:value1,field2:value2,field3:value3" />
By default, log, error, and trace information for all processes and applications on a process server is written to the SystemOut.log, But our requirement is to only log Request and Responses . Is there any setup in admin console to do this?
Thanks in advance.....
When WebSphere starts, it designates the SystemOut.log file as the file into which all System.out prints will go to. Therefore, whenever any code is issuing, for example, System.out.println, the output will end up going into SystemOut.log - and that is true for both your application code and WebSphere's internal code.
To achieve the effect that you're looking for, consider using some logging framework, such as Log4J, SLF4J or Java's standard logging API.
I have just upgraded to WebSphere 7 (from 6.1), and I am having new class loading conflicts which I didn't have before.
I have looked a lot over the internet but couldn't find directions on how:
1. How to enable class loading trace.
2. Where does the output go to?
Thank you very much.
To enable the classloading information you need to go in application server > server_name > Java and Process Management > Process definition > Java Virtual Machine, there you will find the verbose classloading option. Enable it and restart the application server.
You should then find the classloading informations in the native_stderr.log in the <WAS_install_root>/profiles/<profile>/logs/<server> directory.
To add to Olivier.Roger's response (I don't have the permissions to respond inline), the information is in native_stderr.log.
(late, but for anyone else who finds this thread like I did)
I would recommend the class loader MustGather that is used by IBM support when you open a PMR. Expand the "Collecting data manually" section.