I am running an ActiveMQ broker in a RedHat OpenShift cluster, with an EFK (ElasticSearch, Fluentd, Kibana) stack handling logging. In my java components I have configured my loggers to log in JSON format using the LogStash encoder so that they can be properly parsed, stored and displayed in Kibana and I'm looking to do the same for ActiveMQ.
In my Java components my logging configuration looks like and I expected to be able to do something similar for ActiveMQ:
<configuration>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>
<logger name="jsonLogger" additivity="false" level="INFO">
<appender-ref ref="CONSOLE" />
</logger>
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
The ActiveMQ docs don't seem to mention any way of changing the logging appender: http://activemq.apache.org/how-do-i-change-the-logging.html
Has anyone managed to do this successfully before?
Related
I have a small AWS Lambda function written in Java with a Logback attached to it for sending out error log emails. The variables needed for the Logback appender (basic SMTP details) are loaded into the Logback.xml configuration appender SMTPAppender in the usual format of "${smtpPort}" etc. The variables are maintained through the Lambda Function Environment variable: key=JAVA_TOOL_OPTIONS, value=-DsmtpPort=587 -DsmtpHost=xyz.
Here is the logback.xml configuration:
<configuration debug="true">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{ISO8601} %-5p [%t] %c - %m%n</pattern>
</encoder>
</appender>
<appender name="EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
<smtpPort>${smtpPort}</smtpPort>
<STARTTLS>${startTls}</STARTTLS>
<smtpHost>${smtpHost}</smtpHost>
<username>${smtpUser}</username>
<password>${smtpPassword}</password>
<to>${logEmailTo}</to>
<from>xxxx#xxxxxxxx.net</from>
<subject>Error log [Scheduled Task Runner]</subject>
<layout class="ch.qos.logback.classic.html.HTMLLayout">
<pattern>%date%thread%level%logger%msg</pattern>
<cssBuilder class="LambdaHandlers.util.ErrorEmailCssBuilder"/>
</layout>
<cyclicBufferTracker class="ch.qos.logback.core.spi.CyclicBufferTracker">
<!-- send just 10 log entries per email -->
<bufferSize>10</bufferSize>
</cyclicBufferTracker>
</appender>
<logger name="org.apache.commons.httpclient" level="OFF"/>
<logger name="httpclient.wire.header" level="OFF"/>
<logger name="httpclient.wire.content" level="OFF"/>
<logger name="httpclient.wire" level="OFF"/>
<logger name="org.apache.http" level="OFF"/>
<logger name="io.micrometer" level="INFO"/>
<logger name="software.amazon.awssdk" level="OFF"/>
<logger name="io.netty" level="OFF"/>
<logger name="org.apache.catalina.realm" level="DEBUG"/>
<logger name="org.apache.axis" level="INFO"/>
<root level="debug">
<appender-ref ref="STDOUT"/>
<appender-ref ref="EMAIL"/>
</root>
</configuration>
When the values are hardcoded in the logback.xml the email sending works as expected, however; after moving these to the JAVA_TOOL_OPTIONS bucket, the email stop working. The Lambda environment variables are correctly loaded on Lambda startup and can be accessed through the System.getProperty() in the Lambda context, providing the values from the environment variable.
I have this logback setup in spring
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender class="ch.qos.logback.core.ConsoleAppender" name="CONSOLE">
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE"/>
</root>
</configuration>
How can I configure fluentd to collect logs from my spring applications in k8s and send them to elasticsearch?
I guess you have to be more specific. What have you tried? What does not work as expected? ... collecting logs of a spring application with fluentd is not different to using any other technology, as long as is it streamed to stdout/stderr.
It should be pretty straight forward. See this: https://medium.com/avmconsulting-blog/how-to-deploy-an-efk-stack-to-kubernetes-ebc1b539d063 or https://devopscube.com/setup-efk-stack-on-kubernetes/
I have a java package with specialized operations. Specialized in the sense that they are rarely used and i don't want to have them be mixed with normal logging.
I know that adding logging.file=myapplication.log will redirect the logging to this file but is there a way to specify only logging from specific packages to another file? Like logging.file.my.package=special.log ?
It's not possible with the logging configuration Spring Boot provides. However, you can fall back upon the configuration that the logging framework provides. By default, this is Logback, which can be configured to log to multiple logging files.
To do that, you need to add a logback.xml file to your classpath and configure multiple appenders. For example:
<appender name="FILE1" class="ch.qos.logback.core.FileAppender">
<file>log1.log</file>
<append>true</append>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
<appender name="FILE2" class="ch.qos.logback.core.FileAppender">
<file>log2.log</file>
<append>true</append>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
And now you can define a separate logger in case you want to log to a different file. Make sure to add the additivity="false" otherwise, the log message will still be printed in both log files:
<logger name="com.example.apps.special-package" level="INFO" additivity="false">
<appender-ref ref="FILE2" />
</logger>
<root level="INFO">
<appender-ref ref="FILE1" />
</root>
In this case, all logs will be written to log1.log (FILE1 appender), while logs from the package com.example.apps.special-package will be written to log2.log (FILE2 appender).
Spring uses Logback as default logger. According to the official doc you can setup logback.xml yourself to add to the default logging mechanism your 'special' behavior:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}/}spring.log}"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<include resource="org/springframework/boot/logging/logback/console-appender.xml" />
<property name="SPECIAL_FILE_NAME" value="special"/>
<appender name="SPECIAL_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<encoder>
<pattern>%date{yyyy-MM-dd HH:mm:ss.SSS} [%-10.10thread] %-5level %30.30logger{29}:%-4line %msg%n</pattern>
<charset>utf8</charset>
</encoder>
<file>logs/${SPECIAL_FILE_NAME}.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>logs/${SPECIAL_FILE_NAME}-%i.log</fileNamePattern>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>10MB</MaxFileSize>
</triggeringPolicy>
</appender>
<logger name="logging.file.my.package" level="debug" additivity="false">
<appender-ref ref="SPECIAL_FILE"/>
</logger>
<root level="INFO">
<appender-ref ref="FILE" />
<appender-ref ref="CONSOLE" />
</root>
</configuration>
Here we use Spring default FILE and CONSOLE appenders to log app info as usual (except logging.file.my.package), and use SPECIAL_FILE appender to log info from this package to file log/special.log.
It's not so easy to debug custom module deployed to the Spring XD runtime (version 1.3.1-RELEASE).
I'm aware of log sink, however it's something different that I want to achieve.
I'd like to add my own log messages to the XD log (ideally to the STDOUT alongside it's own logs). These log messages are generated in my custom module (processor in this case) using slf4j API.
I've added:
org.slf4j.Logger#info invocation to the processor class
logback-classic dependency to the pom.xml (w/o a version, as it's managed by spring-xd-module-parent dependencyManagement
logback.xml to the resources directory
logback-test.xml to the test resources directory
Log messages are logged into STDOUT during integration test invocation (via SingleNodeIntegrationTestSupport), however they don't appear in the XD log when module is uploaded or stream using it is deployed.
logback.xml contents (identical for -test):
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
<charset>UTF-8</charset>
</encoder>
</appender>
<logger name="com.maxromanovsky" level="debug" />
<logger name="org.springframework" level="warn" />
<logger name="org.apache.zookeeper" level="error" />
<root level="warn">
<appender-ref ref="STDOUT" />
</root>
The container logback configuration files can be found in xd/config (xd-container-logback.groovy and xd-singlenode-logback.groovy).
You need to add your custom logger configuration there.
I want to see the actual parameters of my SQL queries when I use Hibernate. I add this to my logback.xml to see the queries (with question marks):
<logger name="org.hibernate.type" level="TRACE" />
but to no effect.
Is there any special configuration necessary?
OnConsoleStatusListener shows me the correct configuration
23:48:15,246 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.hibernate.type] to TRACE
but no output from org.hibernate.type package.
I'm using Spring with Jpa.
Things you have to make sure:
Are you sure that SLF4J + LogBack is working in your app?
Is your logger pointing to any appender?
Have you configured an appended?
<configuration>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<!-- "application-name" is a variable -->
<File>c:/logs/${application-name}.log</File>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%d %p %t %c - %m%n</Pattern>
</layout>
</appender>
<root level="debug">
<appender-ref ref="FILE"/>
</root>
</configuration>
I'm using this configuration, and it works for me:
<logger name="org.hibernate.type" level="trace" additivity="false">
<appender-ref ref="consoleAppender" />
</logger>
The logger that works for me is the following:
<logger name="org.hibernate.type" level="TRACE" />