It's not so easy to debug custom module deployed to the Spring XD runtime (version 1.3.1-RELEASE).
I'm aware of log sink, however it's something different that I want to achieve.
I'd like to add my own log messages to the XD log (ideally to the STDOUT alongside it's own logs). These log messages are generated in my custom module (processor in this case) using slf4j API.
I've added:
org.slf4j.Logger#info invocation to the processor class
logback-classic dependency to the pom.xml (w/o a version, as it's managed by spring-xd-module-parent dependencyManagement
logback.xml to the resources directory
logback-test.xml to the test resources directory
Log messages are logged into STDOUT during integration test invocation (via SingleNodeIntegrationTestSupport), however they don't appear in the XD log when module is uploaded or stream using it is deployed.
logback.xml contents (identical for -test):
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
<charset>UTF-8</charset>
</encoder>
</appender>
<logger name="com.maxromanovsky" level="debug" />
<logger name="org.springframework" level="warn" />
<logger name="org.apache.zookeeper" level="error" />
<root level="warn">
<appender-ref ref="STDOUT" />
</root>
The container logback configuration files can be found in xd/config (xd-container-logback.groovy and xd-singlenode-logback.groovy).
You need to add your custom logger configuration there.
Related
I have been using the default spring logging configuration where I only specify filename in the application.properties file.
logging.file.name=app.log
But this by default appends logs when I start the application from cmd line "java -jar abc.jar"
I tried to search for the property which clears the file before starting application every time but couldn't find it. How should I clear the log file before starting the app?
Spring Boot uses Logback framework as as a default Logger.
You can use environnement variable via application.properties file to set some logging properties but logback xml configuration provides more powerfull features.
When a file in the classpath has one of the following names, Spring Boot will automatically load it over the default configuration:
logback-spring.xml
logback.xml
logback-spring.groovy
logback.groovy
So You can just put the code snippet below in src/main/resources/logback-spring.xml file
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="Console"
class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>
%black(%d{ISO8601}) %highlight(%-5level) [%blue(%t)] %yellow(%C{1.}): %msg%n%throwable
</Pattern>
</layout>
</appender>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>app.log</file>
<append>false</append>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
<!-- LOG everything at INFO level -->
<root level="info">
<appender-ref ref="FILE" />
<appender-ref ref="Console" />
</root>
</configuration>
The line <append>false</append> does the job.
if you wanna log more information than those avaible with the encoder pattern <pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern> please check logback documention https://logback.qos.ch/manual/layouts.html#logback-access
Using spring boot 2.1.1.RELEASE one can seemingly format logs as JSON by providing a logback-spring.xml file as follows:
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>true</prettyPrint>
</jsonFormatter>
</layout>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="stdout" />
</root>
and adding to the pom.xml
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
<version>0.1.5</version>
</dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-jackson</artifactId>
<version>0.1.5</version>
</dependency>
indeed leading to messages like:
{
"timestamp" : "2018-12-11T18:20:25.641Z",
"level" : "INFO",
"thread" : "main",
"logger" : "com.netflix.config.sources.URLConfigurationSource",
"message" : "To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.",
"context" : "default"
}
Why?
I'm trialing logz.io which appears to behave more favourably when logs are JSON formatted, some o the shippers struggle with multiline logs like we see in java stack traces and when formatting in JSON it can automatically parse fields like level and message and if there is MDC data it automatically gets that.
I had some not so great experiences with a few of the methods of shipping logs to logzio, like their docker image and using rsyslog without using JSON formatted log messages.
Issues With This Approach
It works ok for console appending, but spring boot provides like logging.file=test.log, logging.level.com.example=WARN, logging.pattern.console. I can indeed import the managed configuration from spring-boot-2.1.1.RELEASE.jar!/org/springframework/boot/logging/logback/base.xml which in turn imports a console-appender.xml andfile-appender.xml`.
An example of the console-appender
<included>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>${CONSOLE_LOG_PATTERN}</pattern>
</encoder>
</appender>
</included>
An example of the file appender
<included>
<appender name="FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<encoder>
<pattern>${FILE_LOG_PATTERN}</pattern>
</encoder>
<file>${LOG_FILE}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.%i.gz</fileNamePattern>
<maxFileSize>${LOG_FILE_MAX_SIZE:-10MB}</maxFileSize>
<maxHistory>${LOG_FILE_MAX_HISTORY:-0}</maxHistory>
</rollingPolicy>
</appender>
</included>
These two are exactly what I need to support spring configuration of the properties, but they don't include the encoder/layout I'd need.
It appears in my initial tests that I can't simple name my appender the same as those and provide my layouts. For example:
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml"/>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>true</prettyPrint>
</jsonFormatter>
</layout>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>
leads to the message being logged in both JSON and plain text format.
I can indeed just copy and paste the contents of these 3 files into my custom config rather than import them at all. Then I may override what I want to customise.
However, as spring evolves and new releases are made which may add features, I'd be forever forcing myself to keep up, copy and paste the new files and make my changes and test them.
Is there any better way that I can either:
Just make additive changes to the appenders rather than entirely redefine them, e.g. keep the config from spring but provide my own encoder or layout to be used by those appenders.
Configure spring to JSON log via properties entirely without any config - I doubt this :S
Footnote: logzio do provide a dependency one can import, but I dislike the idea of coupling the logging provider into the code directly. I feel that if the servoce happens to produce JSON logs to stdout or a file, it's easy for any provider to process those and ship them to some destination.
I am not using any dependency.
Simply, doing it via application.yml, that's all.
This solution solves, multiline log issue, too.
logging:
pattern:
console: "{\"time\": \"%d\", \"level\": \"%p\", \"correlation-id\": \"%X{X-Correlation-Id}\", \"source\": \"%logger{63}:%L\", \"message\": \"%replace(%m%wEx{6}){'[\r\n]+', '\\n'}%nopex\"}%n"
I use something like the following, has always worked fine.
Spring Boot recommendation is to name the file logback-spring.xml and place it under src/main/resources/, this enables us to use spring profiles in logback.
So in the file below you will see that for LOCAL profile you can log in the standard fashion but for the deployments on the server or a container you can you a different logging strategy.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %5p [YourApp:%thread:%X{X-B3-TraceId}:%X{X-B3-SpanId}] %logger{40} - %msg%n
</pattern>
</encoder>
</appender>
<appender name="jsonstdout" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<providers>
<timestamp>
<timeZone>EST</timeZone>
</timestamp>
<pattern>
<pattern>
{
"level": "%level",
"service": "YourApp",
"traceId": "%X{X-B3-TraceId:-}",
"spanId": "%X{X-B3-SpanId:-}",
"thread": "%thread",
"class": "%logger{40}",
"message": "%message"
}
</pattern>
</pattern>
<stackTrace>
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<maxDepthPerThrowable>30</maxDepthPerThrowable>
<maxLength>2048</maxLength>
<shortenedClassNameLength>20</shortenedClassNameLength>
<rootCauseFirst>true</rootCauseFirst>
</throwableConverter>
</stackTrace>
</providers>
</encoder>
</appender>
<root level="info">
<springProfile name="LOCAL">
<appender-ref ref="stdout" />
</springProfile>
<springProfile name="!LOCAL">
<appender-ref ref="jsonstdout" />
</springProfile>
</root>
</configuration>
Sounds like you need to copy paste with modifications 3 out of 4 files from here https://github.com/spring-projects/spring-boot/tree/v2.1.1.RELEASE/spring-boot-project/spring-boot/src/main/resources/org/springframework/boot/logging/logback into your configuration.
The good news is that you don't need to copy paste https://github.com/spring-projects/spring-boot/blob/v2.1.1.RELEASE/spring-boot-project/spring-boot/src/main/resources/org/springframework/boot/logging/logback/defaults.xml
That can be included like so <include resource="org/springframework/boot/logging/logback/default.xml"/>
That will get you some of spring's default config without copying and pasting
If you switch to log4j2 using the method specified in the Spring Boot documentation
implementation "org.springframework.boot:spring-boot-starter-log4j2"
modules {
module("org.springframework.boot:spring-boot-starter-logging") {
replacedBy("org.springframework.boot:spring-boot-starter-log4j2", "Use Log4j2 instead of Logback")
}
}
You can have a file log4j2.properties that contains the following.
appender.stdout.type=Console
appender.stdout.name=json
appender.stdout.json.type=JsonTemplateLayout
appender.stdout.json.eventTemplateUri=classpath:LogstashJsonEventLayoutV1.json
appender.console.type=Console
appender.console.name=console
rootLogger.appenderRef.stdout.ref=${env:LOGGING_APPENDER:-json}
I used LOGGING_APPENDER to match my environment level overrides.
Note one odd flaw though... this does not work when you're on JDK17.
I'm currently using SLF4J API for logging.
Whenever an exception is thrown during runtime, the full error stack trace does not log to file, it is only printed to the console. I'm using eclipse.
Here is my code for logback.xml (currently located in classes folder under WEB-INF)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE xml>
<configuration>
<!-- Specify here the path of the folder you want to save your logs -->
<property name="LOGFILE_PATH" value="C:/Logs" />
<!-- All logging will be redirected/ printed to console. -->
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%d{yyyy-MM-dd hh:mm:ss a} [%thread] %-5level %logger{50} - %rEx %msg%n </Pattern>
</layout>
</appender>
<!-- Send log to file -->
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>${LOGFILE_PATH}/spring-mybatis-log.log</File>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%d{yyyy-MM-dd hh:mm:ss a} [%thread] %-5level %logger - %rEx %msg%n</pattern>
</layout>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOGFILE_PATH}/spring-mybatis-log-%d{yyyy-MM-dd}-%i.txt
</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy
class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>2MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<root level="DEBUG">
<appender-ref ref="STDOUT" />
<appender-ref ref="FILE" />
</root>
</configuration>
Is there something missing/wrong with the above file??
Is it possible to log (to a file) all the text that will be printed to console?
How does spring(or the project itself) read the logback.xml file? What if I rename it and place it in another folder?
How to create one root containing all the levels (INFO, DEBUG, ERROR, WARN, etc..) ?
To answer your questions:
Nothing looks obviously wrong to me in the file you've posted, although I didn't try actually running it.
Just the way you did it is fine, with two appenders, one that goes to the file and the other to go to the console.
Logback by default looks in the classpath for the logback.xml file. Refer to the configuration page of the manual for the details. The way it gets there depends on your build system. When using Maven, I'd suggest putting it in src/main/resources. But if it ends up in WEB-INF/classes when deployed in your web app, that should work. If no matter what you put in your logback.xml file you are only getting console output (try adding a syntax error or renaming the file to test), that's what I'd look at first, to ensure that Logback is picking up the file right. It will default to showing everything just on the console if it can't find the file, though I think it shows a warning at the beginning that it's doing so. If it is picking up the file, you can try putting debug="true" in the <configuration> element, to see if there's an error that it's picking up that's causing it to not use the appender the way you're expecting.
Specifying the "DEBUG" level of logging, as you've done, will also get all higher levels as well.
If the issue is that logging from Spring isn't going where you want, while your application's logging is working fine, you may need to redirect Spring (which uses Apache Commons Logging) to use SLF4J instead. To do that, remove the commons-logging dependency and add the jcl-over-slf4j library. This will emulate the commons-logging calls and have them point to SLF4J instead. See the "Using SLF4J" section of this blog post for more details.
Your configuration looks OK, but you can try use your root level INFO instead DEBUG and if you have frameworks like spring, hibernate etc. Logback allow you uses another levels to them something like:
<logger name="org.hibernate" level="OFF" />
<logger name="org.springframework" level="INFO" />
<logger name="org.springframework.web.servlet.mvc" level="INFO" />
I need some help for the following problem.
I have a spring boot application and I would like to configure a fluentd appender using logback.
I've created a file called logback.xml in my src/main/resources with the following content:
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%date - %level - [%thread] - %logger - [%file:%line] - %msg%n</pattern>
</encoder>
</appender>
<appender name="FLUENT_TEXT" class="ch.qos.logback.more.appenders.DataFluentAppender">
<tag>dab</tag>
<label>normal</label>
<remoteHost>localhost</remoteHost>
<port>24224</port>
<maxQueueSize>20</maxQueueSize>
</appender>
<logger name="org.com" level="DEBUG"/>
<root level="DEBUG">
<appender-ref ref="STDOUT" />
<appender-ref ref="FLUENT_TEXT" />
</root>
</configuration>
In my build.gradle I have :
compile 'org.fluentd:fluent-logger:0.3.1'
compile 'com.sndyuk:logback-more-appenders:1.1.0'
When I launch the app using gradle bootRun I have the following message:
10:56:33,020 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - Attempted to append to non started appender [STDOUT].
10:56:33,020 |-WARN in ch.qos.logback.more.appenders.DataFluentAppender[FLUENT_TEXT] - Attempted to append to non started appender [FLUENT_TEXT].
10:56:33,028 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - Attempted to append to non started appender [STDOUT].
Exception in thread "main" 10:56:33,028 |-WARN in ch.qos.logback.more.appenders.DataFluentAppender[FLUENT_TEXT] - Attempted to append to non started appender [FLUENT_TEXT].
java.lang.NullPointerException
at ch.qos.logback.more.appenders.DataFluentAppender$FluentDaemonAppender.close(DataFluentAppender.java:72)
I've found here https://github.com/spring-projects/spring-boot/blob/master/spring-boot-docs/src/main/asciidoc/spring-boot-features.adoc something saying that logback.xml is loaded too early so I need to use a file called logback-spring.xml.
I've did it and it's like the file is never loaded, no error but nothing gets to my fluetd socket.
Any idea how to solve it ?
Thanks.
C.C.
When running your springboot application, load a 'spring' profile.
One way of doing it would be via the command line, see below.
-Dspring.profiles.active=spring
As specified in title I'm using Maven, and Jetty. For logging using SLF4J and Logback. I have 'logback.xml' at 'src/main/resources'.
<configuration>
<appender name="STDOUT"
class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%-4relative [%thread] %-5level %class - %msg%n</pattern>
</layout>
</appender>
<appender name="FILE"
class="ch.qos.logback.core.FileAppender">
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%-4relative [%thread] %-5level %class - %msg%n</pattern>
</layout>
<File>myLog.log</File>
</appender>
<logger name="org.mortbay">
<level value="debug" />
</logger>
<root>
<level value="error" />
<appender-ref ref="STDOUT" />
<appender-ref ref="FILE" />
</root>
</configuration>
But my problem is its not creating the file 'myLog.log' if I run/debug the project. What's the solution to get the log file.
Is there any way to get the log file only with SLF4J?
Sorry! I misunderstood the usage of 'Logback'. I got solution from http://www.mail-archive.com/user#slf4j.org/msg00661.html
i.e.
It appears that you have misunderstood
the purpose of SLF4J. If you place
slf4j-jdk14-1.5.6.jar then slf4j-api
will bind with java.util.logging.
Logback will not be used. Only if you
place logback-core.jar and
logback-classic.jar on your class path
(but not slf4j-jdk14-1.5.6.jar) will
SLF4J API bind with logback. SLF4J
binds with one and only one underlying
logging API (per JVM launch).
HTH,
Thanks to Ceki Gulcu. Now I can able to get logs in my file.
If you are using JBoss 5.1 and you are having the same problem[logback not writing to file] then add the following in jboss-web.xml.
<class-loading>
<loader-repository>
com.hp:classloader=logback-slf4j
<loader-repository-config>java2ParentDelegation=false</loader-repository-config>
</loader-repository>
</class-loading>
This should solve your problem.