Logback configuration issue - spring

I'm using Spring Boot 2.3.2. I'm trying to configure logback to continue outputing to the console EXACTLY how it does today (ansi and all), plus mirror it to a log file in the exact same format (ansi and all). I want ALL the log files to be in the format of ${spring.application.name}.YYYY-MM-DD.log including the active one and I want to keep 7 days worth of files, so I've got it configured like this:
application.properties:
logging.file.name=${spring.application.name}
logging.file.path=/var/logs
logging.pattern.file=%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(%5p) %clr(${PID}){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n%wEx
logback.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/console-appender.xml" />
<timestamp key="byDate" datePattern="yyyy-MM-dd" />
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/${LOG_FILE}.${byDate}.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/${LOG_FILE}.%d{yyyy-MM-dd}.log</fileNamePattern>
<maxHistory>7</maxHistory>
</rollingPolicy>
<encoder>
<pattern>${FILE_LOG_PATTERN}</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE" />
</root>
</configuration>
This results in a collision error between file and fileNamePattern. I get what the error means and I can fix it by removing the date stamp from file, but there is no way to have all the files in a consistent format?
If I remove the timestamp from the active file, then it works except for the next issue...
The other issue that I have is that the logfile never gets written too until I shutdown the application.

This results in a collision error between file and fileNamePattern. I
get what the error means and I can fix it by removing the date stamp
from file, but there is no way to have all the files in a consistent
format?
Here :
<fileNamePattern>${LOG_PATH}/${LOG_FILE}.%d{yyyy-MM-dd}.log</fileNamePattern>
you don't have to specify ${LOG_PATH}. It is a filename pattern not a full path.
That should be that :
<fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.log</fileNamePattern>
The other issue that I have is that the logfile never gets written too
until I shutdown the application.
Not sure but it looks a file locking. Try to ensure that no process locks the file.
As an additional remark : you specified :
logging.file.name=${spring.application.name}
logging.file.path=/var/logs
But the spring boot logging doc specify to reference the one or the other :
you need to set a logging.file.name or logging.file.path property (for
example, in your application.properties).
Valuing logging.file.name such as the following respects better the documentation :
logging.file.name=/var/log/${spring.application.name}

Related

Spring Boot logback-spring.xml creates a log file under path /var/log even if custom log path is defined in the xml and application.properties files

I prepared a custom RollingFileAppender configuration in the logback-spring.xml file and application.properties files. The log file soduncu.log created under path /var/app/sefa/logs. This is expected behavior for the configuration but there is an unexpected situation that soduncu.log created under default Linux log path /var/log/. I tried to prevent this unexpected log file creation but I couldn't. What is the situation here and what I did wrong?
logback-spring.xml:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<springProperty name="LOG_DIR" source="logging.path" defaultValue="/var/app/sefa/logs">
</springProperty>
<appender name="ROLLING"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_DIR}/soduncu.log</file>
<rollingPolicy
class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<!-- rollover daily -->
<fileNamePattern>${LOG_DIR}/soduncu-%d{yyyy-MM-dd}.%i.log
</fileNamePattern>
<maxFileSize>10MB</maxFileSize>
<maxHistory>60</maxHistory>
<totalSizeCap>20GB</totalSizeCap>
</rollingPolicy>
<encoder>
<pattern>%date %-4relative [%thread] %-5level %logger{35} - %msg %n</pattern>
</encoder>
</appender>
<root level="WARN">
<appender-ref ref="ROLLING" />
</root>
</configuration>
application.properties:
logging.path=/var/app/sefa/logs
logging.file=/var/app/sefa/logs/soduncu.log
logging.level.root=INFO
logging.level.org.springframework=ERROR
logging.level.com.nuvia=DEBUG
Here is the one Linux process id link to a log file under two different path:
/proc/14368/fd/1 -> /var/log/soduncu.log
/proc/14368/fd/2 -> /var/log/soduncu.log
/proc/14368/fd/6 -> /var/app/sefa/logs/soduncu.log
I am unable to replicate this behavior with your code.
Try to change
name="LOG_DIR" source="logging.path"
to
name="LOG_DIR_PATH" source="log.dir.path"
and see what happens
make sure to delete soduncu.log from var/log first.
I hope it helps.

Is there a recommended way to get spring boot to JSON format logs with logback

Using spring boot 2.1.1.RELEASE one can seemingly format logs as JSON by providing a logback-spring.xml file as follows:
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>true</prettyPrint>
</jsonFormatter>
</layout>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="stdout" />
</root>
and adding to the pom.xml
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
<version>0.1.5</version>
</dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-jackson</artifactId>
<version>0.1.5</version>
</dependency>
indeed leading to messages like:
{
"timestamp" : "2018-12-11T18:20:25.641Z",
"level" : "INFO",
"thread" : "main",
"logger" : "com.netflix.config.sources.URLConfigurationSource",
"message" : "To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.",
"context" : "default"
}
Why?
I'm trialing logz.io which appears to behave more favourably when logs are JSON formatted, some o the shippers struggle with multiline logs like we see in java stack traces and when formatting in JSON it can automatically parse fields like level and message and if there is MDC data it automatically gets that.
I had some not so great experiences with a few of the methods of shipping logs to logzio, like their docker image and using rsyslog without using JSON formatted log messages.
Issues With This Approach
It works ok for console appending, but spring boot provides like logging.file=test.log, logging.level.com.example=WARN, logging.pattern.console. I can indeed import the managed configuration from spring-boot-2.1.1.RELEASE.jar!/org/springframework/boot/logging/logback/base.xml which in turn imports a console-appender.xml andfile-appender.xml`.
An example of the console-appender
<included>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>${CONSOLE_LOG_PATTERN}</pattern>
</encoder>
</appender>
</included>
An example of the file appender
<included>
<appender name="FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<encoder>
<pattern>${FILE_LOG_PATTERN}</pattern>
</encoder>
<file>${LOG_FILE}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.%i.gz</fileNamePattern>
<maxFileSize>${LOG_FILE_MAX_SIZE:-10MB}</maxFileSize>
<maxHistory>${LOG_FILE_MAX_HISTORY:-0}</maxHistory>
</rollingPolicy>
</appender>
</included>
These two are exactly what I need to support spring configuration of the properties, but they don't include the encoder/layout I'd need.
It appears in my initial tests that I can't simple name my appender the same as those and provide my layouts. For example:
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml"/>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>true</prettyPrint>
</jsonFormatter>
</layout>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>
leads to the message being logged in both JSON and plain text format.
I can indeed just copy and paste the contents of these 3 files into my custom config rather than import them at all. Then I may override what I want to customise.
However, as spring evolves and new releases are made which may add features, I'd be forever forcing myself to keep up, copy and paste the new files and make my changes and test them.
Is there any better way that I can either:
Just make additive changes to the appenders rather than entirely redefine them, e.g. keep the config from spring but provide my own encoder or layout to be used by those appenders.
Configure spring to JSON log via properties entirely without any config - I doubt this :S
Footnote: logzio do provide a dependency one can import, but I dislike the idea of coupling the logging provider into the code directly. I feel that if the servoce happens to produce JSON logs to stdout or a file, it's easy for any provider to process those and ship them to some destination.
I am not using any dependency.
Simply, doing it via application.yml, that's all.
This solution solves, multiline log issue, too.
logging:
pattern:
console: "{\"time\": \"%d\", \"level\": \"%p\", \"correlation-id\": \"%X{X-Correlation-Id}\", \"source\": \"%logger{63}:%L\", \"message\": \"%replace(%m%wEx{6}){'[\r\n]+', '\\n'}%nopex\"}%n"
I use something like the following, has always worked fine.
Spring Boot recommendation is to name the file logback-spring.xml and place it under src/main/resources/, this enables us to use spring profiles in logback.
So in the file below you will see that for LOCAL profile you can log in the standard fashion but for the deployments on the server or a container you can you a different logging strategy.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %5p [YourApp:%thread:%X{X-B3-TraceId}:%X{X-B3-SpanId}] %logger{40} - %msg%n
</pattern>
</encoder>
</appender>
<appender name="jsonstdout" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<providers>
<timestamp>
<timeZone>EST</timeZone>
</timestamp>
<pattern>
<pattern>
{
"level": "%level",
"service": "YourApp",
"traceId": "%X{X-B3-TraceId:-}",
"spanId": "%X{X-B3-SpanId:-}",
"thread": "%thread",
"class": "%logger{40}",
"message": "%message"
}
</pattern>
</pattern>
<stackTrace>
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<maxDepthPerThrowable>30</maxDepthPerThrowable>
<maxLength>2048</maxLength>
<shortenedClassNameLength>20</shortenedClassNameLength>
<rootCauseFirst>true</rootCauseFirst>
</throwableConverter>
</stackTrace>
</providers>
</encoder>
</appender>
<root level="info">
<springProfile name="LOCAL">
<appender-ref ref="stdout" />
</springProfile>
<springProfile name="!LOCAL">
<appender-ref ref="jsonstdout" />
</springProfile>
</root>
</configuration>
Sounds like you need to copy paste with modifications 3 out of 4 files from here https://github.com/spring-projects/spring-boot/tree/v2.1.1.RELEASE/spring-boot-project/spring-boot/src/main/resources/org/springframework/boot/logging/logback into your configuration.
The good news is that you don't need to copy paste https://github.com/spring-projects/spring-boot/blob/v2.1.1.RELEASE/spring-boot-project/spring-boot/src/main/resources/org/springframework/boot/logging/logback/defaults.xml
That can be included like so <include resource="org/springframework/boot/logging/logback/default.xml"/>
That will get you some of spring's default config without copying and pasting
If you switch to log4j2 using the method specified in the Spring Boot documentation
implementation "org.springframework.boot:spring-boot-starter-log4j2"
modules {
module("org.springframework.boot:spring-boot-starter-logging") {
replacedBy("org.springframework.boot:spring-boot-starter-log4j2", "Use Log4j2 instead of Logback")
}
}
You can have a file log4j2.properties that contains the following.
appender.stdout.type=Console
appender.stdout.name=json
appender.stdout.json.type=JsonTemplateLayout
appender.stdout.json.eventTemplateUri=classpath:LogstashJsonEventLayoutV1.json
appender.console.type=Console
appender.console.name=console
rootLogger.appenderRef.stdout.ref=${env:LOGGING_APPENDER:-json}
I used LOGGING_APPENDER to match my environment level overrides.
Note one odd flaw though... this does not work when you're on JDK17.

SpringBoot with LogBack creating LOG_FILE_IS_UNDEFINED folder

I am using SpringBoot with LogBack and I am trying to direct all log-statements of one specific package (here shown as "com.example.somepackagename") to a file. All other log statements should go to stdout.
At first ran into the issue that during startup a file "LOG_FILE_IS_UNDEFINED" was created. I then googled on that issue (found e.g. this on github or this on StackOverflow and a few others), but none of the appends really solved the issue.
Based on the misc. descriptions I came up with the below configuration using a logback-spring.xml-file:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property name="LOG_PATH" value="${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}" />
<property name="LOG_FILE" value="${LOG_FILE:-ts-msgs.log}" />
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<Pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</Pattern>
</encoder>
</appender>
<appender name="MSG_LOG_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/${LOG_FILE}</file>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } --- [%t] %-40.40logger{39} : %m%n%wex</Pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- rollover daily -->
<fileNamePattern>${LOG_PATH}/archived/debug.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>10MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<logger name="com.example.somepackagename" additivity="false" level="debug">
<appender-ref ref="MSG_LOG_FILE" />
</logger>
<logger name="org.springframework" additivity="false" level="info">
<appender-ref ref="STDOUT" />
</logger>
<root level="warn">
<appender-ref ref="STDOUT" />
</root>
</configuration>
With that I don't get a file called "LOG_FILE_IS_UNDEFINED" anymore but LogBack now always first creates an empty file "ts-msgs.log" in the working directory of the application before it creates a file with the same name in the temp-directory (where that file should go). So, somehow during initialization it already creates a file with the correct name but the correct path not being set, yet. In other words, the initialization order of Logback within SpringBoot seems broken (or at least not intuitive) and it seems impossible to avoid a first log-file being created before the correct path for the file-appender is being defined and applied.
How can one prevent that first, initial and empty log file from being created? This is necessary, since the working directory is in general write-protected and the attempt to create a file there will lead to an exception (and typically the program being terminated).
Any hints or suggestions?
BTW: I already tried to rename that file to logback.xml (which seems to be processed earlier during SpringBoot startup) but that didn't solve the issue, so I kept the suggested filename.
I get similar issue, on application start following empty file might be generated:
LOG_PATH_IS_UNDEFINED
LOG_FILE_IS_UNDEFINED
Reason
Logging related config in application.yml, is read after logback-spring.xml is parsed, thus can't read it.
Possible solutions
I found 3 possible solutions, choose one that best fit your case:
Move logging config from application.yml to bootstrap.yml, this will need spring-cloud dependency to make it work.
This works because bootstrap.yml is read before logback-spring.xml.
If you are not using spring-cloud, this probably won't be your best choice, since the extra dependencies is unnecessary.
Define path & file in logback-spring.xml directly.
e.g
<configuration>
<springProperty name="LOG_PATH" source="logging.path" defaultValue="logs/" />
<springProperty name="LOG_FILE" source="logging.file" defaultValue="app.log" />
In this case, might need to add "logback-spring.xml" to each sub project, if the log file names need to be different, and can't simply put the config file in a shared common dependency.
Just keep the config in application.yml, and ignore the generated empty files by setting .gitinore.
e.g
LOG_*_IS_UNDEFINED
In this case, the logs are still written to the file specified by application.yml, though the empty file is generate.
Tips
The logback-spring.xml file mentioned above, might be logback.xml or some other name in your case.
Removing :
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml"/>
from logback.xml fixed the problem for me.

How to make Playframework Evolutions logging more verbose?

I've been trying to see what's going on with them, as I'm facing a problem.
When I click the "apply this script" button, the page asking me to apply the script appears again, and it stucks in that loop. It creates the "play_evolutions" table, but no script is run.
The log in the console doesn't give me any relevant info. I've run manually the script and it's ok... So I wan't to know what the real problem is, therefore I need more logging.
Evolutions are not very verbose in general (when it talks it mostly does that on warn level).
Basically make sure you have this line at you should get everything play-evolutions produce (it should be in logback.xml file):
<logger name="play.api.db.evolutions" level="DEBUG" />
Full file (as an example only):
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%level %logger{15} - %message%n%xException</pattern>
</encoder>
</appender>
<logger name="play.api.db.evolutions" level="DEBUG" />
<root level="INFO">
<appender-ref ref="STDOUT" />
</root>
</configuration>

Spring - Exceptions do not log to a file

I'm currently using SLF4J API for logging.
Whenever an exception is thrown during runtime, the full error stack trace does not log to file, it is only printed to the console. I'm using eclipse.
Here is my code for logback.xml (currently located in classes folder under WEB-INF)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE xml>
<configuration>
<!-- Specify here the path of the folder you want to save your logs -->
<property name="LOGFILE_PATH" value="C:/Logs" />
<!-- All logging will be redirected/ printed to console. -->
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%d{yyyy-MM-dd hh:mm:ss a} [%thread] %-5level %logger{50} - %rEx %msg%n </Pattern>
</layout>
</appender>
<!-- Send log to file -->
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>${LOGFILE_PATH}/spring-mybatis-log.log</File>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%d{yyyy-MM-dd hh:mm:ss a} [%thread] %-5level %logger - %rEx %msg%n</pattern>
</layout>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOGFILE_PATH}/spring-mybatis-log-%d{yyyy-MM-dd}-%i.txt
</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy
class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>2MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<root level="DEBUG">
<appender-ref ref="STDOUT" />
<appender-ref ref="FILE" />
</root>
</configuration>
Is there something missing/wrong with the above file??
Is it possible to log (to a file) all the text that will be printed to console?
How does spring(or the project itself) read the logback.xml file? What if I rename it and place it in another folder?
How to create one root containing all the levels (INFO, DEBUG, ERROR, WARN, etc..) ?
To answer your questions:
Nothing looks obviously wrong to me in the file you've posted, although I didn't try actually running it.
Just the way you did it is fine, with two appenders, one that goes to the file and the other to go to the console.
Logback by default looks in the classpath for the logback.xml file. Refer to the configuration page of the manual for the details. The way it gets there depends on your build system. When using Maven, I'd suggest putting it in src/main/resources. But if it ends up in WEB-INF/classes when deployed in your web app, that should work. If no matter what you put in your logback.xml file you are only getting console output (try adding a syntax error or renaming the file to test), that's what I'd look at first, to ensure that Logback is picking up the file right. It will default to showing everything just on the console if it can't find the file, though I think it shows a warning at the beginning that it's doing so. If it is picking up the file, you can try putting debug="true" in the <configuration> element, to see if there's an error that it's picking up that's causing it to not use the appender the way you're expecting.
Specifying the "DEBUG" level of logging, as you've done, will also get all higher levels as well.
If the issue is that logging from Spring isn't going where you want, while your application's logging is working fine, you may need to redirect Spring (which uses Apache Commons Logging) to use SLF4J instead. To do that, remove the commons-logging dependency and add the jcl-over-slf4j library. This will emulate the commons-logging calls and have them point to SLF4J instead. See the "Using SLF4J" section of this blog post for more details.
Your configuration looks OK, but you can try use your root level INFO instead DEBUG and if you have frameworks like spring, hibernate etc. Logback allow you uses another levels to them something like:
<logger name="org.hibernate" level="OFF" />
<logger name="org.springframework" level="INFO" />
<logger name="org.springframework.web.servlet.mvc" level="INFO" />

Resources