How to enable the spring-boot application to print the logging directory while startup?, like below example:
logging directory - /home/test/studentapp.log
Irrespective of any logging framework we use like LOGBACK, Log4j2, Java Util logging ..e.t.c in Spring, want to print the logging directory while startup.
Is there any property needs to be configured at application.properties to enable this?
Edit:
For example, if I use the below logging logback.xml configuration, then I want to log the logging directory mentioned in the below configuration to the console at application startup, Like If you start kafka server It will give the location of the logging directories in the console at startup. In a similar way can we have this feature in spring?
<appender name="STUDENT_APP_LOG_APPENDER" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/home/test/studentapp.log</file>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>[%d{ dd-MM-yyyy HH:mm:ss.SSS }] [ %5p ] [%thread] [ %logger ] %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>/home/test/archived/studentapp-%d{dd-MM-yyyy HH:mm:ss}.%i.log</fileNamePattern>
<maxFileSize>100MB</maxFileSize>
<totalSizeCap>10GB</totalSizeCap>
<maxHistory>90</maxHistory>
</rollingPolicy>
</appender>
I have a Spring Boot application running in a Kubernetes cluster and a EFK stack (like ELK, but with instead of Logstash, Fluentd is used as a lightweight alternative to collect logs from all kubernetes pods and sends them to elasticsearch).
In order to adapt the logs to a JSON output, I have used a logstash-logback-encoder library:
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>4.11</version>
</dependency>
And out of the box I had my logs converted to JSON (which is great).
I log to STDOUT, everything gets picked up and sent to Elasticsearch.
No special configuration for logging is needed inside the Spring Boot application.
But the problem I have right now is that while reading my logs in realtime from the STDOUT of the Kubernetes pod, they are very hard to read with all the JSON formatting.
Example:
{"#timestamp":"2018-02-08T12:49:06.080+01:00","#version":1,"message":"Mapped \"{[/error],produces=[text/html]}\" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)","logger_name":"org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping","thread_name":"main","level":"INFO","level_value":20000}
{"#timestamp":"2018-02-08T12:49:06.080+01:00","#version":1,"message":"Mapped \"{[/error]}\" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)","logger_name":"org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping","thread_name":"main","level":"INFO","level_value":20000}
{"#timestamp":"2018-02-08T12:49:06.098+01:00","#version":1,"message":"Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]","logger_name":"org.springframework.web.servlet.handler.SimpleUrlHandlerMapping","thread_name":"main","level":"INFO","level_value":20000}
{"#timestamp":"2018-02-08T12:49:06.098+01:00","#version":1,"message":"Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]","logger_name":"org.springframework.web.servlet.handler.SimpleUrlHandlerMapping","thread_name":"main","level":"INFO","level_value":20000}
{"#timestamp":"2018-02-08T12:49:06.137+01:00","#version":1,"message":"Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]","logger_name":"org.springframework.web.servlet.handler.SimpleUrlHandlerMapping","thread_name":"main","level":"INFO","level_value":20000}
{"#timestamp":"2018-02-08T12:49:06.268+01:00","#version":1,"message":"Registering beans for JMX exposure on startup","logger_name":"org.springframework.jmx.export.annotation.AnnotationMBeanExporter","thread_name":"main","level":"INFO","level_value":20000}
{"#timestamp":"2018-02-08T12:49:06.333+01:00","#version":1,"message":"Initializing ProtocolHandler [\"http-nio-8080\"]","logger_name":"org.apache.coyote.http11.Http11NioProtocol","thread_name":"main","level":"INFO","level_value":20000}
{"#timestamp":"2018-02-08T12:49:06.355+01:00","#version":1,"message":"Starting ProtocolHandler [\"http-nio-8080\"]","logger_name":"org.apache.coyote.http11.Http11NioProtocol","thread_name":"main","level":"INFO","level_value":20000}
What I want to do is Log to STDOUT in 'normal non-JSON' format then send logs to Fluentd in JSON format.
I am trying to configure two log appenders (one to STDOUT and another in JSON format for Fluentd) but I am pretty sure that this will duplicate the data (Fluentd will get the JSON format AND the STDOUT).
My plan B is to build one image for deployment (without the JSON format) and another for production but this is more like plan Z, tbh because I want to monitor those pods in production as well.
My question is How can I do this with possibly one log appender OR without duplicating the data in Fluentd. Is there maybe a different approach that I haven't thought of?
I'm assuming you're using Logback and not Log4j that you have tagged since the logstash library you linked to seems to be written for Logback.
The easiest solution would probably be to configure fluentd to read logs from a file and redirect the JSON appender to that file.
There's an article about the tail input plugin, but essentially you configure it like this:
logback.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- JSON appender for log collection -->
<appender name="json" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/some/path/to/your/file.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>/some/path/to/your/file.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>
<!-- Console appender for humans -->
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<!-- Set threshold for the console log here if you want the
log collection to get all log messages regardless of level -->
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg %n</pattern>
</encoder>
</appender>
<!-- Tie it all together -->
<root level="all">
<appender-ref ref="json" />
<appender-ref ref="console" />
</root>
</configuration>
fluentd
<source>
#type tail
path /some/path/to/your/file.log
pos_file /some/path/to/your/file.log
format json
</source>
Accourding to the documenation, fluentd will follow rollovers and start from the beginning of the new file after finishing the old file.
If you want the console output to be the same as a regular Spring Boot application you can copy the pattern from their configuration
Even though I was tempted for a solution that was proposed, at the end I just used jq, a json parser to view my logs on the cli. I did so in order to avoid duplicating log data and not having to create files nor having to specially configure fluentd for reading logs from files.
In application.yml of my spring-boot application, I have the following code:
logging:
file: test.log
pattern:
file: "%d %-5level [%thread] %logger : %msg%n"
I would write something like
logging:
level:
org: hibernate: *
I would to filter and gather all log messages of all levels (ERROR, DEBUG, INFO, etc) ifrom only one specific package (org.hibernate) in "test.log". It is possible?
Yes, you can define log level for each of your/third library packages:
Example for application.yml:
# Fine-tuning a specific logger (for a single class)
logging:
level:
yourpackage: INFO
org:
hibernate: ALL
springframework: INFO
Example for application.properties:
# Fine-tuning a specific logger (for a single class)
logging.level.yourpackage.controllers.HomeController = INFO
# Specify the level for spring boot and hibernate's loggers
logging.level.org.springframework = INFO
logging.level.org.hibernate = ALL
Update 1: Only logging hibernate.* to a log file
If you want to log only one specific Java package to your log file add a logback.xml to your src/main/resources folder and add the following settings:
<configuration>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>hibernateLogs.log</file>
<encoder>
<pattern>%date %level [%thread] %logger{10} [%file:%line] %msg%n</pattern>
</encoder>
</appender>
<logger name="org.hibernate" level="ALL">
<appender-ref ref="FILE" />
</logger>
</configuration>
This config will log all of your hibernate logs to a file.
I am using SpringBoot with LogBack and using the below configuration in my yml file:
logging:
path: C:/var/log/pincode
The logging.path Spring Environment Variable is transferred to the LOG_PATH Environment variable and the log file is placed at the correct place, but there is also a directory called LOG_PATH_IS_UNDEFINED created in the root directory of my project.
This seems to be caused by the different phase used by SpringBoot to configure LogBack with its Environment variables.
17:29:21,325 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
17:29:21,337 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy - Will use the pattern LOG_PATH_IS_UNDEFINED/catalina.out.%d{yyyy-MM-dd} for the active file
17:29:21,340 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - The date pattern is 'yyyy-MM-dd' from file name pattern 'LOG_PATH_IS_UNDEFINED/catalina.out.%d{yyyy-MM-dd}'.
17:29:21,340 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Roll-over at midnight.
17:29:21,343 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Setting initial period to Mon Aug 11 17:24:07 BRT 2014
17:29:21,346 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[serverConsole] - Active log file name: LOG_PATH_IS_UNDEFINED/catalina.out
17:29:21,346 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[serverConsole] - File property is set to [LOG_PATH_IS_UNDEFINED/catalina.out]
...
17:29:21,358 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
And then after that it start configuring logback again but this time using the path i set:
17:29:21,672 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
17:29:21,673 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy - No compression will be used
17:29:21,673 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy - Will use the pattern C:/var/log/pincode//catalina.out.%d{yyyy-MM-dd} for the active file
17:29:21,674 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - The date pattern is 'yyyy-MM-dd' from file name pattern 'C:/var/log/pincode//catalina.out.%d{yyyy-MM-dd}'.
17:29:21,674 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Roll-over at midnight.
17:29:21,674 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Setting initial period to Mon Aug 11 17:29:21 BRT 2014
17:29:21,674 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[serverConsole] - Active log file name: C:/var/log/pincode//catalina.out
17:29:21,674 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[serverConsole] - File property is set to [C:/var/log/pincode//catalina.out]
...
17:29:21,685 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
My logback.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
<include resource="org/springframework/boot/logging/logback/basic.xml" />
<property name="FILE_LOG_PATTERN"
value="%d{yyyy-MM-dd HH:mm:ss.SSS} %5p ${PID:- } [%t] --- %-40.40logger{39} : %m%n%wex" />
<appender name="serverConsole"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<Append>true</Append>
<File>${LOG_PATH}/catalina.out</File>
<encoder>
<pattern>${FILE_LOG_PATTERN}</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/catalina.out.%d{yyyy-MM-dd}
</fileNamePattern>
<maxHistory>15</maxHistory>
</rollingPolicy>
</appender>
<!-- Plain Text Rolling Appender -->
<appender name="server"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<Append>true</Append>
<File>${LOG_PATH}/pincode.log</File>
<filter class="ch.qos.logback.classic.filter.LevelFilter">
<level>INFO</level>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
<encoder>
<pattern>${FILE_LOG_PATTERN}</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/pincode.log.%d{yyyy-MM-dd}
</fileNamePattern>
<maxHistory>15</maxHistory>
</rollingPolicy>
</appender>
<!-- Plain Text Rolling Appender -->
<appender name="server-error"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<Append>true</Append>
<File>${LOG_PATH}/pincode-error.log</File>
<filter class="ch.qos.logback.classic.filter.LevelFilter">
<level>ERROR</level>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
<encoder>
<pattern>${FILE_LOG_PATTERN}</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/pincode-error.log.%d{yyyy-MM-dd}
</fileNamePattern>
<maxHistory>15</maxHistory>
</rollingPolicy>
</appender>
<logger name="com.app" level="INFO">
<appender-ref ref="server" />
<appender-ref ref="server-error" />
</logger>
<root level="INFO">
<appender-ref ref="serverConsole" />
</root>
If I remove my logback.xml file from the project it doesn't create the folder, so somewhere Spring is loading the xml before parsing the yml?
How can I avoid Logback to create this LOG_PATH_IS_UNDEFINED directory?
In your case LOG_PATH is not defined on startup. You should use ${LOG_PATH:-.} instead , See .
But if you define logging.path in your application.properties you will see two log files in . and in ${logging.path} directory.
Spring container set LOG_PATH after Logback initialization... Logback is not supported lazy file creation as far as I know. In this case you should use logback-spring.xml instead logback.xml.
I faced similar issue and its easy to solve it. Basically , concept is that Spring Boot already gives you System property - LOG_PATH for Spring Boot property - logging.path so you define logging.path in your application.properties and simply use LOG_PATH in your logback configuration - logback-spring.xml.
You shouldn't declare logback <property ...> for LOG_PATH , just use it whenever you want.
See at near bottom here
I encountered the same problem.
put an entry in logback.xml
<property resource="application.properties" />
In application.properties
FILE_LOG_PATTERN=###
LOG_FILE=###
when your application starts,the name of the directory created is what you have defined in the properties file.
Might not be your case but if you have bootstrap.properties make sure logging.path is defined there and only there.
if you're on Spring Boot Finchley (2.x), you can define spring.application.name in your application.properties or application.yml file and add the following in your Logback configuration:
<configuration>
<springProperty scope="context" name="springAppName" source="spring.application.name"/>
</configuration>
you will now have ${springAppName} as a variable at your disposal for your log pattern.
Before Spring Boot enviroment is prepared, the Spring Boot main class or the SpringApplication will initialize the loggerfactory, which will detect the default configuration file (logback.groovy, logback.xml, logback-test.xml), but at this time the Spring Boot application is not started yet, which means the variable LOG_PATH is not set. So at first you should alter the name of your logback config file and configure the file name manually in the Spring Boot config as logging.config. By this way the logback will configure a console appender by default instead of creating a file appender, and then when the Spring Boot enviroment is ready it will fire an enviromentready event, which causes a logback reconfig by LoggingApplicationListener. You can find the issue at springboot's page https://github.com/spring-projects/spring-boot/issues/2558
If you upgrade your Spring Boot version into pom.xml, make sure that your replaced
logging.path = your/log/path
by
logging.file.path = your/log/path
into application.properties. That was my case.
I encountered the same problem. I tried to define my own logback.xml and had trouble using the logging.path and logging.file properties defined in my application.properties file. Here is how I resolved (and worked around) the issues.
First, I learned that you cannot define both logging.path and logging.file properties. Since I'm using a RollingFileAppender that will produce multiple files over multiple days, I define logging.file, but use it more like a file prefix.
In application.properties
# Don't add the file type at the end. This will be added in logback.xml
logging.file=logs/my-app-name
In src/main/resources/logback.xml
<configuration>
<property name="FILE_LOG_PATTERN" value="%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"/>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<Pattern>${FILE_LOG_PATTERN}</Pattern>
</encoder>
</appender>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<encoder>
<Pattern>${FILE_LOG_PATTERN}</Pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.log</fileNamePattern>
<maxHistory>30</maxHistory>
</rollingPolicy>
</appender>
<root level="INFO">
<appender-ref ref="STDOUT" />
<appender-ref ref="FILE" />
</root>
</configuration>
This seems to work for the most part. I defined my own FILE_LOG_PATTERN in the file, which I think is optional. The interesting part is the fileNamePattern. It correctly translates logging.file from my application.properties file into the variable LOG_FILE. The only real ugliness here is that on startup Logback still complains about the log file being undefined and creates an empty file called LOG_FILE_IS_UNDEFINED_XXX in the current directory. But the actual log file in my property is created and correctly appended to.
Andrew
somewhere Spring is loading the xml before parsing the yml
so just rename logback.xml to your-logback.xml and add logging.config=classpath:your-logback.xml in your application.properties
Do below to create dev/log directory only. Do not add log.path in application.properties
Add log.path=dev/logs in your bootstrap.properties.
Add below line in your logback-spring.xml.
<springProperty scope="context" name="LOG_PATH" source="log.path"/>
<property name="LOG_FILE" value="${LOG_PATH}/app/current"/>
Note
Make sure you include the below line only.
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
Do not use the below line else console logs will never be disabled and spring.log file will be created in temp directory(if you dont not provide logging.path in application.properties). The check the code of below file to know more.
<include resource="org/springframework/boot/logging/logback/base.xml"/>
put an entry in logback:
<property name="DEV_HOME" value="c:/application_logs/ps-web" />
and reference it:
<fileNamePattern>${DEV_HOME}.%d{yyyy-MM-dd}.log</fileNamePattern>
I had the same problem since I configured logging.path and logging.file on application.properties but some logs was produced before Spring Boot LogBack configuration and so they were written into LOG_PATH_IS_UNDEFINED/LOG_FILE_IS_UNDEFINED file and then the logs switched to the right location.
I found 2 possible solutions:
1) Configure logging.path and logging.file in bootstrap.properties since the configuration in bootstrap take place before
or
2) Set logging.path and logging.file as system properties with -Dlogging.path=... -Dlogging.file=... when launching the application
Based on Spring Boot common properties,
add the following into your application.yml
logging:
file:
path: logs/dev
if using application.properties, it should be
logging.file.path = logs/dev
Declare the property LOG_PATH in your logback.xml
<property name="LOG_PATH" value="" />
is where you must specify the directory where the log files are created. If this field is left empty, logback will create a new directory in the program execution. The name of the directory created is LOG_PATH_IS_UNDEFINED
Try adding the following to your POM file
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<filesets>
<fileset>
<directory>${basedir}/catalina.base_IS_UNDEFINED</directory>
<includes>
<include>**/*.log</include>
</includes>
<followSymlinks>false</followSymlinks>
</fileset>
</filesets>
</configuration>
</plugin>
To call an logback form an external path in a .yml file, it worked for me as:
logging:
config: C:/folder/logback.xml
I suppose you have included an error file.
Please change
<include resource="org/springframework/boot/logging/logback/basic.xml" />
to
<include resource="org/springframework/boot/logging/logback/base.xml"/>
then have a try.
version: 1.5.9
springcloud:
bootstrap.yml
spring:
application:
name: awesome-app
# profile:
# active: dev
logging:
path: /code/awesome-app
spring-logback.xml
${LOG_PATH}/awesome-app.log
springboot:
application.yml
spring:
application:
name: awesome-app
# profile:
# active: dev
logging:
path: /code/awesome-app
spring-logback.xml
${LOG_PATH}/awesome-app.log
custom-log-configuration:https://docs.spring.io/spring-boot/docs/1.5.6.RELEASE/reference/htmlsingle/#boot-features-custom-log-configuration
I was also having similar issue. I resolved it by renaming logback-spring.xml to logback-whatever.xml and added below in application.properties:
logging.config=classpath:logback-whatever.xml
Also, this issue comes when we use user defined properties for logging purposes such as:
log.path=logs
log.archive.path=archived
What worked for me:
having logback-spring.xml under resources folder (the same as yamls)
setting the environment variable in the application.yaml
after modifications, do a ./gradlew clean build
Happy new year everybody!
So, I've updated my spring boot to version (2.4.1) and I started getting the LOG_PATH_IS_UNDEFINED error.
I did a bit of research and I'm guessing there is a problem with the mapping of the properties from logging.path to LOG_PATH. (I manually debugged the logger and the properties were being load)
My solution/Patch:
I added a manual mapping to the logback-spring.xml at the very top:
<springProperty scope="context" name="LOG_PATH" source="logging.path"/>
Now it is working for me...