I create loggback.xml woth thos appender:
<appender name="ROLLING-FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--<file>logs/nfp-server.log</file>-->
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<!-- rollover daily -->
<fileNamePattern>D:/logs/archived/log_%d{yyyy-MM-dd}.%i.txt</fileNamePattern>
<!-- each file should be at most 100MB, keep 60 days worth of history, but at most 20GB -->
<maxFileSize>10KB</maxFileSize>
<maxHistory>60</maxHistory>
<totalSizeCap>20MB</totalSizeCap>
</rollingPolicy>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
<springProfile name="local">
<logger name="nfp" level="WARN" additivity="false">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="ROLLING-FILE"/>
</logger>
</springProfile>
<springProfile name="prod">
<logger name="nfp" level="WARN" additivity="false">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="ROLLING-FILE"/>
</logger>
</springProfile>
and in spring propery:
#loging
logging.file=D:/logs/nfp-server_local.log
I want each log file > 10KB move to archive. But when I start server in D:/logs I have this:
-logs
-nfp-server_local.log (int this file all logs, 48KB)
-archived
-log_2018-04-02.0.txt (0 byte)
Related
these screenshot are my configurations about logback
logback-spring.xml
<root level="info">
<appender-ref ref="FILE"/>
<appender-ref ref="STDOUT"/>
</root>
application.properties
logging.file.path=./logs
logging.level.cn.hsa=debug
logging.level.root=info
logging.level.org.springframework.web=trace
It’s not working when i change logging.level.root to debug or others in application.properties, it's always use this configure in logback-spring.xml
I read spring source code, logback-srping.xml loaded before application.properties
but i dont know why these configures were not override
You can define something like this if you want to customize:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<springProperty scope="context" name="LOG_PATH" source="app.log.path"
defaultValue="your_log_path" />
<timestamp key="dateStamp" datePattern="yyyy-MM-dd" />
<appender name="consoleAppender" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<Pattern>[%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c{1} - %msg%n
</Pattern>
</encoder>
</appender>
<springProfile name="test,prod">
<appender name="Access-FileAppender"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- daily rollover -->
<fileNamePattern>${LOG_PATH}/logs/xyz.%d{yyyy-MM-dd}.log
</fileNamePattern>
<!-- keep 90 days' worth of history -->
<maxHistory>90</maxHistory>
</rollingPolicy>
<encoder>
<Pattern>[%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c{1} - %msg%n
</Pattern>
</encoder>
</appender>
<appender name="Async-Access-FileAppender" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="Access-FileAppender" />
</appender>
</springProfile>
<root>
<level value="INFO" />
<appender-ref ref="consoleAppender" />
</root>
<springProfile name="local">
<logger name="com.xxx.yyy" level="DEBUG" additivity="false">
<appender-ref ref="consoleAppender" />
</logger>
</springProfile>
<springProfile name="dev">
<logger name="com.xxx.yyy" level="DEBUG" additivity="false">
<appender-ref ref="consoleAppender" />
</logger>
</springProfile>
<springProfile name="test,prod">
<logger name="com.xxx.yyy" level="INFO" additivity="true">
<appender-ref ref="consoleAppender" />
</logger>
</springProfile>
</configuration>
As you can see, test/prod have the similar logging configuration. Similarly, you can configure for local and dev as well.
Hope this helps.
I am using logback/slf4j to do my logging. I want to parse my log file to analyze some data, so instead of parsing a great big file (mostly consisting of debug statements) I want to have two logger instances which each log to a separate file; one for analytics and one for all purpose logging. Does anyone know if this is possible with Logback, or any other logger for that matter?
It's very possible to do something like this in logback. Here's an example configuration:
<?xml version="1.0"?>
<configuration>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>logfile.log</file>
<append>true</append>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg %n</pattern>
</encoder>
</appender>
<appender name="ANALYTICS-FILE" class="ch.qos.logback.core.FileAppender">
<file>analytics.log</file>
<append>true</append>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg %n</pattern>
</encoder>
</appender>
<!-- additivity=false ensures analytics data only goes to the analytics log -->
<logger name="analytics" level="DEBUG" additivity="false">
<appender-ref ref="ANALYTICS-FILE"/>
</logger>
<root>
<appender-ref ref="FILE"/>
</root>
</configuration>
Then you'd setup two separate loggers, one for everything and one to log analytics data like so:
Logger analytics = LoggerFactory.getLogger("analytics");
You can have as many loggers as you wish. But, it's better you have one for each package that you need to log differently. Then all the classes in that package and its sub-packages will get the that specific logger. They all can share the root logger and send their log data to root logger appender using additivity="true". Here's an example:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property name="pattern" value="%date{HH:mm:ss.SSS} %-5p %logger{36}
%X{akkaSource} [%file:%line] - %m%n" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%date{HH:mm:ss.SSS} %-5p %logger{36} %X{akkaSource} [%file:%line] - %m%n</pattern>
</encoder>
</appender>
<appender name="abc" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${catalina.base}/logs/worker.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${catalina.base}/logs/worker-%d{yyyy-MM-dd_HH}.log</fileNamePattern>
<maxHistory>360</maxHistory>
</rollingPolicy>
<encoder>
<pattern>${pattern}</pattern>
</encoder>
</appender>
<appender name="xyz" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${catalina.base}/logs/transformer.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${catalina.base}/logs/transformer-%d{yyyy-MM-dd_HH}.log</fileNamePattern>
<maxHistory>360</maxHistory>
</rollingPolicy>
<encoder>
<pattern>${pattern}</pattern>
</encoder>
</appender>
<logger name="com.xxx.yyy.zzz" level="INFO" additivity="true">
<appender-ref ref="xyz"/>
</logger>
<logger name="com.aaa.bbb.ccc" level="INFO" additivity="true">
<appender-ref ref="abc"/>
</logger>
<root>
<level value="INFO" />
<appender-ref ref="STDOUT" />
</root>
in my case I wanted to leave class names as log name
private static final Logger log = LoggerFactory.getLogger(ScheduledPost.class);
and as I had few such classes, so my logback.xml
<!--additivity=false ensures this log data only goes to the this log, and no one more -->
<logger name="xxx.xxx.xxx.ScheduledPost" level="DEBUG" additivity="false">
<appender-ref ref="ASYNC_SCHEDULE_LOG_FILE"/>
</logger>
<logger name="xxx.xxx.xxx.GcmPost" level="DEBUG" additivity="false">
<appender-ref ref="ASYNC_SCHEDULE_LOG_FILE"/>
</logger>
<logger name="xxx.xxx.xxx.PushUtils" level="DEBUG" additivity="false">
<appender-ref ref="ASYNC_SCHEDULE_LOG_FILE"/>
</logger>
NiFi 1.2.0
I am having a custom processor and I wish to have a dedicated log file for the same. Accordingly, I have configured the com.datalake.processors.SQLServerCDCProcessor class to use an appender named 'SQLSERVER-CDC'
Following is the logback.xml :
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="30 seconds">
<contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
<resetJUL>true</resetJUL>
</contextListener>
<appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/var/log/nifi/nifi-app.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!--
For daily rollover, use 'app_%d.log'.
For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
To GZIP rolled files, replace '.log' with '.log.gz'.
To ZIP rolled files, replace '.log' with '.log.zip'.
-->
<fileNamePattern>/var/log/nifi/archive/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>100MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<!-- keep 30 log files worth of history -->
<maxHistory>3</maxHistory>
</rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
<immediateFlush>true</immediateFlush>
</appender>
<appender name="USER_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/var/log/nifi/nifi-user.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!--
For daily rollover, use 'user_%d.log'.
For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
To GZIP rolled files, replace '.log' with '.log.gz'.
To ZIP rolled files, replace '.log' with '.log.zip'.
-->
<fileNamePattern>/var/log/nifi/archive/nifi-user_%d.log</fileNamePattern>
<!-- keep 30 log files worth of history -->
<maxHistory>3</maxHistory>
</rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender>
<appender name="BOOTSTRAP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/var/log/nifi/nifi-bootstrap.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!--
For daily rollover, use 'user_%d.log'.
For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
To GZIP rolled files, replace '.log' with '.log.gz'.
To ZIP rolled files, replace '.log' with '.log.zip'.
-->
<fileNamePattern>/var/log/nifi/archive/nifi-bootstrap_%d.log</fileNamePattern>
<!-- keep 5 log files worth of history -->
<maxHistory>5</maxHistory>
</rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender>
<!-- Start : Added for log for custom processor -->
<appender name="SQLSERVER-CDC" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/var/log/nifi/sqlserver-cdc.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!--
For daily rollover, use 'app_%d.log'.
For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
To GZIP rolled files, replace '.log' with '.log.gz'.
To ZIP rolled files, replace '.log' with '.log.zip'.
-->
<fileNamePattern>/var/log/nifi/archive/sqlserver-cdc_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>25MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<!-- keep 30 log files worth of history -->
<maxHistory>3</maxHistory>
</rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
<immediateFlush>true</immediateFlush>
</encoder>
</appender>
<!-- End : Added for log for custom processor -->
<!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
<logger name="org.apache.nifi" level="INFO"/>
<logger name="org.apache.nifi.processors" level="WARN"/>
<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>
<logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" />
<logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" />
<logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" />
<logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" level="ERROR" />
<logger name="org.apache.zookeeper.server.quorum" level="ERROR" />
<logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" />
<logger name="org.apache.zookeeper.server.PrepRequestProcessor" level="ERROR" />
<logger name="org.apache.calcite.runtime.CalciteException" level="OFF" />
<logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" />
<logger name="org.apache.curator.ConnectionState" level="OFF" />
<!-- Logger for managing logging statements for nifi clusters. -->
<logger name="org.apache.nifi.cluster" level="INFO"/>
<!-- Logger for logging HTTP requests received by the web server. -->
<logger name="org.apache.nifi.server.JettyServer" level="INFO"/>
<!-- Logger for managing logging statements for jetty -->
<logger name="org.eclipse.jetty" level="INFO"/>
<!-- Suppress non-error messages due to excessive logging by class or library -->
<logger name="com.sun.jersey.spi.container.servlet.WebComponent" level="ERROR"/>
<logger name="com.sun.jersey.spi.spring" level="ERROR"/>
<logger name="org.springframework" level="ERROR"/>
<!-- Suppress non-error messages due to known warning about redundant path annotation (NIFI-574) -->
<logger name="com.sun.jersey.spi.inject.Errors" level="ERROR"/>
<!--
Logger for capturing user events. We do not want to propagate these
log events to the root logger. These messages are only sent to the
user-log appender.
-->
<logger name="org.apache.nifi.web.security" level="INFO" additivity="false">
<appender-ref ref="USER_FILE"/>
</logger>
<logger name="org.apache.nifi.web.api.config" level="INFO" additivity="false">
<appender-ref ref="USER_FILE"/>
</logger>
<logger name="org.apache.nifi.authorization" level="INFO" additivity="false">
<appender-ref ref="USER_FILE"/>
</logger>
<logger name="org.apache.nifi.cluster.authorization" level="INFO" additivity="false">
<appender-ref ref="USER_FILE"/>
</logger>
<logger name="org.apache.nifi.web.filter.RequestLogger" level="INFO" additivity="false">
<appender-ref ref="USER_FILE"/>
</logger>
<!--
Logger for capturing Bootstrap logs and NiFi's standard error and standard out.
-->
<logger name="org.apache.nifi.bootstrap" level="INFO" additivity="false">
<appender-ref ref="BOOTSTRAP_FILE" />
</logger>
<logger name="org.apache.nifi.bootstrap.Command" level="INFO" additivity="false">
<appender-ref ref="CONSOLE" />
<appender-ref ref="BOOTSTRAP_FILE" />
</logger>
<!-- Everything written to NiFi's Standard Out will be logged with the logger org.apache.nifi.StdOut at INFO level -->
<logger name="org.apache.nifi.StdOut" level="INFO" additivity="false">
<appender-ref ref="BOOTSTRAP_FILE" />
</logger>
<!-- Everything written to NiFi's Standard Error will be logged with the logger org.apache.nifi.StdErr at ERROR level -->
<logger name="org.apache.nifi.StdErr" level="ERROR" additivity="false">
<appender-ref ref="BOOTSTRAP_FILE" />
</logger>
<!-- Start : Added for log for custom processor -->
<logger name="com.datalake.processors.SQLServerCDCProcessor" level="DEBUG" >
<appender-ref ref="SQLSERVER-CDC"/>
</logger>
<!-- End : Added for log for custom processor -->
<root level="INFO">
<appender-ref ref="APP_FILE"/>
</root>
</configuration>
The strange fact is that the custom processor debug statements are written to both 'nifi-app.log' and the 'sqlserver-cdc.log' but I want these statements to be written only in the latter('sqlserver-cdc.log').
What am I missing ?
I think the problem is that com.datalake.processors.SQLServerCDCProcessor is also subject to the root logger configuration. You may be able to fix this by adding additivity="false":
<logger name="com.datalake.processors.SQLServerCDCProcessor" level="DEBUG" additivity="false" >
<appender-ref ref="SQLSERVER-CDC"/>
</logger>
See Lockback exclude logger from root for more on additivity vs. the root logger.
I am trying to not log debug message in class com.data.march.ClientUtils. For that, I think that I can change the log level of that class in the logback.xml to error.
Here is my logback.xml:
<configuration>
<appender name="dailyRollingFileAppender"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>c:/Logs/logato.log</File>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- daily rollover -->
<FileNamePattern>c:/Logato/Data/Logs/Logato.%d{yyyy-MM-dd}.log</FileNamePattern>
<!-- keep 30 days' worth of history -->
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder>
<Pattern>%d{HH:mm:ss.SSS} %-5level [%thread] %logger{35} - %msg %n
</Pattern>
</encoder>
</appender>
<appender name="minuteRollingFileAppender"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- rollover every minute -->
<FileNamePattern>c:/Data/Logs/minutes/Logato-minute.%d{yyyy-MM-dd_HH-mm}.log
</FileNamePattern>
<!-- keep 30 minutes' worth of history -->
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder>
<Pattern>%-4relative %-5level [%thread] %logger{35} - %msg %n
</Pattern>
</encoder>
</appender>
<appender name="consoleAppender" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<Pattern>.%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg %n
</Pattern>
</encoder>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>TRACE</level>
</filter>
</appender>
<logger name="org" additivity="false">
<level value="INFO"/>
<appender-ref ref="dailyRollingFileAppender"/>
<appender-ref ref="consoleAppender"/>
</logger>
<logger name="com.data" additivity="false">
<level value="DEBUG"/>
<appender-ref ref="dailyRollingFileAppender"/>
<appender-ref ref="consoleAppender"/>
</logger>
<logger name="org" additivity="false">
<level value="INFO"/>
<appender-ref ref="dailyRollingFileAppender"/>
</logger>
<logger name="com.data.march.ClientUtils" additivity="false">
<level value="ERROR"/>
</logger>
<root>
<level value="DEBUG"/>
<appender-ref ref="consoleAppender"/>
</root>
</configuration>
But my attempt failed. Do you have any idea?
Try with adding this:
<logger name="com.data.march.ClientUtils" level="error"/>
I needed my logs to be rolling, so I have created a "logback-spring.xml" file and placed it src/main/resources. Works flawlessly.
The issue is that the actuator endpoint "/logfile" no longer works as I have removed the logging configuration from the "applications.yml" file.
According to documentation, either "logging.path" or "logging.file" needs to be set in order to make the "/logfile" endpoint work. This however seem to conflict with my new 'xml configuration.
Here is my logback-spring.xml configuration for good measure:
<configuration debug="true" scan="true">
<include resource="org/springframework/boot/logging/logback/base.xml"/>
<property name="LOG_PATH" value="logs"/>
<property name="LOG_ARCHIVE" value="${LOG_PATH}/archive"/>
<appender name="RollingFile-Appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_FILE}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_ARCHIVE}/bookingflow-%d{yyyy-MM-dd}.log</fileNamePattern>
<maxHistory>30</maxHistory>
<totalSizeCap>1GB</totalSizeCap>
</rollingPolicy>
<encoder>
<pattern>%d %-5level [%thread] %logger : %msg%n</pattern>
</encoder>
</appender>
<appender name="Async-Appender" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="RollingFile-Appender"/>
</appender>
<logger name="com.novasol.bookingflow.api" level="debug">
<appender-ref ref="Async-Appender"/>
</logger>
<springProfile name="production">
<logger name="com.novasol.bookingflow.api" level="error">
<appender-ref ref="Async-Appender"/>
</logger>
</springProfile>
</configuration>
Any pointers appreciated.
Kind regards
Lars
Solved it in the following manner:
In application.yml
logging:
config:
classpath: "logback-spring.xml"
file: logs/bookingflow.log
The "logback-spring.xml":
<configuration debug="true" scan="true">
<include resource="org/springframework/boot/logging/logback/base.xml"/>
<property name="LOG_ARCHIVE" value="${LOG_PATH}/archive"/>
<appender name="RollingFile-Appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/${LOG_FILE}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_ARCHIVE}/bookingflow-%d{yyyy-MM-dd}.log</fileNamePattern>
<maxHistory>30</maxHistory>
<totalSizeCap>1GB</totalSizeCap>
</rollingPolicy>
<encoder>
<pattern>%d %-5level [%thread] %logger : %msg%n</pattern>
</encoder>
</appender>
<appender name="Async-Appender" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="RollingFile-Appender"/>
</appender>
<logger name="com.novasol.bookingflow.api" level="debug">
<appender-ref ref="Async-Appender"/>
</logger>
<springProfile name="production">
<logger name="com.novasol.bookingflow.api" level="error">
<appender-ref ref="Async-Appender"/>
</logger>
</springProfile>
Finally found the solution:
The following needs to be specified
endpoints:
logfile:
external-file: logs/custom.log