<rollingPolicy
class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- rollover hourly -->
<fileNamePattern>${DEV_HOME}/archived/server.%d{yyyy-MM-dd_HH}.log
</fileNamePattern>
</rollingPolicy>
I m using this method now for hourly rollover but need to change this at 8hr rollover {i.e., we have 24 hr then 3 files will be created of 8hr log value each}.
Is there any solution to it?
By default, logback does not provide a rolling policy implementation for such functionality. You can create your own rolling policy trigger for this by extending DefaultTimeBasedFileNamingAndTriggeringPolicy and overriding the computeNextCheck method to perform the next roll after a specified multiple of the rolling time period it inferred from the file name pattern.
Custom Rolling Trigger Policy Class:
import ch.qos.logback.core.joran.spi.NoAutoStart;
import ch.qos.logback.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy;
#NoAutoStart // #NoAutoStart prevents null pointer exception by stopping Joran from launching the start method before initialization
public class ArbitraryTimeRollingTriggerPolicy<E> extends DefaultTimeBasedFileNamingAndTriggeringPolicy<E> {
// Setters and getters are required to set times inside logback configuration xml file
private int times;
public int getTimes() {
return times;
}
public void setTimes(int times) {
this.times = times;
}
#Override
protected void computeNextCheck() {
nextCheck = rc.getEndOfNextNthPeriod(dateInCurrentPeriod, times).getTime();
}
}
The rolling policy to use inside your appender in logback config:
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${DEV_HOME}/archived/server.%d{yyyy-MM-dd_HH}.log</fileNamePattern>
<!-- Custom Rolling Trigger Policy -->
<timeBasedFileNamingAndTriggeringPolicy class="<YOUR-PACKAGE>.ArbitraryTimeRollingTriggerPolicy">
<!-- Number of hours, after which you want to rotate the file -->
<times>8</times>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
Related
I would like to reduce the size of certain logs using logback (I attempted to use Pattern layout with with a pattern of %.100000msg to limit the max size to one hundred thousand but had no luck), the large(over 1 million characters caused by a few REST GET calls) logs are causing Elastic to slow down when searching for certain information.
What would be the best practice to overcome this?
<include resource="org/springframework/boot/logging/logback/base.xml"/>
<appender name="Console"
class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>
%.30msg
</Pattern>
</layout>
</appender>
<logger name="" level="INFO"/>
I solved this by refering to this documentation to create a custom converter. The property was an extra, it was implemented to allow for configuration within the xml file.
<property scope="context" name="max-message-length" value="100000"/>
<conversionRule conversionWord="boundedMsg"
converterClass="za.co.ksdc.qadee.config.TruncationCustomLogConverter" />
The following java class was implemented to truncate the log message:
public class TruncationCustomLogConverter extends ClassicConverter {
#Override
public String convert(ILoggingEvent event) {
int maxMessageLength = Integer.parseInt(getContext().getProperty("max-
message-length"));
String formattedMessage = event.getFormattedMessage();
if (formattedMessage == null ||
formattedMessage.length() < maxMessageLength) {
return formattedMessage;
}
return new StringBuilder(maxMessageLength)
.append(formattedMessage, 0, maxMessageLength)
.toString();
}
}
Based on this guide about logging AWSRequestId, I can add %X{AWSRequestId} on my logback.xml to print the request id on my lambda function.
On the example, the method that adds this key on MDC is populateMappingDiagnosticContextValues of class MicronautRequestHandler. But since my lambda function has two endpoints, I created it as an Application type and it says it's using the MicronautLambdaHandler as the handler class.
What I did is I created my custom handler that extends the MicronautLambdaHandler and added the populateMappingDiagnosticContextValues method.
#Override
public AwsProxyResponse handleRequest(AwsProxyRequest input, Context context) {
if (context != null) {
populateMappingDiagnosticContextValues(context);
}
return handler.proxy(input, context);
}
logback.xml
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<withJansi>false</withJansi>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss} %X{AWSRequestId} %-5p %c{1} - %m%n</pattern>
</encoder>
</appender>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</configuration>
On the lambda setup I pointed the handler on my custom handler and the app is working.
Except that the logs is not showing the AWSRequestId.
Is this functionality only works on MicronautRequestHandler class and not on MicronautLambdaHandler?
I also tried the aws guide here that uses log4j2 for logging the request id. But log4j doesn't work when building the GraalVM native image, that's why I sticked to logback.
If you copied populateMappingDiagnosticContextValues and mdcput into your Handler (I suppose you did it) it should work without any additional changes.
I am using logback with slf4j for logging in Spring Boot application. I have created a custom layout class because all log statements are to be wrapped as a json. I have configured the logback-spring.xml as show below to take the custom layout. It works!
Issue is I am unable to apply the pattern. Only either the layout works (or) the pattern. What I want is always on log, go to layout class and then apply the pattern before logging.
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${user.home}/logs/sample.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${user.home}/logs/sample_%d{yyyy-MM-dd}.%i.log</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>10MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<!-- how many days to keep the files -->
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.test.test.payment.core.logging.SampleLogLayout" >
</layout>
</encoder>
*<!-- <encoder>
<charset>UTF-8</charset>
<Pattern>{"#timestamp": "%d{yyyy-MM-dd HH:mm:ss.SSS}", "priority": "%p", "application": "payment",
"class": "%C", "file": "%F:%L", "payload": %m }%n
</Pattern>
</encoder>-->*
</appender>
Here's the SampleLogLayout class:
public class SampleLogLayout extends LayoutBase<LoggingEvent> {
#Override
public String doLayout(LoggingEvent event) {
String renderedMessage = event.getMessage();
if (!isJson(renderedMessage)) {
Throwable throwable = null;
if (event.getLevel().equals(Level.ERROR) || event.getLevel().equals(Level.WARN)) {
final IThrowableProxy throwableProxy = event.getThrowableProxy();
if ((throwableProxy != null) && (throwableProxy instanceof ThrowableProxy)) {
ThrowableProxy proxy = (ThrowableProxy) throwableProxy;
throwable = proxy.getThrowable();
}
String message = LogErrorMessage.create(CommonCoreErrors.GENERIC_ERROR)
.message(renderedMessage).exception(throwable).build();
return message;
} else {
return LogMessage.create(renderedMessage).build();
}
}
return renderedMessage;
}
private boolean isJson(String msg) {
if (null == msg) {
return false;
} else {
return msg.startsWith("{") && msg.endsWith("}");
}
}
}
I started to reproduce this in order to provide a complete solution but the absence of LogMessage and LogErrorMessage made that a bit tricky.
However, it looks to me like you just want to log in JSON format, not just the message but also the metadata such as timestamp, priority etc.
This can be achieved by using the JsonLayout. Here's an example of the layout configuration:
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>false</prettyPrint>
</jsonFormatter>
<timestampFormat>yyyy-MM-dd' 'HH:mm:ss.SSS</timestampFormat>
<appendLineSeparator>true</appendLineSeparator>
<includeContextName>false</includeContextName>
</layout>
With that configuration the following log invocation ...
logger.info("{\"a\": 1, \"b\": 2}");
... will emit:
{"timestamp":"2017-10-05 10:51:34.610","level":"INFO","thread":"main","logger":"com.stackoverflow.logback.LogbackTest","message":"{\"a\": 1, \"b\": 2}"}
You can include MDC too, for example ...
MDC.put("application", "payment");
logger.info("{\"a\": 1, \"b\": 2}");
... will emit:
{"timestamp":"2017-10-05 10:52:56.088","level":"INFO","thread":"main","mdc":{"application":"payment"},"logger":"com.stackoverflow.logback.LogbackTest","message":"{\"a\": 1, \"b\": 2}"}
That's quite close to your desired output and JsonLayout is extensible so you could ...
Override toJsonMap() to change names of the keys e.g. replace timestamp with #timestamp, replace message with payload
Implement addCustomDataToJsonMap() to add other key:value pairs to the log event
So, I think you could achieve your desired output using the pre-existing JsonLayout rather than writing your own.
More details on Logback JSON extensions here.
The Maven coordinates are:
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
<version>0.1.5</version>
</dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-core</artifactId>
<version>0.1.5</version>
</dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-jackson</artifactId>
<version>0.1.5</version>
</dependency>
I have a scenario where I send some sensitive data in the ResponseBody of a spring controller. Spring logs the entire response body in DEBUG mode, where I can see the sensitive data also. Now if I configure my app to print only INFO logs or higher, this won't be displayed, but that is not a fool proof method, since its possible that DEBUG mode could be turned on accidentally.
Is there anyway I can disable certain fields in the response body from being logged by Spring?
Thanks
My suggestion would be that you implement a custom converter for logback (if that is your logging library)
http://logback.qos.ch/manual/layouts.html#customConversionSpecifier
It enables you to convert your logging data and blur out any data that should be removed from the message.
From the documentation it is pretty straight forward
public class MySampleConverter extends ClassicConverter {
long start = System.nanoTime();
#Override
public String convert(ILoggingEvent event) {
long nowInNanos = System.nanoTime();
return Long.toString(nowInNanos-start);
}
}
And the config
<configuration>
<conversionRule conversionWord="nanos"
converterClass="chapters.layouts.MySampleConverter" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%-6nanos [%thread] - %msg%n</pattern>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="STDOUT" />
</root>
</configuration>
If you're using Log4j you can implement a custom filter which would mask the data per your layout
http://vozis.blogspot.sg/2012/02/log4j-filter-to-mask-payment-card.html
Edit
The final easier solution turned out to be to override toString of the return object. Spring uses toString to log the return message
I have gone thru the below link from perf4J site and have done same to same: http://perf4j.codehaus.org/devguide.html#Using_Spring_AOP_to_Integrate_Timing_Aspects
Added the following in my spring.xml.
<aop:aspectj-autoproxy/>
<bean id="timingAspect" class="org.perf4j.log4j.aop.TimingAspect"/>
<bean id="wscClientBase" class="com.xyz.csa.core.common.WscClientBase"/>
In class WscClientBase I have the following method with #Profiled annotation.
#Profiled(tag = "SOAPCALLTEST")
public Object sendMessage(Object message) {
String msg = message.toString();
if (msg.indexOf(' ') > 1) {
msg = msg.substring(1, msg.indexOf(' '));
}
try {
Object ret = marshalSendAndReceive(message);
return ret;
} catch (RuntimeException ex) {
throw ex;
}
}
I dont see the perf4j TimingLogger statements in the application log. However if I use it obtrusively (without annotation) as below, I see the log statements successfully.
public Object sendMessage(Object message) {
String msg = message.toString();
if (msg.indexOf(' ') > 1) {
msg = msg.substring(1, msg.indexOf(' '));
}
StopWatch stopWatch = new Slf4JStopWatch();
try {
Object ret = marshalSendAndReceive(message);
stopWatch.stop("PERF_SUCCESS_TAG", msg);
return ret;
} catch (RuntimeException ex) {
stopWatch.stop("PERF_FAILURE_TAG", msg);
throw ex;
}
}
Am I missing something?
Perf4j
This is a performance analysis and checking plugin for application. It can be integrated with spring using spring AOP. It creates a log file that is give to a parser to analyse and produce relevant information. It can provide average,mean, std deviation by default.
For more general information please check http://perf4j.codehaus.org/index.html
How to setup Perf4j.
For normal setup you just need to add perf4j jar and create StopWatch instance for every code sniplet you want to monitor.
StopWatch stopWatch= new StopWatch(“snipletTagName”)
…
//{your code sniplet}
…
stopwatch.stop();
This will create perf4j monitor and you will get logg information on the console.
Main purpose of this documentation is to have a setup by setup understanding of integrating perf4j with spring.
1.Add all of the below Jar files.
1.perf4j-0.9.16-slf4jonly.jar
2.aspectjweaver-1.6.12.jar
3.aopalliance-1.0.jar
4.commons-logging-1.1.1.jar
5.logback-classic-1.0.7.jar
6.logback-core-1.0.7.jar
7.slf4j-api-1.7.1.jar
8.perf4j-0.9.16.jar
9.aspectjrt-1.6.1.jar
10.commons-jexl-1.1.jar
11.asm-1.5.3.jar
12.cglib-2.1_3.jar
Make sure you have all these jars in your classpath along with spring libraries.
2.create your own logback.xml that will be used by perf4j implicitly
the content of the logback.xml will be
<configuration>
<appender name="perf4jFileAppender"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>logs/perf4j.log</File>
<encoder>
<Pattern>%date %-5level [%thread] %logger{36} [%file:%line] %msg%n
</Pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<FileNamePattern>logs/perf4j.%d{yyyy-MM-dd}.log</FileNamePattern>
</rollingPolicy>
</appender>
<appender name="CoalescingStatistics"
class="org.perf4j.logback.AsyncCoalescingStatisticsAppender">
<param name="TimeSlice" value="1" />
<appender-ref ref="perf4jFileAppender" />
</appender>
<appender name="RootConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>debug</level>
</filter>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%date %-5level [%thread] %logger{36} [%file:%line] %msg%n
</pattern>
</layout>
</appender>
<!-- Loggers -->
<!-- The Perf4J logger. Note that org.perf4j.TimingLogger is the value of
the org.perf4j.StopWatch.DEFAULT_LOGGER_NAME constant. Also, note that additivity
is set to false, which is usually what is desired - this means that timing
statements will only be sent to this logger and NOT to upstream loggers. -->
<logger name="org.perf4j.TimingLogger" additivity="false">
<level value="DEBUG" />
<appender-ref ref="CoalescingStatistics" />
<appender-ref ref="perf4jFileAppender" />
<appender-ref ref="RootConsoleAppender" />
</logger>
</configuration>
3.In your spring configuration file you need to add aspectj tag that will enable #Profiled annotation of perf4j.
(Note: What is #Profiled annotation?: you will add this tag to all the
methods in all the classes that are called from spring instance or use
dependency injection. The object basically should be spring context
registered and the method should be invoked by the object that are
registered in spring context. I wasted one day thinking why my method
was not logged then I realized that the object I tested was not part
of spring context.
OK, the code that you need to add to spring configuration xml is
<!-- this is my spring-context.xml -->
<beans>
<aop:aspectj-autoproxy>
<aop:include name="timingAspect" />
</aop:aspectj-autoproxy>
<bean id="timingAspect" class="org.perf4j.slf4j.aop.TimingAspect" />
<!-- this is the class that will be registered with the spring and now we can get this class and call the method that we need to monitor-->
<bean class="com.perf4jexample.Test" />
</beans>
4.Create the Test class that will implement #Profiled annotation.
public class Test {
private String testVal;
public Test() {
// TODO Auto-generated constructor stub
}
#Profiled
public void testing() {
System.out.println("testt" );
}
public String getTestVal() {
return testVal;
}
public void setTestVal(String testVal) {
this.testVal = testVal;
}
}
5.Ok now you have setup every thing just thing remains is test class that will start spring context and with it load perf4j.
public class Test(){
public static void main(){
AbstractApplicationContext context = new ClassPathXmlApplicationContext(
"spring-context.xml");
context.start();
Test bean = context.getBean(Test.class);
bean.testing();
}
I hope by following these setups you should be able to perf4j console appender to display one line on console.
Perf4j Monitoring command on the log:
For Generating Performance statistical information execute on you logger path
java -jar perf4j-0.9.16.jar myLogger.log
For Generating Graphs
java -jar perf4j-0.9.16.jar --graph perfGraphs.out myLogger.log
I hope this tutorial helps you to integrated Spring, perf4j, logback with Profiled annotation.
Try adding <aop:include name="timingAspect"/> inside the <aop:aspectj-autoproxy/>.
Can you also confirm that you are invoking sendMessage on an object that is retrieved from the spring application context (using getBean or injected as a dependency).
Here I have two ways to make perf4j #Profiled work on spring boot project.
Precondition is adding below dependencies
"org.aspectj:aspectjweaver",
"org.perf4j:perf4j:0.9.16",
"commons-jexl:commons-jexl:1.1",
"cglib:cglib:3.2.1",
For normal spring project, probably need to add little more dependencies like spring-aop, aopalliance... those looks included in spring-boot-starter-parent
1. Java Configuration
This is simplest way and mostly works, but I found that somehow not working on spring-data JpaRepository method. It just provides org.perf4j.log4j.aop.TimingAspect bean and do aspectj autoproxing. Same way like xml configuration provided by other people above
#Configuration
#EnableAspectJAutoProxy
public class PerformanceConfig {
#Bean
public TimingAspect timingAspect() {
return new TimingAspect();
}
}
2. Provide Your own Aspect
In this way, the #Profiled annotated spring-data repository interface methods also work fine. But the downside of this is ignoring the tag given in #Profiled(tag='some tag') and using the joinPoint method name as tag.
#Aspect
#Component
public class PerformanceTracker {
#Around(value="execution(#org.perf4j.aop.Profiled * com.mypackage..*(..))")
public Object checkPerformance(ProceedingJoinPoint pjp) throws Throwable {
StopWatch stopWatch = new Log4JStopWatch(pjp.getSignature().toShortString());
Object result = pjp.proceed();
stopWatch.stop();
return result;
}
}
The reason why it could not work is that the profiled method is in the parent class of the Spring-bean. I can tell it by looking at your bean name: WscClientBase. I assume that this is base class from which you have many children classes.
After some research time I found very important note in Spring documentation about #Transactional and #Cacheable.
http://docs.spring.io/spring-framework/docs/current/spring-framework-reference/html/transaction.html#transaction-declarative-annotations. Check “Method visibility and #Transactional” block.
http://docs.spring.io/spring/docs/current/spring-framework-reference/html/cache.html#cache-annotation-enable. Check “Method visibility and cache annotations”
In both cases they say things like this:
“When using proxies, you should apply the <> annotations only to
methods with public visibility. If you do annotate protected, private
or package-visible methods with these annotations, no error is raised,
but the annotated method does not exhibit the configured caching
settings. Consider the use of AspectJ (see below) if you need to
annotate non-public methods as it changes the bytecode itself.”
And below…
“Spring recommends that you only annotate concrete classes (and
methods of concrete classes) with the #Cache* annotation, as opposed
to annotating interfaces. You certainly can place the #Cache*
annotation on an interface (or an interface method), but this works
only as you would expect it to if you are using interface-based
proxies. The fact that Java annotations are not inherited from
interfaces means that if you are using class-based proxies
(proxy-target-class="true") or the weaving-based aspect (
mode="aspectj"), then the caching settings are not recognized by the
proxying and weaving infrastructure, and the object will not be
wrapped in a caching proxy, which would be decidedly bad.”
I assume that #Profiled use similar weaving mechanism therefore you can't put #Profiled on any method in parent classes.
As a matter of fact I had similar issue in my application: I had #Profiled and #Cacheable in parent class and none of those worked: I didn't see any records in my Perf4J log and cache wasn't updated. When I moved #Profiled into methods of children-classes I started seeing the records in the perf4j.log.
put config below in your "servlet-context-config.xml" .have fun!
<aop:aspectj-autoproxy/>
<bean id="timingAspect" class="org.perf4j.log4j.aop.TimingAspect"/>
<bean id="wscClientBase" class="com.xyz.csa.core.common.WscClientBase"/>
For people who have this type of problem they can check that there is not in spring logs (level info), messages like it " is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)".