I have a scheduled task in a fixed rate, that reads a queue.
Each message that comes from the queue has an ID.
I wanna know if it's possible split the log by ID, appending to a different file.
I was thinking about use aspects or a custom appender, one of these can do the job for me?
Thanks.
Well, after some search I've remembered of MDC (Mapped Diagnostic Context) wich can do what I want with almost no workarounds.
I just need to add a SiftingAppender to the logback-spring.xml like this:
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">
<include resource="org/springframework/boot/logging/logback/base.xml"/>
<appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator>
<key>checkoutId</key>
<defaultValue>system</defaultValue>
</discriminator>
<sift>
<appender name="${checkoutId}" class="ch.qos.logback.core.FileAppender">
<file>${checkoutId}.log</file>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%d{HH:mm:ss:SSS} | %-5level | %thread | %logger{20} | %msg%n%rEx</pattern>
</layout>
</appender>
</sift>
</appender>
<root level="INFO">
<appender-ref ref="SIFT" />
</root>
</configuration>
Than I call like that:
#Scheduled(initialDelayString = "${consumeStart:10000}", fixedRateString = "${consumeRate:5000}")
private void task() {
try {
val message = queue.get(timeout);
if (message != null) {
MDC.put("checkoutId", message.toString());
. . .
}
} finally {
MDC.remove("checkoutId");
}
}
Related
I would like to reduce the size of certain logs using logback (I attempted to use Pattern layout with with a pattern of %.100000msg to limit the max size to one hundred thousand but had no luck), the large(over 1 million characters caused by a few REST GET calls) logs are causing Elastic to slow down when searching for certain information.
What would be the best practice to overcome this?
<include resource="org/springframework/boot/logging/logback/base.xml"/>
<appender name="Console"
class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>
%.30msg
</Pattern>
</layout>
</appender>
<logger name="" level="INFO"/>
I solved this by refering to this documentation to create a custom converter. The property was an extra, it was implemented to allow for configuration within the xml file.
<property scope="context" name="max-message-length" value="100000"/>
<conversionRule conversionWord="boundedMsg"
converterClass="za.co.ksdc.qadee.config.TruncationCustomLogConverter" />
The following java class was implemented to truncate the log message:
public class TruncationCustomLogConverter extends ClassicConverter {
#Override
public String convert(ILoggingEvent event) {
int maxMessageLength = Integer.parseInt(getContext().getProperty("max-
message-length"));
String formattedMessage = event.getFormattedMessage();
if (formattedMessage == null ||
formattedMessage.length() < maxMessageLength) {
return formattedMessage;
}
return new StringBuilder(maxMessageLength)
.append(formattedMessage, 0, maxMessageLength)
.toString();
}
}
I am working on a task that I want to mask sensitive data using Log4j2 LogEventPatternConverter Class.
#Plugin(name="SensitiveDataLog", category = "Converter")
#ConverterKeys({"sense"})
public class SensitiveDataLog extends LogEventPatternConverter {
#Value("${ssn}")
private String ssn;
public SensitiveDataLog(String name, String style) {
super(name, style);
}
public static SensitiveDataLog newInstance(String[] options) {
return new SensitiveDataLog("sense","sense");
}
#Override
public void format(LogEvent logEvent, StringBuilder outputMsg) {
String message = logEvent.getMessage().getFormattedMessage();
Matcher matcher = SSN_PATTERN.matcher(message);
if (matcher.find()) {
String maskedMessage = matcher.replaceAll("***-**-****");
outputMsg.append(maskedMessage);
} else {
outputMsg.append(message);
}
}
}
Suppose I want to keep pattern in application.properties, But problem here is we cannot load property value ssn. Always its null.
Here is my log4j2.xml
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="info" monitorInterval="30"
packages="com.virtusa.xlab.fw.logging.component"
xmlns="http://logging.apache.org/log4j/2.0/config">
<Properties>
<Property name="basePath">logs/log4j2</Property>
</Properties>
<Appenders>
<!-- File Appender -->
<RollingFile name="FILE"
fileName="${basePath}/logfile.log" filePattern="${basePath}/logfile.%d{yyyy-MM-dd}-%i.log" append="true">
<PatternLayout
pattern="%-5p | %d{yyyy-MM-dd HH:mm:ss} | [%t] %C{2} (%F:%L) - %sense%n" />
<Policies>
<SizeBasedTriggeringPolicy size="1 KB" />
</Policies>
<DefaultRolloverStrategy max="4" />
</RollingFile>
<!-- Console Appender -->
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout
pattern="%-5p | %d{yyyy-MM-dd HH:mm:ss} | [%t] %C{2} (%F:%L) - %sense%n" />
</Console>
</Appenders>
<Loggers>
<Logger name="com.virtusa.xlab.fw" level="info" />
<Root level="info">
<AppenderRef ref="STDOUT" />
<AppenderRef ref="FILE" />
</Root>
</Loggers>
</Configuration>
Can anyone help me out here?
Thanks.
The problem is that SensitiveDataLog is created via static method newInstance(). Obviously, field ssn is not initialized at that moment. What you can do is to init the field later, e.g. when refreshing Spring context.
Here is my snippet:
private static XmlMaskPatternConverter INSTANCE = new XmlMaskPatternConverter();
public XmlMaskPatternConverter() {
super(NAME, NAME);
}
public static XmlMaskPatternConverter newInstance() {
return INSTANCE;
}
Now you can call static method getInstance() somewhere in your Spring Configuration (I do it in #Bean method) and set the ssn value there. Ofc, you need to create a setter for this field.
P.S. Hope it helps. I faced this problem too, so decided to leave my solution here. My first post on SO btw)
I have created a test project with Spring Boot to learn about about using the logback-spring.xml file. I want to use Spring's default setting for writing to console so I am using the following line
<include resource="org/springframework/boot/logging/logback/base.xml" />
And I also want to log to a file on a rolling basis and keep a maximum number of log files on the system. Writing to console is working as expected. However no logs are written to the log file. The folder named "logs" gets created and the file "logfile.log" also gets created. But nothing gets logged to it.
Below is the fill logback-spring.xml file
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml" />
<property name="LOG_PATH" value="logs" />
<property name="LOG_ARCHIVE" value="${LOG_PATH}/archive" />
<appender name="File-Appender" class="ch.qos.logback.core.FileAppender">
<file>${LOG_PATH}/logfile.log</file>
<encoder>
<pattern>%d{dd-MM-yyyy HH:mm:ss.SSS} %magenta([%thread]) %highlight(%-5level) %logger{36}.%M - %msg%n</pattern>
<outputPatternAsHeader>false</outputPatternAsHeader>
</encoder>
</appender>
<logger name="test" level="DEBUG" />
</configuration>
and below is the TestApplication.java file which is part of the test package
#SpringBootApplication
public class TestApplication {
private static final Logger logger = LoggerFactory.getLogger(TestApplication.class);
public static void main(String[] args) {
SpringApplication.run(TestApplication.class, args);
logger.trace("Trace Message!");
logger.debug("Debug Message!");
logger.info("Info Message!");
logger.warn("Warn Message!");
logger.error("Error Message!");
}
}
Why is nothing being logged to the file?
I think there are a few issues.
First, remove the line <logger name="test" level="DEBUG" /> . This sets up a logger for classes under the package test but defines no appender, so nothing is logged.
Once that's gone, add
<root level="DEBUG">
<appender-ref ref="File-Appender"/>
</root>
This will configure the root logger (which all loggers inherit) on debug level and to output all the logs to the File-Appender.
Also, I cannot recall if logback creates missing directories, so you might need to ensure the logs directory does exist before starting the application.
Im trying to get hot reloading to work with my logging level. The monitorInterval Should do the trick for me, but for some reason it doesnt work.
My log4j2.xml file looks like this:
<Configuration monitorInterval="10">
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout
pattern="%d{ISO8601} [%-12.-12t] %-5p [%12.12X{CorrelationId}] [%-30.-30X{Path}] %logger{36}:%L - %msg%n"/>
</Console>
<File name="anywhere" fileName="anywhere.log" append="false">
<PatternLayout>
<Pattern>%d{ISO8601} [%-12.-12t] %-5p [%12.12X{CorrelationId}] [%-30.-30X{Path}] %logger{36}:%L - %msg%n</Pattern>
</PatternLayout>
</File>
</Appenders>
<Loggers>
<logger name="com.cetrea" level="info"/>
<Root level="warn">
<AppenderRef ref="anywhere"/>
<AppenderRef ref="STDOUT"/>
</Root>
</Loggers>
</Configuration>
Im testing it with my rest api, and when i hit this route, it should print out just LOG.info which it does.
private static final Logger LOG = LogManager.getLogger(TokenController.class);
#RequestMapping(value = "/sensitive-data/{token}", method = RequestMethod.GET, produces = "text/plain;charset=UTF-8")
public ResponseEntity getData(#PathVariable("token") String token) {
if (tokenMap.containsKey(token)) {
return ResponseEntity.ok(tokenMap.get(token));
} else {
Timestamp timestamp = new Timestamp(System.currentTimeMillis());
LOG.info("Hit with wrong or expired token at " + timestamp + "");
LOG.debug("debug thing");
return new ResponseEntity("Token not found, or has expired", HttpStatus.NOT_FOUND);
}
}
Now if i change the level to debug, i would expect it to also print out the LOG.debug, but it doesn't. This doesn't take in effect until i restart the program, instead of it hot reloading 10 seconds later.
As it turns out, when it builds it includes the log4j file, and then reads from that, so the file i was editting wasnt read. I added
-Dlog4j.configurationFile="Path to the actual file"
to the runtime settings and then it worked.
logQuery is called in prepareStatementAndSetParameters mehtod - SQLInsertClause class
protected void logQuery(Logger logger, String queryString, Collection<Object> parameters) {
String normalizedQuery = queryString.replace('\n', ' ');
MDC.put(QueryBase.MDC_QUERY, normalizedQuery);
MDC.put(QueryBase.MDC_PARAMETERS, String.valueOf(parameters));
if (logger.isDebugEnabled()) {
logger.debug(normalizedQuery);
}
}
how can I set debug level to logger ?
That logger there is from SLF4J API. Depending on the logger you have behind the API you use facilities of that underlying logging implementation.
For instance we use Logback Classic (dependency ch.qos.logback:logback-classic) and I can explicitly override what configuration file to use with -Dlogback.configurationFile=devel-logback.xml in JVM parameters. Default mechanism is documented here. My file looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%date %level [%.60thread] %logger{1} %msg%n</pattern>
</encoder>
</appender>
<logger name="com.mysema.query.jpa.impl.JPAQuery" level="DEBUG"/>
<!-- more loggers -->
<root level="DEBUG">
<appender-ref ref="CONSOLE"/>
</root>
</configuration>
Also adding -Dlogback.debug=true to JVM arguments adds some debug output when logback is being initialized.