Add MDC variables in JsonLayout - log4j2 - spring-boot

How to add MDC variables in the json log generated by JsonLayout of log4j2. I've used KeyValuePair tag to add properties like host name into the log, but I didn't found any way to add MDC variables into it. In pattern layout I used %X{traceId} but I'm sure JsonLayout can't parse those conversion chars(As far as I know conversion chars are used by pattern layout only). I went into source code of JsonLayout but didn't found function which actually puts all of the data into the log message.
Thank you.

What you're looking for is a log4j2 lookup. It sounds like you're interested specifically in the Context Map Lookup as you mentioned MDC (which is now called ThreadContext in log4j2 by the way).
Here is a simple example:
package example;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.ThreadContext;
public class ThreadContextExample {
private static final Logger log = LogManager.getLogger();
public static void main(String[] args) {
ThreadContext.put("myKey", "myValue");
log.info("Here's a message!");
}
}
Here is the log4j2.xml configuration:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<JsonLayout compact="false" eventEol="false" stacktraceAsString="true">
<KeyValuePair key="myJsonKey" value="${ctx:myKey}"/>
</JsonLayout>
</Console>
</Appenders>
<Loggers>
<Root level="debug">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>
and finally some sample output (shortened for readability):
{
"thread" : "main",
"level" : "INFO",
"loggerName" : "example.ThreadContextExample",
"message" : "Here's a message!",
...
"myJsonKey" : "myValue"
}

Related

Log4j log file not getting created at specified location springboot

I am trying to configure log4j2 in springboot.I have removed(excluded) the logback dependency already from pom.xml.I am using this xml under resource folder named log4j2.xml
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="DEBUG">
<Appenders>
<Console name="LogToConsole" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</Console>
<File name="LogToFile" fileName="logs/app.log">
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
</PatternLayout>
</File>
</Appenders>
<Loggers>
<Logger name="com.ashish" level="debug" additivity="false">
<AppenderRef ref="LogToFile"/>
<AppenderRef ref="LogToConsole"/>
</Logger>
<Logger name="org.springframework.boot" level="error" additivity="false">
<AppenderRef ref="LogToConsole"/>
</Logger>
<Root level="error">
<AppenderRef ref="LogToFile"/>
<AppenderRef ref="LogToConsole"/>
</Root>
</Loggers>
</Configuration>
This is my controller class.
package com.ashish;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.GetMapping;
import java.util.Arrays;
import java.util.List;
#Controller
public class HelloController {
private static final Logger logger = LogManager.getLogger(HelloController.class);
private List<Integer> num = Arrays.asList(1, 2, 3, 4, 5);
#GetMapping("/")
public String main(Model model) {
if (logger.isDebugEnabled()) {
logger.debug("Hello from Log4j 2 - num : {}", num);
}
logger.debug("Hello from Log4j 2 - num : {}", () -> num);
model.addAttribute("tasks", num);
return "welcome";
}
private int getNum() {
return 100;
}
}
Am i missing anything here?I tried to set it using application.properties too using latest version of Log4j2.But still it's not getting created.When i run the application i can't see any log file getting dynamically created at the path specified in xml.
First, you have status="DEBUG" specified in your configuration. So you will see Log4j configure itself on the console (or wherever system.out is getting routed to). If you do not then you aren't really using Log4j.
If you do see the output then check the debug lines. I have a suspicion your log file is either not getting created due to a permissions problem or it isn't being written where you expect it.
Your configuration specifies a relative directory named "logs". Whatever directory is the working directory when the app is started should contain your logs directory. Frequently on Linux that will end up being "/". You almost certainly won't have permission to create a logs directory there so configuration will fail.
I can't comment on the content of the log4j2 config file itself, probably it will make sense to print on console first and make sure that its driven by log4j2 indeed.
However, I'll refer to the beginning of the question:
I am trying to configure log4j2 in springboot.I have removed(excluded) the logback dependency already from pom.xml
You don't present the pom.xml but in general in order to switch spring boot to work with log4j2 you should:
"Exclude" the default logging mechanism of spring boot:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
Add a log4j2 starter that will in turn (transitively) add log4j2 dependency of the versions compatible with your spring boot version:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
Quick googling has revealed this tutorial that contains all the steps including end-to-end example of such an integration.

logback-spring.xml log to file while using org/springframework/boot/logging/logback/base.xml

I have created a test project with Spring Boot to learn about about using the logback-spring.xml file. I want to use Spring's default setting for writing to console so I am using the following line
<include resource="org/springframework/boot/logging/logback/base.xml" />
And I also want to log to a file on a rolling basis and keep a maximum number of log files on the system. Writing to console is working as expected. However no logs are written to the log file. The folder named "logs" gets created and the file "logfile.log" also gets created. But nothing gets logged to it.
Below is the fill logback-spring.xml file
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml" />
<property name="LOG_PATH" value="logs" />
<property name="LOG_ARCHIVE" value="${LOG_PATH}/archive" />
<appender name="File-Appender" class="ch.qos.logback.core.FileAppender">
<file>${LOG_PATH}/logfile.log</file>
<encoder>
<pattern>%d{dd-MM-yyyy HH:mm:ss.SSS} %magenta([%thread]) %highlight(%-5level) %logger{36}.%M - %msg%n</pattern>
<outputPatternAsHeader>false</outputPatternAsHeader>
</encoder>
</appender>
<logger name="test" level="DEBUG" />
</configuration>
and below is the TestApplication.java file which is part of the test package
#SpringBootApplication
public class TestApplication {
private static final Logger logger = LoggerFactory.getLogger(TestApplication.class);
public static void main(String[] args) {
SpringApplication.run(TestApplication.class, args);
logger.trace("Trace Message!");
logger.debug("Debug Message!");
logger.info("Info Message!");
logger.warn("Warn Message!");
logger.error("Error Message!");
}
}
Why is nothing being logged to the file?
I think there are a few issues.
First, remove the line <logger name="test" level="DEBUG" /> . This sets up a logger for classes under the package test but defines no appender, so nothing is logged.
Once that's gone, add
<root level="DEBUG">
<appender-ref ref="File-Appender"/>
</root>
This will configure the root logger (which all loggers inherit) on debug level and to output all the logs to the File-Appender.
Also, I cannot recall if logback creates missing directories, so you might need to ensure the logs directory does exist before starting the application.

Logback and JsonLayout: can't pass custom fields

I have this configuration in my logback.xml into a Spring Web Application (NO Spring Boot).
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>true</prettyPrint>
</jsonFormatter>
</layout>
<customFields>{"appname":"foobar"}</customFields>
</encoder>
</appender>
<!-- LOG everything at INFO level -->
<root level="INFO">
<appender-ref ref="Console" />
</root>
</configuration>
The JSON layout works fine but custom fields as "appname": "foobar" are not printed:
{
"timestamp" : "2020-06-10T14:55:25.534Z",
"level" : "INFO",
"thread" : "Catalina-utility-1",
"logger" : "org.springframework.web.servlet.DispatcherServlet",
"message" : "FrameworkServlet 'dispatcher': initialization completed in 72 ms",
"context" : "default"
}
What am I doing wrong?
SOLUTION
I was using the wrong libraries for my needs:
logback-jackson
logback-json-classic
Because of the fact that I need to process logs through Logstash I've corrected my configuration like this:
pom.xml
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>6.4</version>
</dependency>
logback.xml
<appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<customFields>{"customer":"X", "appname":"Y", "environment":"dev"}</customFields>
</encoder>
</appender>
and now It works fine.
I just stumbled this question because I had the same problem, and I found a solution, with logback-jackson and logback-json-classic.
Option 1: Per-Thread via Mapped Diagnostic Context (MDC)
SLF4j's Mapped Diagnostic Context is a per-thread key-value store that we can use to write custom structured data to the log output.
MDC.put("customKey", "customValue");
Logback's JsonLayout will automatically print this value under a special mdc JSON object without any further configuration.
{ [...], "mdc": {"customKey", "customValue"}}
Note that the MDC is constructed per thread and if it is empty, no mdc field is printed to the log output.
Option 2: Global (for all threads)
If you want custom fields to appear at the JSON output's root, you need to create a custom, but simple Layout class that extends JsonLayout. JsonLayout provides us with a addCustomDataToJsonMap we can override.
package com.mypackage;
import ch.qos.logback.contrib.json.classic.JsonLayout;
public class CustomJsonLayout extends JsonLayout {
#Override
protected void addCustomDataToJsonMap(Map<String, Object> map, ILoggingEvent event) {
map.put("customKey", "customValue");
}
}
Now, you just need to tell Logback to use CustomJsonLayout instead of JsonLayout in your logback.xml file and keep the rest the same.
<layout class="com.mypackage.CustomJsonLayout">
...
</layout>
Now, any log message will have the following output:
{ ..., "customKey": "customValue"}

Print intLevel in log4j2

You can see in https://logging.apache.org/log4j/2.x/manual/customloglevels.html the numerical values corresponding to built-in log4j2 logging levels, for example INFO->400. How can you refer to it in patternlayout resp. in JDBC Logger configuration?
I have an old log4j 1.x config for JDBC where it was referred as %iprio.
A workaround is to use
level{OFF=0,FATAL=100,ERROR=200,WARN=300,INFO=400,DEBUG=500,TRACE=600,ALL=1000}
but I am not very happy about that.
It sounds like you want to log the integer value of the log level instead of the name and you don't want to use the labeling feature of the PatternLayout's level parameter.
One possible solution would be to create a custom Lookup, here is some sample code:
package example;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.lookup.StrLookup;
#Plugin(name = "level", category = "Lookup")
public class LevelLookup implements StrLookup{
/**
* Lookup the value for the key.
* #param key the key to be looked up, may be null
* #return The value for the key.
*/
public String lookup(String key) {
return null;
}
/**
* Lookup the value for the key using the data in the LogEvent.
* #param event The current LogEvent.
* #param key the key to be looked up, may be null
* #return The value associated with the key.
*/
public String lookup(LogEvent event, String key) {
return String.valueOf(event.getLevel().intLevel());
}
}
Next, here is a sample configuration using the new lookup - note the ${level:}:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] ${level:} %logger{36} - %msg%n" />
</Console>
</Appenders>
<Loggers>
<Root level="debug">
<AppenderRef ref="Console" />
</Root>
</Loggers>
</Configuration>
Here is some sample code that performs some logging:
package example;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class SomeClass {
private static final Logger log = LogManager.getLogger();
public static void main(String[] args){
log.debug("This is some debug!");
log.info("Here's some info!");
log.error("Some erorr happened!");
}
}
and finally here is the output:
22:49:29.438 [main] 500 example.SomeClass - This is some debug!
22:49:29.440 [main] 400 example.SomeClass - Here's some info!
22:49:29.440 [main] 200 example.SomeClass - Some erorr happened!
Hope this helps!

Play! 2.5 log access to stdout in development

Is there easy way to get development access logs on my console with Play 2.5?
Something I could read as "GET /foo/123 routed to FooController's show action with id=123"?
I've found how to get netty access log ( btw, option play.server.netty.log.wire=true in application.conf doesn't work for me for some reason, but -Dplay.server.netty.log.wire=true does ), but it's too low-level.
You can create a logger.xml file in the conf directory. It should follow logback's file format.
for instance a default configuration could look like :
<configuration scan="true" scanPeriod="5 seconds">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%level %logger{15} - %message%n%xException{5}</pattern>
</encoder>
</appender>
<logger name="play" level="INFO" />
<logger name="application" level="INFO" />
<root level="ERROR">
<appender-ref ref="STDOUT" />
</root>
</configuration>
You can then enable loglevels : FATAL, ERROR, WARNING, INFO, DEBUG, TRACE selectively for any package or as a default on the root leve.
This is described in the playframework configuration configuring logging
Note that this behaviour was changed in 2.4.x from the previous versions where you could configure loggers through application.conf
Once you have logging working you can use the sample logging filter provided in the documentation to log all requests to your server.
import javax.inject.Inject
import akka.stream.Materializer
import play.api.Logger
import play.api.mvc._
import scala.concurrent.{ExecutionContext, Future}
class LoggingFilter #Inject() (implicit val mat: Materializer, ec: ExecutionContext) extends Filter {
def apply(nextFilter: RequestHeader => Future[Result])
(requestHeader: RequestHeader): Future[Result] = {
val startTime = System.currentTimeMillis
nextFilter(requestHeader).map { result =>
val endTime = System.currentTimeMillis
val requestTime = endTime - startTime
Logger.info(s"${requestHeader.method} ${requestHeader.uri} took ${requestTime}ms and returned ${result.header.status}")
result.withHeaders("Request-Time" -> requestTime.toString)
}
}
}
you will have to activate it by setting the play.http.filters:
play.http.filters=com.example.LoggingFilter

Resources