Open Telemetry Logger MDC auto-instrumentation - spring-boot

The following Open Telemetry starters have been added to the Spring Boot project (v2.7.2) to instrument the application:
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-spring-boot-starter</artifactId>
<version>1.22.1-alpha</version>
</dependency>
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-jaeger-spring-boot-starter</artifactId>
<version>1.22.1-alpha</version>
</dependency>
Traces and spans are successfully exported to a Jaeger collector. The problem is those traces and spans cannot be correlated with log statements because logs don't contain the current trace_id and span_id.
By following the documentation I added the logging.pattern.level property to application.properties but it seems like information about the current span is not injected into the logging event's MDC copy.
logging.pattern.level = trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %5p
For example:
log.info(
"traceId: {}, spanId: {}",
Span.current().getSpanContext().getTraceId(),
Span.current().getSpanContext().getSpanId());
2023-01-25 12:21:36.774 trace_id= span_id= trace_flags= INFO 34272 --- [nio-8080-exec-2] h.c.DemoController : traceId: 1bccb6a4fea8345026ca87a202f0091f, spanId: c59a5d44ee40cd2c
Have I missed anything?

opentelemetry-spring-boot-starter is missing this feature right now (version 1.22.1-alpha), but it can be added very easily.
Firstly, MDC Instrumentation for Logback dependency should be added to the project.
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-logback-mdc-1.0</artifactId>
<version>OPENTELEMETRY_VERSION</version>
</dependency>
Then add a custom logback.xml configuration to the classpath, which will automatically inject context information from the span context to logging events.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- Default logback configuration -->
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/spring.log}"/>
<include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml"/>
<!-- -->
<appender name="OTEL" class="io.opentelemetry.instrumentation.logback.v1_0.OpenTelemetryAppender">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="FILE"/>
</appender>
<root level="INFO">
<appender-ref ref="OTEL"/>
</root>
</configuration>
Finally, add the property to application.properties (as mentioned in the question):
logging.pattern.level = trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %5p

Related

Logback configuration in custom Spring Boot starter

The purpose of a custom Spring Boot starter is to provide a default Logback XML configuration, e.g.:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/spring.log}"/>
<include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml"/>
<root level="INFO">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="FILE"/>
</root>
</configuration>
It also provides a default logging.pattern.level defined in the starter's properties loaded by EnvironmentPostProcessor. For instance, let's assume it appends spring.application.name to the logging level:
logging.pattern.level=%5p [${spring.application.name}]
The problems that occur:
Property spring.application.name is not recognized:
2023-02-07 09:44:00.704 INFO [spring.application.name_IS_UNDEFINED] 70495 ---...
The default logging.pattern.level cannot be overridden in the application properties. For example, if the application.properties contains the property:
logging.pattern.level=%5p stackoverflow
#Bean
public String test(#Value("${logging.pattern.level}") String loggingPattern) {
Logger logger = Logger.getLogger("LogDemo");
logger.info(loggingPattern);
return loggingPattern;
}
2023-02-07 09:57:50.124 INFO [spring.application.name_IS_UNDEFINED] 70880 --- [ main] LogDemo : %5p stackoverflow
If logback.xml is not defined in the custom starter the spring.application.name is recognized successfully. If logging.pattern.level is moved from the starter's properties to the application.properties it also works.
It seems like logging is being configured right after loading the starter's properties but before loading the application properties, in case the starter's properties contain the logging.pattern.level.
Is there a way to define a default logging configuration without the problems listed above?
I managed to define a default logging configuration in a custom starter using Spring Boot's extensions to Logback, without additional properties and EnvironmentPostProcessor.
Because the standard logback.xml configuration file is loaded too
early, you cannot use extensions in it. You need to either use
logback-spring.xml or define a logging.config property.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<springProperty scope="context" name="SPRING_APPLICATION_NAME" source="spring.application.name"/>
<springProperty scope="context" name="SPRING_LOG_LEVEL_PATTERN" source="logging.pattern.level"/>
<property name="DEFAULT_LOG_LEVEL_PATTERN"
value="%5p [${SPRING_APPLICATION_NAME}]"/>
<property name="LOG_LEVEL_PATTERN"
value="${SPRING_LOG_LEVEL_PATTERN:-${DEFAULT_LOG_LEVEL_PATTERN}}"/>
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/spring.log}"/>
<include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml"/>
<root level="INFO">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="FILE"/>
</root>
</configuration>

GCP and Spring logback. Severity is always info

When logging errors to stackdriver, every message is logged as INFO, even when using log.error or log.warn, etc., but the payload is correct.
I'd like to be able to filter by severity and get email on error.
I'm using Spring Boot and Logback. The app has been deployed on a Kubernetes Cluster on GCP.
Here is my logback-spring.xml
<configuration>
<include resource="org/springframework/cloud/gcp/autoconfigure/logging/logback-appender.xml" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d{HH:mm:ss, UTC} %-5level %logger{35} - %msg %n</pattern>
</encoder>
</appender>
<springProfile name="prod,qa">
<root level="WARN">
<appender-ref ref="STACKDRIVER" />
</root>
</springProfile>
</configuration>
And here is the dep added in Maven
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-gcp-starter-logging</artifactId>
</dependency>
Spring Boot version: 2.1.3.RELEASE
Spring Cloud version: Greenwich.RELEASE
What is wrong with this config? Is there any other solution?
EDIT: Just realized that the STACKDRIVER appender above is not the one logging to Stackdriver, but STDOUT is enough (maybe bc it's a Kubernetes cluster?), but the issue persists
The Stackdriver logging agent configuration for Kubernetes defaults to INFO for any logs written to the container's stdout and ERROR for logs written to stderr. If you want finer-grained control over severity, you can configure Spring to log as single-line JSON (e.g., via JsonLayout1) and let the logging agent pick up the severity from the JSON object (see https://cloud.google.com/logging/docs/agent/configuration#process-payload).
1By default, JsonLayout will use "level" for the log level, while the Stackdriver logging agent recognizes "severity", so you may have to override addCustomDataToJsonMap.
See also GKE & Stackdriver: Java logback logging format?
Directly using google cloud logging logback appender takes the severity from the log level on each case:
In Maven:
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-logging-logback</artifactId>
<version>0.116.0-alpha</version>
</dependency>
In logback.xml:
<appender name="Cloud" class="com.google.cloud.logging.logback.LoggingAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<log>YOUR_LOG_NAME</log>
<resourceType>container</resourceType>
<flushLevel>INFO</flushLevel>
</appender>
...
<logger name="org.springframework" level="WARN" additivity="true">
<appender-ref ref="Cloud"/>
</logger>
It is not the spring cloud component but solves the problem.

Spring boot Microservices logging with Graylog

I want to use Graylog/RabbitMQ for logging with my spring boot microservices. As per my understanding I have to send my logs to RabbitMQ and have to integrate it with Graylog. I want to know the workflow and how to implement it like how to send the logs to RabbitMQ, do I need to use any other logging framework.
You can use Logback appender to send logs from spring-boot app. Add following dependency to your pom.xml
<dependency>
<groupId>de.siegmar</groupId>
<artifactId>logback-gelf</artifactId>
<version>1.1.0</version>
</dependency>
Then you need to add a logback configuration file to your classpath.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml" />
<appender name="GELF" class="de.siegmar.logbackgelf.GelfUdpAppender">
<graylogHost>localhost</graylogHost>
<graylogPort>12201</graylogPort>
<maxChunkSize>508</maxChunkSize>
<useCompression>true</useCompression>
<layout class="de.siegmar.logbackgelf.GelfLayout">
<originHost>localhost</originHost>
<includeRawMessage>false</includeRawMessage>
<includeMarker>true</includeMarker>
<includeMdcData>true</includeMdcData>
<includeCallerData>false</includeCallerData>
<includeRootCauseData>false</includeRootCauseData>
<includeLevelName>false</includeLevelName>
<shortPatternLayout class="ch.qos.logback.classic.PatternLayout">
<pattern>%m%nopex</pattern>
</shortPatternLayout>
<fullPatternLayout class="ch.qos.logback.classic.PatternLayout">
<pattern>%m</pattern>
</fullPatternLayout>
<staticField>app_name:backend</staticField>
<staticField>os_arch:${os.arch}</staticField>
<staticField>os_name:${os.name}</staticField>
<staticField>os_version:${os.version}</staticField>
</layout>
</appender>
<root level="debug">
<appender-ref ref="GELF" />
<appender-ref ref="CONSOLE" />
</root>
</configuration>
For more information: logback-gelf

Spring Boot and Logging with External Tomcat only outputs startup log

My Spring boot 2 app is not showing any log message when is running. I can only see the startup log. This app is deployed as WAR in the production server and I configured the log to output to a file:
logging.file = app.log
logging.level.root=WARN
logging.level.org.springframework.web=DEBUG
logging.level.org.hibernate=ERROR
logging.pattern.console= %d{yyyy-MM-dd HH:mm:ss} - %msg%n
In my local I can see whatever debug message I include in my code but in the server I can't. I only see the application startup trace.
My config to generate the file is the provided by official guideance. And the tomcat dependency in the app.war:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<scope>provided</scope>
</dependency>
Do you have any idea of what is happening? It is strange. The log file is generating in the server ( we deploy it in a docker container ) but after the app is running, no more log is output to the file.
By default Spring Boot logs on INFO level, which should include ERROR.
As per Spring boot logging guide 79.1.1 Configure Logback for File-only Output
If you want to disable console logging and write output only to a file, you need a custom logback-spring.xml that imports file-appender.xml but not console-appender.xml, as shown in the following example:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}/}spring.log}"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<root level="INFO">
<appender-ref ref="FILE" />
</root>
</configuration>
You also need to add logging.file to your application.properties which you have alraedy added.

How to set max number of archived logs in spring boot

I tried following settings in application.properties:
logging.file=foo/bar.log
logging.file.max-history=2
logging.file.max-size=1KB
Still, its not limiting the number of archive logs to 2.
As per application properties documentation reference, only supported when you setup logback.
logging.file.max-history=0 # Maximum of archive log files to keep. Only supported with the default logback setup.
So to add support of logback please see section 79.1 Configure Logback for Logging & 79.1.1 Configure Logback for File-only Output of Spring Boot Logging Guide
If you want to disable console logging and write output only to a
file, you need a custom logback-spring.xml that imports
file-appender.xml but not console-appender.xml, as shown in the
following example:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}/}spring.log}"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<root level="INFO">
<appender-ref ref="FILE" />
</root>
</configuration>

Resources