I want to use Graylog/RabbitMQ for logging with my spring boot microservices. As per my understanding I have to send my logs to RabbitMQ and have to integrate it with Graylog. I want to know the workflow and how to implement it like how to send the logs to RabbitMQ, do I need to use any other logging framework.
You can use Logback appender to send logs from spring-boot app. Add following dependency to your pom.xml
<dependency>
<groupId>de.siegmar</groupId>
<artifactId>logback-gelf</artifactId>
<version>1.1.0</version>
</dependency>
Then you need to add a logback configuration file to your classpath.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml" />
<appender name="GELF" class="de.siegmar.logbackgelf.GelfUdpAppender">
<graylogHost>localhost</graylogHost>
<graylogPort>12201</graylogPort>
<maxChunkSize>508</maxChunkSize>
<useCompression>true</useCompression>
<layout class="de.siegmar.logbackgelf.GelfLayout">
<originHost>localhost</originHost>
<includeRawMessage>false</includeRawMessage>
<includeMarker>true</includeMarker>
<includeMdcData>true</includeMdcData>
<includeCallerData>false</includeCallerData>
<includeRootCauseData>false</includeRootCauseData>
<includeLevelName>false</includeLevelName>
<shortPatternLayout class="ch.qos.logback.classic.PatternLayout">
<pattern>%m%nopex</pattern>
</shortPatternLayout>
<fullPatternLayout class="ch.qos.logback.classic.PatternLayout">
<pattern>%m</pattern>
</fullPatternLayout>
<staticField>app_name:backend</staticField>
<staticField>os_arch:${os.arch}</staticField>
<staticField>os_name:${os.name}</staticField>
<staticField>os_version:${os.version}</staticField>
</layout>
</appender>
<root level="debug">
<appender-ref ref="GELF" />
<appender-ref ref="CONSOLE" />
</root>
</configuration>
For more information: logback-gelf
Related
The following Open Telemetry starters have been added to the Spring Boot project (v2.7.2) to instrument the application:
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-spring-boot-starter</artifactId>
<version>1.22.1-alpha</version>
</dependency>
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-jaeger-spring-boot-starter</artifactId>
<version>1.22.1-alpha</version>
</dependency>
Traces and spans are successfully exported to a Jaeger collector. The problem is those traces and spans cannot be correlated with log statements because logs don't contain the current trace_id and span_id.
By following the documentation I added the logging.pattern.level property to application.properties but it seems like information about the current span is not injected into the logging event's MDC copy.
logging.pattern.level = trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %5p
For example:
log.info(
"traceId: {}, spanId: {}",
Span.current().getSpanContext().getTraceId(),
Span.current().getSpanContext().getSpanId());
2023-01-25 12:21:36.774 trace_id= span_id= trace_flags= INFO 34272 --- [nio-8080-exec-2] h.c.DemoController : traceId: 1bccb6a4fea8345026ca87a202f0091f, spanId: c59a5d44ee40cd2c
Have I missed anything?
opentelemetry-spring-boot-starter is missing this feature right now (version 1.22.1-alpha), but it can be added very easily.
Firstly, MDC Instrumentation for Logback dependency should be added to the project.
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-logback-mdc-1.0</artifactId>
<version>OPENTELEMETRY_VERSION</version>
</dependency>
Then add a custom logback.xml configuration to the classpath, which will automatically inject context information from the span context to logging events.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- Default logback configuration -->
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/spring.log}"/>
<include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml"/>
<!-- -->
<appender name="OTEL" class="io.opentelemetry.instrumentation.logback.v1_0.OpenTelemetryAppender">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="FILE"/>
</appender>
<root level="INFO">
<appender-ref ref="OTEL"/>
</root>
</configuration>
Finally, add the property to application.properties (as mentioned in the question):
logging.pattern.level = trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %5p
I'm using spring-boot-starter-web like framework. These, use Logback (without implement other libraries) to manage logs with SLF4J like a facade. I need to use it with rsyslog but the official doc refers only to Syslog.
I tried to use the Syslog implementation since Syslog inherit for rsyslog but not found ( i attach my logback-spring.xml below ).
Any idea?
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="30 seconds">
<property resource="application.properties" />
<!-- Syslog , make sure that syslog are enabled in the OS-->
<appender name="RSYSLOG" class="ch.qos.logback.classic.net.SyslogAppender">
<syslogHost>127.0.0.1</syslogHost>
<facility>LOCAL0</facility>
<port>514</port>
<throwableExcluded>true</throwableExcluded>
<suffixPattern>%package.yes.rest %m thread:%t priority:%p category:%c
exception:%exception:%msg</suffixPattern>
</appender>
<logger name="package.yes.rest" level="info" additivity="false">
<appender-ref ref="RSYSLOG"/>
</logger>
</configuration>
Bonus clip: I see the choose of change Logback to Log4j2 too, but it's more stable use the inherit Logback.
When logging errors to stackdriver, every message is logged as INFO, even when using log.error or log.warn, etc., but the payload is correct.
I'd like to be able to filter by severity and get email on error.
I'm using Spring Boot and Logback. The app has been deployed on a Kubernetes Cluster on GCP.
Here is my logback-spring.xml
<configuration>
<include resource="org/springframework/cloud/gcp/autoconfigure/logging/logback-appender.xml" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d{HH:mm:ss, UTC} %-5level %logger{35} - %msg %n</pattern>
</encoder>
</appender>
<springProfile name="prod,qa">
<root level="WARN">
<appender-ref ref="STACKDRIVER" />
</root>
</springProfile>
</configuration>
And here is the dep added in Maven
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-gcp-starter-logging</artifactId>
</dependency>
Spring Boot version: 2.1.3.RELEASE
Spring Cloud version: Greenwich.RELEASE
What is wrong with this config? Is there any other solution?
EDIT: Just realized that the STACKDRIVER appender above is not the one logging to Stackdriver, but STDOUT is enough (maybe bc it's a Kubernetes cluster?), but the issue persists
The Stackdriver logging agent configuration for Kubernetes defaults to INFO for any logs written to the container's stdout and ERROR for logs written to stderr. If you want finer-grained control over severity, you can configure Spring to log as single-line JSON (e.g., via JsonLayout1) and let the logging agent pick up the severity from the JSON object (see https://cloud.google.com/logging/docs/agent/configuration#process-payload).
1By default, JsonLayout will use "level" for the log level, while the Stackdriver logging agent recognizes "severity", so you may have to override addCustomDataToJsonMap.
See also GKE & Stackdriver: Java logback logging format?
Directly using google cloud logging logback appender takes the severity from the log level on each case:
In Maven:
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-logging-logback</artifactId>
<version>0.116.0-alpha</version>
</dependency>
In logback.xml:
<appender name="Cloud" class="com.google.cloud.logging.logback.LoggingAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<log>YOUR_LOG_NAME</log>
<resourceType>container</resourceType>
<flushLevel>INFO</flushLevel>
</appender>
...
<logger name="org.springframework" level="WARN" additivity="true">
<appender-ref ref="Cloud"/>
</logger>
It is not the spring cloud component but solves the problem.
I have to put all the log data (i.e., debug, info, error) into mysql database instead of to file/console.
I read the spring boot documentation but I didn't see any configuration related to database for logging.
https://docs.spring.io/spring-boot/docs/current/reference/html/howto-logging.html
Also tried the following link but its also not working.
https://www.tutorialspoint.com/log4j/log4j_logging_database.htm
Can anyone help me to do this.
Thanks.
I read the spring boot documentation but I didn't see any
configuration related to database for logging.
Because spring boot hands off that functionality to logging framework (slf4j + logback/log4j etc). So you need to configure your logging framework accordingly using it's configuration file (eg: logback.xml, logback-spring.xml, logback.groovy etc). Default logging frameworks in Spring boot is slf4j+logback. So checkout how you can use DBAppender.
For Logback
https://logback.qos.ch/manual/appenders.html#DBAppender
http://learningviacode.blogspot.com/2014/01/writing-logs-to-database.html
Log to database with LogBack
https://medium.com/#chakrar27/storing-log-data-in-postgresql-using-logback-db-appender-292891a9918
1. create logback.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{5} - %msg%n
</pattern>
</encoder>
</appender>
<appender name="db" class="ch.qos.logback.classic.db.DBAppender">
<connectionSource
class="ch.qos.logback.core.db.DriverManagerConnectionSource">
<driverClass>org.postgresql.Driver</driverClass>
<url>jdbc:postgresql://localhost:5432/simple</url>
<user>postgres</user>
<password>root</password> <!-- no password -->
</connectionSource>
</appender>
<!-- the level of the root level is set to DEBUG by default. -->
<root level="TRACE">
<appender-ref ref="stdout" />
<appender-ref ref="db" />
</root>
</configuration>
2. Create the 3 tables
logging_event
logging_event_property
logging_event_exception
They must exist before DBAppender can be used
For Log4J
https://logging.apache.org/log4j/2.x/manual/appenders.html#JDBCAppender
For Log4J2
http://smasue.github.io/log4j2-spring-database-appender
I am trying to configure Log4j 2 in Spring XML configuration file for the first time (but unsuccessfully). I need to create two appenders - one for logging into console (>=DEBUG) and another for logging into database via JDBCAppender (>= INFO).
There is a problem because I don't know how to set another log level logger that differs from root logger.
Thank you for sharing some XML configuration sample. Thanks in advance!
You can set the level on the appender ref.
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="WARN">
<appenders>
<appender name="A">
...
</appender>
<appender name="B">
...
</appender>
</appenders>
<loggers>
<root level="trace">
<appender-ref ref="A" level="info"/>
<appender-ref ref="B" level="debug"/>
</root>
</loggers>
</configuration>