GCP and Spring logback. Severity is always info - spring-boot

When logging errors to stackdriver, every message is logged as INFO, even when using log.error or log.warn, etc., but the payload is correct.
I'd like to be able to filter by severity and get email on error.
I'm using Spring Boot and Logback. The app has been deployed on a Kubernetes Cluster on GCP.
Here is my logback-spring.xml
<configuration>
<include resource="org/springframework/cloud/gcp/autoconfigure/logging/logback-appender.xml" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d{HH:mm:ss, UTC} %-5level %logger{35} - %msg %n</pattern>
</encoder>
</appender>
<springProfile name="prod,qa">
<root level="WARN">
<appender-ref ref="STACKDRIVER" />
</root>
</springProfile>
</configuration>
And here is the dep added in Maven
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-gcp-starter-logging</artifactId>
</dependency>
Spring Boot version: 2.1.3.RELEASE
Spring Cloud version: Greenwich.RELEASE
What is wrong with this config? Is there any other solution?
EDIT: Just realized that the STACKDRIVER appender above is not the one logging to Stackdriver, but STDOUT is enough (maybe bc it's a Kubernetes cluster?), but the issue persists

The Stackdriver logging agent configuration for Kubernetes defaults to INFO for any logs written to the container's stdout and ERROR for logs written to stderr. If you want finer-grained control over severity, you can configure Spring to log as single-line JSON (e.g., via JsonLayout1) and let the logging agent pick up the severity from the JSON object (see https://cloud.google.com/logging/docs/agent/configuration#process-payload).
1By default, JsonLayout will use "level" for the log level, while the Stackdriver logging agent recognizes "severity", so you may have to override addCustomDataToJsonMap.
See also GKE & Stackdriver: Java logback logging format?

Directly using google cloud logging logback appender takes the severity from the log level on each case:
In Maven:
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-logging-logback</artifactId>
<version>0.116.0-alpha</version>
</dependency>
In logback.xml:
<appender name="Cloud" class="com.google.cloud.logging.logback.LoggingAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<log>YOUR_LOG_NAME</log>
<resourceType>container</resourceType>
<flushLevel>INFO</flushLevel>
</appender>
...
<logger name="org.springframework" level="WARN" additivity="true">
<appender-ref ref="Cloud"/>
</logger>
It is not the spring cloud component but solves the problem.

Related

How to ignore logback RollingFileAppender FileNotFoundException

I am using logback in my spring boot project, but logback log file must locate in /home/xxx/logs folder.
spring boot cannot start caused by RollingFileAppender's FileNotFoundException exception in my MacOS machine, because of MacOS cannot create folder /home/xxx/logs.
How to ignore this Exception in my spring boot?
As much I know logging framework don't provide any exception handling capability. It's not their job. Either you Correct log location and syntax or just remove config which you don't need.
Include a file logback-spring.xml in your classpath or resources folder and then you will be able to explicitly configure the location of the log file or you may have it print to the console instead. The config below would do just that and override the default config that comes with spring boot.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{dd-MM-yyyy HH:mm:ss.SSS} %magenta([%thread]) %highlight(%-5level) %logger{36}.%M - %msg%n</pattern>
</encoder>
</appender>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</configuration>

Spring boot Microservices logging with Graylog

I want to use Graylog/RabbitMQ for logging with my spring boot microservices. As per my understanding I have to send my logs to RabbitMQ and have to integrate it with Graylog. I want to know the workflow and how to implement it like how to send the logs to RabbitMQ, do I need to use any other logging framework.
You can use Logback appender to send logs from spring-boot app. Add following dependency to your pom.xml
<dependency>
<groupId>de.siegmar</groupId>
<artifactId>logback-gelf</artifactId>
<version>1.1.0</version>
</dependency>
Then you need to add a logback configuration file to your classpath.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml" />
<appender name="GELF" class="de.siegmar.logbackgelf.GelfUdpAppender">
<graylogHost>localhost</graylogHost>
<graylogPort>12201</graylogPort>
<maxChunkSize>508</maxChunkSize>
<useCompression>true</useCompression>
<layout class="de.siegmar.logbackgelf.GelfLayout">
<originHost>localhost</originHost>
<includeRawMessage>false</includeRawMessage>
<includeMarker>true</includeMarker>
<includeMdcData>true</includeMdcData>
<includeCallerData>false</includeCallerData>
<includeRootCauseData>false</includeRootCauseData>
<includeLevelName>false</includeLevelName>
<shortPatternLayout class="ch.qos.logback.classic.PatternLayout">
<pattern>%m%nopex</pattern>
</shortPatternLayout>
<fullPatternLayout class="ch.qos.logback.classic.PatternLayout">
<pattern>%m</pattern>
</fullPatternLayout>
<staticField>app_name:backend</staticField>
<staticField>os_arch:${os.arch}</staticField>
<staticField>os_name:${os.name}</staticField>
<staticField>os_version:${os.version}</staticField>
</layout>
</appender>
<root level="debug">
<appender-ref ref="GELF" />
<appender-ref ref="CONSOLE" />
</root>
</configuration>
For more information: logback-gelf

Spring boot logging into mysql database

I have to put all the log data (i.e., debug, info, error) into mysql database instead of to file/console.
I read the spring boot documentation but I didn't see any configuration related to database for logging.
https://docs.spring.io/spring-boot/docs/current/reference/html/howto-logging.html
Also tried the following link but its also not working.
https://www.tutorialspoint.com/log4j/log4j_logging_database.htm
Can anyone help me to do this.
Thanks.
I read the spring boot documentation but I didn't see any
configuration related to database for logging.
Because spring boot hands off that functionality to logging framework (slf4j + logback/log4j etc). So you need to configure your logging framework accordingly using it's configuration file (eg: logback.xml, logback-spring.xml, logback.groovy etc). Default logging frameworks in Spring boot is slf4j+logback. So checkout how you can use DBAppender.
For Logback
https://logback.qos.ch/manual/appenders.html#DBAppender
http://learningviacode.blogspot.com/2014/01/writing-logs-to-database.html
Log to database with LogBack
https://medium.com/#chakrar27/storing-log-data-in-postgresql-using-logback-db-appender-292891a9918
1. create logback.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{5} - %msg%n
</pattern>
</encoder>
</appender>
<appender name="db" class="ch.qos.logback.classic.db.DBAppender">
<connectionSource
class="ch.qos.logback.core.db.DriverManagerConnectionSource">
<driverClass>org.postgresql.Driver</driverClass>
<url>jdbc:postgresql://localhost:5432/simple</url>
<user>postgres</user>
<password>root</password> <!-- no password -->
</connectionSource>
</appender>
<!-- the level of the root level is set to DEBUG by default. -->
<root level="TRACE">
<appender-ref ref="stdout" />
<appender-ref ref="db" />
</root>
</configuration>
2. Create the 3 tables
logging_event
logging_event_property
logging_event_exception
They must exist before DBAppender can be used
For Log4J
https://logging.apache.org/log4j/2.x/manual/appenders.html#JDBCAppender
For Log4J2
http://smasue.github.io/log4j2-spring-database-appender

How can I use YAML to configure multiple log files for Logback/Springboot?

I'm rewriting a small DropWizard application to run on SpringBoot.
My DW app has the following logging config which works:
logging:
level: INFO
appenders:
- type: file
currentLogFilename: /var/log/paas/console.log
archivedLogFilenamePattern: /var/log/paas/console.log-%d.gz
archivedFileCount: 7
loggers:
com.myorg:
level: DEBUG
appenders:
- type: file
currentLogFilename: /var/log/paas/paas.log
archivedLogFilenamePattern: /var/log/paas/paas.log-%d.gz
archivedFileCount: 7
This config separates my application and console messages into two separate logs.
When I try using this same configuration with SpringBoot, it has no effect. I'm able to write everything to a single log with the following config, but I really need to have two separate logs:
logging:
level:
org.springframework.web: INFO
com.myorg: DEBUG
file: /var/log/paas/paas.log
Is it not possible to do this with LogBack and YAML? Or is there an alternate syntax that will give me the same result I'm getting for my DropWizard app?
Spring Boot's YAML configuration of Logback only allows a single file appender. In order to configure Logback to use multiple file appenders you'll have to provide an explicit logback.xml or logback-spring.xml. If you remove the logging confoguraiotn from your application.yaml file and then add either a logback.xml or a logback-spring.xml to the root of your runtime classpath then Logback will be configured from that file.
Here's an example, using a logback.xml:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="INFO_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!-- deny all events with a level below INFO, that is TRACE and DEBUG -->
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<file>/var/log/paas/console.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>/var/log/paas/console.log-%d.gz</fileNamePattern>
<maxHistory>7</maxHistory>
</rollingPolicy>
<encoder>
<pattern>...</pattern>
</encoder>
</appender>
<appender name="DEBUG_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/var/log/paas/paas.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>/var/log/paas/paas.log-%d.gz</fileNamePattern>
<maxHistory>7</maxHistory>
</rollingPolicy>
<encoder>
<pattern>...</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="INFO_FILE"/>
<appender-ref ref="DEBUG_FILE"/>
</root>
</configuration>

How to implement logging in custom module in Spring XD?

It's not so easy to debug custom module deployed to the Spring XD runtime (version 1.3.1-RELEASE).
I'm aware of log sink, however it's something different that I want to achieve.
I'd like to add my own log messages to the XD log (ideally to the STDOUT alongside it's own logs). These log messages are generated in my custom module (processor in this case) using slf4j API.
I've added:
org.slf4j.Logger#info invocation to the processor class
logback-classic dependency to the pom.xml (w/o a version, as it's managed by spring-xd-module-parent dependencyManagement
logback.xml to the resources directory
logback-test.xml to the test resources directory
Log messages are logged into STDOUT during integration test invocation (via SingleNodeIntegrationTestSupport), however they don't appear in the XD log when module is uploaded or stream using it is deployed.
logback.xml contents (identical for -test):
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
<charset>UTF-8</charset>
</encoder>
</appender>
<logger name="com.maxromanovsky" level="debug" />
<logger name="org.springframework" level="warn" />
<logger name="org.apache.zookeeper" level="error" />
<root level="warn">
<appender-ref ref="STDOUT" />
</root>
The container logback configuration files can be found in xd/config (xd-container-logback.groovy and xd-singlenode-logback.groovy).
You need to add your custom logger configuration there.

Resources