I have this logback setup in spring
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender class="ch.qos.logback.core.ConsoleAppender" name="CONSOLE">
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE"/>
</root>
</configuration>
How can I configure fluentd to collect logs from my spring applications in k8s and send them to elasticsearch?
I guess you have to be more specific. What have you tried? What does not work as expected? ... collecting logs of a spring application with fluentd is not different to using any other technology, as long as is it streamed to stdout/stderr.
It should be pretty straight forward. See this: https://medium.com/avmconsulting-blog/how-to-deploy-an-efk-stack-to-kubernetes-ebc1b539d063 or https://devopscube.com/setup-efk-stack-on-kubernetes/
Related
I'm using spring-boot-starter-web like framework. These, use Logback (without implement other libraries) to manage logs with SLF4J like a facade. I need to use it with rsyslog but the official doc refers only to Syslog.
I tried to use the Syslog implementation since Syslog inherit for rsyslog but not found ( i attach my logback-spring.xml below ).
Any idea?
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="30 seconds">
<property resource="application.properties" />
<!-- Syslog , make sure that syslog are enabled in the OS-->
<appender name="RSYSLOG" class="ch.qos.logback.classic.net.SyslogAppender">
<syslogHost>127.0.0.1</syslogHost>
<facility>LOCAL0</facility>
<port>514</port>
<throwableExcluded>true</throwableExcluded>
<suffixPattern>%package.yes.rest %m thread:%t priority:%p category:%c
exception:%exception:%msg</suffixPattern>
</appender>
<logger name="package.yes.rest" level="info" additivity="false">
<appender-ref ref="RSYSLOG"/>
</logger>
</configuration>
Bonus clip: I see the choose of change Logback to Log4j2 too, but it's more stable use the inherit Logback.
When logging errors to stackdriver, every message is logged as INFO, even when using log.error or log.warn, etc., but the payload is correct.
I'd like to be able to filter by severity and get email on error.
I'm using Spring Boot and Logback. The app has been deployed on a Kubernetes Cluster on GCP.
Here is my logback-spring.xml
<configuration>
<include resource="org/springframework/cloud/gcp/autoconfigure/logging/logback-appender.xml" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d{HH:mm:ss, UTC} %-5level %logger{35} - %msg %n</pattern>
</encoder>
</appender>
<springProfile name="prod,qa">
<root level="WARN">
<appender-ref ref="STACKDRIVER" />
</root>
</springProfile>
</configuration>
And here is the dep added in Maven
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-gcp-starter-logging</artifactId>
</dependency>
Spring Boot version: 2.1.3.RELEASE
Spring Cloud version: Greenwich.RELEASE
What is wrong with this config? Is there any other solution?
EDIT: Just realized that the STACKDRIVER appender above is not the one logging to Stackdriver, but STDOUT is enough (maybe bc it's a Kubernetes cluster?), but the issue persists
The Stackdriver logging agent configuration for Kubernetes defaults to INFO for any logs written to the container's stdout and ERROR for logs written to stderr. If you want finer-grained control over severity, you can configure Spring to log as single-line JSON (e.g., via JsonLayout1) and let the logging agent pick up the severity from the JSON object (see https://cloud.google.com/logging/docs/agent/configuration#process-payload).
1By default, JsonLayout will use "level" for the log level, while the Stackdriver logging agent recognizes "severity", so you may have to override addCustomDataToJsonMap.
See also GKE & Stackdriver: Java logback logging format?
Directly using google cloud logging logback appender takes the severity from the log level on each case:
In Maven:
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-logging-logback</artifactId>
<version>0.116.0-alpha</version>
</dependency>
In logback.xml:
<appender name="Cloud" class="com.google.cloud.logging.logback.LoggingAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<log>YOUR_LOG_NAME</log>
<resourceType>container</resourceType>
<flushLevel>INFO</flushLevel>
</appender>
...
<logger name="org.springframework" level="WARN" additivity="true">
<appender-ref ref="Cloud"/>
</logger>
It is not the spring cloud component but solves the problem.
I want to use Graylog/RabbitMQ for logging with my spring boot microservices. As per my understanding I have to send my logs to RabbitMQ and have to integrate it with Graylog. I want to know the workflow and how to implement it like how to send the logs to RabbitMQ, do I need to use any other logging framework.
You can use Logback appender to send logs from spring-boot app. Add following dependency to your pom.xml
<dependency>
<groupId>de.siegmar</groupId>
<artifactId>logback-gelf</artifactId>
<version>1.1.0</version>
</dependency>
Then you need to add a logback configuration file to your classpath.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml" />
<appender name="GELF" class="de.siegmar.logbackgelf.GelfUdpAppender">
<graylogHost>localhost</graylogHost>
<graylogPort>12201</graylogPort>
<maxChunkSize>508</maxChunkSize>
<useCompression>true</useCompression>
<layout class="de.siegmar.logbackgelf.GelfLayout">
<originHost>localhost</originHost>
<includeRawMessage>false</includeRawMessage>
<includeMarker>true</includeMarker>
<includeMdcData>true</includeMdcData>
<includeCallerData>false</includeCallerData>
<includeRootCauseData>false</includeRootCauseData>
<includeLevelName>false</includeLevelName>
<shortPatternLayout class="ch.qos.logback.classic.PatternLayout">
<pattern>%m%nopex</pattern>
</shortPatternLayout>
<fullPatternLayout class="ch.qos.logback.classic.PatternLayout">
<pattern>%m</pattern>
</fullPatternLayout>
<staticField>app_name:backend</staticField>
<staticField>os_arch:${os.arch}</staticField>
<staticField>os_name:${os.name}</staticField>
<staticField>os_version:${os.version}</staticField>
</layout>
</appender>
<root level="debug">
<appender-ref ref="GELF" />
<appender-ref ref="CONSOLE" />
</root>
</configuration>
For more information: logback-gelf
I am running an ActiveMQ broker in a RedHat OpenShift cluster, with an EFK (ElasticSearch, Fluentd, Kibana) stack handling logging. In my java components I have configured my loggers to log in JSON format using the LogStash encoder so that they can be properly parsed, stored and displayed in Kibana and I'm looking to do the same for ActiveMQ.
In my Java components my logging configuration looks like and I expected to be able to do something similar for ActiveMQ:
<configuration>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>
<logger name="jsonLogger" additivity="false" level="INFO">
<appender-ref ref="CONSOLE" />
</logger>
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
The ActiveMQ docs don't seem to mention any way of changing the logging appender: http://activemq.apache.org/how-do-i-change-the-logging.html
Has anyone managed to do this successfully before?
In my spring boot app the following configuration is used for adding JSON logs via logback-spring.xml:
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml"/>
<appender name="consoleAppender" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<fieldNames>
<timestamp>time</timestamp>
<message>msg</message>
<thread>thread</thread>
<logger>logger</logger>
<version>[ignore]</version>
<levelValue>[ignore]</levelValue>
</fieldNames>
</encoder>
</appender>
<root level="all">
<appender-ref ref="consoleAppender" />
</root>
</configuration>
However, when running the app both versions of logs (non-JSON & JSON) are traced:
2017-08-08 07:31:49.718 INFO 6849 --- [ main] .s.b.c.e.j.JettyEmbeddedServletContainer : Jetty started on port(s) 10000 (http/1.1)
{"time":"2017-08-08T07:31:49.718+02:00","msg":"Jetty started on port(s) 10000 (http/1.1)","logger":"org.springframework.boot.context.embedded.jetty.JettyEmbeddedServletContainer","thread":"main","level":"INFO","HOSTNAME":"ilya-ThinkPad-X1-Carbon-4th","caller_class_name":"org.springframework.boot.context.embedded.jetty.JettyEmbeddedServletContainer","caller_method_name":"start","caller_file_name":"JettyEmbeddedServletContainer.java","caller_line_number":144}
Is is possible to configure logback / logstash encoder, so that only JSON version would be available in the logs and non-JSON would be skipped ?
oh, stupid me - one should just remove the line from logback config which adds base logging:
<include resource="org/springframework/boot/logging/logback/base.xml"/>