How to Log AWSRequestId with Micronaut Lambda Application Native Image - aws-lambda

Based on this guide about logging AWSRequestId, I can add %X{AWSRequestId} on my logback.xml to print the request id on my lambda function.
On the example, the method that adds this key on MDC is populateMappingDiagnosticContextValues of class MicronautRequestHandler. But since my lambda function has two endpoints, I created it as an Application type and it says it's using the MicronautLambdaHandler as the handler class.
What I did is I created my custom handler that extends the MicronautLambdaHandler and added the populateMappingDiagnosticContextValues method.
#Override
public AwsProxyResponse handleRequest(AwsProxyRequest input, Context context) {
if (context != null) {
populateMappingDiagnosticContextValues(context);
}
return handler.proxy(input, context);
}
logback.xml
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<withJansi>false</withJansi>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss} %X{AWSRequestId} %-5p %c{1} - %m%n</pattern>
</encoder>
</appender>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</configuration>
On the lambda setup I pointed the handler on my custom handler and the app is working.
Except that the logs is not showing the AWSRequestId.
Is this functionality only works on MicronautRequestHandler class and not on MicronautLambdaHandler?
I also tried the aws guide here that uses log4j2 for logging the request id. But log4j doesn't work when building the GraalVM native image, that's why I sticked to logback.

If you copied populateMappingDiagnosticContextValues and mdcput into your Handler (I suppose you did it) it should work without any additional changes.

Related

Logback and JsonLayout: can't pass custom fields

I have this configuration in my logback.xml into a Spring Web Application (NO Spring Boot).
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>true</prettyPrint>
</jsonFormatter>
</layout>
<customFields>{"appname":"foobar"}</customFields>
</encoder>
</appender>
<!-- LOG everything at INFO level -->
<root level="INFO">
<appender-ref ref="Console" />
</root>
</configuration>
The JSON layout works fine but custom fields as "appname": "foobar" are not printed:
{
"timestamp" : "2020-06-10T14:55:25.534Z",
"level" : "INFO",
"thread" : "Catalina-utility-1",
"logger" : "org.springframework.web.servlet.DispatcherServlet",
"message" : "FrameworkServlet 'dispatcher': initialization completed in 72 ms",
"context" : "default"
}
What am I doing wrong?
SOLUTION
I was using the wrong libraries for my needs:
logback-jackson
logback-json-classic
Because of the fact that I need to process logs through Logstash I've corrected my configuration like this:
pom.xml
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>6.4</version>
</dependency>
logback.xml
<appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<customFields>{"customer":"X", "appname":"Y", "environment":"dev"}</customFields>
</encoder>
</appender>
and now It works fine.
I just stumbled this question because I had the same problem, and I found a solution, with logback-jackson and logback-json-classic.
Option 1: Per-Thread via Mapped Diagnostic Context (MDC)
SLF4j's Mapped Diagnostic Context is a per-thread key-value store that we can use to write custom structured data to the log output.
MDC.put("customKey", "customValue");
Logback's JsonLayout will automatically print this value under a special mdc JSON object without any further configuration.
{ [...], "mdc": {"customKey", "customValue"}}
Note that the MDC is constructed per thread and if it is empty, no mdc field is printed to the log output.
Option 2: Global (for all threads)
If you want custom fields to appear at the JSON output's root, you need to create a custom, but simple Layout class that extends JsonLayout. JsonLayout provides us with a addCustomDataToJsonMap we can override.
package com.mypackage;
import ch.qos.logback.contrib.json.classic.JsonLayout;
public class CustomJsonLayout extends JsonLayout {
#Override
protected void addCustomDataToJsonMap(Map<String, Object> map, ILoggingEvent event) {
map.put("customKey", "customValue");
}
}
Now, you just need to tell Logback to use CustomJsonLayout instead of JsonLayout in your logback.xml file and keep the rest the same.
<layout class="com.mypackage.CustomJsonLayout">
...
</layout>
Now, any log message will have the following output:
{ ..., "customKey": "customValue"}

Logback.xml configuration by environment

I have logback configuration which is like bellow, as you can see i have an appender that appends all the logs to the standalone logging service.
<appender name="digitalGelfAppender"
class="de.siegmar.logbackgelf.GelfUdpAppender">
<graylogHost>testserver.loggingservice.com</graylogHost>
<graylogPort>1234</graylogPort>
<useCompression>true</useCompression>
<layout class="de.siegmar.logbackgelf.GelfLayout">
<originHost>originHost</originHost>
<includeRawMessage>false</includeRawMessage>
<includeMarker>true</includeMarker>
<includeMdcData>true</includeMdcData>
<includeCallerData>false</includeCallerData>
<includeRootCauseData>false</includeRootCauseData>
<includeLevelName>false</includeLevelName>
<shortPatternLayout class="ch.qos.logback.classic.PatternLayout">
<pattern>${log.pattern}</pattern>
</shortPatternLayout>
<fullPatternLayout class="ch.qos.logback.classic.PatternLayout">
<pattern>${log.pattern}</pattern>
</fullPatternLayout>
<staticField>application:DAP</staticField>
<staticField>environment:UAT</staticField>
</layout>
</appender>
<root level="#logback.loglevel#">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="digitalGelfAppender"/>
</root>
My question is that i need to change
<graylogHost>testserver.loggingservice.com</graylogHost>
<graylogPort>1234</graylogPort>
by environment value. So it will be like bellow on production server.
<graylogHost>prodserver.loggingservice.com</graylogHost>
<graylogPort>prodPort</graylogPort>
How can i pass the environment value to the logback.xml file by environment?
I tried to add these variables and use it like property inside xml. Its not working at all.
<property name="GRAYLOG_SERVER" value="${graylog-hostname}" />
<property name="GRAYLOG_SERVER_PORT" value="{graylog-port}" />
Usage
<graylogHost>${GRAYLOG_SERVER}</graylogHost>
<graylogPort>${GRAYLOG_SERVER_PORT}</graylogPort>
Hi i solve my problem by implementing listener that retrieve data from System parameter of environment;
#Controller
public class LoggerContextListener extends ContextAwareBase implements
LoggerContextListener, LifeCycle {
#Override
public void start() {
Map<String, String> property = System.getenv();
Context context = getContext();
context.putProperty("GRAYLOG_SERVER", property.get("graylog-hostname"));
context.putProperty("GRAYLOG_SERVER_PORT", property.get("graylog-port"));
}
After that i add that listener as a contextListener to the lockback.xml
<contextListener class="com.anadolusigorta.dap.config.DapLoggerContextListener"/>
Then problem solved! Now my xml can use these parameter that set from listener;
<graylogHost>${GRAYLOG_SERVER}</graylogHost>
<graylogPort>${GRAYLOG_SERVER_PORT}</graylogPort>
You don't need to create the listener. It is enough that use the environment variables as follows.
<graylogHost>${graylog-hostname}</graylogHost>
<graylogPort>${graylog-port}</graylogPort>

Access to request parameter in custom logback appender

I have custom logback appender that is used in my project. The project is based on REST controllers.
Is there a possibility to have access to current request parameters in the logback appender that is used by logger in spring controller? I need to write something to logs in basis on the request parameter.
What I can see is that logback is initialized before spring boot beans which is not good for me.
Use the org.slf4j.MDC: You can put values in the MDC at any time. Before anything else.
Example :
#ResponseBody
#RequestMapping(value="/myRequest", method = RequestMethod.GET)
public String doAnyAction(#RequestParam(value="Key", required = true) long key)
{
MDC.put("key", key);
logger.info(key);
}
And change your Logback.xml appender to add the "key" as well :
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<layout>
<Pattern>%X{key} - %m%n</Pattern>
</layout>
</appender>

Debug logging of ResponseBody by Springframework

I have a scenario where I send some sensitive data in the ResponseBody of a spring controller. Spring logs the entire response body in DEBUG mode, where I can see the sensitive data also. Now if I configure my app to print only INFO logs or higher, this won't be displayed, but that is not a fool proof method, since its possible that DEBUG mode could be turned on accidentally.
Is there anyway I can disable certain fields in the response body from being logged by Spring?
Thanks
My suggestion would be that you implement a custom converter for logback (if that is your logging library)
http://logback.qos.ch/manual/layouts.html#customConversionSpecifier
It enables you to convert your logging data and blur out any data that should be removed from the message.
From the documentation it is pretty straight forward
public class MySampleConverter extends ClassicConverter {
long start = System.nanoTime();
#Override
public String convert(ILoggingEvent event) {
long nowInNanos = System.nanoTime();
return Long.toString(nowInNanos-start);
}
}
And the config
<configuration>
<conversionRule conversionWord="nanos"
converterClass="chapters.layouts.MySampleConverter" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%-6nanos [%thread] - %msg%n</pattern>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="STDOUT" />
</root>
</configuration>
If you're using Log4j you can implement a custom filter which would mask the data per your layout
http://vozis.blogspot.sg/2012/02/log4j-filter-to-mask-payment-card.html
Edit
The final easier solution turned out to be to override toString of the return object. Spring uses toString to log the return message

Querydsl logging queries with bindings

logQuery is called in prepareStatementAndSetParameters mehtod - SQLInsertClause class
protected void logQuery(Logger logger, String queryString, Collection<Object> parameters) {
String normalizedQuery = queryString.replace('\n', ' ');
MDC.put(QueryBase.MDC_QUERY, normalizedQuery);
MDC.put(QueryBase.MDC_PARAMETERS, String.valueOf(parameters));
if (logger.isDebugEnabled()) {
logger.debug(normalizedQuery);
}
}
how can I set debug level to logger ?
That logger there is from SLF4J API. Depending on the logger you have behind the API you use facilities of that underlying logging implementation.
For instance we use Logback Classic (dependency ch.qos.logback:logback-classic) and I can explicitly override what configuration file to use with -Dlogback.configurationFile=devel-logback.xml in JVM parameters. Default mechanism is documented here. My file looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%date %level [%.60thread] %logger{1} %msg%n</pattern>
</encoder>
</appender>
<logger name="com.mysema.query.jpa.impl.JPAQuery" level="DEBUG"/>
<!-- more loggers -->
<root level="DEBUG">
<appender-ref ref="CONSOLE"/>
</root>
</configuration>
Also adding -Dlogback.debug=true to JVM arguments adds some debug output when logback is being initialized.

Resources