splunk logback/log4j appender does not seem to log splunk - aws-lambda

Question

I need to log to splunk from AWS Lambda using Java8 runtime. It uses spring framework and I added logback splunk appender to the project. There are no errors and the logs don't seem to show up in splunk. The splunk admin mentioned that there are no requests received on splunk server. When I tried to invoke the REST API manually, the log show up in splunk. So the connectivity from AWS Lambda to splunk server is good. The splunk appender seem to be invoking the API in async fashion and I added a 50seconds sleep at the end of the AWS Lambda code to see whether it is an issue with VM exiting before the async step completes. No luck yet. How do I debug further?
Code snippet:-
public class LambdaApp implements RequestHandler<String, Object>
{
private static final Logger LOGGER = LoggerFactory.getLogger(LambdaApp.class);
private static final Logger SPLUNK_LOGGER = LoggerFactory.getLogger("splunk.logger");
#Override
public Object handleRequest(String event, Context context)
{
SPLUNK_LOGGER.info("AWS Lambda start");
try {
Thread.sleep(50000);
} catch(InterruptedException ex) {
Thread.currentThread().interrupt();
}
return "handled";
}
Maven dependency:-
<dependency>
<groupId>com.splunk.logging</groupId>
<artifactId>splunk-library-javalogging</artifactId>
<version>1.5.2</version>
</dependency>
Logback configuration:-
<appender name="http" class="com.splunk.logging.HttpEventCollectorLogbackAppender">
    <url>https://a.b.c.d:8088</url>
    <token>valid-token</token>
    <disableCertificateValidation>true</disableCertificateValidation>
    <layout class="ch.qos.logback.classic.PatternLayout">
        <pattern>{%msg}</pattern>
    </layout>
</appender>
<logger name ="splunk.logger" level="DEBUG">
<appender-ref ref="http" />
</logger>

Answer 1

The first step is to add batch_size_count to rule out any issues with the HttpEventCollectorLogbackAppender not flushing to Splunk.
<appender name="http" class="com.splunk.logging.HttpEventCollectorLogbackAppender">
<url>https://a.b.c.d:8088</url>
<token>valid-token</token>
<batch_size_count>1</batch_size_count>
<disableCertificateValidation>true</disableCertificateValidation>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>{%msg}</pattern>
</layout>
</appender>
You should also verify that you are using Splunk 6.3+ on the receiving end since HTTP Event Collector requires a minimum of v6.3

Answer 2

You have to specify a valid / existing Splunk index or the log entry will be dropped silently on the floor.
Setting batch_size_count to 1 will make sure that every log entry gets flushed. Unfortunately everything's written asynchronously no matter how you configure it (even though "sequential" is the default sendMode), so flushing isn't sufficient to make sure it's actually flushed. For that you need to set terminationTimeout to a value (in milliseconds) that's "long enough", but not too long to significantly impact your lambda.
Note that terminationTimeout is applied on each flush and not just during whatever they think is "termination". They also use busy polling to implement it. It was apparently implemented by someone that doesn't really understand how to write multi-threaded code or the implications of a thread spinning every few milliseconds to poll the state of another thread.