Fluentd logs not sent to Elasticsearch - pattern not match - elasticsearch

Question

I can't get log messages to be processed by Fluentd and sent to Elasticsearch. I'm tailing the container of my service, it will pick up the log, but it can't parse it, it always fails with the error pattern not match. I understand that something is wrong with my parsing setup, but I can't see what.
The service writes to Stdout with Serilog, using the ElasticsearchJsonFormatter. My understanding is that it will write valid json to the console. This appears to be happening if I view the logs of the running container. When I view the Fluentd logs, it looks as if it has all been escaped.
If I view the logs of the service pod I can see the message, if then view the logs of the fluentd pod I can see the pattern not match error. Both of these are included below.
Any help or pointers will be greatly appreciated, as I've been stuck on this for days now.
Serilog Setup
var loggerConfig = new LoggerConfiguration()
.WriteTo.Console(new ElasticsearchJsonFormatter())
.MinimumLevel.Information()
.MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
.MinimumLevel.Override("System", LogEventLevel.Warning);
Fluentd Config
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
data:
fluent.conf: |
<source>
#type tail
path /var/log/containers/my-service-*.log
pos_file /var/log/app.log.pos
tag kubernetes.*
read_from_head true
<parse>
#type json
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
<match **>
#type elasticsearch
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
index_name fluentd
type_name fluentd
</match>
Example log message
This is what I can see if I view the logs for the running container. In my case this is a Kubernetes pod.
{
"#timestamp": "2021-12-29T13:23:28.2692128+00:00",
"level": "Information",
"messageTemplate": "Get Record - User Claim Contracts: {contracts} All Claims: {allclaims}",
"message": "Get Record - User Claim Contracts: [1, 2, 3, 4] All Claims: \"Application.Claims\"",
"fields": {
"contracts": [
1,
2,
3,
4
],
"allclaims": "Application.Claims",
"ddsource": "application",
"RecordId": null,
"UserId": null,
"TransactionId": null,
"env": "UAT",
}
}
Fluentd pattern not match
This is what I see when I view the logs for the fluentd container. Again, this is a pod.
2021-12-29 13:37:48 +0000 [warn]: #0 pattern not match: "2021-12-29T13:37:47.852294372Z stdout F {\"#timestamp\":\"2021-12-29T13:37:47.8518242+00:00\",\"level\":\"Information\",\"messageTemplate\":\"Get Record - User Claim Contracts: {contracts} All Claims: {allclaims}\",\"message\":\"Get Record - User Claim Contracts: [1, 2, 3, 4] All Claims: \\\"Application.Claims\\\"\",\"fields\":{\"contracts\":[1,2,3,4],\"allclaims\":\"\",\"ddsource\":\"\",\"RecordId\":null,\"UserId\":null,\"TransactionId\":null,\"env\":\"UAT\",\"\":\"\",\"\":\"\"}}"

Answer 1

Your log message is not valid JSON, since it contains a comma in line "env": "UAT",. The Fluentd log writes out two more empty fields "":"" as part of your record.
To parse time fields, you have to tell Fluentd the name of the time_key, in your case with time_key: #timestamp. You can use %iso8601 as time_format. For details see the Fluentd documentation on time parameters.