ElasticSearch 8 errors with Action/metadata line [1] contains an unknown parameter [_type] status:400 - elasticsearch

Question

I am trying to setup EFK (ElasticSearch 8, FluentD and Kibana) stack on K8S cluster (on-premises)
I followed this link to install elasticsearch and installed it using helm charts and followed this link to install fluentd
Output of fluentd and elasticsearch pods
[root#ctrl01 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 1/1 Running 0 136m
[root#ctrl01 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
fluentd-cnb7p 1/1 Running 0 107m
fluentd-dbxjk 1/1 Running 0 107m
However, elasticsearch log was piled up with the following warning messages
2021-10-18 12:13:12 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2021-10-18 12:13:42 +0000 error_class="Elasticsearch::Transport::Transport::Errors::BadRequest" error="[400] {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}" plugin_id="out_es"
2021-10-18 12:13:12 +0000 [warn]: suppressed same stacktrace
Conf file (tailored output)
2021-10-18 12:09:10 +0000 [info]: using configuration file: <ROOT>
<match fluent.**>
#type null
</match>
<source>
#type tail
#id in_tail_container_logs
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
read_from_head true
format json
time_format %Y-%m-%dT%H:%M:%S.%NZ
</source>
<source>
#type tail
#id in_tail_minion
path /var/log/salt/minion
pos_file /var/log/fluentd-salt.pos
tag salt
format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
time_format %Y-%m-%d %H:%M:%S
</source>
I am not sure which 'type' field it refers to. I am unable to find an example of ElasticSearch 8 for match and source directives to compare
It seems type field is not supported from ES 8 onwards but I am not sure on that. Kindly let me know the reason for the error

Answer 1

I faced similar errors when I tried to use elasticsearch 8.2.3 with fluentBit 1.9.5. I could see elastic was sending logs but could not see any data in kibana webpage due to which could not create indices and saw the above error in fluent-bit pod logs. I followed this github issue and added Suppress_Type_Name On under outputs: section in my fluent-bit helm chart values.yaml file and it worked fine after that.
[OUTPUT]
Name es
Match *
Host {{ .Values.global.backend.es.host }}
Port {{ .Values.global.backend.es.port }}
Logstash_Format Off
Retry_Limit False
Type _doc
Time_Key #timestamp
Replace_Dots On
Suppress_Type_Name On
Index {{ .Values.global.backend.es.index }}
{{ .Values.extraEntries.output }}

Answer 2

I was working on the same issue for a few days and I found a solution but just a workaround, not the optimal solution.
If you set TypeName as null for ElasticsearchSinkOptions, you don't face this issue.
Unfortunately, you can't set it from appsettings.json. At least I couldn't find a way.
In background, Serilog.Sinks.ElasticSearch library use this property as _type in HTTP header. But the '_type' header,
as leandrojmp pointed out in the comment, it is no longer available in version 8.2 of ElasticSearch.