Why are my parser types not picked up in my elastic index? - elasticsearch

Question

I'm using fluent-bit to forward logs to an elastic db. All my fields are being indexed in elastic under the default string type but I want some fields indexed as numbers.
I've attempted to set the types in my fluent-bit config by adding a types entry to both the docker parser and the json parser (not sure which one is being used here, these are container logs from a k8s cluster):
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
Types my_float_field:float my_integer_field:integer
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
Types my_float_field:float my_integer_field:integer
But these fields continue to appear as string types in fresh elastic indexes under the ids log_processed.my_float_field and log_processed.my_integer_field. I'm sure I'm doing something obviously wrong but I can see what.
Any pointers would be greatly appreciated.

Answer 1

Use Elasticsearch index templates.
AFAIK the JSON parser plugin doesn't support "Type" parameter. It keeps the original JSON data types, so if my_float_field and my_integer_field contain quoted values, JSON parser will interpret them as strings as well. See this example from the docs:
A simple configuration that can be found in the default parsers
configuration file, is the entry to parse Docker log files (when the
tail input plugin is used):
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S %z
The following log entry is a valid content for the parser defined
above:
{"key1": 12345, "key2": "abc", "time": "2006-07-28T13:22:04Z"}
After processing, it internal representation will be:
[1154103724, {"key1"=>12345, "key2"=>"abc"}]
If you are using Logstash format in Elasticsearch output plugin, you can define an Elasticsearch index template containing the desired type mapping. The template will be applied to each newly created index (not to existing indices). The format changed in Elasticsearch 7, so be sure to check the correct documentation version. For the 7.7 version:
PUT _template/template_1
{
"index_patterns": ["fluent-bit-*"],
"settings": {
"number_of_shards": 1
},
"mappings": {
"properties": {
"my_float_field": {
"type": "float"
},
"my_integer_field": {
"type": "integer"
}
}
}
}