Fluentd not able to upload logs to Elasticsearch - elasticsearch

Question

I have downloaded the elasticsearch & kibana on my Ubuntu18.04 machine. Both of them are running fine and I can access them. Below is the elasticsearch details:
{
"name" : "TX-G1-000",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "AobC_iiNSyyNftYl3pUJ7w",
"version" : {
"number" : "7.14.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "66b55ebfa59c92c15db3f69a335d500018b3331e",
"build_date" : "2021-08-26T09:01:05.390870785Z",
"build_snapshot" : false,
"lucene_version" : "8.9.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
I have also installed Fluentd and its service td-agent is running fine.
● td-agent.service - td-agent: Fluentd based data collector for Treasure Data
Loaded: loaded (/lib/systemd/system/td-agent.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-09-20 21:18:12 IST; 7min ago
Docs: https://docs.treasuredata.com/display/public/PD/About+Treasure+Data%27s+Server-Side+Agent
Process: 5486 ExecStop=/bin/kill -TERM ${MAINPID} (code=exited, status=0/SUCCESS)
Process: 5491 ExecStart=/opt/td-agent/bin/fluentd --log $TD_AGENT_LOG_FILE --daemon /var/run/td-agent/td-agent.pid $TD_AGENT_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 5498 (fluentd)
Tasks: 12 (limit: 4915)
CGroup: /system.slice/td-agent.service
├─5498 /opt/td-agent/bin/ruby /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid
└─5501 /opt/td-agent/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid --under-supervisor
Sep 20 21:18:11 TX-G1-000 systemd[1]: Starting td-agent: Fluentd based data collector for Treasure Data...
Sep 20 21:18:12 TX-G1-000 systemd[1]: Started td-agent: Fluentd based data collector for Treasure Data.
Below is my td-agent.conf file:
<source>
#type tail
path /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json
pos_file /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json.pos
format json
time_format %Y-%m-%d %H:%M:%S
tag health01
</source>
<source>
#type tail
path /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json
pos_file /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json.pos
format json
time_format %Y-%m-%d %H:%M:%S
tag cycle01
</source>
<match health*>
#type elasticsearch
hosts http://localhost:9200/
index_name health_skl_device
type_name health
</match>
<match cycle*>
#type elasticsearch
hosts http://localhost:9200/
index_name cycle_skl_device
type_name cycle
</match>
When running the td-agent, below are its logs:
2021-09-20 21:18:12 +0530 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.5'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-kafka' version '0.16.3'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-prometheus' version '2.0.1'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-s3' version '1.6.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-td' version '1.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluentd' version '1.13.3'
2021-09-20 21:18:12 +0530 [info]: using configuration file: <ROOT>
<source>
#type tail
path "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json"
pos_file "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json.pos"
format json
time_format %Y-%m-%d %H:%M:%S
tag "health01"
<parse>
time_format %Y-%m-%d %H:%M:%S
#type json
unmatched_lines
time_type string
</parse>
</source>
<source>
#type tail
path "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json"
pos_file "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json.pos"
format json
time_format %Y-%m-%d %H:%M:%S
tag "cycle01"
<parse>
time_format %Y-%m-%d %H:%M:%S
#type json
unmatched_lines
time_type string
</parse>
</source>
<match health*>
#type elasticsearch
hosts "http://localhost:9200/"
index_name "health_skl_device"
type_name "health"
</match>
<match cycle*>
#type elasticsearch
hosts "http://localhost:9200/"
index_name "cycle_skl_device"
type_name "cycle"
</match>
</ROOT>
2021-09-20 21:18:12 +0530 [info]: starting fluentd-1.13.3 pid=5491 ruby="2.7.4"
2021-09-20 21:18:12 +0530 [info]: spawn command to main: cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2021-09-20 21:18:13 +0530 [info]: adding match pattern="health*" type="elasticsearch"
2021-09-20 21:18:13 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2021-09-20 21:18:13 +0530 [info]: adding match pattern="cycle*" type="elasticsearch"
2021-09-20 21:18:13 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2021-09-20 21:18:13 +0530 [info]: adding source type="tail"
2021-09-20 21:18:13 +0530 [info]: adding source type="tail"
2021-09-20 21:18:13 +0530 [info]: #0 starting fluentd worker pid=5501 ppid=5498 worker=0
2021-09-20 21:18:13 +0530 [info]: #0 following tail of /home/thingtrax/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json
2021-09-20 21:18:13 +0530 [info]: #0 following tail of /home/thingtrax/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json
2021-09-20 21:18:13 +0530 [info]: #0 fluentd worker is now running worker=0
I do not see any error logs but not sure why its not able to upload data. I try to create index pattern, it doesnt matches on Kibana. Can anyone please help me in debugging this issue. Thanks
Logs after adding debug
2021-09-23 07:41:50 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'index_name health_skl_device' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: health_skl_device' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: Need substitution: false
2021-09-23 07:41:50 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'index_name cycle_skl_device' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: cycle_skl_device' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: Need substitution: false
2021-09-23 07:41:50 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: No fluent logger for internal event

Answer 1

I think you have incorrect match tags. Nowhere in documentation does it mention that asterisks can be used that way, they should either take a place of a whole tag part or be used inside a regular expression. According to this section, Fluentd accepts all non-period characters as a part of a tag. So in fact health* is a valid name for a tag, fluentd expects exact matches of that string.
You should try using /health.*/ and /cycle.*/ instead.
Better yet, you can go the intended way, change the tag names to health.01 and cycle.01 and use health.** and cycle.** for matching.