I have installed td-agent and I am trying to upload data to Elasticsearch. Below is the td-agent.conf file:
<source>
#type tail
path /home/rocket/PycharmProjects/EFK/log.json
pos_file /home/rocket/PycharmProjects/EFK/log.json.pos
format json
time_format %Y-%m-%d %H:%M:%S
tag log
</source>
<match *log*>
#type elasticsearch
host 35.171.30.19
port 9200
user elastic
password XXXXXX
index_name test
</match>
Below is the error I am getting:
2023-01-30 14:13:47 +0000 [info]: starting fluentd-1.15.3 pid=5105 ruby="2.7.6"
2023-01-30 14:13:47 +0000 [info]: spawn command to main: cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2023-01-30 14:13:47 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-01-30 14:13:48 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-01-30 14:13:48 +0000 [info]: adding match pattern="*log*" type="elasticsearch"
2023-01-30 14:13:48 +0000 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.4.0 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server."
2023-01-30 14:13:48 +0000 [error]: Worker 0 finished unexpectedly with status 2
2023-01-30 14:13:48 +0000 [info]: Received graceful stop
2023-01-30 14:13:49 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-01-30 14:13:49 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-calyptia-monitoring' version '0.1.3'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-metrics-cmetrics' version '0.1.2'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.8'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-td' version '1.2.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-01-30 14:13:49 +0000 [info]: using configuration file: <ROOT>
<source>
#type tail
path "/home/rocket/PycharmProjects/EFK/log.json"
pos_file "/home/rocket/PycharmProjects/EFK/log.json.pos"
format json
time_format %Y-%m-%d %H:%M:%S
tag "log"
<parse>
time_format %Y-%m-%d %H:%M:%S
#type json
unmatched_lines
time_type string
</parse>
</source>
<match *log*>
#type elasticsearch
host "35.179.40.29"
port 9200
user "elastic"
password xxxxxx
index_name "test"
</match>
</ROOT>
2023-01-30 14:13:49 +0000 [info]: starting fluentd-1.15.3 pid=5116 ruby="2.7.6"
2023-01-30 14:13:49 +0000 [info]: spawn command to main: cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2023-01-30 14:13:49 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-01-30 14:13:49 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-01-30 14:13:49 +0000 [info]: adding match pattern="*log*" type="elasticsearch"
2023-01-30 14:13:50 +0000 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.4.0 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server."
2023-01-30 14:13:50 +0000 [error]: Worker 0 finished unexpectedly with status 2
2023-01-30 14:13:50 +0000 [info]: Received graceful stop
So the error says error_class=Fluent::ConfigError error="Using Elasticsearch client 8.4.0 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server."
So it's an issue between the Elastic plugin version and Elasticsearch server version. But I am unable to find anywhere which version is supported and how to install it.
Below is how I have installed td-agent in Ubuntu 18.04.
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent4.sh | sh
Related
I have downloaded the elasticsearch & kibana on my Ubuntu18.04 machine. Both of them are running fine and I can access them. Below is the elasticsearch details:
{
"name" : "TX-G1-000",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "AobC_iiNSyyNftYl3pUJ7w",
"version" : {
"number" : "7.14.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "66b55ebfa59c92c15db3f69a335d500018b3331e",
"build_date" : "2021-08-26T09:01:05.390870785Z",
"build_snapshot" : false,
"lucene_version" : "8.9.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
I have also installed Fluentd and its service td-agent is running fine.
● td-agent.service - td-agent: Fluentd based data collector for Treasure Data
Loaded: loaded (/lib/systemd/system/td-agent.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-09-20 21:18:12 IST; 7min ago
Docs: https://docs.treasuredata.com/display/public/PD/About+Treasure+Data%27s+Server-Side+Agent
Process: 5486 ExecStop=/bin/kill -TERM ${MAINPID} (code=exited, status=0/SUCCESS)
Process: 5491 ExecStart=/opt/td-agent/bin/fluentd --log $TD_AGENT_LOG_FILE --daemon /var/run/td-agent/td-agent.pid $TD_AGENT_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 5498 (fluentd)
Tasks: 12 (limit: 4915)
CGroup: /system.slice/td-agent.service
├─5498 /opt/td-agent/bin/ruby /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid
└─5501 /opt/td-agent/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid --under-supervisor
Sep 20 21:18:11 TX-G1-000 systemd[1]: Starting td-agent: Fluentd based data collector for Treasure Data...
Sep 20 21:18:12 TX-G1-000 systemd[1]: Started td-agent: Fluentd based data collector for Treasure Data.
Below is my td-agent.conf file:
<source>
#type tail
path /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json
pos_file /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json.pos
format json
time_format %Y-%m-%d %H:%M:%S
tag health01
</source>
<source>
#type tail
path /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json
pos_file /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json.pos
format json
time_format %Y-%m-%d %H:%M:%S
tag cycle01
</source>
<match health*>
#type elasticsearch
hosts http://localhost:9200/
index_name health_skl_device
type_name health
</match>
<match cycle*>
#type elasticsearch
hosts http://localhost:9200/
index_name cycle_skl_device
type_name cycle
</match>
When running the td-agent, below are its logs:
2021-09-20 21:18:12 +0530 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.5'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-kafka' version '0.16.3'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-prometheus' version '2.0.1'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-s3' version '1.6.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-td' version '1.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluentd' version '1.13.3'
2021-09-20 21:18:12 +0530 [info]: using configuration file: <ROOT>
<source>
#type tail
path "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json"
pos_file "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json.pos"
format json
time_format %Y-%m-%d %H:%M:%S
tag "health01"
<parse>
time_format %Y-%m-%d %H:%M:%S
#type json
unmatched_lines
time_type string
</parse>
</source>
<source>
#type tail
path "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json"
pos_file "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json.pos"
format json
time_format %Y-%m-%d %H:%M:%S
tag "cycle01"
<parse>
time_format %Y-%m-%d %H:%M:%S
#type json
unmatched_lines
time_type string
</parse>
</source>
<match health*>
#type elasticsearch
hosts "http://localhost:9200/"
index_name "health_skl_device"
type_name "health"
</match>
<match cycle*>
#type elasticsearch
hosts "http://localhost:9200/"
index_name "cycle_skl_device"
type_name "cycle"
</match>
</ROOT>
2021-09-20 21:18:12 +0530 [info]: starting fluentd-1.13.3 pid=5491 ruby="2.7.4"
2021-09-20 21:18:12 +0530 [info]: spawn command to main: cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2021-09-20 21:18:13 +0530 [info]: adding match pattern="health*" type="elasticsearch"
2021-09-20 21:18:13 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2021-09-20 21:18:13 +0530 [info]: adding match pattern="cycle*" type="elasticsearch"
2021-09-20 21:18:13 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2021-09-20 21:18:13 +0530 [info]: adding source type="tail"
2021-09-20 21:18:13 +0530 [info]: adding source type="tail"
2021-09-20 21:18:13 +0530 [info]: #0 starting fluentd worker pid=5501 ppid=5498 worker=0
2021-09-20 21:18:13 +0530 [info]: #0 following tail of /home/thingtrax/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json
2021-09-20 21:18:13 +0530 [info]: #0 following tail of /home/thingtrax/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json
2021-09-20 21:18:13 +0530 [info]: #0 fluentd worker is now running worker=0
I do not see any error logs but not sure why its not able to upload data. I try to create index pattern, it doesnt matches on Kibana. Can anyone please help me in debugging this issue. Thanks
Logs after adding debug
2021-09-23 07:41:50 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'index_name health_skl_device' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: health_skl_device' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: Need substitution: false
2021-09-23 07:41:50 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'index_name cycle_skl_device' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: cycle_skl_device' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: Need substitution: false
2021-09-23 07:41:50 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: No fluent logger for internal event
I think you have incorrect match tags. Nowhere in documentation does it mention that asterisks can be used that way, they should either take a place of a whole tag part or be used inside a regular expression. According to this section, Fluentd accepts all non-period characters as a part of a tag. So in fact health* is a valid name for a tag, fluentd expects exact matches of that string.
You should try using /health.*/ and /cycle.*/ instead.
Better yet, you can go the intended way, change the tag names to health.01 and cycle.01 and use health.** and cycle.** for matching.
i tried to set up an EFK Stack. While E+K work fine in the default namespace, the Fluentd container can't connect to elasticsearch.
kubectl get services -n default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch-master ClusterIP 10.43.40.136 <none> 9200/TCP,9300/TCP 92m
elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 92m
kibana-kibana ClusterIP 10.43.152.189 <none> 5601/TCP 74m
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 14d
I've installed fluentd from this repo and changed the url to elasticsearch
https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch-rbac.yaml
kubectl -n kube-system get pods | grep fluentd
fluentd-4fd2s 1/1 Running 0 51m
fluentd-7t2v5 1/1 Running 0 49m
fluentd-dfnfg 1/1 Running 0 50m
fluentd-lvrsv 1/1 Running 0 48m
fluentd-rv4td 1/1 Running 0 50m
but the log is telling me:
2021-07-23 21:38:59 +0000 [info]: starting fluentd-1.13.2 pid=7 ruby="2.6.8"
2021-07-23 21:38:59 +0000 [info]: spawn command to main: cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "-r", "/fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]
2021-07-23 21:39:01 +0000 [info]: adding match in #FLUENT_LOG pattern="fluent.**" type="null"
2021-07-23 21:39:01 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2021-07-23 21:39:01 +0000 [warn]: #0 [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2021-07-23 21:39:01 +0000 [info]: adding match pattern="**" type="elasticsearch"
2021-07-23 21:39:09 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. connect_write timeout reached
2021-07-23 21:39:09 +0000 [warn]: #0 [out_es] Remaining retry: 14. Retry to communicate after 2 second(s).
2021-07-23 21:39:18 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. connect_write timeout reached
2021-07-23 21:39:18 +0000 [warn]: #0 [out_es] Remaining retry: 13. Retry to communicate after 4 second(s).
2021-07-23 21:39:31 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. connect_write timeout reached
2021-07-23 21:39:31 +0000 [warn]: #0 [out_es] Remaining retry: 12. Retry to communicate after 8 second(s).
2021-07-23 21:39:52 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. connect_write timeout reached
2021-07-23 21:39:52 +0000 [warn]: #0 [out_es] Remaining retry: 11. Retry to communicate after 16 second(s).
2021-07-23 21:40:29 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. connect_write timeout reached
2021-07-23 21:40:29 +0000 [warn]: #0 [out_es] Remaining retry: 10. Retry to communicate after 32 second(s).
2021-07-23 21:41:38 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. connect_write timeout reached
I installed dig and it resolved the service:
root#fluentd-dfnfg:/home/fluent# nslookup elasticsearch-master.default.svc.cluster.local
Server: 10.43.0.10
Address: 10.43.0.10#53
Name: elasticsearch-master.default.svc.cluster.local
Address: 10.43.40.136
I'm out of ideas.
PS: Im using a hardened RKE2. (https://github.com/rancherfederal/rke2-ansible)
There are 0 error messages when bringing up the Fluentd docker container, so it makes it hard to debug.
curl http://elasticsearch:9200/_cat/indices from the fluentd-container shows indices, but however doesn't show the fluentd-index.
docker logs 7b
2018-06-29 13:56:41 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2018-06-29 13:56:41 +0000 [info]: starting fluentd-0.12.19
2018-06-29 13:56:41 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.4.0'
2018-06-29 13:56:41 +0000 [info]: gem 'fluent-plugin-rename-key' version '0.1.3'
2018-06-29 13:56:41 +0000 [info]: gem 'fluentd' version '0.12.19'
2018-06-29 13:56:41 +0000 [info]: gem 'fluentd' version '0.10.61'
2018-06-29 13:56:41 +0000 [info]: adding filter pattern="**" type="record_transformer"
2018-06-29 13:56:41 +0000 [info]: adding match pattern="docker.*" type="rename_key"
2018-06-29 13:56:41 +0000 [info]: Added rename key rule: rename_rule1 {:key_regexp=>/^log$/, :new_key=>"message"}
2018-06-29 13:56:41 +0000 [info]: adding match pattern="**" type="elasticsearch"
2018-06-29 13:56:41 +0000 [info]: adding source type="forward"
2018-06-29 13:56:41 +0000 [info]: adding source type="monitor_agent"
2018-06-29 13:56:41 +0000 [info]: using configuration file: <ROOT>
<source>
#type forward
</source>
<source>
#type monitor_agent
bind 0.0.0.0
port 24220
</source>
<filter **>
type record_transformer
<record>
node /
role app
environment dev
tenant xxx
tag ${tag}
</record>
</filter>
<match docker.*>
type rename_key
rename_rule1 ^log$ message
append_tag message
</match>
<match **>
type elasticsearch
host elasticsearch
port 9200
index_name fluentd
type_name fluentd
include_tag_key true
logstash_format true
</match>
</ROOT>
2018-06-29 13:56:41 +0000 [info]: listening fluent socket on 0.0.0.0:24224
...
2018-06-29 14:16:38 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2018-06-29 14:20:56 +0000 [warn]: incoming chunk is broken: source="host: 172.18.42.1, addr: 172.18.42.1, port: 48704" msg=49
2018-06-29 14:20:56 +0000 [warn]: incoming chunk is broken: source="host: 172.18.42.1, addr: 172.18.42.1, port: 48704" msg=50
2018-06-29 14:20:56 +0000 [warn]: incoming chunk is broken: source="host: 172.18.42.1, addr: 172.18.42.1, port: 48704" msg=51
... many repeats
2018-07-01 06:21:52 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-07-01 08:39:07 +0000 error_class="MultiJson::ParseError" error="Yajl::ParseError" plugin_id="object:2ac58fef2200"
2018-07-01 06:21:52 +0000 [warn]: suppressed same stacktrace
2018-07-01 08:39:07 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-07-01 13:02:17 +0000 error_class="MultiJson::ParseError" error="Yajl::ParseError" plugin_id="object:2ac58fef2200"
2018-07-01 08:39:07 +0000 [warn]: suppressed same stacktrace
2018-07-01 13:02:17 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-07-01 21:04:48 +0000 error_class="MultiJson::ParseError" error="Yajl::ParseError" plugin_id="object:2ac58fef2200"
2018-07-01 13:02:17 +0000 [warn]: suppressed same stacktrace
2018-07-01 21:04:48 +0000 [warn]: failed to flush the buffer. error_class="MultiJson::ParseError" error="Yajl::ParseError" plugin_id="object:2ac58fef2200"
2018-07-01 21:04:48 +0000 [warn]: retry count exceededs limit.
2018-07-01 21:04:48 +0000 [warn]: suppressed same stacktrace
2018-07-01 21:04:48 +0000 [error]: throwing away old logs.
I am able to successfully insert data in a test-index in ElasticSearch by curling. How do I troubleshoot where fluentd fails?
I am unable to comment so adding couple of observations here.
Documentation says to use #type elasticsearch. Also if both elastic and fluentd are running as docker containers, please make sure to run them with proper network so they can talk to each other(try IPs first maybe).
Also, what is your Dockerfile looks like so we can pass verbosity to fluentd command?.
I successfully used this configuration for fluentd+elastisearch:
<source>
#type forward
#label #mainstream
bind 0.0.0.0
port 24224
</source>
<label #mainstream>
<match **>
#type copy
<store>
#type elasticsearch
host elasticsearch
port 9200
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key #log_name
<buffer>
flush_mode interval
flush_interval 1s
retry_type exponential_backoff
flush_thread_count 2
retry_forever true
retry_max_interval 30
chunk_limit_size 2M
queue_limit_length 8
overflow_action block
</buffer>
</store>
</match>
</label>
For debugging you could use tcpdump:
sudo tcpdump -i eth0 tcp port 24224 -X -s 0 -nn
**note: removed the leading slash form the first source tag
Accordingly to the fluentd documentation you can use different log level, aka order of verbosity
https://docs.fluentd.org/deployment/logging
my english is not good ,sorry ...
these days ,i always test td-agent on AWS EC2(Amazon Linux AMI release 2014.03).
input log from nginx and output log to S3 is OK, this very good!
then i want praser log to json ,when i made the source format "nginx" ,it says
" 2015-03-31 00:12:24 +0800 [warn]: pattern not match: "95.218.58.120 - [2015-03-30T16:11:40+08:00] \"GET /provider.txt HTTP/1.1\" 200 726 \"-\" \"FreePP 2.9.5.159 (iPhone; iPhone OS 8.1.3; ar_SA) [65075070]\" "
i think mybe my nginx log is not default.then,i made regexp for this ,
format /^(?<remote>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: (?<path>[^\"]*)+\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"?$/
and my log is:
178.152.203.39 - [2015-03-31T00:18:10+08:00] "GET /provider.txt HTTP/1.1" 200 726 "-" "Dalvik/1.6.0 (Linux; U; Android 4.3; GT-I9300 Build/JSS15J)"
restart td-agent ,td-agent logs seems OK.but no output found ,and my cpu 100%,any one can help me ? many thanks !!!
top - 00:26:17 up 4 days, 1:10, 2 users, load average: 0.20, 0.35, 0.57
Tasks: 82 total, 3 running, 79 sleeping, 0 stopped, 0 zombie
Cpu(s): 48.3%us, 2.2%sy, 0.0%ni, 47.5%id, 0.0%wa, 0.3%hi, 1.7%si, 0.0%st
Mem: 7401596k total, 6612324k used, 789272k free, 151200k buffers
Swap: 0k total, 0k used, 0k free, 5911168k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4724 td-agent 20 0 222m 23m 2800 R 96.5 0.3 0:11.70 ruby
473 nginx 20 0 119m 12m 1668 S 3.7 0.2 5:58.61 nginx
2015-03-31 00:26:04 +0800 [info]: process finished code=0
2015-03-31 00:26:05 +0800 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2015-03-31 00:26:05 +0800 [info]: starting fluentd-0.10.61
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-mongo' version '0.7.6'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-s3' version '0.5.6'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-td' version '0.10.26'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.0'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-03-31 00:26:05 +0800 [info]: gem 'fluentd' version '0.10.61'
2015-03-31 00:26:05 +0800 [info]: using configuration file: <ROOT>
<source>
type tail
format /^(?<remote>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: (?<path>[^\"]*)+\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"?$/
time_format %d/%b/%YT%H:%M:%S%z
path /var/log/nginx/provider.access.log
log_level debug
pos_file /var/log/td-agent/access.log.pos
tag s3.nginx.access
</source>
<match s3.nginx.access>
type file
path /var/log/td-agent/nginxAccess
time_slice_format %Y%m%d
time_slice_wait 10m
format single_value
</match>
</ROOT>
2015-03-31 00:26:05 +0800 [info]: adding source type="tail"
2015-03-31 00:26:05 +0800 [info]: adding match pattern="s3.nginx.access" type="file"
2015-03-31 00:26:05 +0800 [info]: following tail of /var/log/nginx/provider.access.log
restart td-agent ,td-agent logs seems OK.but no output found ,
From you configuration, you use single_value in file output.
single_value is almost for none parser.
If you want to use single_value, you should use none in tail input.
See format section in this article: http://docs.fluentd.org/articles/out_file#format
the regexp has some wrong，
i modify it
format /^(?[^ ]) (?[^ ]) [(?[^]])] "(?\S+) (?[^"])" (?[^ ]) (?[^ ]) "(?[^\"])" "(?[^\"])"/
the pb is sloved. thanks !
I am setting up fluentd and elasticsearch on a local VM in order to try the fluentd and ES stack.
OS: centos (recent)
[root#localhost data]# cat /etc/redhat-release
CentOS release 6.5 (Final)
I am elasticsearch up and running on localhost (I used it with logstash with no issue)
[root#localhost data]# curl -X GET http://localhost:9200/
{
"status" : 200,
"name" : "Simon Williams",
"version" : {
"number" : "1.2.1",
"build_hash" : "6c95b759f9e7ef0f8e17f77d850da43ce8a4b364",
"build_timestamp" : "2014-06-03T15:02:52Z",
"build_snapshot" : false,
"lucene_version" : "4.8"
},
"tagline" : "You Know, for Search"
}
I have installed td-agent following the installation notes from fluentd website.
I am using that configuration file:
<source>
type tail
path /tmp/data/log
pos_file /tmp/data/log.pos
format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[\
^\"]*)" "(?<agent>[^\"]*)")?/
time_format %d/%b/%Y:%H:%M:%S %z
tag front.nginx.access
</source>
<match front.nginx.access>
type elasticsearch
host localhost
port 9200
index_name fluentd
type_name nginx
include_tag_key
# buffering
buffer_type file
buffer_path /tmp/fluentd/buffer/
flush_interval 10s
buffer_chunk_limit 16m
buffer_queue_limit 4096
retry_wait 15s
</match>
Here is the start-up log:
2014-07-24 13:39:58 +0200 [info]: starting fluentd-0.10.50
2014-07-24 13:39:58 +0200 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-mixin-config-placeholders' version '0.2.4'
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-plugin-elasticsearch' version '0.3.1'
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-plugin-flume' version '0.1.1'
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-plugin-mongo' version '0.7.3'
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-plugin-parser' version '0.3.4'
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-plugin-s3' version '0.4.0'
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-plugin-scribe' version '0.10.10'
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-plugin-td' version '0.10.20'
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-plugin-td-monitoring' version '0.1.2'
2014-07-24 13:39:58 +0200 [info]: gem 'fluent-plugin-webhdfs' version '0.2.2'
2014-07-24 13:39:58 +0200 [info]: gem 'fluentd' version '0.10.50'
2014-07-24 13:39:58 +0200 [info]: using configuration file: <ROOT>
<source>
type tail
path /tmp/data/log
pos_file /tmp/data/log.pos
format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?/
time_format %d/%b/%Y:%H:%M:%S %z
tag front.nginx.access
</source>
<match front.nginx.access>
type elasticsearch
host localhost
port 9200
index_name fluentd
type_name nginx
include_tag_key
buffer_type file
buffer_path /tmp/fluentd/buffer/
flush_interval 10s
buffer_chunk_limit 16m
buffer_queue_limit 4096
retry_wait 15s
</match>
</ROOT>
2014-07-24 13:39:58 +0200 [info]: adding source type="tail"
2014-07-24 13:39:58 +0200 [info]: adding match pattern="front.nginx.access" type="elasticsearch"
2014-07-24 13:39:58 +0200 [info]: following tail of /tmp/data/log
I get that error:
2014-07-24 13:40:00 +0200 [warn]: temporarily failed to flush the buffer. next_retry=2014-07-24 13:40:13 +0200 error_class="Elasticsearch::Transport::Transport::Errors::ServiceUnavailable" error="[503] " instance=70247139359260
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/elasticsearch-transport-0.4.11/lib/elasticsearch/transport/transport/base.rb:132:in `__raise_transport_error'
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/elasticsearch-transport-0.4.11/lib/elasticsearch/transport/transport/base.rb:227:in `perform_request'
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/elasticsearch-transport-0.4.11/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/elasticsearch-transport-0.4.11/lib/elasticsearch/transport/client.rb:92:in `perform_request'
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/elasticsearch-api-0.4.11/lib/elasticsearch/api/actions/ping.rb:19:in `ping'
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.3.1/lib/fluent/plugin/out_elasticsearch.rb:46:in `client'
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.3.1/lib/fluent/plugin/out_elasticsearch.rb:103:in `send'
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-elasticsearch-0.3.1/lib/fluent/plugin/out_elasticsearch.rb:98:in `write'
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/buffer.rb:296:in `write_chunk'
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/buffer.rb:276:in `pop'
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/output.rb:310:in `try_flush'
2014-07-24 13:40:00 +0200 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/output.rb:132:in `run'
running tcpdump on port 9200, I get nothing...
tcpdump -x -X -i any 'port 9200'
I've found the problem.
Actually, I had not modified the default cluster name in ES.
Another ES cluster existed on the same network.
The clients used in this cluster where sending packets to my ES cluster with an ancient protocol.
I have corrected all issues by changing the ES cluster name.