failed to publish events: temporary bulk send failure - elasticsearch

Question

When i try to create multiple index in filebeat.yml and output to elasticsearch, i am getting temporary bulk send failure error. This is coming only when i introduce ilm as disable. can anyone help
Below is the filebeat config
filebeat.inputs:
- type: filestream
id: denali
enabled: true
paths:
- /var/log/denali/denali.log
parsers:
- multiline:
type: pattern
pattern: '^(\d{4}-\d{2}-\d{2})'
negate: true
match: after
fields:
app_id: denali
- type: filestream
id: freeswitch
enabled: true
paths:
- /var/log/freeswitch/freeswitch.log
parsers:
- multiline:
type: pattern
pattern: '^((\d|[a-z]|-)+ \d{4}-\d{2}-\d{2}|\d{4}-\d{2}-\d{2})'
negate: true
match: after
fields:
app_id: freeswitch
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.enabled: true
setup.ilm.enabled: false
setup.template.overwrite: true
setup.template.name: "index-%{[agent.version]}"
setup.template.pattern: "index-%{[agent.version]}-*"
output.elasticsearch:
hosts: ["ip:port"]
index: "index-%{[agent.version]}-%{[fields.app_id]:other}-%{+yyyy.MM.dd}"
protocol: "http"
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
- drop_fields:
fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.name", "agent.type", "agent.version", "cloud.account.id", "cloud.provider", "cloud.service.name", "container.id", "container.image.name", "container.labels.COMMIT", "container.labels.PIPELINE_URL", "container.labels.PROJECT_NAME", "container.labels.PROJECT_URL", "container.labels.SOURCE_BRANCH", "container.labels.TimeStamp", "container.labels.RELEASEARTIFACT_VERSION", "container.labels.com_docker_compose_config-hash", "container.labels.com_docker_compose_container-number", "container.labels.com_docker_compose_oneoff", "container.labels.com_docker_compose_project", "container.labels.com_docker_compose_project_config_files", "container.labels.com_docker_compose_project_working_dir", "container.labels.com_docker_compose_service", "container.labels.com_docker_compose_version", "ecs.version", "host.architecture", "host.containerized", "host.id", "host.mac", "host.os.codename", "host.os.family", "host.os.kernel", "host.os.name", "host.os.platform", "host.os.type", "host.os.version", "log.offset"]

Answer 1

#Ramanichandran can you please provide the error logs from filebeat? Also, do you see any errors on ES logs when filebeat is trying to send logs for ingestion?
I don't believe it's due to creation of multiple indices since you are essentially creating only 3 indices. I have configured filebeat to create about 15 indices in my use case and it works just fine where my config is similar to yours with ILM disabled.
It's worth trying to set following attributes for output.elasticsearch:
bulk_max_size: 25
bulk_max_bytes: 104857600