Can't communicate with Elasticsearch endpoint from FluentBit - elasticsearch

Question

Problem:
Connection by AWS Elasticsearch endpoint is refused when pushing Kubernetes logs through a fluentBit forwarder.
Here is the fluentBit set up:
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
labels:
k8s-app: fluent-bit
data:
# Configuration files: server, input, filters and output
# ======================================================
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
#INCLUDE input-kubernetes.conf
#INCLUDE filter-kubernetes.conf
#INCLUDE output-elasticsearch.conf
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
filter-kubernetes.conf: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
Merge_Log On
K8S-Logging.Parser On
output-elasticsearch.conf: |
[OUTPUT]
Name es
Match *
Host ${FLUENT_ELASTICSEARCH_HOST}
Port ${FLUENT_ELASTICSEARCH_PORT}
Logstash_Format On
Retry_Limit False
tls Off
tls.verify Off
#----
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: logging
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
template:
metadata:
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2020"
prometheus.io/path: /api/v1/metrics/prometheus
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:1.5
imagePullPolicy: Always
ports:
- containerPort: 2020
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "https://vpc-tf-test2-xyzxyzxyzxyz.eu-west-2.es.amazonaws.com"
- name: FLUENT_ELASTICSEARCH_PORT
value: "443"
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: fluent-bit-config
configMap:
name: fluent-bit-config
serviceAccountName: fluent-bit
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
The fluentBit forwarder logs show this:
[2021/02/01 09:09:11] [error] [io] connection #46 failed to: https://vpc-tf-test2-xyzxyzxyz.eu-west-2.es.amazonaws.com:443
[2021/02/01 09:09:11] [ warn] [engine] failed to flush chunk '1-1611849613.623397482.flb', retry in 1521 seconds: task_id=1980, input=tail.0 > output=es.0
[2021/02/01 09:09:11] [ warn] [engine] failed to flush chunk '1-1611849347.548817423.flb', retry in 1806 seconds: task_id=1623, input=tail.0 > output=es.0
[2021/02/01 09:09:11] [ warn] [engine] failed to flush chunk '1-1611849095.485002520.flb', retry in 1286 seconds: task_id=1284, input=tail.0 > output=es.0
[2021/02/01 09:09:13] [ warn] net_tcp_fd_connect: getaddrinfo(host='https://vpc-tf-test2-xyzxyzxyzxyz.eu-west-2.es.amazonaws.com'): Name or service not known
[2021/02/01 09:09:13] [error] [io] connection #46 failed to: https://vpc-tf-test2-xyzxyzxyz.eu-west-2.es.amazonaws.com:443
[2021/02/01 09:09:13] [ warn] [engine] failed to flush chunk '1-1611849450.549250742.flb', retry in 799 seconds: task_id=1766, input=tail.0 > output=es.0
I am trying to trace where the access is getting blocked.
Access to the ES endpoint is protected by Security Group with this inbound rules:
Type: All traffic
Protocol: All
Port range: All
Source: sg-xyzxyzxyz (eks-cluster-sg-vrs2-eks-dev-xyzxyzyxz)

Answer 1

Change FLUENT_ELASTICSEARCH_HOST value to vpc-tf-test2-xyzxyzxyzxyz.eu-west-2.es.amazonaws.com.