Filebeat: How to export logs of specific pods - elasticsearch

This is my filebeat config map.
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: $${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
setup.ilm.enabled: false
processors:
- add_cloud_metadata:
- add_host_metadata:
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
This sends logs from every pod to AWS ElasticSearch. How I can restrict it to send logs from specific pods by name and/or by the label?

Related

Any service on my Istio service don't registered at my Jaeger

Istio configuration:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
enableTracing: true
defaultConfig:
tracing:
sampling: 100
zipkin:
address: "jaeger-collector.jaeger-system:9411"
Jaeger chart value:
storage:
type: elasticsearch
elasticsearch:
host: 10.8.60.128
port: 9200
scheme: https
user: elastic
password: some-password
provisionDataStore:
cassandra: false
elasticsearch: false
query:
service:
type: NodePort
cmdlineParams:
es.tls.ca: "/tls/ca-crt.pem"
extraConfigmapMounts:
- name: jaeger-tls
mountPath: /tls
subPath: ""
configMap: jaeger-tls
readOnly: true
collector:
service:
zipkin:
port: 9411
cmdlineParams:
es.tls.ca: "/tls/ca-crt.pem"
extraConfigmapMounts:
- name: jaeger-tls
mountPath: /tls
subPath: ""
configMap: jaeger-tls
readOnly: true
spark:
enabled: true
cmdlineParams:
java.opts: "-Djavax.net.ssl.trustStore=/tls/trust.store -Djavax.net.ssl.trustStorePassword=some-password"
extraConfigmapMounts:
- name: jaeger-tls
mountPath: /tls
subPath: ""
configMap: jaeger-tls
readOnly: true
Jaeger service:
$ kubectl get svc -n jaeger-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jaeger-agent ClusterIP 10.152.183.115 <none> 5775/UDP,6831/UDP,6832/UDP,5778/TCP,14271/TCP 22m
jaeger-collector ClusterIP 10.152.183.50 <none> 14250/TCP,14268/TCP,9411/TCP,14269/TCP 22m
jaeger-query NodePort 10.152.183.46 <none> 80:31976/TCP,16685:30366/TCP,16687:30921/TCP 22m
The service I mean is an example on Istio Documentation: https://istio.io/latest/docs/examples/bookinfo/
What's wrong with my configuration? I check everything is work fine without any trouble.

Minikube: Issue running Kibana

I am currently learning Kubernetes and I am using Minikube on MacOS using Docker Desktop, I am facing issues with running Kibana which seems to be failing to start and to also enable it through my nginx ingress controller.
Regarding Kibana, it doesn't move to ready stage, it seems to be stuck and restarts several times. Everything lives inside the default namespace, except for fluentd that I use a persistent volume and persistent volume claim to access the shared /data/logs folder.
I have added my fluentd, kibana, es and ingress yaml configuration. And also kibana logs below.
Fluentd
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluentd
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: fluentd
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-system
labels:
k8s-app: fluentd-logging
version: v1
spec:
selector:
matchLabels:
k8s-app: fluentd-logging
version: v1
template:
metadata:
labels:
k8s-app: fluentd-logging
version: v1
spec:
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "elasticsearch"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "http"
- name: FLUENT_UID
value: "0"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- mountPath: /var/logs
name: logs
terminationGracePeriodSeconds: 30
volumes:
- name: logs
persistentVolumeClaim:
claimName: chi-kube-pvc
Kibana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana
spec:
selector:
matchLabels:
run: kibana
template:
metadata:
labels:
run: kibana
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:7.14.2
readinessProbe:
httpGet:
path: /kibana
port: 5601
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /kibana
port: 5601
initialDelaySeconds: 15
periodSeconds: 20
env:
- name: XPACK_SECURITY_ENABLED
value: "true"
- name: SERVER_BASEPATH
value: "/kibana"
ports:
- containerPort: 5601
volumeMounts:
- mountPath: /var/logs
name: logs
volumes:
- name: logs
persistentVolumeClaim:
claimName: chi-pvc
---
apiVersion: v1
kind: Service
metadata:
name: kibana
labels:
service: kibana
spec:
type: NodePort
selector:
run: kibana
ports:
- port: 5601
targetPort: 5601
Kibana logs:
{"type":"log","#timestamp":"2021-09-22T09:54:47+00:00","tags":["info","plugins-service"],"pid":1216,"message":"Plugin \"metricsEntities\" is disabled."}
{"type":"log","#timestamp":"2021-09-22T09:54:47+00:00","tags":["warning","config","deprecation"],"pid":1216,"message":"You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning."}
{"type":"log","#timestamp":"2021-09-22T09:54:47+00:00","tags":["warning","config","deprecation"],"pid":1216,"message":"Support for setting server.host to \"0\" in kibana.yml is deprecated and will be removed in Kibana version 8.0.0. Instead use \"0.0.0.0\" to bind to all interfaces."}
{"type":"log","#timestamp":"2021-09-22T09:54:47+00:00","tags":["warning","config","deprecation"],"pid":1216,"message":"plugins.scanDirs is deprecated and is no longer used"}
{"type":"log","#timestamp":"2021-09-22T09:54:47+00:00","tags":["warning","config","deprecation"],"pid":1216,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","#timestamp":"2021-09-22T09:54:47+00:00","tags":["warning","config","deprecation"],"pid":1216,"message":"\"xpack.reporting.roles\" is deprecated. Granting reporting privilege through a \"reporting_user\" role will not be supported starting in 8.0. Please set \"xpack.reporting.roles.enabled\" to \"false\" and grant reporting privileges to users using Kibana application privileges **Management > Security > Roles**."}
{"type":"log","#timestamp":"2021-09-22T09:54:48+00:00","tags":["info","http","server","NotReady"],"pid":1216,"message":"http server running at http://0.0.0.0:5601"}
{"type":"log","#timestamp":"2021-09-22T09:54:48+00:00","tags":["info","plugins-system"],"pid":1216,"message":"Setting up [106] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApiGuard,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,telemetry,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,savedObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presentationUtil,timelion,home,searchprofiler,painlessLab,grokdebugger,graph,visTypeVega,management,watcher,licenseManagement,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,savedObjectsManagement,spaces,security,transform,savedObjectsTagging,lens,reporting,canvas,lists,ingestPipelines,fileUpload,maps,dataVisualizer,encryptedSavedObjects,dataEnhanced,timelines,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,osquery,ml,cases,securitySolution,observability,uptime,infra,monitoring,logstash,console,apmOss,apm]"}
{"type":"log","#timestamp":"2021-09-22T09:54:48+00:00","tags":["info","plugins","taskManager"],"pid":1216,"message":"TaskManager is identified by the Kibana UUID: 4f523c36-da1f-46e2-a071-84ee400bb9e7"}
{"type":"log","#timestamp":"2021-09-22T09:54:48+00:00","tags":["warning","plugins","security","config"],"pid":1216,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","#timestamp":"2021-09-22T09:54:48+00:00","tags":["warning","plugins","security","config"],"pid":1216,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","#timestamp":"2021-09-22T09:54:48+00:00","tags":["warning","plugins","reporting","config"],"pid":1216,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","#timestamp":"2021-09-22T09:54:48+00:00","tags":["warning","plugins","reporting","config"],"pid":1216,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 8.4.2105\n OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","#timestamp":"2021-09-22T09:54:48+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":1216,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","#timestamp":"2021-09-22T09:54:48+00:00","tags":["warning","plugins","actions","actions"],"pid":1216,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","#timestamp":"2021-09-22T09:54:48+00:00","tags":["warning","plugins","alerting","plugins","alerting"],"pid":1216,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","#timestamp":"2021-09-22T09:54:48+00:00","tags":["info","plugins","ruleRegistry"],"pid":1216,"message":"Write is disabled, not installing assets"}
{"type":"log","#timestamp":"2021-09-22T09:54:49+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","#timestamp":"2021-09-22T09:54:49+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"Starting saved objects migrations"}
{"type":"log","#timestamp":"2021-09-22T09:54:49+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 226ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:49+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 192ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:49+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 118ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:49+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 536ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:49+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 86ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:49+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 86ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:50+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 64ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:50+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 112ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:50+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 49ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:50+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 29ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:50+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 106ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:50+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana] Migration completed after 840ms"}
{"type":"log","#timestamp":"2021-09-22T09:54:50+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 104ms."}
{"type":"log","#timestamp":"2021-09-22T09:54:50+00:00","tags":["info","savedobjects-service"],"pid":1216,"message":"[.kibana_task_manager] Migration completed after 869ms"}
{"type":"log","#timestamp":"2021-09-22T09:54:50+00:00","tags":["info","plugins-system"],"pid":1216,"message":"Starting [106] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApiGuard,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,telemetry,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,savedObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presentationUtil,timelion,home,searchprofiler,painlessLab,grokdebugger,graph,visTypeVega,management,watcher,licenseManagement,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,savedObjectsManagement,spaces,security,transform,savedObjectsTagging,lens,reporting,canvas,lists,ingestPipelines,fileUpload,maps,dataVisualizer,encryptedSavedObjects,dataEnhanced,timelines,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,osquery,ml,cases,securitySolution,observability,uptime,infra,monitoring,logstash,console,apmOss,apm]"}
{"type":"log","#timestamp":"2021-09-22T09:54:50+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":1216,"message":"config sourced from: production cluster"}
{"type":"log","#timestamp":"2021-09-22T09:54:51+00:00","tags":["info","http","server","Kibana"],"pid":1216,"message":"http server running at http://0.0.0.0:5601"}
{"type":"log","#timestamp":"2021-09-22T09:54:52+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":1216,"message":"Starting monitoring stats collection"}
{"type":"log","#timestamp":"2021-09-22T09:54:52+00:00","tags":["info","plugins","securitySolution"],"pid":1216,"message":"Dependent plugin setup complete - Starting ManifestTask"}
{"type":"log","#timestamp":"2021-09-22T09:54:52+00:00","tags":["info","status"],"pid":1216,"message":"Kibana is now degraded"}
{"type":"log","#timestamp":"2021-09-22T09:54:52+00:00","tags":["info","plugins","reporting"],"pid":1216,"message":"Browser executable: /usr/share/kibana/x-pack/plugins/reporting/chromium/headless_shell-linux_x64/headless_shell"}
{"type":"log","#timestamp":"2021-09-22T09:54:52+00:00","tags":["warning","plugins","reporting"],"pid":1216,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","#timestamp":"2021-09-22T09:54:55+00:00","tags":["info","status"],"pid":1216,"message":"Kibana is now available (was degraded)"}
{"type":"response","#timestamp":"2021-09-22T09:54:58+00:00","tags":[],"pid":1216,"method":"get","statusCode":404,"req":{"url":"/kibana","method":"get","headers":{"host":"172.17.0.3:5601","user-agent":"kube-probe/1.22","accept":"*/*","connection":"close"},"remoteAddress":"172.17.0.1","userAgent":"kube-probe/1.22"},"res":{"statusCode":404,"responseTime":36,"contentLength":60},"message":"GET /kibana 404 36ms - 60.0B"}
{"type":"response","#timestamp":"2021-09-22T09:55:08+00:00","tags":[],"pid":1216,"method":"get","statusCode":404,"req":{"url":"/kibana","method":"get","headers":{"host":"172.17.0.3:5601","user-agent":"kube-probe/1.22","accept":"*/*","connection":"close"},"remoteAddress":"172.17.0.1","userAgent":"kube-probe/1.22"},"res":{"statusCode":404,"responseTime":25,"contentLength":60},"message":"GET /kibana 404 25ms - 60.0B"}
{"type":"response","#timestamp":"2021-09-22T09:55:08+00:00","tags":[],"pid":1216,"method":"get","statusCode":404,"req":{"url":"/kibana","method":"get","headers":{"host":"172.17.0.3:5601","user-agent":"kube-probe/1.22","accept":"*/*","connection":"close"},"remoteAddress":"172.17.0.1","userAgent":"kube-probe/1.22"},"res":{"statusCode":404,"responseTime":22,"contentLength":60},"message":"GET /kibana 404 22ms - 60.0B"}
{"type":"response","#timestamp":"2021-09-22T09:55:18+00:00","tags":[],"pid":1216,"method":"get","statusCode":404,"req":{"url":"/kibana","method":"get","headers":{"host":"172.17.0.3:5601","user-agent":"kube-probe/1.22","accept":"*/*","connection":"close"},"remoteAddress":"172.17.0.1","userAgent":"kube-probe/1.22"},"res":{"statusCode":404,"responseTime":26,"contentLength":60},"message":"GET /kibana 404 26ms - 60.0B"}
{"type":"response","#timestamp":"2021-09-22T09:55:28+00:00","tags":[],"pid":1216,"method":"get","statusCode":404,"req":{"url":"/kibana","method":"get","headers":{"host":"172.17.0.3:5601","user-agent":"kube-probe/1.22","accept":"*/*","connection":"close"},"remoteAddress":"172.17.0.1","userAgent":"kube-probe/1.22"},"res":{"statusCode":404,"responseTime":27,"contentLength":60},"message":"GET /kibana 404 27ms - 60.0B"}
{"type":"response","#timestamp":"2021-09-22T09:55:28+00:00","tags":[],"pid":1216,"method":"get","statusCode":404,"req":{"url":"/kibana","method":"get","headers":{"host":"172.17.0.3:5601","user-agent":"kube-probe/1.22","accept":"*/*","connection":"close"},"remoteAddress":"172.17.0.1","userAgent":"kube-probe/1.22"},"res":{"statusCode":404,"responseTime":22,"contentLength":60},"message":"GET /kibana 404 22ms - 60.0B"}
{"type":"response","#timestamp":"2021-09-22T09:55:38+00:00","tags":[],"pid":1216,"method":"get","statusCode":404,"req":{"url":"/kibana","method":"get","headers":{"host":"172.17.0.3:5601","user-agent":"kube-probe/1.22","accept":"*/*","connection":"close"},"remoteAddress":"172.17.0.1","userAgent":"kube-probe/1.22"},"res":{"statusCode":404,"responseTime":25,"contentLength":60},"message":"GET /kibana 404 25ms - 60.0B"}
Elasticsearch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: elasticsearch
spec:
selector:
matchLabels:
component: elasticsearch
template:
metadata:
labels:
component: elasticsearch
spec:
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.14.2
env:
- name: discovery.type
value: single-node
ports:
- containerPort: 9200
protocol: TCP
resources:
limits:
cpu: 2
memory: 4Gi
requests:
cpu: 500m
memory: 4Gi
volumeMounts:
- mountPath: /var/logs
name: logs
volumes:
- name: logs
persistentVolumeClaim:
claimName: chi-pvc
---
apiVersion: v1
kind: Service
metadata:
name: elasticsearch
labels:
service: elasticsearch
spec:
type: NodePort
selector:
component: elasticsearch
ports:
- port: 9200
targetPort: 9200
Ingress-resource.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: chi-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /healthz
pathType: Prefix
backend:
service:
name: chi-svc
port:
number: 3000
- path: /kibana
pathType: Prefix
backend:
service:
name: kibana
port:
number: 5601
- path: /elasticsearch
pathType: Prefix
backend:
service:
name: elasticsearch
port:
number: 9200

I ended up sorting out the issue by having different ingresses and removing the Nginx rewrite-target annotation, I went one step ahead and created a special namespace for the logging infrastructure.
Namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: kube-logging
Persistent-volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: chi-pv
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
hostPath:
path: /data/logs/
type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: chi-pvc
spec:
storageClassName: manual
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi
selector:
matchLabels:
type: local
Elastic-search.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: elasticsearch
namespace: kube-logging
labels:
k8s-app: elasticsearch
version: v1
spec:
selector:
matchLabels:
k8s-app: elasticsearch
version: v1
template:
metadata:
labels:
k8s-app: elasticsearch
version: v1
spec:
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.15.0
env:
- name: discovery.type
value: single-node
- name: ES_JAVA_OPTS
value: "-Xms512m -Xmx512m"
ports:
- containerPort: 9200
resources:
limits:
cpu: 500m
memory: 4Gi
requests:
cpu: 500m
memory: 4Gi
---
apiVersion: v1
kind: Service
metadata:
name: elasticsearch
namespace: kube-logging
labels:
k8s-app: elasticsearch
version: v1
spec:
type: NodePort
selector:
k8s-app: elasticsearch
ports:
- port: 9200
Fluentd.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: kube-logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluentd
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: fluentd
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: kube-logging
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-logging
labels:
k8s-app: fluentd-logging
version: v1
spec:
selector:
matchLabels:
k8s-app: fluentd-logging
version: v1
template:
metadata:
labels:
k8s-app: fluentd-logging
version: v1
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
# Don't need this for Minikubes
# tolerations:
# - key: node-role.kubernetes.io/master
# effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
env:
- name: FLUENT_ELASTICSEARCH_HOST
# <hostname>.<namespace>.svc.cluster.local
value: "elasticsearch.kube-logging.svc.cluster.local"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "http"
- name: FLUENTD_SYSTEMD_CONF
value: 'disable'
- name: FLUENT_LOGSTASH_FORMAT
value: "true"
# # X-Pack Authentication
# # =====================
# - name: FLUENT_ELASTICSEARCH_USER
# value: "elastic"
# - name: FLUENT_ELASTICSEARCH_PASSWORD
# value: "changeme"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
# When actual pod logs in /var/lib/docker/containers, the following lines should be used.
- name: dockercontainerlogdirectory
mountPath: /var/lib/docker/containers
readOnly: true
# When actual pod logs in /var/log/pods, the following lines should be used.
# - name: dockercontainerlogdirectory
# mountPath: /var/log/pods
# readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
# When actual pod logs in /var/lib/docker/containers, the following lines should be used.
- name: dockercontainerlogdirectory
hostPath:
path: /var/lib/docker/containers
# When actual pod logs in /var/log/pods, the following lines should be used.
# - name: dockercontainerlogdirectory
# hostPath:
# path: /var/log/pods
Kibana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana
namespace: kube-logging
labels:
k8s-app: kibana
version: v1
spec:
selector:
matchLabels:
k8s-app: kibana
version: v1
template:
metadata:
labels:
k8s-app: kibana
version: v1
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:7.15.0
env:
- name: SERVER_NAME
value: kibana
- name: SERVER_BASEPATH
value: /kibana
- name: SERVER_REWRITEBASEPATH
value: "true"
# - name: XPACK_SECURITY_ENABLED
# value: "true"
readinessProbe:
httpGet:
path: /kibana/api/status
port: 5601
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /kibana/api/status
port: 5601
initialDelaySeconds: 15
periodSeconds: 20
ports:
- containerPort: 5601
---
apiVersion: v1
kind: Service
metadata:
name: kibana
namespace: kube-logging
labels:
k8s-app: kibana
version: v1
spec:
type: NodePort
selector:
k8s-app: kibana
ports:
- port: 5601
targetPort: 5601
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kibana-ingress
namespace: kube-logging
spec:
rules:
- host: logging.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kibana
port:
number: 5601

Filbeat 7.3.2 serviceaccount with clusterrole issue

My Kubernetes user is not admin in the cluster. So I just cannot create a cluster role binding for filebeat service account. I am using auto discover in filebeat. Can someone help how can I achieve this without clusterrole.
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: logging
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
data:
filebeat.yml: |-
setup.dashboards.enabled: false
setup.template.enabled: true
setup.template.settings:
index.number_of_shards: 1
filebeat.modules:
- module: system
syslog:
enabled: true
#var.paths: ["/var/log/syslog"]
auth:
enabled: true
#var.paths: ["/var/log/authlog"]
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
equals:
kubernetes.namespace: microsrv-test
config:
- type: docker
json.keys_under_root: true
json.add_error_key: true
json.message_key: log
containers:
ids:
- "${data.kubernetes.container.id}"
processors:
- drop_event:
when.or:
- and:
- regexp:
message: '^\d+\.\d+\.\d+\.\d+ '
- equals:
fileset.name: error
- and:
- not:
regexp:
message: '^\d+\.\d+\.\d+\.\d+ '
- equals:
fileset.name: access
- add_cloud_metadata:
- add_kubernetes_metadata:
- add_docker_metadata:
output.elasticsearch:
hosts: ["elasticsearch:9200"]
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: filebeat
namespace: logging
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
spec:
template:
metadata:
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.3.2
imagePullPolicy: Always
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: ELASTICSEARCH_HOST
value: elasticsearch
- name: ELASTICSEARCH_PORT
value: "9200"
- name: KIBANA_HOST
value: kibana
- name: KIBANA_PORT
value: "5601"
securityContext:
runAsUser: 0
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: data
mountPath: /usr/share/filebeat/data
- name: varlog
mountPath: /var/log
readOnly: true
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: dockersock
mountPath: /var/run/docker.sock
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: dockersock
hostPath:
path: /var/run/docker.sock
- name: data
emptyDir: {}
---
Cluster Roles and role bindings
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: logging
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: logging
labels:
k8s-app: filebeat
I have tried creating non cluster role and rolebinding as below,
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: logging
labels:
k8s-app: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: filebeat
namespace: logging
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- pods
verbs:
- get
- watch
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: filebeat
namespace: logging
subjects:
- kind: ServiceAccount
name: filebeat
namespace: logging
roleRef:
kind: Role
name: filebeat
apiGroup: rbac.authorization.k8s.io
but I am getting error.
Performing a resource sync err kubernetes api: Failure 403 pods is
forbidden: User "system:serviceaccount:xxxxx:filebeat" cannot list
resource "pods" in API group "" at the cluster scope for *v1.PodList|

Unfortunately, it will not work the way you want it and the error you are getting indicates it perfectly:
Performing a resource sync err kubernetes api: Failure 403 pods is forbidden: User "system:serviceaccount:xxxxx:filebeat" cannot list resource "pods" in API group "" at the cluster scope for *v1.PodList|
Notice the most important part which is: at the cluster scope. You could also check whether an action is allowed by executing the kubectl auth can-i command. More about that can be found in the Authorization Overview.
This brings us to the differences between Role and ClusterRole:
An RBAC Role or ClusterRole contains rules that represent a set of
permissions. Permissions are purely additive (there are no "deny"
rules).
A Role always sets permissions within a particular namespace;
when you create a Role, you have to specify the namespace it belongs
in.
ClusterRole, by contrast, is a non-namespaced resource. The
resources have different names (Role and ClusterRole) because a
Kubernetes object always has to be either namespaced or not
namespaced; it can't be both.
ClusterRoles have several uses. You can use a ClusterRole to:
define permissions on namespaced resources and be granted within individual namespace(s)
define permissions on namespaced resources and be granted across all namespaces
define permissions on cluster-scoped resources
If you want to define a role within a namespace, use a Role; if you
want to define a role cluster-wide, use a ClusterRole.
And between RoleBinding and ClusterRoleBinding:
A role binding grants the permissions defined in a role to a user or
set of users. It holds a list of subjects (users, groups, or service
accounts), and a reference to the role being granted. A RoleBinding
grants permissions within a specific namespace whereas a
ClusterRoleBinding grants that access cluster-wide.
A RoleBinding may reference any Role in the same namespace.
Alternatively, a RoleBinding can reference a ClusterRole and bind that
ClusterRole to the namespace of the RoleBinding. If you want to bind a
ClusterRole to all the namespaces in your cluster, you use a
ClusterRoleBinding.
So it is impossible to get the cluster scope permissions by using Role and RoleBinding.
You will most likely have to ask your Admin to help you solve this issue.

Kubernetes Traefik v2.3.0 - Web UI 404 Not Found after removing --api.insecure

I'm running Traefik v2.3.0 in a AKS (Azure Kubernetes Service) Cluster and i'm currently trying to setup a Basic Authentication on my Traefik UI.
The dashboard (Traefik UI) works fine without any authentication but i'm getting the server not found page when I try to access with a Basic Authentication.
Here is my configuration.
IngressRoute, Middleware for BasicAuth, Secret and Service :
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-ui
namespace: ingress-basic
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: Host(`traefik-ui.domain.com`) && PathPrefix(`/`) || PathPrefix(`/dashboard`)
services:
- name: traefik-ui
port: 80
middlewares:
- name: traefik-ui-auth
namespace: ingress-basic
tls:
secretName: traefik-ui-cert
---
apiVersion: v1
kind: Secret
metadata:
name: traefik-secret
namespace: ingress-basic
data:
users: |2
dWlhZG06JGFwcjEkanJMZGtEb1okaS9BckJmZzFMVkNIMW80bGtKWFN6LwoK
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: traefik-ui-auth
namespace: ingress-basic
spec:
basicAuth:
secret: traefik-secret
---
apiVersion: v1
kind: Service
metadata:
name: traefik-ui
namespace: ingress-basic
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
selector:
app: traefik-ingress-lb
sessionAffinity: None
type: ClusterIP
DaemonSet and Service:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: traefik-ingress
namespace: ingress-basic
spec:
selector:
matchLabels:
app: traefik-ingress-lb
template:
metadata:
labels:
app: traefik-ingress-lb
name: traefik-ingress-lb
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: size
operator: In
values:
- small
containers:
- args:
- --api.dashboard=true
- --accesslog
- --accesslog.fields.defaultmode=keep
- --accesslog.fields.headers.defaultmode=keep
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --entrypoints.metrics.address=:8082
- --providers.kubernetesIngress.ingressClass=traefik-cert-manager
- --certificatesresolvers.default.acme.email=info#domain.com
- --certificatesresolvers.default.acme.storage=acme.json
- --certificatesresolvers.default.acme.tlschallenge
- --providers.kubernetescrd
- --ping=true
- --pilot.token=xxxxxx-xxxx-xxxx-xxxxx-xxxxx-xx
- --metrics.statsd=true
- --metrics.statsd.address=localhost:8125
- --metrics.statsd.addEntryPointsLabels=true
- --metrics.statsd.addServicesLabels=true
image: traefik:v2.3.0
imagePullPolicy: IfNotPresent
name: traefik-ingress-lb
ports:
- containerPort: 80
name: web
protocol: TCP
- containerPort: 8080
name: admin
protocol: TCP
- containerPort: 443
name: websecure
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /acme/acme.json
name: acme
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: traefik-ingress
serviceAccountName: traefik-ingress
terminationGracePeriodSeconds: 60
tolerations:
- effect: NoSchedule
key: size
operator: Equal
value: small
volumes:
- hostPath:
path: /srv/configs/acme.json
type: ""
name: acme
With this configuration :
kubectl exec -it -n ingress-basic traefik-ingress-2m88q -- curl http://localhost:8080/dashboard/
404 page not found
When removing the Middleware and adding "--api.insecure" in the DaemonSet config :
kubectl exec -it -n ingress-basic traefik-ingress-1hf4q -- curl http://localhost:8080/dashboard/
<!DOCTYPE html><html><head><title>Traefik</title><meta charset=utf-8><meta name=description content="Traefik UI"><meta name=format-detection content="telephone=no"><meta name=msapplication-tap-highlight content=no><meta name=viewport content="user-scalable=no,initial-scale=1,maximum-scale=1,minimum-scale=1,width=device-width"><link rel=icon type=image/png href=statics/app-logo-128x128.png><link rel=icon type=image/png sizes=16x16 href=statics/icons/favicon-16x16.png><link rel=icon[...]</body></html>
Please let me know what I am doing wrong here? Is there any other way of doing it ?
Regards,

Here's another take on the IngressRoute, adapted to your environment.
I think 99% of the issue is actual route matching, especially if you say --api.insecure works.
Also as a rule of a thumb, logging & access log would help a lot in the DaemonSet definition.
- --log
- --log.level=DEBUG
- --accesslog
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-ui
namespace: ingress-basic
spec:
entryPoints:
- websecure
routes:
- match: Host(`traefik-ui.domain.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
kind: Rule
services:
- name: api#internal
kind: TraefikService
middlewares:
- name: traefik-basic-auth
tls:
secretName: traefik-ui-cert

How can I edit path.data and path.log for elasticsearch on Kubernetes?

I made an es-deploy.yml file then I typed the path.log and path.data values.
After creating the pod, I checked that directory then there was nothing.
The setting did not work!
How can I edit path.data and path.log for elasticsearch on Kubernetes!
I also tried using PATH_DATA
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: es
labels:
component: elasticsearch
spec:
replicas: 1
template:
metadata:
labels:
component: elasticsearch
spec:
serviceAccount: elasticsearch
initContainers:
- name: init-sysctl
image: busybox
imagePullPolicy: IfNotPresent
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
containers:
- name: es
securityContext:
capabilities:
add:
- IPC_LOCK
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.0
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: "CLUSTER_NAME"
value: "myesdb"
- name: "DISCOVERY_SERVICE"
value: "elasticsearch"
- name: NODE_MASTER
value: "true"
- name: NODE_DATA
value: "true"
- name: HTTP_ENABLE
value: "true"
- name: ES_JAVA_OPTS
value: "-Xms256m -Xmx256m"
- name: "path.data"
value: "/data/elk/data"
- name: "path.logs"
value: "/data/elk/log"
ports:
- containerPort: 9200
name: http
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
volumeMounts:
- mountPath: /data/elk/

Those values path.data and path.logs are not environment variables. They are config options.
The default path.data for the official elasticsearch image is /usr/share/elasticsearch/data based on the default value of ES_HOME=/usr/share/elasticsearch/ If you don't want to use that path you have to override it in the elasticsearch.yaml config.
You will have to create a ConfigMap containing your elasticsearch.yaml with something like this:
apiVersion: v1
kind: ConfigMap
metadata:
name: elasticsearch-config
namespace: es
data:
elasticsearch.yml: |
cluster:
name: ${CLUSTER_NAME:elasticsearch-default}
node:
master: ${NODE_MASTER:true}
data: ${NODE_DATA:true}
name: ${NODE_NAME}
ingest: ${NODE_INGEST:true}
max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES:1}
processors: ${PROCESSORS:1}
network.host: ${NETWORK_HOST:_site_}
path:
data: ${DATA_PATH:"/data/elk"}
repo: ${REPO_LOCATIONS:[]}
bootstrap:
memory_lock: ${MEMORY_LOCK:false}
http:
enabled: ${HTTP_ENABLE:true}
compression: true
cors:
enabled: true
allow-origin: "*"
discovery:
zen:
ping.unicast.hosts: ${DISCOVERY_SERVICE:elasticsearch-discovery}
minimum_master_nodes: ${NUMBER_OF_MASTERS:1}
xpack:
license.self_generated.type: basic
(Note that the above ConfigMap will also allow you to use the DATA_PATH environment variable)
Then mount your volumes in your Pod with something like this:
volumeMounts:
- name: storage
mountPath: /data/elk
- name: config-volume
mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
subPath: elasticsearch.yml
volumes:
- name: config-volume
configMap:
name: elasticsearch-config
- name: storage
<add-whatever-volume-you-are-using-for-data>

Resources