Access Kafka installed on a minikube cluster on Windows 10 - windows

I'm trying to install Kafka with Strimzy on a local munikube cluster running on Windows 10, to test the impact of different parameters (especially the TLS configuration). Before moving to TLS, i'd simply like to connect to my cluster :)
Here is my yaml configuration :
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 2.3.0
replicas: 1
listeners:
external:
type: nodeport
tls: false
config:
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
log.message.format.version: "2.3"
storage:
type: persistent-claim
size: 1Gi
zookeeper:
replicas: 1
storage:
type: persistent-claim
size: 2Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}
For the listener, I firstly started with plain: {} but this only gives me services of type ClusterIP, not accessible from outside minikube (i really need to connect from outside).
I then moved to a listener of kind external.
You can fin below the configuration of the cluster:
kubectl get all -n kafka
NAME READY STATUS RESTARTS AGE
pod/my-cluster-entity-operator-9657c9d79-8hknc 3/3 Running 0 17m
pod/my-cluster-kafka-0 2/2 Running 0 18m
pod/my-cluster-zookeeper-0 2/2 Running 0 18m
pod/strimzi-cluster-operator-f77b7d544-hq5pq 1/1 Running 0 5h22m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-cluster-kafka-0 NodePort 10.99.3.204 <none> 9094:30117/TCP 18m
service/my-cluster-kafka-bootstrap ClusterIP 10.106.176.111 <none> 9091/TCP 18m
service/my-cluster-kafka-brokers ClusterIP None <none> 9091/TCP 18m
service/my-cluster-kafka-external-bootstrap NodePort 10.109.235.156 <none> 9094:32372/TCP 18m
service/my-cluster-zookeeper-client ClusterIP 10.97.2.69 <none> 2181/TCP 18m
service/my-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 18m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/my-cluster-entity-operator 1/1 1 1 17m
deployment.apps/strimzi-cluster-operator 1/1 1 1 5h22m
The IP address of the minikube cluster is 192.168.49.2 (given by minikube ip)
For the while, is everything correct on my configuration ? I cannot connect on the cluster with a producer (i get a timeout error when i try to publish data).
I tried to connect to 192.168.49.2:32372 & 192.168.49.2:30117 and I always get the same timeout error. I also tryed to run
minikube service -n kafka my-cluster-kafka-external-bootstrap
and
minikube service -n kafka my-cluster-kafka-0
and i still get the same error.
What is wrong in what i'm trying to do?
Thanks!

Ok, I got the answser.
I changed the type of the service to LoadBalancer and started minikube tunnel
One other point, as I'm running this on windows, I noticed that if I run everything using powershell it works, if I used an other command line tool (like Moba) it does not work, I don't explain this.

Related

ECK Operator stuck at "Skipping pod because it has no IP yet"

I am trying to deploy a two node Elastic cluster on my 3 node Kubernetes cluster using the ECK operator (version 2.4.0). I have deployed the operator by applying the CRDS and manifest. I then followed the quickstart steps to get started and planned to add persistant storage after.
I have applied following Elasticsearch definition:
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
namespace: elasticsearch
spec:
version: 8.4.3
nodeSets:
- name: default
count: 2
config:
node.store.allow_mmap: false
A get all on the namespace:
kube#k8s-master-01:~$ kubectl get all -n elasticsearch
NAME READY STATUS RESTARTS AGE
pod/quickstart-es-default-0 0/1 Pending 0 5m47s
pod/quickstart-es-default-1 0/1 Pending 0 5m47s
pod/quickstart-kb-679bf8c9f7-psvn6 1/1 Running 0 5m45s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/quickstart-es-default ClusterIP None <none> 9200/TCP 5m47s
service/quickstart-es-http ClusterIP 10.101.171.44 <none> 9200/TCP 5m47s
service/quickstart-es-internal-http ClusterIP 10.105.134.217 <none> 9200/TCP 5m47s
service/quickstart-es-transport ClusterIP None <none> 9300/TCP 5m47s
service/quickstart-kb-http ClusterIP 10.111.195.203 <none> 5601/TCP 5m46s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/quickstart-kb 1/1 1 1 5m45s
NAME DESIRED CURRENT READY AGE
replicaset.apps/quickstart-kb-679bf8c9f7 1 1 1 5m45s
NAME READY AGE
statefulset.apps/quickstart-es-default 0/2 5m47s
Events output of a describe pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m8s (x3 over 7m18s) default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Logs of the ECK operator:
{"log.level":"info","#timestamp":"2022-10-25T15:18:49.749Z","log.logger":"elasticsearch-controller","message":"Starting reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":"10557","namespace":"elasticsearch","es_name":"quickstart"}
{"log.level":"info","#timestamp":"2022-10-25T15:18:49.982Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","namespace":"elasticsearch","pod_name":"quickstart-es-default-0"}
{"log.level":"info","#timestamp":"2022-10-25T15:18:49.982Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","namespace":"elasticsearch","pod_name":"quickstart-es-default-1"}
{"log.level":"info","#timestamp":"2022-10-25T15:18:49.984Z","log.logger":"driver","message":"Elasticsearch cannot be reached yet, re-queuing","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","namespace":"elasticsearch","es_name":"quickstart"}
{"log.level":"info","#timestamp":"2022-10-25T15:18:49.984Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":"10557","namespace":"elasticsearch","es_name":"quickstart","took":0.23531655}
What am I misunderstanding from the Elastic docs? If relevant I use Traefik as Ingress, Calico as CNI and Metallb as LB.

Gitea: dial tcp: lookup gitea-postgresql.default.svc.cluster.local

I see this error when trying to use Gitea with microk8s on Ubuntu 21.10:
$ k logs gitea-0 -c configure-gitea
Wait for database to become avialable...
gitea-postgresql (10.152.183.227:5432) open
...
2021/11/20 05:49:40 ...om/urfave/cli/app.go:277:Run() [I] PING DATABASE postgres
2021/11/20 05:49:45 cmd/migrate.go:38:runMigrate() [F] Failed to initialize ORM engine: dial tcp: lookup gitea-postgresql.default.svc.cluster.local: Try again
I am looking for some clues as to how to debug this please.
The other pods seem to be running as expected:
$ k get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system hostpath-provisioner-5c65fbdb4f-nfx7d 1/1 Running 0 11h
kube-system calico-node-h8tpk 1/1 Running 0 11h
kube-system calico-kube-controllers-f7868dd95-dpp8n 1/1 Running 0 11h
kube-system coredns-7f9c69c78c-cnpkj 1/1 Running 0 11h
default gitea-memcached-584956987c-zb8kp 1/1 Running 0 20s
default gitea-postgresql-0 1/1 Running 0 20s
default gitea-0 0/1 Init:1/2 1 20s
The services are not as expected, since gitea-0 is not starting:
$ k get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 11h
kube-system kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 11h
default gitea-postgresql-headless ClusterIP None <none> 5432/TCP 3m25s
default gitea-ssh ClusterIP None <none> 22/TCP 3m25s
default gitea-http ClusterIP None <none> 3000/TCP 3m25s
default gitea-memcached ClusterIP 10.152.183.15 <none> 11211/TCP 3m25s
default gitea-postgresql ClusterIP 10.152.183.227 <none> 5432/TCP 3m25s
Also see:
https://github.com/ubuntu/microk8s/issues/2741
https://gitea.com/gitea/helm-chart/issues/249
I worked through to the point where I had the logs below, specifically:
cmd/migrate.go:38:runMigrate() [F] Failed to initialize ORM engine: dial tcp: lookup gitea-postgresql.default.svc.cluster.local: Try again
Using k cluster-info dump I saw:
[ERROR] plugin/errors: 2 gitea-postgresql.default.svc.cluster.local.cisco.com. A: read udp 10.1.147.194:56647->8.8.8.8:53: i/o timeout
That led me to test the DNS with dig and 8.8.8.8. That test didn't reveal any errors, in that DNS seemed to work. Even so, DNS seemed suspect.
So then I tried microk8s enable storage dns:<IP address of DNS in lab>, whereas I was previously only using microk8s storage dns. The storage part enables the persistent volumes that the database needs.
The key piece here is the lab DNS server IP address argument when enabling DNS with microk8s.

NGINX Ingress 404 not found using docker desktop on windows (not minikube)

I am trying to make this basic example work on docker desktop on windows, I am not using minikube.
I managed to reach the service using NodePort with:
http://localhost:31429
But when I try http://hello-world.info (made sure to add it in hosts) - 404 not found.
kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 20m
default web NodePort 10.111.220.81 <none> 8080:31429/TCP 6m47s
ingress-nginx ingress-nginx-controller LoadBalancer 10.107.29.182 localhost 80:30266/TCP,443:32426/TCP 19m
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.101.138.244 <none> 443/TCP 19m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 20m
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
example-ingress <none> hello-world.info 80 21m
I am lost, can someone please help ?
I also noticed that ADDRESS is empty.
Many thanks.
Reproduced this case on Docker Desktop 4.1.1, Windows 10 Pro
Install Ingress Controller for Docker Desktop:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/cloud/deploy.yaml
As I understand it, #dev1334 used an example from Set up Ingress on Minikube with the NGINX Ingress Controller article. I also tried it with some modifications to the original example.
In the example for the example-ingress.yaml file in the spec.rules section, the host hello-world.info is specified. Since Docker Desktop for Windows adds to a hosts file in C:\Windows\System32\drivers\etc\hosts during installation the following entry: 127.0.0.1 kubernetes.docker.internal I changed the host in the example-ingress.yaml from hello-world.info to kubernetes.docker.internal
But Ingress still didn't work as expected due to the following error:
"Ignoring ingress because of error while validating ingress class" ingress="default/example-ingress" error="ingress does not contain a valid IngressClass"
I added this line kubernetes.io/ingress.class: "nginx" to the annotations section in example-ingress.yaml
So, the final version of the example-ingress.yaml file is below.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: kubernetes.docker.internal
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web
port:
number: 8080
- path: /v2
pathType: Prefix
backend:
service:
name: web2
port:
number: 8080
Test results
C:\Users\Andrew_Skorkin>kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default web-79d88c97d6-c8xnf 1/1 Running 0 112m
default web2-5d47994f45-cxtzm 1/1 Running 0 94m
ingress-nginx ingress-nginx-admission-create-sjdcq 0/1 Completed 0 114m
ingress-nginx ingress-nginx-admission-patch-wccc9 0/1 Completed 1 114m
ingress-nginx ingress-nginx-controller-5c8d66c76d-jb4w9 1/1 Running 0 114m
...
C:\Users\Andrew_Skorkin>kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7d15h
default web NodePort 10.101.43.157 <none> 8080:32651/TCP 114m
default web2 NodePort 10.100.4.84 <none> 8080:30081/TCP 96m
ingress-nginx ingress-nginx-controller LoadBalancer 10.106.138.217 localhost 80:30287/TCP,443:32664/TCP 116m
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.111.208.242 <none> 443/TCP 116m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 7d15h
C:\Users\Andrew_Skorkin>curl kubernetes.docker.internal
Hello, world!
Version: 1.0.0
Hostname: web-79d88c97d6-c8xnf
C:\Users\Andrew_Skorkin>curl kubernetes.docker.internal/v2
Hello, world!
Version: 2.0.0
Hostname: web2-5d47994f45-cxtzm

Why can't I Access a Service Exposed from Minikube on Windows?

I'm new to Kubernetes. I successfully created a deployment with 2 replicas of my Angular frontend application, but when I expose it with a service and try to access the service with 'minikube service service-name', the browser can't show me the application.
This is my docker file
FROM registry.gitlab.informatica.aci.it/ccsc/images/nodejs/10_15
LABEL maintainer="d.vaccaro#informatica.aci.it" name="assistenza-fo" version="v1.0.0" license=""
WORKDIR /usr/src/app
ARG PRODUCTION_MODE="false"
ENV NODE_ENV='development'
ENV HTTP_PORT=4200
COPY package*.json ./
RUN if [ "${PRODUCTION_MODE}" = "true" ] || [ "${PRODUCTION_MODE}" = "1" ]; then \
echo "Build di produzione"; \
npm ci --production ; \
else \
echo "Build di sviluppo"; \
npm ci ; \
fi
RUN npm audit fix
RUN npm install -g #angular/cli
COPY dockerize /usr/local/bin
RUN chmod +x /usr/local/bin/dockerize
COPY . .
EXPOSE 4200
CMD ng serve --host 0.0.0.0
pod description
Name: assistenza-fo-674f85c547-bzf8g
Namespace: default
Priority: 0
Node: minikube/172.17.0.2
Start Time: Sun, 19 Apr 2020 12:41:06 +0200
Labels: pod-template-hash=674f85c547
run=assistenza-fo
Annotations: <none>
Status: Running
IP: 172.18.0.6
Controlled By: ReplicaSet/assistenza-fo-674f85c547
Containers:
assistenza-fo:
Container ID: docker://ef2bfb66d22dea56b2dc0e49e875376bf1edff369274015445806451582703a0
Image: registry.gitlab.informatica.aci.it/apra/sta-r/assistenza/assistenza-fo:latest
Image ID: docker-pullable://registry.gitlab.informatica.aci.it/apra/sta-r/assistenza/assistenza-fo#sha256:8d02a3e69d6798c1ac88815ef785e05aba6e394eb21f806bbc25fb761cca5a98
Port: 4200/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 19 Apr 2020 12:41:08 +0200
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-zdrwg (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-zdrwg:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-zdrwg
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
my deployment description
Name: assistenza-fo
Namespace: default
CreationTimestamp: Sun, 19 Apr 2020 12:41:06 +0200
Labels: run=assistenza-fo
Annotations: deployment.kubernetes.io/revision: 1
Selector: run=assistenza-fo
Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: run=assistenza-fo
Containers:
assistenza-fo:
Image: registry.gitlab.informatica.aci.it/apra/sta-r/assistenza/assistenza-fo:latest
Port: 4200/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: assistenza-fo-674f85c547 (2/2 replicas created)
Events: <none>
and my service description
Name: assistenza-fo
Namespace: default
Labels: run=assistenza-fo
Annotations: <none>
Selector: run=assistenza-fo
Type: LoadBalancer
IP: 10.97.3.206
Port: <unset> 4200/TCP
TargetPort: 4200/TCP
NodePort: <unset> 30375/TCP
Endpoints: 172.18.0.6:4200,172.18.0.7:4200
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
When i run the command
minikube service assistenza-fo
I get the following output:
|-----------|---------------|-------------|-------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|---------------|-------------|-------------------------|
| default | assistenza-fo | 4200 | http://172.17.0.2:30375 |
|-----------|---------------|-------------|-------------------------|
* Opening service default/assistenza-fo in default browser...
but Chrome prints out: "unable to reach the site" for timeout.
Thank you
EDIT
I create again the service, this time as a NodePort service. Still not working. This is the service description:
Name: assistenza-fo
Namespace: default
Labels: run=assistenza-fo
Annotations: <none>
Selector: run=assistenza-fo
Type: NodePort
IP: 10.107.46.43
Port: <unset> 4200/TCP
TargetPort: 4200/TCP
NodePort: <unset> 30649/TCP
Endpoints: 172.18.0.7:4200,172.18.0.8:4200
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
I was able to reproduce your issue.
It's actually a bug on latest version of Minikube for Windows running Docker Driver: --driver=docker
You can see it here: Issue - minikube service not working with Docker driver on Windows 10 Pro #7644
it was patched with the merge: Pull - docker driver: Add Service & Tunnel features to windows
it is available now on Minikube v1.10.0-beta.0
In order to make it work, download the beta version from the website:
https://github.com/kubernetes/minikube/releases/download/v1.10.0-beta.0/minikube-windows-amd64.exe
move it to your working folder and rename it to minikube.exe
C:\Kubernetes>rename minikube-windows-amd64.exe minikube.exe
C:\Kubernetes>dir
22/04/2020 21:10 <DIR> .
22/04/2020 21:10 <DIR> ..
22/04/2020 21:04 55.480.832 minikube.exe
22/04/2020 20:05 489 nginx.yaml
2 File(s) 55.481.321 bytes
If you haven't yet, stop and uninstall the older version, then start Minikube with the new binary:
C:\Kubernetes>minikube.exe start --driver=docker
* minikube v1.10.0-beta.0 on Microsoft Windows 10 Pro 10.0.18363 Build 18363
* Using the docker driver based on existing profile
* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Restarting existing docker container for "minikube" ...
* Preparing Kubernetes v1.18.0 on Docker 19.03.2 ...
- kubeadm.pod-network-cidr=10.244.0.0/16
* Enabled addons: dashboard, default-storageclass, storage-provisioner
* Done! kubectl is now configured to use "minikube"
C:\Kubernetes>kubectl get all
NAME READY STATUS RESTARTS AGE
pod/nginx-76df748b9-t6q59 1/1 Running 1 78m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 85m
service/nginx-svc NodePort 10.100.212.15 <none> 80:31027/TCP 78m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 1/1 1 1 78m
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-76df748b9 1 1 1 78m
Minikube is now running on version v1.10.0-beta.0, now you can run the service as intended (and note the command will be unavailable because it will be tunneling the connection:
The browser will open automatically and your service will be available:
If you have any doubts let me know in the comments.

StackDriver Monitoring for Elasticsearch in GKE

I need to monitor Elasticsearch(2.4) that installed on top of the k8s cluster. I have 2 clients, 3 masters and several data nodes run in pods. Following the "how to" of Stackdriver and the post "Can I run Google Monitoring Agent inside a Kubernetes Pod?", I deployed an agent in its own Pod. Suddenly, after all, have no Elasticsearch metrics in StackDriver. The Only Zeros.
Any suggestion are more than welcome.
This is my configuration:
Elastic service:
$kubectl describe svc elasticsearch
Name: elasticsearch
Namespace: default
Labels: component=elasticsearch
role=client
Selector: component=elasticsearch,role=client
Type: NodePort
IP: <IP>
Port: http 9200/TCP
NodePort: http <PORT>/TCP
Endpoints: <IP>:9200,<IP>:9200
Session Affinity: None
No events.
Stackdriver deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: stackagent
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
component: monitoring
role: stackdriver-agent
spec:
containers:
- name: hslab-data-agent
image: StackDriverAgent:version1
StackDriverAgent:version1 Docker:
FROM ubuntu
WORKDIR /stackdriver
RUN apt-get update
RUN apt-get install curl lsb-release libyajl2 -y
RUN apt-get clean
COPY ./stackdriver/run.sh run.sh
COPY ./stackdriver/elasticsearch.conf elasticsearch.conf
RUN chmod 755 ./run.sh
CMD ["./run.sh"]
run.sh:
#!/bin/bash
curl -O https://repo.stackdriver.com/stack-install.sh
chmod 755 stack-install.sh
bash stack-install.sh --write-gcm
cp ./elasticsearch.conf /opt/stackdriver/collectd/etc/collectd.d/
service stackdriver-agent restart
while true; do
sleep 60
agent_pid=$(cat /var/run/stackdriver-agent.pid 2>/dev/null)
ps -p $agent_pid > /dev/null 2>&1
if [ $? != 0 ]; then
echo "Stackdriver agent pid not found!"
break;
fi
done
elasticsearch.conf:
Taken from https://raw.githubusercontent.com/Stackdriver/stackdriver-agent-service-configs/master/etc/collectd.d/elasticsearch.conf
# This is the monitoring configuration for Elasticsearch 1.0.x and later.
# Look for ELASTICSEARCH_HOST and ELASTICSEARCH_PORT to adjust your configuration file.
LoadPlugin curl_json
<Plugin "curl_json">
# When using non-standard Elasticsearch configurations, replace the below with
#<URL "http://ELASTICSEARCH_HOST:ELASTICSEARCH_PORT/_nodes/_local/stats/">
# PREVIOUSE LINE
# <URL "http://localhost:9200/_nodes/_local/stats/">
<URL "http://elasticsearch:9200/_nodes/_local/stats/">
Instance "elasticsearch"
....
Running state:
NAME READY STATUS RESTARTS AGE
esclient-4231471109-bd4tb 1/1 Running 0 23h
esclient-4231471109-k5pnw 1/1 Running 0 23h
esdata-1-2524076994-898r0 1/1 Running 0 23h
esdata-2-2426789420-zhz7j 1/1 Running 0 23h
esmaster-1-4205943399-zj2pn 1/1 Running 0 23h
esmaster-2-4248445829-pwq46 1/1 Running 0 23h
esmaster-3-3967126695-w0tp2 1/1 Running 0 23h
stackagent-3122989159-15vj1 1/1 Running 0 18h
The problem with API URL of plugin configuration.The http://elasticsearch:9200/_nodes/**_local**/stats/"> will returns info only for the _local node which is a client that have no documents.
In addition, the Stackdriver data will be collected under k8s cluster node and not under the pod name.
The partial solution is to setup sidecar in data node and patch the elasticsearch.conf query with the corresponding ES node name:
get the curl [elasticsearch]:9200/_nodes/stats
find the ES node name by the $(hostname)
patch the configuration <URL "http://elasticsearch:9200/_nodes/<esnode_name>/stats/">
This will collect the information of ES data node under the k8s data node name.

Resources