Metricbeats on GKE not working as expected - Missing deployments, nodes, etc - elasticsearch

Question

I followed everything as the tutorial is saying: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html
But I still don't get my nodes and deployments status:
https://aws1.discourse-cdn.com/elastic/optimized/3X/a/1/a15b848aa799b87e1ddcbb9377df1333da5b44de_2_690x339.png
I have kube-state-metrics running, when I see the logs on the metricbeat pod I get the following error:
"github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: Failed to list *v1beta1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:metricbeat" cannot list resource "replicasets" in API group "apps" at the cluster scope"

Answer 1

How did u deploy the kube-state-metrics? Through helm chart or k8s default example manifest located in github?
You should see the log entry like this from metricbeat pod:
# k logs -n kube-system pod/metricbeat-5f97c8ccc5-884k8
2020-01-21T06:34:27.162Z INFO instance/beat.go:610 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2020-01-21T06:34:27.181Z INFO instance/beat.go:618 Beat ID: 8e09f938-b2b9-4d7d-b610-8256e1f9dbfb
2020-01-21T06:34:27.181Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-01-21T06:34:27.181Z INFO [beat] instance/beat.go:941 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "8e09f938-b2b9-4d7d-b610-8256e1f9dbfb"}}}
2020-01-21T06:34:27.181Z INFO [beat] instance/beat.go:950 Build info {"system_info": {"build": {"commit": "60dd883ca29e1fdd5b8b075bd5f3698948b1d44d", "libbeat": "7.5.1", "time": "2019-12-16T22:02:52.000Z", "version": "7.5.1"}}}
2020-01-21T06:34:27.181Z INFO [beat] instance/beat.go:953 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.12.12"}}}
2020-01-21T06:34:27.185Z INFO [beat] instance/beat.go:957 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-01-16T12:03:08Z","containerized":false,"name":"aks-agentpool-36903858-vmss00000R","ip":["127.0.0.1/8","::1/128","10.240.0.8/16","fe80::20d:3aff:fe80:38a2/64","172.17.0.1/16","10.244.5.1/24","fe80::dcfd:15ff:fe92:efd/64","fe80::f84f:ff:fe17:383d/64","fe80::386d:7eff:fe7c:6b42/64","fe80::b499:e3ff:fe26:c48b/64","fe80::f047:24ff:fe3c:bb95/64","fe80::4483:19ff:fece:9762/64","fe80::e802:42ff:fe90:1b78/64","fe80::ccab:6aff:feed:a3de/64","fe80::b88a:a3ff:feab:3b99/64","fe80::54f8:9bff:fe33:391f/64","fe80::a078:ebff:fef6:d195/64","fe80::9c9a:1eff:fe44:ec4f/64","fe80::3c08:3fff:feef:9758/64"],"kernel_version":"4.15.0-1064-azure","mac":["00:0d:3a:80:38:a2","02:42:f7:db:ad:17","de:fd:15:92:0e:fd","fa:4f:00:17:38:3d","3a:6d:7e:7c:6b:42","b6:99:e3:26:c4:8b","f2:47:24:3c:bb:95","46:83:19:ce:97:62","ea:02:42:90:1b:78","ce:ab:6a:ed:a3:de","ba:8a:a3:ab:3b:99","56:f8:9b:33:39:1f","a2:78:eb:f6:d1:95","9e:9a:1e:44:ec:4f","3e:08:3f:ef:97:58"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-01-21T06:34:27.185Z INFO [beat] instance/beat.go:986 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-01-21T06:34:26.070Z"}}}
2020-01-21T06:34:27.185Z INFO instance/beat.go:297 Setup Beat: metricbeat; Version: 7.5.1
2020-01-21T06:34:27.185Z INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'metricbeat-7.5.1' as ILM is enabled.
2020-01-21T06:34:27.186Z INFO elasticsearch/client.go:171 Elasticsearch url: https://elasticsearch-clsuter-es-http.elastic-system:9200
2020-01-21T06:34:27.186Z INFO [publisher] pipeline/module.go:97 Beat name: aks-agentpool-36903858-vmss00000R
2020-01-21T06:34:27.188Z INFO add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata: hosting provider type detected as az, metadata={"instance":{"id":"3849a291-883a-4d02-8fbd-8ce6910a3a8f","name":"aks-agentpool-36903858-vmss_27"},"machine":{"type":"Standard_DS2_v2"},"provider":"az","region":"eastasia"}