StackDriver Monitoring for Elasticsearch in GKE - elasticsearch

Question

I need to monitor Elasticsearch(2.4) that installed on top of the k8s cluster. I have 2 clients, 3 masters and several data nodes run in pods. Following the "how to" of Stackdriver and the post "Can I run Google Monitoring Agent inside a Kubernetes Pod?", I deployed an agent in its own Pod. Suddenly, after all, have no Elasticsearch metrics in StackDriver. The Only Zeros.
Any suggestion are more than welcome.
This is my configuration:
Elastic service:
$kubectl describe svc elasticsearch
Name: elasticsearch
Namespace: default
Labels: component=elasticsearch
role=client
Selector: component=elasticsearch,role=client
Type: NodePort
IP: <IP>
Port: http 9200/TCP
NodePort: http <PORT>/TCP
Endpoints: <IP>:9200,<IP>:9200
Session Affinity: None
No events.
Stackdriver deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: stackagent
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
component: monitoring
role: stackdriver-agent
spec:
containers:
- name: hslab-data-agent
image: StackDriverAgent:version1
StackDriverAgent:version1 Docker:
FROM ubuntu
WORKDIR /stackdriver
RUN apt-get update
RUN apt-get install curl lsb-release libyajl2 -y
RUN apt-get clean
COPY ./stackdriver/run.sh run.sh
COPY ./stackdriver/elasticsearch.conf elasticsearch.conf
RUN chmod 755 ./run.sh
CMD ["./run.sh"]
run.sh:
#!/bin/bash
curl -O https://repo.stackdriver.com/stack-install.sh
chmod 755 stack-install.sh
bash stack-install.sh --write-gcm
cp ./elasticsearch.conf /opt/stackdriver/collectd/etc/collectd.d/
service stackdriver-agent restart
while true; do
sleep 60
agent_pid=$(cat /var/run/stackdriver-agent.pid 2>/dev/null)
ps -p $agent_pid > /dev/null 2>&1
if [ $? != 0 ]; then
echo "Stackdriver agent pid not found!"
break;
fi
done
elasticsearch.conf:
Taken from https://raw.githubusercontent.com/Stackdriver/stackdriver-agent-service-configs/master/etc/collectd.d/elasticsearch.conf
# This is the monitoring configuration for Elasticsearch 1.0.x and later.
# Look for ELASTICSEARCH_HOST and ELASTICSEARCH_PORT to adjust your configuration file.
LoadPlugin curl_json
<Plugin "curl_json">
# When using non-standard Elasticsearch configurations, replace the below with
#<URL "http://ELASTICSEARCH_HOST:ELASTICSEARCH_PORT/_nodes/_local/stats/">
# PREVIOUSE LINE
# <URL "http://localhost:9200/_nodes/_local/stats/">
<URL "http://elasticsearch:9200/_nodes/_local/stats/">
Instance "elasticsearch"
....
Running state:
NAME READY STATUS RESTARTS AGE
esclient-4231471109-bd4tb 1/1 Running 0 23h
esclient-4231471109-k5pnw 1/1 Running 0 23h
esdata-1-2524076994-898r0 1/1 Running 0 23h
esdata-2-2426789420-zhz7j 1/1 Running 0 23h
esmaster-1-4205943399-zj2pn 1/1 Running 0 23h
esmaster-2-4248445829-pwq46 1/1 Running 0 23h
esmaster-3-3967126695-w0tp2 1/1 Running 0 23h
stackagent-3122989159-15vj1 1/1 Running 0 18h

Answer 1

The problem with API URL of plugin configuration.The http://elasticsearch:9200/_nodes/**_local**/stats/"> will returns info only for the _local node which is a client that have no documents.
In addition, the Stackdriver data will be collected under k8s cluster node and not under the pod name.
The partial solution is to setup sidecar in data node and patch the elasticsearch.conf query with the corresponding ES node name:
get the curl [elasticsearch]:9200/_nodes/stats
find the ES node name by the $(hostname)
patch the configuration <URL "http://elasticsearch:9200/_nodes/<esnode_name>/stats/">
This will collect the information of ES data node under the k8s data node name.