Configuring Prometheus JMX exporter for Hadoop2 - hadoop

Question

I am trying to scrape metrics from following Hadoop2 daemons running on ec2 instance using Prometheus JMX exporter:
hadoop namenode
hadoop datanode
yarn resourcemanager
yarn nodemanager
I am trying to run JMX exporter as a java agent with all the four daemons. For this I have added EXTRA_JAVA_OPTS in hadoop-env.sh and yarn-env.sh :
export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"
export HADOOP_DATANODE_OPTS="$HADOOP_DATANODE_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"
export YARN_RESOURCEMANAGER_OPTS="$YARN_RESOURCEMANAGER_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"
export YARN_NODEMANAGER_OPTS="$YARN_NODEMANAGER_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"
Sample prometheus_config.yml for a resourcemanager metric NumAllSources is as follows :
rules:
- pattern: Hadoop<service=ResourceManager, name=MetricsSystem, sub=Stats><>NumAllSources
name: sources
labels:
app_id: "hadoop_rm"
I am getting the following exception when I restart the resourcemanager or other daemons with new configs and java_opts :
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:382)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:397)
Caused by: java.lang.IllegalArgumentException: Collector already registered that provides name: jmx_scrape_duration_seconds
at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:54)
at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:128)
Any suggestions how to fix this?

Answer 1

While #chanhou's solution will work, I wanted to keep my edits in hadoop-env.sh, so I went with
if ! grep -q <<<"$HADOOP_NAMENODE_OPTS" jmx_prometheus_javaagent; then
HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -javaagent:/home/caesarli/platform/jmx_prometheus_javaagent-0.12.0.jar=11099:/home/caesarli/platform/hadoop-2.8.4/etc/hadoop/jmx-name.yaml"
fi
and similar for the HADOOP_DATANODE_OPTS.

Answer 2

That is because -javaagent opts is declare multiple times in $HADOOP_OPTS when you call /usr/local/hadoop/sbin/hadoop-daemon.sh start datanode and hadoop-daemon.sh will eventually call /usr/local/hadoop/bin/hdfs to start related service.
During the process, it will source multiple times of hadoop-config.sh and if you echo $HADOOP_OPTS in the shell script /usr/local/hadoop/bin/hdfs then you will find multiple -javaagent there.
A workaround is declare HADOOP_OPTS=$HADOOP_OPTS -javaagent:... in /usr/local/hadoop/bin/hdfs to ensure only one -javaagent appear in HADOOP_OPTS

Answer 3

I think this is because you are using same ports (9102) for all the registrations, changing ports will help.