How to configure OTEL collector with Spring app using micrometer library and java instrumentation agent to get metrics - spring

Question

Issue
Otel collector instrumentation agent does not forward prometheus metrics from Spring app correctly
Local reproduction steps
Simple java springboot app with gradle
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation 'org.springframework.boot:spring-boot-starter-actuator'
}
OTEL Collector running as a docker container with the following files
docker-compose.yml
otel-collector:
image: otel/opentelemetry-collector
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./config.yaml:/etc/otel-collector-config.yaml
ports:
- "1888:1888" # pprof extension
- "8888:8888" # Prometheus metrics exposed by the collector
- "8889:8889" # Prometheus exporter metrics
- "13133:13133" # health_check extension
- "4317:4317" # OTLP gRPC receiver
- "4318:4318" # OTLP http receiver
- "55679:55679" # zpages extension
config.yml
receivers:
prometheus:
config:
scrape_configs:
- job_name: "Book"
scrape_interval: 5s
static_configs:
- targets: ["localhost:9080"]
processors:
batch:
exporters:
prometheus:
endpoint: "localhost:9090"
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [prometheus]
Prometheus running as docker container
docker-compose.yml
prometheus:
image: prom/prometheus:v2.17.1
container_name: prometheus
volumes:
- ./prometheus:/etc/prometheus
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.retention.time=200h'
- '--web.enable-lifecycle'
restart: unless-stopped
expose:
- 9090
prometheus.yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
Scenario
I start the OTEL Collector and Prometheus using docker
I run the following command below to auto instrument my spring java app
java -javaagent:./opentelemetry-javaagent.jar \
-Dotel.javaagent.extensions=./opentelemetry-micrometer-1.5-1.13.0-alpha.jar \
-Dotel.instrumentation.spring-boot-actuator-autoconfigure.enabled=false \
-Dotel.javaagent.debug=true \
-Dotel.metrics.exporter=prometheus \
-Dotel.traces.exporter=none \
-Dotel.exporter.prometheus.metrics.endpoint=http://127.0.0.1:4317 \
-Dotel.exporter.prometheus.port=9080 \
-Dotel.resource.attributes="service.name=helloapp" \
-Dotel.instrumentation.micrometer.base-time-unit=s \
-Dotel.instrumentation.micrometer.prometheus-mode.enabled=true \
-jar build/libs/hello-0.0.1-SNAPSHOT.jar
Issue:
Prometheus metrics exported by Instrumentation agent are different compared to Metrics received by collector and subsequently forwarded to Prometheus docker app
Expected prometheus metrics
# TYPE process_runtime_jvm_system_cpu_utilization gauge
# HELP process_runtime_jvm_system_cpu_utilization Recent cpu utilization for the whole system
process_runtime_jvm_system_cpu_utilization 0.0 1660037708660
# TYPE process_runtime_jvm_memory_usage gauge
# HELP process_runtime_jvm_memory_usage Measure of memory used process_runtime_jvm_memory_usage{pool="Metaspace",type="non_heap"}
4.289408E7 1660037708660 process_runtime_jvm_memory_usage{pool="G1 Eden Space",type="heap"} 6291456.0 1660037708660
process_runtime_jvm_memory_usage{pool="G1 Old Gen",type="heap"}
2.0279808E7 1660037708660 process_runtime_jvm_memory_usage{pool="Compressed Class
Space",type="non_heap"} 5806728.0 1660037708660
process_runtime_jvm_memory_usage{pool="CodeHeap 'profiled
nmethods'",type="non_heap"} 1.2102912E7 1660037708660
process_runtime_jvm_memory_usage{pool="CodeHeap
'non-nmethods'",type="non_heap"} 1271552.0 1660037708660
process_runtime_jvm_memory_usage{pool="G1 Survivor Space",type="heap"}
5671744.0 1660037708660 process_runtime_jvm_memory_usage{pool="CodeHeap 'non-profiled
nmethods'",type="non_heap"} 3989888.0 1660037708660
# TYPE process_runtime_jvm_threads_count gauge
# HELP process_runtime_jvm_threads_count Number of executing threads process_runtime_jvm_threads_count 12.0 1660037708660
# TYPE process_runtime_jvm_memory_limit gauge
# HELP process_runtime_jvm_memory_limit Measure of max obtainable memory process_runtime_jvm_memory_limit{pool="G1 Old Gen",type="heap"}
4.294967296E9 1660037708660 process_runtime_jvm_memory_limit{pool="Compressed Class
Space",type="non_heap"} 1.073741824E9 1660037708660
process_runtime_jvm_memory_limit{pool="CodeHeap 'profiled
nmethods'",type="non_heap"} 1.22908672E8 1660037708660
process_runtime_jvm_memory_limit{pool="CodeHeap
'non-nmethods'",type="non_heap"} 5840896.0 1660037708660
process_runtime_jvm_memory_limit{pool="CodeHeap 'non-profiled
nmethods'",type="non_heap"} 1.22908672E8 1660037708660
# TYPE process_runtime_jvm_memory_init gauge
# HELP process_runtime_jvm_memory_init Measure of initial memory requested
process_runtime_jvm_memory_init{pool="Metaspace",type="non_heap"} 0.0
1660037708660 process_runtime_jvm_memory_init{pool="G1 Eden
Space",type="heap"} 2.5165824E7 1660037708660
process_runtime_jvm_memory_init{pool="G1 Old Gen",type="heap"}
2.43269632E8 1660037708660 process_runtime_jvm_memory_init{pool="Compressed Class
Space",type="non_heap"} 0.0 1660037708660
process_runtime_jvm_memory_init{pool="CodeHeap 'profiled
nmethods'",type="non_heap"} 2555904.0 1660037708660
process_runtime_jvm_memory_init{pool="CodeHeap
'non-nmethods'",type="non_heap"} 2555904.0 1660037708660
process_runtime_jvm_memory_init{pool="G1 Survivor Space",type="heap"}
0.0 1660037708660 process_runtime_jvm_memory_init{pool="CodeHeap 'non-profiled nmethods'",type="non_heap"} 2555904.0 1660037708660
# TYPE process_runtime_jvm_classes_loaded_total counter
# HELP process_runtime_jvm_classes_loaded_total Number of classes loaded since JVM start process_runtime_jvm_classes_loaded_total 9284.0
1660037708660
# TYPE process_runtime_jvm_memory_committed gauge
# HELP process_runtime_jvm_memory_committed Measure of memory committed
process_runtime_jvm_memory_committed{pool="Metaspace",type="non_heap"}
4.3384832E7 1660037708660 process_runtime_jvm_memory_committed{pool="G1 Eden Space",type="heap"}
5.24288E7 1660037708660 process_runtime_jvm_memory_committed{pool="G1 Old Gen",type="heap"} 3.7748736E7 1660037708660
process_runtime_jvm_memory_committed{pool="Compressed Class
Space",type="non_heap"} 6029312.0 1660037708660
process_runtime_jvm_memory_committed{pool="CodeHeap 'profiled
nmethods'",type="non_heap"} 1.212416E7 1660037708660
process_runtime_jvm_memory_committed{pool="CodeHeap
'non-nmethods'",type="non_heap"} 2555904.0 1660037708660
process_runtime_jvm_memory_committed{pool="G1 Survivor
Space",type="heap"} 6291456.0 1660037708660
process_runtime_jvm_memory_committed{pool="CodeHeap 'non-profiled
nmethods'",type="non_heap"} 3997696.0 1660037708660
# TYPE process_runtime_jvm_classes_current_loaded gauge
# HELP process_runtime_jvm_classes_current_loaded Number of classes currently loaded process_runtime_jvm_classes_current_loaded 9310.0
1660037708660
# TYPE process_runtime_jvm_cpu_utilization gauge
# HELP process_runtime_jvm_cpu_utilization Recent cpu utilization for the process process_runtime_jvm_cpu_utilization 0.0 1660037708660
# TYPE process_runtime_jvm_classes_unloaded_total counter
# HELP process_runtime_jvm_classes_unloaded_total Number of classes unloaded since JVM start process_runtime_jvm_classes_unloaded_total
1.0 1660037708660
# TYPE process_runtime_jvm_system_cpu_load_1m gauge
# HELP process_runtime_jvm_system_cpu_load_1m Average CPU load of the whole system for the last minute
process_runtime_jvm_system_cpu_load_1m 2.15087890625 1660037708660
# TYPE system_cpu_usage gauge
# HELP system_cpu_usage The "recent cpu usage" of the system the application is running in system_cpu_usage 0.4666666666666667
1660037708660
# TYPE jvm_threads_states_threads gauge
# HELP jvm_threads_states_threads The current number of threads having NEW state jvm_threads_states_threads{state="runnable"} 6.0
1660037708660 jvm_threads_states_threads{state="timed-waiting"} 4.0
1660037708660 jvm_threads_states_threads{state="terminated"} 0.0
1660037708660 jvm_threads_states_threads{state="new"} 0.0
1660037708660 jvm_threads_states_threads{state="blocked"} 0.0
1660037708660 jvm_threads_states_threads{state="waiting"} 2.0
1660037708660
# TYPE jvm_gc_max_data_size_bytes gauge
# HELP jvm_gc_max_data_size_bytes Max size of long-lived heap memory pool jvm_gc_max_data_size_bytes 4.294967296E9 1660037708660
# TYPE system_cpu_count gauge
# HELP system_cpu_count The number of processors available to the Java virtual machine system_cpu_count 8.0 1660037708660
# TYPE jvm_threads_live_threads gauge
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads jvm_threads_live_threads
12.0 1660037708660
# TYPE jvm_threads_daemon_threads gauge
# HELP jvm_threads_daemon_threads The current number of live daemon threads jvm_threads_daemon_threads 8.0 1660037708660
# TYPE jvm_memory_usage_after_gc_percent gauge
# HELP jvm_memory_usage_after_gc_percent The percentage of long-lived heap pool used after the last GC event, in the range [0..1]
jvm_memory_usage_after_gc_percent{area="heap",pool="long-lived"}
0.0047217607498168945 1660037708660
# TYPE jvm_gc_memory_allocated_bytes_total counter
# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the (young) heap memory pool after one GC to before the
next jvm_gc_memory_allocated_bytes_total 5.0331648E7 1660037708660
# TYPE jvm_gc_overhead_percent gauge
# HELP jvm_gc_overhead_percent An approximation of the percent of CPU time used by GC activities over the last lookback period or since
monitoring began, whichever is shorter, in the range [0..1]
jvm_gc_overhead_percent 0.0034007331860589907 1660037708660
# TYPE disk_free_bytes gauge
# HELP disk_free_bytes Usable space for path disk_free_bytes{path="/Users/anuragk/workspace/ts/demo/otel/java/hello/."} 1.89101027328E11 1660037708660
# TYPE jvm_memory_committed_bytes gauge
# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
jvm_memory_committed_bytes{area="nonheap",id="Compressed Class Space"}
6029312.0 1660037708660 jvm_memory_committed_bytes{area="nonheap",id="Metaspace"} 4.358144E7
1660037708660 jvm_memory_committed_bytes{area="nonheap",id="CodeHeap
'profiled nmethods'"} 1.212416E7 1660037708660
jvm_memory_committed_bytes{area="heap",id="G1 Survivor Space"}
6291456.0 1660037708660 jvm_memory_committed_bytes{area="heap",id="G1 Eden Space"} 5.24288E7 1660037708660
jvm_memory_committed_bytes{area="nonheap",id="CodeHeap
'non-nmethods'"} 2555904.0 1660037708660
jvm_memory_committed_bytes{area="heap",id="G1 Old Gen"} 3.7748736E7
1660037708660 jvm_memory_committed_bytes{area="nonheap",id="CodeHeap
'non-profiled nmethods'"} 4063232.0 1660037708660
# TYPE jvm_gc_live_data_size_bytes gauge
# HELP jvm_gc_live_data_size_bytes Size of long-lived heap memory pool after reclamation jvm_gc_live_data_size_bytes 0.0 1660037708660
# TYPE system_load_average_1m gauge
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable
entities running on the available processors averaged over a period of
time system_load_average_1m 2.15087890625 1660037708660
# TYPE process_uptime_seconds gauge
# HELP process_uptime_seconds The uptime of the Java virtual machine process_uptime_seconds 11.542 1660037708660
# TYPE jvm_buffer_memory_used_bytes gauge
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
jvm_buffer_memory_used_bytes{id="mapped - 'non-volatile memory'"} 0.0
1660037708660 jvm_buffer_memory_used_bytes{id="mapped"} 0.0
1660037708660 jvm_buffer_memory_used_bytes{id="direct"} 16384.0
1660037708660
# TYPE jvm_buffer_count_buffers gauge
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool jvm_buffer_count_buffers{id="mapped - 'non-volatile
memory'"} 0.0 1660037708660 jvm_buffer_count_buffers{id="mapped"} 0.0
1660037708660 jvm_buffer_count_buffers{id="direct"} 2.0 1660037708660
# TYPE process_cpu_usage gauge
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process process_cpu_usage 0.3913746438746439 1660037708660
# TYPE jvm_classes_unloaded_classes_total counter
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
jvm_classes_unloaded_classes_total 1.0 1660037708660
# TYPE process_files_max_files gauge
# HELP process_files_max_files The maximum file descriptor count process_files_max_files 10240.0 1660037708660
# TYPE process_files_open_files gauge
# HELP process_files_open_files The open file descriptor count process_files_open_files 19.0 1660037708660
# TYPE jvm_threads_peak_threads gauge
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
jvm_threads_peak_threads 12.0 1660037708660
# TYPE process_start_time_seconds gauge
# HELP process_start_time_seconds Start time of the process since unix epoch. process_start_time_seconds 1.66003769719E9 1660037708660
# TYPE jvm_memory_max_bytes gauge
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
jvm_memory_max_bytes{area="nonheap",id="Compressed Class Space"}
1.073741824E9 1660037708660 jvm_memory_max_bytes{area="nonheap",id="Metaspace"} -1.0 1660037708660
jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'"}
1.22908672E8 1660037708660 jvm_memory_max_bytes{area="heap",id="G1 Survivor Space"} -1.0 1660037708660
jvm_memory_max_bytes{area="heap",id="G1 Eden Space"} -1.0
1660037708660 jvm_memory_max_bytes{area="nonheap",id="CodeHeap
'non-nmethods'"} 5840896.0 1660037708660
jvm_memory_max_bytes{area="heap",id="G1 Old Gen"} 4.294967296E9
1660037708660 jvm_memory_max_bytes{area="nonheap",id="CodeHeap
'non-profiled nmethods'"} 1.22908672E8 1660037708660
# TYPE jvm_gc_pause_seconds histogram
# HELP jvm_gc_pause_seconds Time spent in GC pause jvm_gc_pause_seconds_count{action="end of minor GC",cause="G1
Evacuation Pause"} 1.0 1660037708660
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="G1 Evacuation
Pause"} 0.003 1660037708660 jvm_gc_pause_seconds_bucket{action="end of
minor GC",cause="G1 Evacuation Pause",le="5.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="10.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="25.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="50.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="75.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="100.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="250.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="500.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="750.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="1000.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="2500.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="5000.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="7500.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="10000.0"} 1.0 1660037708660
jvm_gc_pause_seconds_bucket{action="end of minor GC",cause="G1
Evacuation Pause",le="+Inf"} 1.0 1660037708660
# TYPE jvm_classes_loaded_classes gauge
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
jvm_classes_loaded_classes 9409.0 1660037708660
# TYPE jvm_memory_used_bytes gauge
# HELP jvm_memory_used_bytes The amount of used memory jvm_memory_used_bytes{area="nonheap",id="Compressed Class Space"}
5853584.0 1660037708660 jvm_memory_used_bytes{area="nonheap",id="Metaspace"} 4.315976E7
1660037708660 jvm_memory_used_bytes{area="nonheap",id="CodeHeap
'profiled nmethods'"} 1.2106752E7 1660037708660
jvm_memory_used_bytes{area="heap",id="G1 Survivor Space"} 5671744.0
1660037708660 jvm_memory_used_bytes{area="heap",id="G1 Eden Space"}
8388608.0 1660037708660 jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-nmethods'"}
1277696.0 1660037708660 jvm_memory_used_bytes{area="heap",id="G1 Old Gen"} 2.0279808E7 1660037708660
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-profiled
nmethods'"} 4001920.0 1660037708660
# TYPE disk_total_bytes gauge
# HELP disk_total_bytes Total space for path disk_total_bytes{path="/Users/anuragk/workspace/ts/demo/otel/java/hello/."}
4.94384795648E11 1660037708660
# TYPE jvm_buffer_total_capacity_bytes gauge
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
jvm_buffer_total_capacity_bytes{id="mapped - 'non-volatile memory'"}
0.0 1660037708660 jvm_buffer_total_capacity_bytes{id="mapped"} 0.0 1660037708660 jvm_buffer_total_capacity_bytes{id="direct"} 16384.0
1660037708660 ```
Received metrics
# HELP otelcol_exporter_enqueue_failed_log_records Number of log records failed to be added to the sending queue.
# TYPE otelcol_exporter_enqueue_failed_log_records counter otelcol_exporter_enqueue_failed_log_records{exporter="prometheus",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
0
# HELP otelcol_exporter_enqueue_failed_metric_points Number of metric points failed to be added to the sending queue.
# TYPE otelcol_exporter_enqueue_failed_metric_points counter otelcol_exporter_enqueue_failed_metric_points{exporter="prometheus",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
0
# HELP otelcol_exporter_enqueue_failed_spans Number of spans failed to be added to the sending queue.
# TYPE otelcol_exporter_enqueue_failed_spans counter otelcol_exporter_enqueue_failed_spans{exporter="prometheus",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
0
# HELP otelcol_exporter_sent_metric_points Number of metric points successfully sent to destination.
# TYPE otelcol_exporter_sent_metric_points counter otelcol_exporter_sent_metric_points{exporter="prometheus",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
5576
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds counter otelcol_process_cpu_seconds{service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
2.17
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge otelcol_process_memory_rss{service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
6.5077248e+07
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge otelcol_process_runtime_heap_alloc_bytes{service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
9.784448e+06
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes counter otelcol_process_runtime_total_alloc_bytes{service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
3.9931936e+07
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
2.944308e+07
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter otelcol_process_uptime{service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
388.883801875
# HELP otelcol_processor_batch_batch_send_size Number of units in the batch
# TYPE otelcol_processor_batch_batch_send_size histogram otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="10"}
26
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="25"}
28
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="50"}
32
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="75"}
32
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="100"}
32
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="250"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="500"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="750"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="1000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="2000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="3000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="4000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="5000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="6000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="7000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="8000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="9000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="10000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="20000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="30000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="50000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="100000"}
75
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",le="+Inf"}
75
otelcol_processor_batch_batch_send_size_sum{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
5576.000000000002 otelcol_processor_batch_batch_send_size_count{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
75
# HELP otelcol_processor_batch_timeout_trigger_send Number of times the batch was sent due to a timeout trigger
# TYPE otelcol_processor_batch_timeout_trigger_send counter otelcol_processor_batch_timeout_trigger_send{processor="batch",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0"}
75
# HELP otelcol_receiver_accepted_metric_points Number of metric points successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_metric_points counter otelcol_receiver_accepted_metric_points{receiver="prometheus",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",transport="http"}
5576
# HELP otelcol_receiver_refused_metric_points Number of metric points that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_metric_points counter otelcol_receiver_refused_metric_points{receiver="prometheus",service_instance_id="57f37a9a-5825-48b3-a8a2-240ee4c52656",service_version="0.56.0",transport="http"}
0 ```

Answer 1

I also added io.opentelemetry.instrumentation:opentelemetry-runtime-metrics as a runtime library in my application. Then I get some metrics like:
process_runtime_jvm_memory_committed{otel_scope_name="io.opentelemetry.runtime-metrics",otel_scope_version="1.22.1-alpha",pool="CodeHeap 'non-nmethods'",type="non_heap"} 2555904.0 1675781958779
...
process_runtime_jvm_memory_init{otel_scope_name="io.opentelemetry.runtime-metrics",otel_scope_version="1.22.1-alpha",pool="CodeHeap 'non-nmethods'",type="non_heap"} 2555904.0 1675781958779
...
process_runtime_jvm_memory_limit{otel_scope_name="io.opentelemetry.runtime-metrics",otel_scope_version="1.22.1-alpha",pool="CodeHeap 'non-nmethods'",type="non_heap"} 5836800.0 1675781958779
...
process_runtime_jvm_system_cpu_load_1m{otel_scope_name="io.opentelemetry.runtime-metrics",otel_scope_version="1.22.1-alpha"} 0.63 1675781958779
process_runtime_jvm_system_cpu_utilization{otel_scope_name="io.opentelemetry.runtime-metrics",otel_scope_version="1.22.1-alpha"} 0.009925090776389015 1675781958779
process_runtime_jvm_threads_count{otel_scope_name="io.opentelemetry.runtime-metrics",otel_scope_version="1.22.1-alpha",daemon="false"} 1.0 1675781958779
process_runtime_jvm_threads_count{otel_scope_name="io.opentelemetry.runtime-metrics",otel_scope_version="1.22.1-alpha",daemon="true"} 21.0 1675781958779
Still missing metrics like jvm_memory_used_bytes though