"INVALID" is not a valid start token - spring-boot

Question

Can't for the life of me figure out why I'm getting this error for one of the applications I'm trying to scrape. I have the following prometheus.yml:
# prometheus.yml
global:
scrape_interval: 5s
external_labels:
monitor: 'my-monitor'
scrape_configs:
- job_name: 'prometheus'
metrics_path: /metrics
static_configs:
- targets: ['localhost:9090']
- job_name: 'app'
metrics_path: /custom_metrics
static_configs:
- targets: ['host.docker.internal:8888']
basic_auth:
username: user
password: pass
- job_name: 'otherapp'
metrics_path: /custom_metrics
static_configs:
- targets: ['host.docker.internal:8081']
basic_auth:
username: admin
password: admin
I'm running prometheus with docker and both applications are running on my local host. 'app' is being scraped fine but I'm seeing the following in the targets section of the Prometheus UI:
I've run the above configuration against promtool check config with no complaints and I've also run the metrics from both of my applications through promtool check metrics with no complaints other than warnings about missing 'help'
I can hit the urls from the screen shot locally just fine and use the basic_auth username and pass from the config to see the metrics.
At this point I'm at a loss as to how to proceed. Is there something wrong with my config? Is there something that might be wrong with my metrics that promtool wouldn't catch?

Answer 1

One of my applications was configuring Spring Security to use basic auth and the other wasn't. The one that wasn't was showing
“INVALID” is not a valid start token
Until I added the following to my security configuration
.httpBasic()
I was surprised an authentication issue would have surfaced this way based on everything I had read about this error so maybe this will help others who are also surprised by this in the future.

Answer 2

If anyone else encounters this issue, check if your app gzip compresses the metrics response. I didn't notice this, since curl does not add the Accept-Encoding header to the request by default.
After adding the encoding header, I realized that the metrics were actually compressed and the promtool check failed:
curl -s -H 'Accept-Encoding: gzip,deflate' http://localhost:8000/metrics | promtool check metrics
I removed the gzip compression from the /metrics endpoint and it worked.