Why is the "readinessState" detail of /health different from the readiness probe status? - spring-boot

In Spring-Boot 2.4, I have this problem with the Actuator health endpoint and readiness probe. When one of my custom key components is down, the /health/readiness endpoint says DOWN and the /health endpoint too, but the readinessState detail of /health still says UP.
Why is it that way? Shouldn't readinessState say DOWN too?
None of the many tutorials I found online seem to address this question.
My hypothesis: the readinessState has nothing to do with readiness and exposes another piece of information. I hope I'm wrong, because it would be nonesense and what I understand of the code seems to indicate otherwise.
More about my configuration:
Relevant excerpt from application.yml
management:
endpoints:
web:
base-path: /
endpoint:
health:
show-details: ALWAYS
probes:
enabled: true
group:
readiness:
include: db, myCustom, diskSpace
And when I make myCustom go DOWN, following results appear:
GET /health
{
"status": "DOWN",
"components": {
..., // other components
"myCustom": {
"status": "DOWN"
},
"readinessState": {
"status": "UP" // here UP
}
},
"groups": [
"liveness",
"readiness"
]
}
GET /health/readiness
{
"status": "DOWN", // but here DOWN
"components": {
..., // other components
"myCustom": {
"status": "DOWN"
}
}
}

The readiness state only monitors specific health groups. It needs to be told about your custom component.
By default, Spring Boot does not add other Health Indicators to these groups.
https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready-kubernetes-probes-external-state

Related

How to import actuator httptrace in actuator prometheus? (actuator, spring boot, grafana)

Imagine this is my http://localhost:8080/actuator ouotput:
{
"_links": {
"self": {
"href": "http://localhost:8080/actuator",
"templated": false
},
"health": {
"href": "http://localhost:8080/actuator/health",
"templated": false
},
"prometheus": {
"href": "http://localhost:8080/actuator/prometheus",
"templated": false
},
"httptrace": {
"href": "http://localhost:8080/actuator/httptrace",
"templated": false
}
}
}
Now I've hooked up my prometheus environment to /actuator/prometheus and that works fine. I als want prometheus to read my httptrace so I also added /actuator/httptrace to my prometheus config. However this does not work. The formatting on the httptrace endpoint is in json and the formatting in prometheus is in yaml, I think I need the httptrace in the prometheus yaml. Prometheus eats the yaml just fine, the json not so much.
How can I pass my httptrace to actuator/prometheus from my spring boot project? In the end my goal is to get the timeTaken value for every request in grafana.
Spring's HttpTraceRepository exposes the recent traces on an endpoint but not in prometheus as percentiles. So I can suggest two paths:
You implement your own HttpTraceRepository that wraps the one your using (default is InMemory....) and then override the method to fire off some custom Timer from [io.micrometer.core] with the timing (https://micrometer.io/docs/concepts#_timers) which will get aggregated as percentiles if you also enable via https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.metrics.customizing.per-meter-properties
In the end my goal is to get the timeTaken value for every request in grafana.
Just use http_server_requests_seconds_* that are captured per endpoint (not request)
http_server_requests_seconds_count
is the total number of requests your application received at this endpoint
http_server_requests_seconds_sum
is the sum of the the duration of every request your application received at this endpoint
and this great post explains how to use it in prometheus, https://tomgregory.com/spring-boot-default-metrics/

Spring actuator CompositeHealthIndicator - how to automatically use/display all indicators on /health [duplicate]

This question already has answers here:
spring-boot health not showing details (withDetail info)
(7 answers)
Closed 4 years ago.
Spring being all auto auto-magic, I was a bit suprised at how I am observing the HealthIndicator to function.
I expected that, when I have a HealthIndicator bean in context that it will aggregate it along with the other ones it has and produce a summary in /actuator/health. Instead what it does it ignore all the ones but my custom one. I ended up implementing my own endpoint aggregating them.
I must be missing something. Either I am not enabling this functionality correctly (and I admit it's confusing) or I am expecting behavior that isn't what it actually should do.
My Question: Are my expectations correct? If So, how do I make it auto-aggregate? If my expectations are incorrect, what's the idiomatic way for actuator to work as I desire?
Here is my sample code:
#Component
public class HelloWorldHealthIndicator implements HealthIndicator {
#Override
public Health health() {
return Health.up().withDetail("hello","world").build();
}
}
/actuator/health
{
"status": "UP"
}
It's also not printing "hello" : "world" as I would have expected but that's another matter.
Here is my custom controller and output:
#RestController
public class CustomController {
private final Map<String,HealthIndicator> indicators;
#Autowired
public HealthController(Map<String, HealthIndicator> indicators) {
this.indicators = indicators;
}
#GetMapping("/health")
public Health getHealth(#RequestParam("deep") Boolean deep){
if(deep != null && deep)
return new CompositeHealthIndicator(new OrderedHealthAggregator(),indicators).health();
return Health.up().build();
}
}
/health?deep=true
{
"status": "UP",
"details": {
"helloWorldHealthIndicator": {
"status": "UP",
"details": {
"hello": "world"
}
},
"diskSpaceHealthIndicator": {
"status": "UP",
"details": {
"total": 74865782784,
"free": 65754009600,
"threshold": 10485760
}
},
"dbHealthIndicator": {
"status": "UP",
"details": {
"database": "MySQL",
"hello": 1
}
}
}
}
What makes you think your custom HealthIndicator is not invoked? Have you tried to run an app with the debugger to check if it was invoked?
That output doesn't tell us anything: you may simply get the summarized view of all the health indicators (no detail at all)
The doc states
Health information is collected from all HealthIndicator beans defined in your ApplicationContext. Spring Boot includes a number of auto-configured HealthIndicators and you can also write your own.
That code above should work just fine. My best guess is that the details aren't displayed (check the doc to enable that and please note that they are differences between Spring Boot 1 and Spring Boot 2).
Edit: To demonstrate that, I've created a sample project that uses the exact same code as yours and I get this when I invoke health (with security disabled so that details are shown to anonymous users):
{
"status": "UP",
"helloWorld": {
"status": "UP",
"hello": "world"
},
"diskSpace": {
"status": "UP",
"total": 499963170816,
"free": 217149034496,
"threshold": 10485760
}
}

What does the spring boot /health top status: UP indicates

There is a health endpoint which indicates the status of other health indicatiors and as well the main status. My question is:
Is the top status: "UP" just a summary of other health indicators, or it actually can indicate "DOWN" for some other reason?
Is this the actual application health?
{
status: "UP",
jms: {
status: "UP",
provider: "ActiveMQ"
},
diskSpace: {
status: "UP",
total: 255179702272,
free: 78310952960,
threshold: 10485760
},
db: {
status: "UP",
database: "Oracle",
hello: "Hello"
}
}
It simply aggregates (via the configured HealthAggregator) the statuses of all the configured HealthIndicators.
You can provide a custom implementation if you wanted it to do something else.

Spring Boot Actuator endpoint configuration doesn't seem to be working as expected

I have a very simple spring boot application. It is just a zuul reverse proxy. there is no security or anything other than basic settings to discover our services via eureka and path mapping on a per service basis. I'm trying to prevent our actuator endpoints from being publicly exposed but still want the health check endpoint to be used for our ELB but want do not want it to report on the health of all the services it is aware of (i want it to be sensitive). While trying to figure out what properties i need to set to get the expected behavior, i am experiencing very unexpected behavior.
For example, when i set the property endpoints.sensitive=true, this DOES NOT change the default value of the health check endpoint to be sensitive. This seems to go against what the documentation says.
http://docs.spring.io/spring-boot/docs/1.4.2.RELEASE/reference/htmlsingle/#production-ready-customizing-endpoints
Likewise, you can also choose to globally set the “sensitive” flag of
all endpoints. By default, the sensitive flag depends on the type of
endpoint (see the table above). For example, to mark all endpoints as
sensitive except info:
endpoints.sensitive=true
endpoints.info.sensitive=false
In fact, when running in debug, i never see the org.springframework.boot.actuate.endpoint.EndpointProperties#isSensitive get called.
To get health endpoint to be sensitive, i need to explicitly set the property endpoints.health.sensitive=true. Oddly, when this setting is provided, now org.springframework.boot.actuate.endpoint.EndpointProperties#isSensitive gets called.
So this is great, my health check endpoint is now just reporting UP or DOWN and nothing else. But now I want the health check endpoint to be the ONLY endpoint enabled. So i set endpoints.enabled=false and endpoints.health.enabled=true which should disable all the endpoints except health. However, this does not seem to be the case. In my instance, I am able to hit /routes, /resume, /pause, /hystrix.stream, and others. I was only able to determine this when i disabled all endpoints with endpoints.enabled=false and then enabled the actuator endpoint with endpoints.actuator.enabled=true and that allowed me to hit the actuator endpoint which then reported that these endpoints were enabled.
{
"links": [
{
"rel": "self",
"href": "http://localhost:9200/actuator"
},
{
"rel": "resume",
"href": "http://localhost:9200/resume"
},
{
"rel": "pause",
"href": "http://localhost:9200/pause"
},
{
"rel": "hystrix.stream",
"href": "http://localhost:9200/hystrix.stream"
},
{
"rel": "env",
"href": "http://localhost:9200/env"
},
{
"rel": "routes",
"href": "http://localhost:9200/routes"
},
{
"rel": "health",
"href": "http://localhost:9200/health"
},
{
"rel": "refresh",
"href": "http://localhost:9200/refresh"
},
{
"rel": "restart",
"href": "http://localhost:9200/restart"
}
]
}
I would have expected to ONLY see the two endpoints I explicitly enabled.
{
"links": [
{
"rel": "self",
"href": "http://localhost:9200/actuator"
},
{
"rel": "health",
"href": "http://localhost:9200/health"
}
]
}
disabling each endpoint individually does not seem remove them from the actuator endpoint but now when attempted to access, i get a "This endpoint is disabled" message which is an improvement. I however don't seem to be able to disable the routes or `hystrix.stream* endpoints as there seems to be no configuration that exposes this ability.
All this said, I am wondering if this is the expected behavior or is this a bug?
I ran into the same problem as you described here. Please check your spring boot version first! There was a bug that global 'endpoints.sensitive' settings did NOT take effect at some specified spring boot version. (Not sure about the version number exactly. It seems a refactor regression in spring boot.)
Here are some references.
Allow global sensitive override #4419
Spring Boot Actuator: setting all endpoints as sensitive makes all accessible #4368
After I updated my spring boot to version 1.3.0 RELEASE, the setting 'endpoints.sensitive = true' works for me correctly. Hopes it will also work for you. Good luck, man.

HATEOAS paths are invalid when using an API Gateway in a Spring Boot app

I have two spring boot applications where one of them is acting as an API Gateway (as discussed here Spring Example). The other which is wired into the first one is exposing a profile service using spring-data-rest (spring-data-neo4j-rest).
The first application is starting on port 8080 and is using zuul to route requests to the second as follows:
zuul:
routes:
profiles:
path: /profiles/**
url: http://localhost:8083/profiles/
This all works fine and requests to http://localhost:8080/profiles are being served from the second app. The problem though is that the HATEOAS links in the response are incorrect. The response from calling that second service are correct:
{
"_links": {
"self": {
"href": "http://localhost:8083/profiles{?page,size,sort}",
"templated": true
},
"search": {
"href": "http://localhost:8083/profiles/search"
}
},
"_embedded": {
"profiles": [
{
"name": "Andrew Rutter",
"_links": {
"self": {
"href": "http://localhost:8083/profiles/0"
}
}
},
{
"name": "Andrew Rutter",
"_links": {
"self": {
"href": "http://localhost:8083/profiles/1"
}
}
}
]
},
"page": {
"size": 20,
"totalElements": 2,
"totalPages": 1,
"number": 0
}
}
But when this comes back to my API Gateway, the links are being rewritten to
{
"name": "Andrew Rutter",
"_links": {
"self": {
"href": "http://localhost:8080/profiles/profiles/0"
}
}
}
Which is the gateway path alias plus the actual service base Uri. Am I missing a zuul option to disable that behavior and just leave the hateoas uri in place with a host adjustment. Or is there a way for my service behind the gateway to be wired to / rather then the default resource endpoint of /profiles (in this case) which would avoid the undesirable path being added in.
Thanks!
Zuul or Spring-Cloud adds the "X-Forwarded-Host" header to all the forwarded requests, which Spring-hateoas respects and modifies the links appropriately. To quote from Spring-Cloud docs:
The X-Forwarded-Host header added to the forwarded requests by
default. To turn it off set zuul.addProxyHeaders = false. The prefix
path is stripped by default, and the request to the backend picks up a
header "X-Forwarded-Prefix" ("/myusers" in the examples above).
You can try the recommended fix, which is to set the zuul.addProxyHeaders=false
I had exactly the same problem. Change your config as follows:
zuul:
routes:
profiles:
path: /profiles/**
url: http://localhost:8083
stripPrefix: false
This routes all requests going to the gateway matching "/profiles/**" to your back end server "http://localhost:8083" and leaves the prefix (in your case "/profiles" since that's what matched the route).
Zuul forwards to the /profiles contextPath.
Try setting this as configuration:
zuul:
routes:
profiles:
path: /profiles/**
url: http://localhost:8083/
After struggling some time with the same problem, finally I've tried zuul.addProxyHeaders = true and it works!
Links are not broken anymore.
In the demo app I used for SpringOne in my talk about Spring Data REST, I have the following configuration to both handle URI rewrites as well as adjust prefix headers set properly.
zuul:
routes:
api:
path: /api/**
serviceId: spring-a-gram-backend
stripPrefix: false
files:
path: /files/**
serviceId: spring-a-gram-mongodb-fileservice
stripPrefix: false
See it in full at https://github.com/gregturn/spring-a-gram/blob/master/spring-a-gram-frontend/src/main/resources/application.yml#L23-L32

Resources