There is a health endpoint which indicates the status of other health indicatiors and as well the main status. My question is:
Is the top status: "UP" just a summary of other health indicators, or it actually can indicate "DOWN" for some other reason?
Is this the actual application health?
{
status: "UP",
jms: {
status: "UP",
provider: "ActiveMQ"
},
diskSpace: {
status: "UP",
total: 255179702272,
free: 78310952960,
threshold: 10485760
},
db: {
status: "UP",
database: "Oracle",
hello: "Hello"
}
}
It simply aggregates (via the configured HealthAggregator) the statuses of all the configured HealthIndicators.
You can provide a custom implementation if you wanted it to do something else.
Related
I've configured a heartbeat with elasticsearch and I would like to know if it's possible to "catch" the data returned by the json response.
Heartbeat :
- type: http
urls: ["https://myurl.com"]
fields_under_root: true
fields:
application: myapplication
schedule: '#every 1m'
check.response:
status: [200]
Json response :
{
"status": "UP",
"components": {
"ping": {
"status": "UP"
},
"MyService": {
"status": "UP"
}
}
}
In addition to the result of the "check response" is it possible with the heartbeat to add a field with the value "UP" of the ping status for example ?
Thanks for your help
In Spring-Boot 2.4, I have this problem with the Actuator health endpoint and readiness probe. When one of my custom key components is down, the /health/readiness endpoint says DOWN and the /health endpoint too, but the readinessState detail of /health still says UP.
Why is it that way? Shouldn't readinessState say DOWN too?
None of the many tutorials I found online seem to address this question.
My hypothesis: the readinessState has nothing to do with readiness and exposes another piece of information. I hope I'm wrong, because it would be nonesense and what I understand of the code seems to indicate otherwise.
More about my configuration:
Relevant excerpt from application.yml
management:
endpoints:
web:
base-path: /
endpoint:
health:
show-details: ALWAYS
probes:
enabled: true
group:
readiness:
include: db, myCustom, diskSpace
And when I make myCustom go DOWN, following results appear:
GET /health
{
"status": "DOWN",
"components": {
..., // other components
"myCustom": {
"status": "DOWN"
},
"readinessState": {
"status": "UP" // here UP
}
},
"groups": [
"liveness",
"readiness"
]
}
GET /health/readiness
{
"status": "DOWN", // but here DOWN
"components": {
..., // other components
"myCustom": {
"status": "DOWN"
}
}
}
The readiness state only monitors specific health groups. It needs to be told about your custom component.
By default, Spring Boot does not add other Health Indicators to these groups.
https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready-kubernetes-probes-external-state
Imagine this is my http://localhost:8080/actuator ouotput:
{
"_links": {
"self": {
"href": "http://localhost:8080/actuator",
"templated": false
},
"health": {
"href": "http://localhost:8080/actuator/health",
"templated": false
},
"prometheus": {
"href": "http://localhost:8080/actuator/prometheus",
"templated": false
},
"httptrace": {
"href": "http://localhost:8080/actuator/httptrace",
"templated": false
}
}
}
Now I've hooked up my prometheus environment to /actuator/prometheus and that works fine. I als want prometheus to read my httptrace so I also added /actuator/httptrace to my prometheus config. However this does not work. The formatting on the httptrace endpoint is in json and the formatting in prometheus is in yaml, I think I need the httptrace in the prometheus yaml. Prometheus eats the yaml just fine, the json not so much.
How can I pass my httptrace to actuator/prometheus from my spring boot project? In the end my goal is to get the timeTaken value for every request in grafana.
Spring's HttpTraceRepository exposes the recent traces on an endpoint but not in prometheus as percentiles. So I can suggest two paths:
You implement your own HttpTraceRepository that wraps the one your using (default is InMemory....) and then override the method to fire off some custom Timer from [io.micrometer.core] with the timing (https://micrometer.io/docs/concepts#_timers) which will get aggregated as percentiles if you also enable via https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.metrics.customizing.per-meter-properties
In the end my goal is to get the timeTaken value for every request in grafana.
Just use http_server_requests_seconds_* that are captured per endpoint (not request)
http_server_requests_seconds_count
is the total number of requests your application received at this endpoint
http_server_requests_seconds_sum
is the sum of the the duration of every request your application received at this endpoint
and this great post explains how to use it in prometheus, https://tomgregory.com/spring-boot-default-metrics/
Trying to restore an v5.3 ES snapshot from S3 to ES 6.2. Snapshot bucket is in us-east-1 and i'm trying to restore it to a Amazon ES cluster in us-west-2. It's a cross-account, cross-region restore operation.
Registered snapshot repository in the us-west-2 ES cluster as below
{
"type": "s3",
"settings": {
"bucket": "valid-bucket-name",
"server_side_encryption": "true",
"endpoint": "s3.amazonaws.com",
"region" : "us-east-1",
"role_arn": "valid-role"
}
}
Got the response as
{
"acknowledged": true
}
But, then when i try to restore a specific snapshot, S3 throws 301
{
"error": {
"root_cause": [
{
"type": "amazon_s3_exception",
"reason": "amazon_s3_exception: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: D72EFE8A89F76F57; S3 Extended Request ID: in03KW452re297MDp3GQQRFjJhMRXeP4md+FU99CHZ7D4TQKz8PBuSZKoO3+IFd+wAxNApztG5Y=)"
}
],
"type": "amazon_s3_exception",
"reason": "amazon_s3_exception: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: D72EFE8A89F76F57; S3 Extended Request ID: in03KW452re297MDp3GQQRFjJhMRXeP4md+FU99CHZ7D4TQKz8PBuSZKoO3+IFd+wAxNApztG5Y=)"
},
"status": 500
}
Repository is already configured with region us-east-1. Error message is not useful.
If i just specify the endpoint as #Michael-sqlbot suggested, then it will throw the following error
{
"error": {
"root_cause": [
{
"type": "amazon_s3_exception",
"reason": "amazon_s3_exception: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: CC1853D0EF68B5F7; S3 Extended Request ID: nrbWmI3OiPLdrMcRT6FiOHJineYv6clmSf+GcXtBBwKSzfIEV2gmMZjWEDtyCIRQUg+dM/Vmawg=)"
}
],
"type": "blob_store_exception",
"reason": "Failed to check if blob [master.dat-temp] exists",
"caused_by": {
"type": "amazon_s3_exception",
"reason": "amazon_s3_exception: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: CC1853D0EF68B5F7; S3 Extended Request ID: nrbWmI3OiPLdrMcRT6FiOHJineYv6clmSf+GcXtBBwKSzfIEV2gmMZjWEDtyCIRQUg+dM/Vmawg=)"
}
},
"status": 500
}
Update: I can confirm that it's a region/endpoint related issue with s3-snapshot-plugin. Created another cluster in us-east-1 (same region as that of bucket) and it worked with out any issue.
This seems potentially relevant:
Important
If the S3 bucket is in the us-east-1 region, you need to use "endpoint": "s3.amazonaws.com" instead of "region": "us-east-1". (emphasis added)
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-snapshots.html
Cannot comment on Michael's answer due to lack of reputation :-)
I tried their suggested workaround to set up a snapshot repo on a new 7.9 cluster in us-west-2 with a bucket located in eu-west-2 and it worked! Doesn't seem to be documented anymore though on Amazon's developer guide. I'll submit feedback to them.
BTW, if you're trying to migrate data from another cluster, don't forget to add "readonly": true to the settings object. This is just to prevent accidentally writing to that repo.
This question already has answers here:
spring-boot health not showing details (withDetail info)
(7 answers)
Closed 4 years ago.
Spring being all auto auto-magic, I was a bit suprised at how I am observing the HealthIndicator to function.
I expected that, when I have a HealthIndicator bean in context that it will aggregate it along with the other ones it has and produce a summary in /actuator/health. Instead what it does it ignore all the ones but my custom one. I ended up implementing my own endpoint aggregating them.
I must be missing something. Either I am not enabling this functionality correctly (and I admit it's confusing) or I am expecting behavior that isn't what it actually should do.
My Question: Are my expectations correct? If So, how do I make it auto-aggregate? If my expectations are incorrect, what's the idiomatic way for actuator to work as I desire?
Here is my sample code:
#Component
public class HelloWorldHealthIndicator implements HealthIndicator {
#Override
public Health health() {
return Health.up().withDetail("hello","world").build();
}
}
/actuator/health
{
"status": "UP"
}
It's also not printing "hello" : "world" as I would have expected but that's another matter.
Here is my custom controller and output:
#RestController
public class CustomController {
private final Map<String,HealthIndicator> indicators;
#Autowired
public HealthController(Map<String, HealthIndicator> indicators) {
this.indicators = indicators;
}
#GetMapping("/health")
public Health getHealth(#RequestParam("deep") Boolean deep){
if(deep != null && deep)
return new CompositeHealthIndicator(new OrderedHealthAggregator(),indicators).health();
return Health.up().build();
}
}
/health?deep=true
{
"status": "UP",
"details": {
"helloWorldHealthIndicator": {
"status": "UP",
"details": {
"hello": "world"
}
},
"diskSpaceHealthIndicator": {
"status": "UP",
"details": {
"total": 74865782784,
"free": 65754009600,
"threshold": 10485760
}
},
"dbHealthIndicator": {
"status": "UP",
"details": {
"database": "MySQL",
"hello": 1
}
}
}
}
What makes you think your custom HealthIndicator is not invoked? Have you tried to run an app with the debugger to check if it was invoked?
That output doesn't tell us anything: you may simply get the summarized view of all the health indicators (no detail at all)
The doc states
Health information is collected from all HealthIndicator beans defined in your ApplicationContext. Spring Boot includes a number of auto-configured HealthIndicators and you can also write your own.
That code above should work just fine. My best guess is that the details aren't displayed (check the doc to enable that and please note that they are differences between Spring Boot 1 and Spring Boot 2).
Edit: To demonstrate that, I've created a sample project that uses the exact same code as yours and I get this when I invoke health (with security disabled so that details are shown to anonymous users):
{
"status": "UP",
"helloWorld": {
"status": "UP",
"hello": "world"
},
"diskSpace": {
"status": "UP",
"total": 499963170816,
"free": 217149034496,
"threshold": 10485760
}
}