We are using spring boot actuator to get health status of an application, my understanding is that request for health check will be handled by thread out of thread pool that is used to serve actual service requests.
Is there a way to limit number of requests for health endpoint to prevent a DDOS type starvation.
You can use Spring Boot Throttling community library. I think you could restrict DDOS access to your endpoints (Actuator or otherwise) using it's configuration.
https://github.com/weddini/spring-boot-throttling
Another possibility to reduce DDOS vulnerability on the /health endpoint is to have your health checks run on a separate thread pool.
This ensures that:
no more than one health indicator concurrently runs at any given time against an underlying service
your /health endpoint returns instantly (as it returns healths pre-calculated on different threads).
For this purpose, and if you are using Spring Boot >= 2.2, you can use the separate library spring-boot-async-health-indicator to run your healthchecks on a separate thread pool by simply annotating them with #AsyncHealth.
Disclaimer: I created this library to address this issue (among others)
Related
We have a Spring Boot microservice which should get some data from old / legacy system. This microservice exposes external modern REST API. Sometimes we have to issue 7-10 requests to the legacy system in order to get all the data we need for single API call. Unfortunately we can't use Reactor / WebClient and have to stick with WebServiceTemplate to issue those "legacy" calls. We can't also use Reactive Spring WebClient - Making a SOAP call
What is the best way to scale such a miroservice in Kubernetes? We have very big concerns that Thread Pool used for parallel WebServiceTemplate invocation will be depleted very fast, but I'm not sure that creating and exposing custom metric based on active threads count / thread pool size is a good idea.
Any advice will be helpful.
Enable Prometheus exporter in Spring
Make sure metrics are scraped. You're going to watch for a threadpool_size metric. Refer your k8s/prometheus distro docs to get prometheus service discovery working for you.
Write a horizontal pod autoscaler (HPA) based on a Prometheus metric:
Setup Prometheus-Adapter and follow the HPA walkthrough.
Or follow this guide https://github.com/stefanprodan/k8s-prom-hpa
Depending on what k8s distro you are using, you might have different ways to get the Prometheus and prometheus discovery:
(example platform built-in) https://cloud.google.com/stackdriver/docs/solutions/gke/prometheus
(example product) https://docs.datadoghq.com/integrations/prometheus/
(example opensource) https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
any other prometheus solution
First time I'm trying out the actuator dependency, I configured it to be opt-in
management.endpoints.enabled-by-default=false
management.endpoint.info.enabled=true
management.endpoint.health.enabled=true
When I call /actuator/health locally it takes around 1.4 seconds to respond. Keep in mind this is a local call, from the same machine of the server.
If I create a regular endpoint that replies with an empty response, the request would take just a couple of milliseconds.
Is this normal? Can I make it reply faster?
Original Answer: https://stackoverflow.com/a/63666118/1861769
Basically health endpoint is implemented in a way that it contains a
list of all Spring beans that implement the interface HealthIndicator.
Each health indicator is responsible for supplying a health
information about one subsystem (examples of such subsystem are:disk,
postgres, mongo, etc.), spring boot comes with some predefined
HealthIndicators.
So that when the health endpoint is invoked, it iterates through this
list and gets the information about each subsystem and then constructs
the answer.
Hence you can place a break point in relevant health indicators
(assuming you know which subsystems are checked) and see what happens.
If you're looking for the HTTP entry point - the code that gets called
when you call http:///actuator/health (can vary depending
on your settings but you get the idea)`, it can be found here
Yet another approach that comes to mind is disabling "suspicious"
health check and finding the slow one by elimination.
For example, if you have an elastricsearch and would like to disable
it, use in the application.properties:
management.health.elasticsearch.enabled = false
Is it possible to handle actuator requests like health within a separate thread pool from the "main" application?
Why am I asking?
I've got an application that might sometimes use up all available threads, and the Kubernetes health check is failing due to the unavailability of a thread to compute the health endpoint request.
I want to make sure that every health request is processed no matter how much load the application is under.
I was thinking about maybe defining a separate thread pool for the actuators to operate with, but I am not sure how to do this.
We had a similar problem with some of our apps when running in Kubernetes. We looked at different ways of creating multiple tomcat connectors and changing the spring management port to get the desired affect, but never quite got it.
In the end, we attacked the root of the problem, which was resource starvation within the pod. We found that the apps experiencing the health check timeouts had lots of extra threads for various 3rd party thread pools. In some cases we had apps with close to 500 threads, so even under what we considered moderate load, the tomcat pools would get starved and couldn't handle new requests.
FWIW, the biggest culprit we found was the effect of CPU request on a pod and the JDK. When we didn't set any request, the JDK would see every CPU on the node when it queried for numbers of processors. We found there are lots of places in the Java ecosystem where number of processors is used to initialize different thread pools.
In our case, each node had 36 processors, and we found around 10-12 thread pools using this number to determine size...not hard to see to how an app could quickly grow to 500 threads.
I believe that switching to the nonblocking stack (Webflux) could solve your issue, should this be an option for you. If you rely on some blocking API (e.g. JDBC) you can publish it on a separate thread pool (e.g. Schedulers.elastic()). Thus, the HTTP request threads should always be available for processing the incoming trafic (including health check) and the long-running, blocking operations would be processed in a dedicated thread pool. I believe that similar effect should be possible using the asynchronous servlets API or anything that builds on top of it.
If you are using Spring Boot >= 2.2, you can use the separate library spring-boot-async-health-indicator to run your healthchecks on a separate thread pool.
Simply annotate your HealthIndicator with #AsyncHealth:
#AsyncHealth
#Component
public class AsynchronousHealthCheck implements HealthIndicator {
#Override
public Health health() { //will be executed on a separate thread pool
actualCheck();
return Health.up().build();
}
}
Disclaimer: I created this library for this exact purpose
I'm using Spring Boot for microservices, and I came accross and issue with load balancing.
Spring Actuator adds special health and metrics endpoint to the apps; with this, some basic information can be acquired from the running instances.
What I would like to do, is to a create a (reverse)proxy (e.g. with Zuul and/or Ribbon?), which creates a centralized load balancer, that selects instances by their health status.
For example, I have the following microservices
client
proxy (<- I would like to implement this)
server 1
server 2
When the client sends an http request to the proxy, the proxy should be able to decide, which of the to server instances has the least load, and forward request to that one.
Is there an easy way to do this?
Thanks,
krisy
If you want to make a choice on various load-data, you could implement custom HealthIndicators that accumulate some kind of 'load over time' data, use this in your load balancer to decide where to send traffic.
All custom health indicators will be picked up by spring-boot, and invoked on the actuator /health endpoint.
#Component
class LoadIndicator implements HealthIndicator {
#Override
Health health() {
def loadData = ... do stuff to gather whatever load
return Health.up()
.withDetail("load", loadData)
.build();
}
}
Perhaps you could already use some of spring-boots metrics already, there's multiple endpoints in the actuator. /beans, /trace, /metrics. Should be possible to find that data in your application too.
We typically ping /health very frequently in our highly available applications to determine when failover needs to happen. Spring Boot Actuator works well for this if the health indicators that are used don't make expensive calls to external dependencies like a database or web service. However, we like the ease of writing health indicators and how it plugs into the /health endpoint.
Is there any way to configure the Spring Boot Actuator such that only a subset of the indicators are executed in certain circumstances? If so, how?
Thanks!
You can control which health indicators are enabled using the management.health.<service>.enabled properties. For example, to switch off the database health check:
management.health.db.enabled: false
The full list of properties is available here. At the time of writing they are:
management.health.db.enabled
management.health.diskspace.enabled
management.health.mongo.enabled
management.health.rabbit.enabled
management.health.redis.enabled
management.health.solr.enabled
I'm in a similar situation right now. I just implemented a custom Endpoint which does the expensive health checks.
If you need the comfort of an HTTP endpoint you can also implement an AbstractEndpointMvcAdapter which does a similar HTTP status code mapping as Spring's HealthMvcEndpoint.
2021 Update
Health Groups provide the exact functionality you're asking about.
https://docs.spring.io/spring-boot/docs/2.5.3/reference/html/actuator.html#actuator.endpoints.health.groups
If you are using Spring Boot >= 2.2, you can use the separate library spring-boot-async-health-indicator to make your expensive healthchecks run on a separate thread by simply annotating them with #AsyncHealth.
This will ensure that your /health endpoint always return very fast and does not wait on those expensive healthchecks to complete.
Example:
#AsyncHealth
#Component
public class MyExpensiveHealthCheck implements HealthIndicator {
#Override
public Health health() {
verySlowCheck(); // This method does not run when /health is called
return Health.up().build();
}
}
Disclaimer: I created this library for this exact purpose