I have resilience4j config and have managementconfig also.
management.endpoint.circuitbreakerevents.enabled=true
But I can't see any events because URL not found. http://localhost:8080/circuitbreakerevents
{"timestamp":"2022-09-07T13:23:29.661+00:00","status":404,"error":"Not Found","path":"/circuitbreakerevents"}
http://localhost:8080/health is working and I can see Circuit breaker is up
Related
I recently upgraded CAS to 6.4.6.x and noticed that the liveness/readiness probes will intermittently throw 403 error codes. It appears to be a threading issue in the Spring Security Filter Chain. I have validated with the barebone CAS images that this does not happen in the 6.3.x version but can repeat it rather easily with the 6.4.x version. My configuration has not changed after the upgrade and I'm following the documentation.
Endpoint Configuration:
# allow all by default
cas.monitor.endpoints.endpoint.defaults.access[0]=PERMIT
# enable the health endpoint
management.endpoints.enabled-by-default=true
management.endpoints.web.base-path=/actuator
management.endpoints.web.exposure.include=health
management.endpoint.health.enabled=true
Running load tests against the instance if I send 1 request at a time I get 200 responses. If I bump up the concurrency to 2 or more I'm able to reproduce the threading issue and some of the responses return with a 403 after getting picked up by the Spring Default Error Controller.
Setting a breakpoint on the Error Controller, I'm able to see the same thread in the logs essentially jump to two different points in the code path.
I've gone through the Pull Requests from 6.3.x to 6.4.x and nothing jumped out to me that might be causing this issue. I haven't seen any issues raised up in Spring Boot around the Actuator Health Points failing. I've bumped up Spring and Tomcat to the latest patch versions. Any thoughts on what could be causing this or other things I could try to determine how to fix it?
I was looking for the difference between spring hystrix and retryable in spring boot microservices as both are used to implement circuit breaker pattern.Thanks.
AFAIK, a circuit breaker is a component does not retry an operation that it's going to fail anyway.
Spring boot's #Retryable was not designed to work as a circuit breaker, but as an exponential-backoff retry mechanism.
Hystrix is a circuit breaker library, but unfortunately it's not longer maintained. You can use Resilience4J. It's inspired from Hystrix, but much better.
I have an existing project with the following microservice architect. Client --> API Gateway(Spring cloud using Hystrix as circuit breaker) --> UploadService. When uploading small file(POST /upload/video) everything is fine. But when the file is larger then the upload time is very long and Hystrix will be OPEN and return fallback.
Does anyone have practice for my case or how can I set up the timeout for only POST /upload/video request on Hystrix?
It appears that you need to configure a larger timeout in the Hystric client;
in your example this is the "API Gateway (Spring cloud using Hystrix as circuit breaker)"
I imagine that your code will look something like this:
HystrixCommand.Setter yourHystrixCommand; ... blah your HystrixCommand
HystrixCommandProperties.Setter hystrixCommandPropertiesSetter = HystrixCommandProperties.Setter();
hystrixCommandPropertiesSetter.withExecutionTimeoutInMilliseconds(yourDesiredTimeoutValue);
yourHystrixCommand.andCommandPropertiesDefaults(commandProperties);
Here is an introduction to Hystrix at Baeldung
Edit:
"Hystrix Client" here means the client software that is using Hystrix.
I am using Zuul as an api-gateway in a spring-cloud micro-service app, so that every access to api-gateway/some-service/a_route is redirected to /a_route in a generic way (the discovery is backed by consul).
I am trying to use Jaeger to instrument this system. And at this point I am using opentracing-spring-web-autoconfigure, because I cannot upgrade my spring boot/cloud version easily (I am using1.4.5.RELEASE Camden.SR7). So I just added the dependency, created the Jaeger tracer and redirect it to the docker all in one collector.
I have begin by instrumenting the gateway and It somewhat works => It generate span on the gateway, but all the route are marked :
apigateway-service: GET
and there is no information concerning the forwarded route at this level, the full route itself is store in a tag : http.url
"http://localhost:8080/collection-service/collections/projects/"
To be useful I would prefer to have span named :
apigateway-service: GET collection-service/collections/projects/
Can this be configured somewhere ?
No, it cannot, but it wouldn't hurt to open an issue there with this suggestion.
I want the following features in spring-cloud-Eureka backed microservices application.
1) Load balancing - if I have 3 nodes for one service, load balancing should happen between them
2)Retry logic - if one of the nodes did not respond, retry should happen for certain number ( eg 3. should be configurable) before falling back to another node.
3)circuit breaker - if for some reasons, all the 3 nodes of service is having some issue accessing db and throwing exceptions or not responding, the circuit should get open, fall back method called and circuit automatically closes after the services recovers.
Looking at many examples of Spring-cloud, I figured out
1) RestTemplate will help with option 1. but when RestTemplate access one instance of service and if the node fails, will it try with other two nodes?
2) Hystix will help with circuit breaker option (3 above). but if just one node is not responding, will it try other nodes, before opening up circuit and call fallback method. and will it automatically close circuit once the service recovers?
3) how to get retryLogic with spring-cloud? I do know about #Retryable annotation. But will it help in the following situation?
Retry with one node for 3 times and after it fails, try the next node 3 times and the last node 3 times before circuit breaker kicks in.
I see that all these configurations are available in spring cloud. but having a hard-time understanding how to configure for all these for efficient solution.
Here is one proposed:
#HystrixCommand
#Retryable
public Object doSomething() {
// use your RestTemplate here
}
But I don't totally know if it is going to help me with all the subtleties I mentioned above.
I do see there is a #FeignClient. But from this blog, I understand that it provides a high level feature for HTTP client requests. Does it help with retry and circuit breaker and load balancing all-in-one?
Thanks
I do see there is a #FeignClient. Does it help with retry and circuit breaker and load balancing all-in-one?
If you are using the full spring-cloud stack, it actually solves everything you mentioned.
The netflix components in this scenario are the following in spring-cloud:
Eureka - Service Registry
Let's you dyanmically register your services so you only need to fix one host in your app (eureka).
Ribbon - Load balancer
Out of the box it's providing you with round robin loadbalancing, but you can implement your own #RibbonClient (even for a specific service) and design your custom loadbalancing for example based on eureka metadata. The loadbalancing happens on the client side.
Feign - Http client
With #FeignClient you can rapidly develop clients for you other services (or services outside of your infrastructure). It is integrated with ribbon and eureka so you can refer to your services #FeignClient(yourServiceNameInEureka) and what you end up with is a client which loadbalances between the registered instances with your preferred logic. If you are using spring you can use the familiar #RequestMapping annotation to describe the endpoint you are using.
Hystrix - Circuit breaker
By default your feign clients will use hystrix, every request will be wrapped in a hystrix command. You can of course create hytrix commands by hand and configure them for your needs.
You have to configure a little to get thees working (actually just a few #Enable annotation on your configuration).
I highly recommend reading the provided spring documentation because it wraps up almost all of your aspects in a fairly quick read.
http://cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.html