resilience4j + Spring instance of CircuitBreaker - spring

I would like to use Resilience4j to deal with fault tolerance, I am using CircuitBreaker and TimerLimit.
I would like to separate business logic of fault tolerance behavior, to not "dirty" my business code.
So, I am thinking to use Command pattern to execute my methods that will be treat, like Hystrix do with HystrixCommand.
Example:
public class MyCommand {
private static final CircuitBreaker circuitBreaker;
private Long param1, param2;
private MyService myService;
private static final TimeLimiter timeLimiter;
static {
long ttl = 50000;
TimeLimiterConfig configTimerLimit
= TimeLimiterConfig.custom().timeoutDuration(Duration.ofMillis(ttl)).build();
timeLimiter = TimeLimiter.of(configTimerLimit);
// I got the configuration from a class that I created.
circuitBreaker = CircuitBreaker.of("my", CircuitBreakerConfigOptions.defaultForExternalService());
}
public MyCommand(Long param1, Long param2, MyService myService) {
this.param1 = param1;
this.param2 = param2;
this.myService = myService;
}
public String run() {
Callable<String> stringCallable = TimeLimiter.decorateFutureSupplier(timeLimiter,
() -> CompletableFuture.supplyAsync(() -> myService.hello(param1, param2)));
Callable<String> callable = CircuitBreaker.decorateCallable(circuitBreaker, stringCallable);
return Try.of(callable::call).recover(t -> fallback(t)).get();
}
protected String fallback(Throwable throwable) {
Callable<String> stringCallable = TimeLimiter.decorateFutureSupplier(timeLimiter,
() -> CompletableFuture.supplyAsync(() -> myService.otherHello(param1, param2)));
return Try.of(stringCallable::call).getOrElse("Fallback");
}
}
Calling in my Controller:
#ApiOperation(value = "Only to test")
#GetMapping(value = "/execute", produces = MediaType.APPLICATION_JSON)
public String execute() {
return new MyCommand(1L, 2L, new MyService()).run();
}
My doubts:
1 - In this case, circuitBreaker really needs to be static, because I understood that the same object needs to be shared between the same method that you want to threat, I am right?
2 - How I have many instances of this application, circuitBreaker works individually for each instance? I am right?

What I understand from your question is - You need a Resilience4j's circuit breaker which should be sperate i.e. not messed with your business logic.
So I would suggest put circuit breaker protection around run() method. below code will elaborate -
Your Controller -
#ApiOperation(value = "Only to test")
#GetMapping(value = "/execute", produces = MediaType.APPLICATION_JSON)
public String execute() {
return new MyCommand().run();
}
Now Write your run() method of MyCommand class with #CircuitBreaker
public class MyCommand {
#CircuitBreaker(name = "RUN_METHOD_PROTECTION") // <---- here is our circuit breaker annotation code top of below your business code... and that’s it.
Your_Response run(Your_Request){
// Your business logic written here...
}
Further add circuit breaker configuration in your YAML property file as below (Count based I have used instead of time-based) –
resilience4j.circuitbreaker:
backends:
RUN_METHOD_PROTECTION:
registerHealthIndicator: true
slidingWindowSize: 100 # start rate calc after 100 calls
minimumNumberOfCalls: 100 # minimum calls before the CircuitBreaker can calculate the error rate.
permittedNumberOfCallsInHalfOpenState: 10 # number of permitted calls when the CircuitBreaker is half open
waitDurationInOpenState: 10s # time that the CircuitBreaker should wait before transitioning from open to half-open
failureRateThreshold: 50 # failure rate threshold in percentage
slowCallRateThreshold: 100 # consider all transactions under interceptor for slow call rate
slowCallDurationThreshold: 2s # if a call is taking more than 2s then increase the error rate
recordExceptions: # increment error rate if following exception occurs
- org.springframework.web.client.HttpServerErrorException
- java.io.IOException
- org.springframework.web.client.ResourceAccessException
Now if you are unable to use annotation #CircuitBreaker in your project then you can also do things in functional way i.e.
Let say we you have defined a bean in your configuration,
#Bean
public CircuitBreaker MyCircuitBreaker(){
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
.slidingWindow(100,100, CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
.failureRateThreshold(50)
.build();
CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(config);
CircuitBreaker circuitBreaker = registry.circuitBreaker("your_run_method_CircuitBreker"); // here you are registering your circuit breaker with a unique tag. And in future you refer this tag you get a same circuit breaker.
return circuitBreaker;
}
Now Your controller code would be below -
private CircuitBreaker circuitBreaker; // Assume you have injected the value from CircuitBreaker bean
#ApiOperation(value = "Only to test")
#GetMapping(value = "/execute", produces = MediaType.APPLICATION_JSON)
public String execute() {
Function<Your_Request, Your_Response> decorated = CircuitBreaker
.decorateFunction(circuitBreaker, new MyCommand().run());
return decorated.apply();
}
This way too you have never interfered with your business logic.

Since it seems that you are using Spring Boot, you could use the resilience4j-spring-boot-2 starter module which also has support for annotations.
https://resilience4j.readme.io/docs/getting-started-3

Related

Resilience4j How to route to fallback method then return back to original method after specific amount of time

I am working with resilience4j and spring boot,
I need to accomplish the below scenario,
When I have a failure in the originalMethod
After 5 attempts route to the fallback method
After a specific time like 5 minutes return back to the originalMethod
I tried with retry as below but does not fit the problem ,
#Retry(name = "retryService", fallbackMethod = "fallback")
public String originalMethod(String data) throws InterruptedException {
//..... call external service
}
public String fallback(String data, Throwable t) {
logger.error("Inside retryfallback, cause – {}", t.toString());
return "Inside retryfallback method. Some error occurred ";
}
Added properties
resilience4j.retry:
instances:
retryService:
maxRetryAttempts: 5
waitDuration: 50000
I think you can use a circuit breaker for sometime when a failure limit reached to achieve the behavior you want.
By adding #CircuitBreaker(...) annotation and specifying the failureRateThreshold, waitDurationInOpenState and the other needed config properties for that instance.

Circuit Breaker prints that it is OPEN even when it was disabled manually

We have put up #CircuitBreaker annotation on a method in a class and also specified a fallback method.
#Component
public class Example {
#Autowired
#Qualifier(value = Constants.PNO_CIRCUIT_BREAKER_QUALIFIER)
private io.github.resilience4j.circuitbreaker.CircuitBreaker circuitBreaker;
#CircuitBreaker(name = Constants.CIRCUIT_BREAKER_PNO, fallbackMethod = "fallback")
public Mono<Response> fetchPrices(List<PriceAndOfferRequest> priceAndOfferReqList) {
circuitBreaker.transitionToDisabledState();
return webClient.post().uri("someurl")
.body(Mono.just(priceAndOfferReqList), List.class).retrieve()
.bodyToMono(Response.class).publishOn(Schedulers.boundedElastic()).elapsed()
.transform(CircuitBreakerOperator.of(circuitBreaker))
.timeout(Duration.ofMillis(100))
.onErrorResume(e -> fallback(priceAndOfferReqList, e));
}
}
We are running a performance test on it. So even while we have disabled the circuit breaker, we see logs of this form:
CircuitBreaker ‘cb_pno’ is OPEN and does not permit further calls
Can somebody help me in understanding why the CB was opened while it was forcefully disabled?
Also, what is the different between using #CirbuitBreaker annotation and .transform(CircuitBreakerOpearator.of(circuitBreaker))

Running Tasks in different thread in Spring Webflux Annotated controller

I have a spring Webflux Annotated controller as below,
#RestController
public class TestBlockingController {
Logger log = LoggerFactory.getLogger(this.getClass().getName());
#GetMapping()
public Mono<String> blockForXSeconds(#RequestParam("block-seconds") Integer blockSeconds) {
return getStringMono();
}
private Mono<String> getStringMono() {
Integer blockSeconds = 5;
String type = new String();
try {
if (blockSeconds % 2 == 0) {
Thread.sleep(blockSeconds * 1000);
type = "EVEN";
} else {
Thread.sleep(blockSeconds * 1000);
type = "ODD";
}
} catch (Exception e) {
log.info("Got Exception");
}
log.info("Type of block-seconds: " + blockSeconds);
return Mono.just(type);
}
}
How do I make getStringMono run in a different thread than Netty server threads. The problem I am facing is that as I am running in server thread I am getting basically less throughput (2 requests per second). How do I go about making running getStringMono in a separate thread.
You can use subscribeOn operator to delegate the task to a different threadpool:
Mono.defer(() -> getStringMono()).subscribeOn(Schedulers.elastic());
Although, you have to note that this type of blocking should be avoided in a reactive application at any cost. If possible, use a client which supports non-blocking IO and returns a promise type (Mono, CompletableFuture, etc.). If you just want to have an artificial delay, then use Mono.delay instead.
You can use Mono.defer() method.
The method signature is as:
public static <T> Mono<T> defer(Supplier<? extends Mono<? extends T>> supplier)
Your Rest API should look like this.
#GetMapping()
public Mono<String> blockForXSeconds(#RequestParam("block-seconds") Integer blockSeconds) {
return Mono.defer(() -> getStringMono());
}
The defer operator is there to make this source lazy, re-evaluating the content of the lambda each time there is a new subscriber. This will increase your API throughput.
Here you can view the detailed analysis.

Validating Spring Kafka payloads

I am trying to set up a service that has both a REST (POST) endpoint and a Kafka endpoint, both of which should take a JSON representation of the request object (let's call it Foo). I would want to make sure that the Foo object is valid (via JSR-303 or whatever). So Foo might look like:
public class Foo {
#Max(10)
private int bar;
// Getter and setter boilerplate
}
Setting up the REST endpoint is easy:
#PostMapping(value = "/", produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<String> restEndpoint(#Valid #RequestBody Foo foo) {
// Do stuff here
}
and if I POST, { "bar": 9 } it processes the request, but if I post: { "bar": 99 } I get a BAD REQUEST. All good so far!
The Kafka endpoint is easy to create (along with adding a StringJsonMessageConverter() to my KafkaListenerContainerFactory so that I get JSON->Object conversion:
#KafkaListener(topics = "fooTopic")
public void kafkaEndpoint(#Valid #Payload Foo foo) {
// I shouldn't get here with an invalid object!!!
logger.debug("Successfully processed the object" + foo);
// But just to make sure, let's see if hand-validating it works
Validator validator = localValidatorFactoryBean.getValidator();
Set<ConstraintViolation<SlackMessage>> errors = validator.validate(foo);
if (errors.size() > 0) {
logger.debug("But there were validation errors!" + errors);
}
}
But no matter what I try, I can still pass invalid requests in and they process without error.
I've tried both #Valid and #Validated. I've tried adding a MethodValidationPostProcessor bean. I've tried adding a Validator to the KafkaListenerEndpointRegistrar (a la the EnableKafka javadoc):
#Configuration
public class MiscellaneousConfiguration implements KafkaListenerConfigurer {
private Logger logger = LoggerFactory.getLogger(this.getClass());
#Autowired
LocalValidatorFactoryBean validatorFactory;
#Override
public void configureKafkaListeners(KafkaListenerEndpointRegistrar registrar) {
logger.debug("Configuring " + registrar);
registrar.setMessageHandlerMethodFactory(kafkaHandlerMethodFactory());
}
#Bean
public MessageHandlerMethodFactory kafkaHandlerMethodFactory() {
DefaultMessageHandlerMethodFactory factory = new DefaultMessageHandlerMethodFactory();
factory.setValidator(validatorFactory);
return factory;
}
}
I've now spent a few days on this, and I'm running out of other ideas. Is this even possible (without writing validation into every one of my kakfa endpoints)?
Sorry for the delay; we are at SpringOne Platform this week.
The infrastructure currently does not pass a Validator into the payload argument resolver. Please open an issue on GitHub.
Spring kafka listener by default do not scan for #Valid for non Rest controller classes. For more details please refer this answer
https://stackoverflow.com/a/71859991/13898185

Any samples to unit test fallback using Hystrix Spring Cloud

I wish to test the following scenarios:
Set the hystrix.command.default.execution.isolation.thread.timeoutInMillisecond value to a low value, and see how my application behaves.
Check my fallback method is called using Unit test.
Please can someone provide me with link to samples.
A real usage can be found bellow. The key to enable Hystrix in the test class are these two annotations:
#EnableCircuitBreaker
#EnableAspectJAutoProxy
class ClipboardService {
#HystrixCommand(fallbackMethod = "getNextClipboardFallback")
public Task getNextClipboard(int numberOfTasks) {
doYourExternalSystemCallHere....
}
public Task getNextClipboardFallback(int numberOfTasks) {
return null;
}
}
#RunWith(SpringRunner.class)
#EnableCircuitBreaker
#EnableAspectJAutoProxy
#TestPropertySource("classpath:test.properties")
#ContextConfiguration(classes = {ClipboardService.class})
public class ClipboardServiceIT {
private MockRestServiceServer mockServer;
#Autowired
private ClipboardService clipboardService;
#Before
public void setUp() {
this.mockServer = MockRestServiceServer.createServer(restTemplate);
}
#Test
public void testGetNextClipboardWithBadRequest() {
mockServer.expect(ExpectedCount.once(), requestTo("https://getDocument.com?task=1")).andExpect(method(HttpMethod.GET))
.andRespond(MockRestResponseCreators.withStatus(HttpStatus.BAD_REQUEST));
Task nextClipboard = clipboardService.getNextClipboard(1);
assertNull(nextClipboard); // this should be answered by your fallBack method
}
}
Fore open the circuit in your unit test case just before you call the client. Make sure fall back is called. You can have a constant returned from fallback or add some log statements.
Reset the circuit.
#Test
public void testSendOrder_openCircuit() {
String order = null;
ServiceResponse response = null;
order = loadFile("/order.json");
// use this in case of feign hystrix
ConfigurationManager.getConfigInstance()
.setProperty("hystrix.command.default.circuitBreaker.forceOpen", "true");
// use this in case of just hystrix
System.setProperty("hystrix.command.default.circuitBreaker.forceOpen", "true");
response = client.sendOrder(order);
assertThat(response.getResultStatus()).isEqualTo("Fallback");
// DONT forget to reset
ConfigurationManager.getConfigInstance()
.setProperty("hystrix.command.default.circuitBreaker.forceOpen", "false");
// use this in case of just hystrix
System.setProperty("hystrix.command.default.circuitBreaker.forceOpen", "false");
}

Resources