I would like to use Resilience4j to deal with fault tolerance, I am using CircuitBreaker and TimerLimit.
I would like to separate business logic of fault tolerance behavior, to not "dirty" my business code.
So, I am thinking to use Command pattern to execute my methods that will be treat, like Hystrix do with HystrixCommand.
Example:
public class MyCommand {
private static final CircuitBreaker circuitBreaker;
private Long param1, param2;
private MyService myService;
private static final TimeLimiter timeLimiter;
static {
long ttl = 50000;
TimeLimiterConfig configTimerLimit
= TimeLimiterConfig.custom().timeoutDuration(Duration.ofMillis(ttl)).build();
timeLimiter = TimeLimiter.of(configTimerLimit);
// I got the configuration from a class that I created.
circuitBreaker = CircuitBreaker.of("my", CircuitBreakerConfigOptions.defaultForExternalService());
}
public MyCommand(Long param1, Long param2, MyService myService) {
this.param1 = param1;
this.param2 = param2;
this.myService = myService;
}
public String run() {
Callable<String> stringCallable = TimeLimiter.decorateFutureSupplier(timeLimiter,
() -> CompletableFuture.supplyAsync(() -> myService.hello(param1, param2)));
Callable<String> callable = CircuitBreaker.decorateCallable(circuitBreaker, stringCallable);
return Try.of(callable::call).recover(t -> fallback(t)).get();
}
protected String fallback(Throwable throwable) {
Callable<String> stringCallable = TimeLimiter.decorateFutureSupplier(timeLimiter,
() -> CompletableFuture.supplyAsync(() -> myService.otherHello(param1, param2)));
return Try.of(stringCallable::call).getOrElse("Fallback");
}
}
Calling in my Controller:
#ApiOperation(value = "Only to test")
#GetMapping(value = "/execute", produces = MediaType.APPLICATION_JSON)
public String execute() {
return new MyCommand(1L, 2L, new MyService()).run();
}
My doubts:
1 - In this case, circuitBreaker really needs to be static, because I understood that the same object needs to be shared between the same method that you want to threat, I am right?
2 - How I have many instances of this application, circuitBreaker works individually for each instance? I am right?
What I understand from your question is - You need a Resilience4j's circuit breaker which should be sperate i.e. not messed with your business logic.
So I would suggest put circuit breaker protection around run() method. below code will elaborate -
Your Controller -
#ApiOperation(value = "Only to test")
#GetMapping(value = "/execute", produces = MediaType.APPLICATION_JSON)
public String execute() {
return new MyCommand().run();
}
Now Write your run() method of MyCommand class with #CircuitBreaker
public class MyCommand {
#CircuitBreaker(name = "RUN_METHOD_PROTECTION") // <---- here is our circuit breaker annotation code top of below your business code... and that’s it.
Your_Response run(Your_Request){
// Your business logic written here...
}
Further add circuit breaker configuration in your YAML property file as below (Count based I have used instead of time-based) –
resilience4j.circuitbreaker:
backends:
RUN_METHOD_PROTECTION:
registerHealthIndicator: true
slidingWindowSize: 100 # start rate calc after 100 calls
minimumNumberOfCalls: 100 # minimum calls before the CircuitBreaker can calculate the error rate.
permittedNumberOfCallsInHalfOpenState: 10 # number of permitted calls when the CircuitBreaker is half open
waitDurationInOpenState: 10s # time that the CircuitBreaker should wait before transitioning from open to half-open
failureRateThreshold: 50 # failure rate threshold in percentage
slowCallRateThreshold: 100 # consider all transactions under interceptor for slow call rate
slowCallDurationThreshold: 2s # if a call is taking more than 2s then increase the error rate
recordExceptions: # increment error rate if following exception occurs
- org.springframework.web.client.HttpServerErrorException
- java.io.IOException
- org.springframework.web.client.ResourceAccessException
Now if you are unable to use annotation #CircuitBreaker in your project then you can also do things in functional way i.e.
Let say we you have defined a bean in your configuration,
#Bean
public CircuitBreaker MyCircuitBreaker(){
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
.slidingWindow(100,100, CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
.failureRateThreshold(50)
.build();
CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(config);
CircuitBreaker circuitBreaker = registry.circuitBreaker("your_run_method_CircuitBreker"); // here you are registering your circuit breaker with a unique tag. And in future you refer this tag you get a same circuit breaker.
return circuitBreaker;
}
Now Your controller code would be below -
private CircuitBreaker circuitBreaker; // Assume you have injected the value from CircuitBreaker bean
#ApiOperation(value = "Only to test")
#GetMapping(value = "/execute", produces = MediaType.APPLICATION_JSON)
public String execute() {
Function<Your_Request, Your_Response> decorated = CircuitBreaker
.decorateFunction(circuitBreaker, new MyCommand().run());
return decorated.apply();
}
This way too you have never interfered with your business logic.
Since it seems that you are using Spring Boot, you could use the resilience4j-spring-boot-2 starter module which also has support for annotations.
https://resilience4j.readme.io/docs/getting-started-3
Related
I am working with resilience4j and spring boot,
I need to accomplish the below scenario,
When I have a failure in the originalMethod
After 5 attempts route to the fallback method
After a specific time like 5 minutes return back to the originalMethod
I tried with retry as below but does not fit the problem ,
#Retry(name = "retryService", fallbackMethod = "fallback")
public String originalMethod(String data) throws InterruptedException {
//..... call external service
}
public String fallback(String data, Throwable t) {
logger.error("Inside retryfallback, cause – {}", t.toString());
return "Inside retryfallback method. Some error occurred ";
}
Added properties
resilience4j.retry:
instances:
retryService:
maxRetryAttempts: 5
waitDuration: 50000
I think you can use a circuit breaker for sometime when a failure limit reached to achieve the behavior you want.
By adding #CircuitBreaker(...) annotation and specifying the failureRateThreshold, waitDurationInOpenState and the other needed config properties for that instance.
We have put up #CircuitBreaker annotation on a method in a class and also specified a fallback method.
#Component
public class Example {
#Autowired
#Qualifier(value = Constants.PNO_CIRCUIT_BREAKER_QUALIFIER)
private io.github.resilience4j.circuitbreaker.CircuitBreaker circuitBreaker;
#CircuitBreaker(name = Constants.CIRCUIT_BREAKER_PNO, fallbackMethod = "fallback")
public Mono<Response> fetchPrices(List<PriceAndOfferRequest> priceAndOfferReqList) {
circuitBreaker.transitionToDisabledState();
return webClient.post().uri("someurl")
.body(Mono.just(priceAndOfferReqList), List.class).retrieve()
.bodyToMono(Response.class).publishOn(Schedulers.boundedElastic()).elapsed()
.transform(CircuitBreakerOperator.of(circuitBreaker))
.timeout(Duration.ofMillis(100))
.onErrorResume(e -> fallback(priceAndOfferReqList, e));
}
}
We are running a performance test on it. So even while we have disabled the circuit breaker, we see logs of this form:
CircuitBreaker ‘cb_pno’ is OPEN and does not permit further calls
Can somebody help me in understanding why the CB was opened while it was forcefully disabled?
Also, what is the different between using #CirbuitBreaker annotation and .transform(CircuitBreakerOpearator.of(circuitBreaker))
I have a spring Webflux Annotated controller as below,
#RestController
public class TestBlockingController {
Logger log = LoggerFactory.getLogger(this.getClass().getName());
#GetMapping()
public Mono<String> blockForXSeconds(#RequestParam("block-seconds") Integer blockSeconds) {
return getStringMono();
}
private Mono<String> getStringMono() {
Integer blockSeconds = 5;
String type = new String();
try {
if (blockSeconds % 2 == 0) {
Thread.sleep(blockSeconds * 1000);
type = "EVEN";
} else {
Thread.sleep(blockSeconds * 1000);
type = "ODD";
}
} catch (Exception e) {
log.info("Got Exception");
}
log.info("Type of block-seconds: " + blockSeconds);
return Mono.just(type);
}
}
How do I make getStringMono run in a different thread than Netty server threads. The problem I am facing is that as I am running in server thread I am getting basically less throughput (2 requests per second). How do I go about making running getStringMono in a separate thread.
You can use subscribeOn operator to delegate the task to a different threadpool:
Mono.defer(() -> getStringMono()).subscribeOn(Schedulers.elastic());
Although, you have to note that this type of blocking should be avoided in a reactive application at any cost. If possible, use a client which supports non-blocking IO and returns a promise type (Mono, CompletableFuture, etc.). If you just want to have an artificial delay, then use Mono.delay instead.
You can use Mono.defer() method.
The method signature is as:
public static <T> Mono<T> defer(Supplier<? extends Mono<? extends T>> supplier)
Your Rest API should look like this.
#GetMapping()
public Mono<String> blockForXSeconds(#RequestParam("block-seconds") Integer blockSeconds) {
return Mono.defer(() -> getStringMono());
}
The defer operator is there to make this source lazy, re-evaluating the content of the lambda each time there is a new subscriber. This will increase your API throughput.
Here you can view the detailed analysis.
I am trying to set up a service that has both a REST (POST) endpoint and a Kafka endpoint, both of which should take a JSON representation of the request object (let's call it Foo). I would want to make sure that the Foo object is valid (via JSR-303 or whatever). So Foo might look like:
public class Foo {
#Max(10)
private int bar;
// Getter and setter boilerplate
}
Setting up the REST endpoint is easy:
#PostMapping(value = "/", produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<String> restEndpoint(#Valid #RequestBody Foo foo) {
// Do stuff here
}
and if I POST, { "bar": 9 } it processes the request, but if I post: { "bar": 99 } I get a BAD REQUEST. All good so far!
The Kafka endpoint is easy to create (along with adding a StringJsonMessageConverter() to my KafkaListenerContainerFactory so that I get JSON->Object conversion:
#KafkaListener(topics = "fooTopic")
public void kafkaEndpoint(#Valid #Payload Foo foo) {
// I shouldn't get here with an invalid object!!!
logger.debug("Successfully processed the object" + foo);
// But just to make sure, let's see if hand-validating it works
Validator validator = localValidatorFactoryBean.getValidator();
Set<ConstraintViolation<SlackMessage>> errors = validator.validate(foo);
if (errors.size() > 0) {
logger.debug("But there were validation errors!" + errors);
}
}
But no matter what I try, I can still pass invalid requests in and they process without error.
I've tried both #Valid and #Validated. I've tried adding a MethodValidationPostProcessor bean. I've tried adding a Validator to the KafkaListenerEndpointRegistrar (a la the EnableKafka javadoc):
#Configuration
public class MiscellaneousConfiguration implements KafkaListenerConfigurer {
private Logger logger = LoggerFactory.getLogger(this.getClass());
#Autowired
LocalValidatorFactoryBean validatorFactory;
#Override
public void configureKafkaListeners(KafkaListenerEndpointRegistrar registrar) {
logger.debug("Configuring " + registrar);
registrar.setMessageHandlerMethodFactory(kafkaHandlerMethodFactory());
}
#Bean
public MessageHandlerMethodFactory kafkaHandlerMethodFactory() {
DefaultMessageHandlerMethodFactory factory = new DefaultMessageHandlerMethodFactory();
factory.setValidator(validatorFactory);
return factory;
}
}
I've now spent a few days on this, and I'm running out of other ideas. Is this even possible (without writing validation into every one of my kakfa endpoints)?
Sorry for the delay; we are at SpringOne Platform this week.
The infrastructure currently does not pass a Validator into the payload argument resolver. Please open an issue on GitHub.
Spring kafka listener by default do not scan for #Valid for non Rest controller classes. For more details please refer this answer
https://stackoverflow.com/a/71859991/13898185
I wish to test the following scenarios:
Set the hystrix.command.default.execution.isolation.thread.timeoutInMillisecond value to a low value, and see how my application behaves.
Check my fallback method is called using Unit test.
Please can someone provide me with link to samples.
A real usage can be found bellow. The key to enable Hystrix in the test class are these two annotations:
#EnableCircuitBreaker
#EnableAspectJAutoProxy
class ClipboardService {
#HystrixCommand(fallbackMethod = "getNextClipboardFallback")
public Task getNextClipboard(int numberOfTasks) {
doYourExternalSystemCallHere....
}
public Task getNextClipboardFallback(int numberOfTasks) {
return null;
}
}
#RunWith(SpringRunner.class)
#EnableCircuitBreaker
#EnableAspectJAutoProxy
#TestPropertySource("classpath:test.properties")
#ContextConfiguration(classes = {ClipboardService.class})
public class ClipboardServiceIT {
private MockRestServiceServer mockServer;
#Autowired
private ClipboardService clipboardService;
#Before
public void setUp() {
this.mockServer = MockRestServiceServer.createServer(restTemplate);
}
#Test
public void testGetNextClipboardWithBadRequest() {
mockServer.expect(ExpectedCount.once(), requestTo("https://getDocument.com?task=1")).andExpect(method(HttpMethod.GET))
.andRespond(MockRestResponseCreators.withStatus(HttpStatus.BAD_REQUEST));
Task nextClipboard = clipboardService.getNextClipboard(1);
assertNull(nextClipboard); // this should be answered by your fallBack method
}
}
Fore open the circuit in your unit test case just before you call the client. Make sure fall back is called. You can have a constant returned from fallback or add some log statements.
Reset the circuit.
#Test
public void testSendOrder_openCircuit() {
String order = null;
ServiceResponse response = null;
order = loadFile("/order.json");
// use this in case of feign hystrix
ConfigurationManager.getConfigInstance()
.setProperty("hystrix.command.default.circuitBreaker.forceOpen", "true");
// use this in case of just hystrix
System.setProperty("hystrix.command.default.circuitBreaker.forceOpen", "true");
response = client.sendOrder(order);
assertThat(response.getResultStatus()).isEqualTo("Fallback");
// DONT forget to reset
ConfigurationManager.getConfigInstance()
.setProperty("hystrix.command.default.circuitBreaker.forceOpen", "false");
// use this in case of just hystrix
System.setProperty("hystrix.command.default.circuitBreaker.forceOpen", "false");
}