CircuitBreaker will not open R4J - spring

I am using kotlin, with Spring Boot and resilience4j.
I am trying to set up a circuitbreaker to trigger when my other service is down, but now when i send request in Postman, it goes into the "getCatById" still and keeps returning me 500 internal server error, instead of going into the callback i have set.
#Service
class CatGateway {
val logger = LoggerFactory.getLogger(CatGateway::class.java)
#Value("\${catservice.baseurl}")
lateinit var baseUrl: String
val GET_ENDPOINT = "/cat"
val restClient = RestTemplate();
#CircuitBreaker(name = "catGateway", fallbackMethod = "dogFallback")
fun getCatById(id: Int): Cat?{
val result = restClient.getForObject("$baseUrl/$GET_ENDPOINT/$id", ByteArray::class.java)
return jacksonObjectMapper().readValue(result, Cat::class.java )
}
fun dogFallback(t: Throwable): Cat?{
logger.error("Circuitbreaker opened")
throw CatNotFoundException("Cat not found, cause service broke");
}
}
data class Cat(val id: Int)
application.yml
catservice:
baseurl: "http://localhost:8080/api"
server:
port: 8085
resilience4j:
circuitbreaker:
configs:
shared:
register-health-indicator: true
sliding-window-type: count_based
sliding-window-size: 5
failure-rate-threshold: 40
permitted-number-of-calls-in-half-open-state: 1
max-wait-duration-in-half-open-state: 10s
wait-duration-in-open-state: 10s
slow-call-duration-threshold: 2s
writable-stack-trace-enabled: true
automatic-transition-from-open-to-half-open-enabled: true
instances:
catGateway:
base-config: shared
management:
health:
circuitbreakers:
enabled: true
endpoint:
health:
show-details: always
Dependencies:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-circuitbreaker-resilience4j</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<version>2.7.5</version>
</dependency>

I study the same thing today and this is my demo project.
You may just run user-service application to see the effect of circuit breaker
After searching through the internet, the annotation fallback method in resilience4j fail to work as the one in Java due to the exception handling mechanism of Kotlin(please correct me if I am wrong)
So we have to implement another mechanism to catch the exception and trigger the circuit breaker(you may see the circuit breaker status from the api return message caught by the exception handler).
If there is a better solution. Please let me know. Thanks
https://github.com/billchau/circuit-breaker-kotlin/tree/master

Related

Spring Boot, Sleuth, OTEL, and Honeycomb

I have a scenario where I have Spring Boot integrated with OTEL and shipping to Honeycomb.io. I am trying to add an environment tag to each trace. I have created a class:
#Component
public class EnvironmentSpanProcessor implements SpanProcessor {
#Value("${ENVIRONMENT")
private String environment;
Queue<SpanData> spans = new LinkedBlockingQueue<>(50);
#Override
public void onStart(Context context, ReadWriteSpan readWriteSpan) {
readWriteSpan.setAttribute("env", environment);
}
#Override
public boolean isStartRequired() {
return false;
}
#Override
public void onEnd(ReadableSpan readableSpan) {
this.spans.add(readableSpan.toSpanData());
}
#Override
public boolean isEndRequired() {
return true;
}
}
I have set break points in this class, and they never hit on startup, even though the bean can be seen in actuator. I have put breakpoints on:
SdkTracerProvider otelTracerProvider(SpanLimits spanLimits, ObjectProvider<List<SpanProcessor>> spanProcessors,
SpanExporterCustomizer spanExporterCustomizer, ObjectProvider<List<SpanExporter>> spanExporters,
Sampler sampler, Resource resource, SpanProcessorProvider spanProcessorProvider) {
SdkTracerProviderBuilder sdkTracerProviderBuilder = SdkTracerProvider.builder().setResource(resource)
.setSampler(sampler).setSpanLimits(spanLimits);
List<SpanProcessor> processors = spanProcessors.getIfAvailable(ArrayList::new);
processors.addAll(spanExporters.getIfAvailable(ArrayList::new).stream()
.map(e -> spanProcessorProvider.toSpanProcessor(spanExporterCustomizer.customize(e)))
.collect(Collectors.toList()));
processors.forEach(sdkTracerProviderBuilder::addSpanProcessor);
return sdkTracerProviderBuilder.build();
}
in OtelAutoConfiguration and am not seeing them firing either on startup.
My pom.xml relevant section is:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-sleuth-brave</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-sleuth-otel-autoconfigure</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-extension-trace-propagators</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty-shaded</artifactId>
<version>1.47.0</version>
</dependency>
And my configuration from application.yaml
sleuth:
enabled: true
web:
additional-skip-pattern: /readiness|/liveness
client.skip-pattern: /readiness
sampler:
probability: 1.0
rate: 100
propagation:
type: OT_TRACER
otel:
config:
trace-id-ratio-based: 1.0
log.exporter.enabled: true
exporter:
otlp:
endpoint: https://api.honeycomb.io
headers:
x-honeycomb-team: ${TELEMETRY_API_KEY}
x-honeycomb-dataset: app-telemetry
sleuth-span-filter:
enabled: true
resource:
enabled: true
I am getting traces, so it appears the system itself is working, however I cannot get my env tag added.
Preemptive thank you to #marcingrzejszczak for the help so far on my gist: https://gist.github.com/fpmoles/b880ccfdef2d2138169ed398e87ec396
I'm unsure why your span processor is not being picked up by Spring and being added to your list of processors being registered with the tracer provider.
An alternative way to set process consistent values, like environment, would be to set it as a resource attribute. This is more desireable because it's set once and delivered once per batch of spans sent to the configured backend (eg Honeycomb). Using a span processor adds the same attribute to every span.
This can be done in a few different ways:
If using AutoConfigure, you can set via system property or environment variable
Set directly on the resource during your otelTracerProvider method:
resource.setAttribute("environment", "${environment}");
FYI Honeycomb has OTel Java SDK & Agent distros to help simplify sending data that reduces required configuration and sets sensible defaults.

Load balancer does not contain an instance for the service

I want to use Eureka client with spring-cloud-starter-loadbalancer. But when I added configuration I get error. When I remove #LoadBalancerClient(name = "mail-service", configuration = LoadBalancerConfiguration.class) and LoadBalancerConfiguration class configuration it's working fine. I tried this code:
#FeignClient(name = "mail-service")
#LoadBalancerClient(name = "mail-service", configuration = LoadBalancerConfiguration.class)
public interface EmailClient {
#RequestMapping(method = RequestMethod.POST, value = "/engine/emails/register")
void setUserRegistration(CreateUserDTO createUserDTO);
}
#Configuration
public class LoadBalancerConfiguration {
#Bean
public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(ConfigurableApplicationContext context) {
return ServiceInstanceListSupplier.builder()
.withBlockingDiscoveryClient()
.withSameInstancePreference()
.withHealthChecks()
.build(context);
}
}
application.yml:
feign:
client:
config:
default:
connectTimeout: 5000
readTimeout: 5000
loggerLevel: basic
eureka:
client:
serviceUrl:
defaultZone: ${EUREKA_URI:http://localhost:8761/eureka}
fetchRegistry: true
healthcheck:
enabled: true
instance:
preferIpAddress: true
leaseRenewalIntervalInSeconds: 10
POM.xml dependencies
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<dependency>
<groupId>com.netflix.eureka</groupId>
<artifactId>eureka-core</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-loadbalancer</artifactId>
</dependency>
I get this error when I have only one target service.
[503] during [POST] to [http://mail-service/engine/emails/register] [EmailClient#setUserRegistration(CreateUserDTO)]: [Load balancer does not contain an instance for the service mail-service]
I use Release Train Version: 2020.0.3
Do you know what could be the problem?
Any application using load balancer should follow the below order
Start the Eureka Server
Start the instances of the Service one by one which have dependency
Any application relies on information from a service registry (i.e. Eureka). Until the application is registered with it's instances by the serviceId , the Eureka server will not be able to pick that instance while load-balancing.
In the code you have shared, the bean ServiceInstanceListSupplier is created in the configuration class along with the health checks ( .withHealthChecks() ). This is causing the application to fail as service has not been registered yet.
Also, the LoadBalancer config should not be in a #Configuration-annotated class instead, it should be a class passed for config via #LoadBalancerClient or #LoadBalancerClients annotation, as described here.
The only bean you need to instantiate is the ServiceInstanceListSupplier (if you add spring-cloud-starter-loadbalancer, LoadBalancerClientFactory etc. will be instantiated by the starter itself).
So your LoadBalancer configuration class should look like code below.
It should not be in the #Configuration class:
public class LoadBalancerConfiguration {
#Bean
public ServiceInstanceListSupplier instanceSupplier(ConfigurableApplicationContext context) {
return ServiceInstanceListSupplier.builder()
.withDiscoveryClient()
.withHealthChecks()
.build(context);
}
}
As explained in this link, the actual #Configuration class , will have the following annotation: #LoadBalancerClients(defaultConfiguration = LoadBalancerConfiguration .class).
Then, if you enable health-checks in the instances, it should work without any issues.

How to Shutdown/Stop RabbitMQ queue of Spring Cloud stream bindings

I want to stop the consumers in rabbitmq of creating a queue from spring cloud stream bindings while hitting endpoint /prepare-for-shutdown. Please find below the configuration,
Added dependency in pom.xml
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>
Application.yml:
spring:
cloud:
stream:
bindings:
produceChannel:
binder: rabbit
content-type: application/json
destination: internal-exchange
consumeChannel:
binder: rabbit
content-type: application/json
destination: internal-exchange
group: small-queue
rabbit:
bindings:
consumeChannel:
consumer:
autoBindDlq: true
durableSubscription: true
requeueRejected: false
republishToDlq: true
bindingRoutingKey: admin
produceChannel:
producer:
routingKeyExpression: '"admin"'
sample.java
import org.springframework.cloud.stream.annotation.Input;
import org.springframework.messaging.SubscribableChannel;
public interface Sample{
#Input("consumeChannel")
SubscribableChannel consumeChannel();
#Output("produceChannel")
SubscribableChannel produceChannel();
}
The integration with RabbitMQ has been achieved using Spring Cloud's #StreamLinster and #EnableBinding abstractions as shown below:
#EnableBinding(Sample.class)
#StreamListener("consumeChannel")
public void sampleMessage(String message) {
// code
}
Looking forward to stop a consumer of a RabbitMQ queue programmatically.
Thanks in Advance
I analyzed the issue why am getting empty values by invoking the actuator endpoint '/actuator/bindings'
When hitting actuator binding endpoint, it invokes the method gatherInputBindings()in BindingsEndpoint.class.
In BindingsEndpoint.java, fetching the binding values from inputBindingLifecycle
(Collection<Binding<?>>) new DirectFieldAccessor(inputBindingLifecycle).getPropertyValue("inputBindings");
In below methods, setting empty bindings list to inputBindings
In InputBindingLifecycle.java,
void doStartWithBindable(Bindable bindable) {
this.inputBindings = bindable.createAndBindInputs(bindingService);
}
In Bindable.java,
default Collection<Binding<Object>> createAndBindInputs(BindingService adapter) {
return Collections.<Binding<Object>>emptyList();
}
Pls suggest me to fix these issues whether need to change any dependency or any code configuration

Why X-RateLimit-Remaining -1 in response header while using spring cloud api gateway ratelimit with redis?

I implemented ratelimit with redis in my spring cloud api gateway. Here is part of application.yml:
spring:
cloud:
gateway:
httpclient:
ssl:
useInsecureTrustManager: true
discovery:
locator:
enabled: true
routes:
- id: test-rest-service
uri: lb://test-rest-service
predicates:
- Path=/test/**
filters:
- RewritePath=/test/(?<path>.*), /$\{path}
- name: RequestRateLimiter
args:
key-resolver: "#{#userRemoteAddressResolver}"
redis-rate-limiter.replenishRate: 2
redis-rate-limiter.burstCapacity: 3
I called a GET API via postman and checked response header.
X-RateLimit-Remaining -1
X-RateLimit-Burst-Capacity 3
X-RateLimit-Replenish-Rate 2
The rate limit is not working. Why am I getting negative value for X-RateLimit-Remaining? What does it mean? How do I fix it?
This happened to me because there was no Redis instance launched. You have two options:
1) Download and run a Redis instance using docker:
docker run --name redis -d redis
2) You can use in testing an Embedded Redis Server as it is explained in the following article by adding the maven dependency:
<dependency>
<groupId>it.ozimov</groupId>
<artifactId>embedded-redis</artifactId>
<version>0.7.2</version>
<scope>test</scope>
</dependency>
And including the following snippet:
#TestConfiguration
public class TestRedisConfiguration {
private RedisServer redisServer;
public TestRedisConfiguration() {
this.redisServer = new RedisServer(6379);
}
#PostConstruct
public void postConstruct() {
redisServer.start();
}
#PreDestroy
public void preDestroy() {
redisServer.stop();
}
}
I faced the same issue recently. In my case, there was an older version of Redis installed which caused X-RateLimit-Remaining to be set to -1 constantly.
redis-cli shutdown

Spring Cloud Feign/Ribbon with corporate proxy

I want to consume a REST service from the outside world behind a corporate proxy with authentication.
How do I configure Spring Boot + Spring Cloud Feign/Ribbon to use our proxy?
I believe you're looking for something like this:
import feign.Feign;
import okhttp3.OkHttpClient;
import java.net.InetSocketAddress;
import java.net.Proxy;
...
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("proxy-url", 1234));
OkHttpClient okHttpClient = new OkHttpClient.Builder().proxy(proxy).build();
Feign.builder()
.client(new feign.okhttp.OkHttpClient(okHttpClient))
.target(...);
You just have to additionally add compile 'io.github.openfeign:feign-okhttp:9.5.0' to your project.
The target clause contains your defined Interface. Further reference: https://github.com/OpenFeign/feign
Turns out there is actually a much easier solution.
The following information will be helpful (also for more advanced use cases):
Spring Cloud Commons HTTP Factories
Overriding Feign Defaults
OpenFeign Client can run with several HTTP Clients.
By default it uses java.net.URLConnection, but you can also use ApacheHttpClient or OkHttpClient.
Using Apache Http Client
Here is what you can do to set a proxy using ApacheHttpClient:
Add the following two dependencies to your pom.xml:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
<!-- Dependency to switch HttpClient implementation from java.net.URLConnection to Apache HTTP Client -->
<!-- See also: FeignAutoConfiguration for details. -->
<!-- See also: https://cloud.spring.io/spring-cloud-commons/reference/html/#http-clients -->
<!-- See also: https://cloud.spring.io/spring-cloud-openfeign/reference/html/#spring-cloud-feign-overriding-defaults -->
<dependency>
<groupId>io.github.openfeign</groupId>
<artifactId>feign-httpclient</artifactId>
</dependency>
In your app expose the following bean:
// see: https://cloud.spring.io/spring-cloud-commons/reference/html/#http-clients
#Bean
public HttpClientBuilder proxiedHttpClient() {
String proxyHost = "client-envoy";
Integer proxyPort = 80
String proxyScheme = "http";
return HttpClientBuilder.create()
.setProxy(new HttpHost(proxyHost, proxyPort, proxyScheme));
}
That's it - nothing else needs to be configured in application.yaml since ApacheHttpClient will be used by default, if it is on the classpath.
Using Ok Http Client
To set a proxy using OkHttpClient you do a similar thing:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
<dependency>
<groupId>io.github.openfeign</groupId>
<artifactId>feign-okhttp</artifactId>
</dependency>
In your application.yml make sure to enable OkHttpClient and disable ApacheHttpClient:
spring:
cloud:
httpclientfactories:
ok:
enabled: true
apache:
enabled: false
feign:
okhttp:
enabled: true
httpclient:
enabled: false
Instead of HttpClientBuilder expose a bean of type OkHttpClient.Builder.
Spring cloud feign supports three underlying implementations:
Default
Apache HttpClient
OkHttpClient
If using Default:
Create this spring bean (say by defining inside class with #Configuration annotation), no changes required in application properties/yml:
#Bean
public Client feignClient() {
return new Client.Proxied(
null, null, new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)));
}
If using Apache HttpClient:
that means you have feign.httpclient.enabled: true in application.yml and below in your pom.xml or build.gradle:
pom.xml
<dependency>
<groupId>io.github.openfeign</groupId>
<artifactId>feign-httpclient</artifactId>
</dependency>
build.gradle
implementation 'io.github.openfeign:feign-httpclient'
Create this spring bean (say by defining inside class with #Configuration annotation):
#Bean
public CloseableHttpClient feignClient() {
return HttpClientBuilder.create().setProxy(new HttpHost(proxyHost, proxyPort)).build();
}
If using OkHttpClient:
that means you have feign.okhttp.enabled: true in application.yml and below in your pom.xml or build.gradle:
pom.xml
<dependency>
<groupId>io.github.openfeign</groupId>
<artifactId>feign-okhttp</artifactId>
</dependency>
build.gradle
implementation 'io.github.openfeign:feign-okhttp'
Create this spring bean (say by defining inside class with #Configuration annotation):
#Bean
public OkHttpClient feignClient() {
return new OkHttpClient.Builder()
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)))
.build();
}

Resources