How to exclude Hystrix Metrics from the Actuator /metrics endpoint? - spring

I have a spring-boot-app with Actuator and Hystrix enabled.
Spring-Boot-Version: 1.3.1.RELEASE
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-eureka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-hystrix</artifactId>
</dependency>
When I add #HystrixCommandto some methods, the /metrics Endpoint shows all the metrics from hystrix:
gauge.hystrix.HystrixCommand.RestEndpoints.TestController.test.errorPercentage: 0,
gauge.hystrix.HystrixThreadPool.RestEndpoints.rollingMaxActiveThreads: 1,
...
Question:
How can I exclude the hystrix-metrics completely from the /metricsendpoint?
Update 1
I tried to exclude the ServoMetrics and SpectatorMetrics with these approaches:
1)
#EnableAutoConfiguration(exclude={ServoMetricsAutoConfiguration.class,
SpectatorMetricsAutoConfiguration.class} )
2)
#SpringBootApplication(exclude={ServoMetricServices.class, SpectatorMetricServices.class})
But both did not achieved the desired effect.

This issue was created and fixed. In snapshots you can now set the following:
hystrix.metrics.enabled=false.

If you don't want any other metrics, better solution would be to create your own MetricsRegistry. This way, any other future code changes (adding more jars with more in built metrics) would not affect.
#Configuration
public class MetricsConfiguration {
private MetricRegistry metricRegistry;
MetricsConfiguration() {
//avoid other metrics already registered
metricRegistry = new MetricRegistry();
}
}
Note: Creating a #Bean of type MetricRegistry would not help, as Spring autoconfigure will use this Bean object instead of MetricsRegistry object from MetricsDropwizardAutoConfiguration.java

Related

Migrating from OpenTracing to OpenTelemetry

So I was using Open Tracing to trace calls to the microservice. I used to do it the following way:
<dependency>
<groupId>io.opentracing</groupId>
<artifactId>opentracing-api</artifactId>
<version>0.31.0</version>
</dependency>
<dependency>
<groupId>io.opentracing.contrib</groupId>
<artifactId>opentracing-spring-cloud-starter</artifactId>
<version>0.1.13</version>
</dependency>
<dependency>
<groupId>io.jaegertracing</groupId>
<artifactId>jaeger-client</artifactId>
<version>0.31.0</version>
</dependency>
Then add following bean:
#Bean
public Tracer initTracer() {
Configuration.SamplerConfiguration samplerConfig = new Configuration.SamplerConfiguration().withType("const").withParam(1);
Configuration.ReporterConfiguration reporterConfig = Configuration.ReporterConfiguration.fromEnv().withLogSpans(true);
return Configuration.fromEnv("my-service").withSampler(samplerConfig).withReporter(reporterConfig).getTracer();
}
When running I usually get these span reported logs whenever an external request is given to the application.
But with open telemetry I cannot simply do this anymore. What is the correct way to simulate the behaviour?

Using Spring StateMachine in an EJB container

I'm trying to use Spring's state machine library inside of a JavaEE application running in a container to manage instances of a class named RunInfo. But I'm not sure how to integrate the two together.
I tried adding the following dependencies:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-autoconfigure</artifactId>
<version>2.2.11.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.statemachine</groupId>
<artifactId>spring-statemachine-core</artifactId>
<version>2.4.1</version>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-core</artifactId>
<version>5.7.2</version>
</dependency>
on spring-boot-autoconfigure and then adding the following configuration class:
#Configuration
#EnableAutoConfiguration
#ComponentScan(basePackages = {"com.ourdomain"})
public class StateConfiguration {}
I then wrote a simple state machine configuration:
#Configuration
#EnableStateMachine(name = "runInfoStateMachine")
public class RunInfoStateMachineConfiguration extends EnumStateMachineConfigurerAdapter<RunInfoState, RunInfoEvent> {
and state handler:
#Component
public class RunInfoStateHandler extends LifecycleObjectSupport {
// ...
#Autowired private StateMachine<RunInfoState, RunInfoEvent> stateMachine;
}
based on another project that I manage (which is a pure Spring Boot app).
The state handler is created, but the state machine reference is never injected; i.e., stateMachine in the above code is always null.
How does one integrate the Spring StateMachine framework into an EJB application?

Resource leak detected when using spring-data-redis on cloud foundry

We develop a spring-boot service, which offers a rest api (spring-webflux) and sends data via RabbitMQ (spring-rabbit). The service is deployed on cloud foundry and we use spring-boot in version 2.1.4. We added spring-boot-starter-data-redis to use redis to cache some data and we got the following error:
[io.netty.util.ResourceLeakDetector] [] LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:284)
io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:217)
io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:196)
io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:178)
io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:162)
io.lettuce.core.resource.DefaultClientResources.<init>(DefaultClientResources.java:169)
io.lettuce.core.resource.DefaultClientResources$Builder.build(DefaultClientResources.java:532)
io.lettuce.core.resource.DefaultClientResources.create(DefaultClientResources.java:233)
io.lettuce.core.AbstractRedisClient.<init>(AbstractRedisClient.java:98)
io.lettuce.core.RedisClient.<init>(RedisClient.java:87)
io.lettuce.core.RedisClient.create(RedisClient.java:124)
org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.lambda$createClient$7(LettuceConnectionFactory.java:971)
java.base/java.util.Optional.orElseGet(Unknown Source)
org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.createClient(LettuceConnectionFactory.java:971)
org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.afterPropertiesSet(LettuceConnectionFactory.java:273)
org.springframework.cloud.service.keyval.RedisConnectionFactoryCreator.create(RedisConnectionFactoryCreator.java:88)
org.springframework.cloud.service.keyval.RedisConnectionFactoryCreator.create(RedisConnectionFactoryCreator.java:31)
org.springframework.cloud.Cloud.getServiceConnector(Cloud.java:288)
org.springframework.cloud.Cloud.getSingletonServiceConnector(Cloud.java:202)
org.springframework.cloud.config.java.CloudServiceConnectionFactory.redisConnectionFactory(CloudServiceConnectionFactory.java:260)
org.springframework.cloud.config.java.CloudServiceConnectionFactory.redisConnectionFactory(CloudServiceConnectionFactory.java:242)
...
This error only happens when we run the service on cloud foundry, if we run it locally, we don't get any error.
We don't do any configuration of the connection factory or the stringRedisTemplate on our side and only use stringRedisTemplate, which is configured by the spring-autoconfiguration.
We use following configuration for redis on cloud foundry:
#Configuration
#Profile( "cloud" )
public class CloudSpecificConfig extends AbstractCloudConfig {
#Bean
public RedisConnectionFactory redisConnectionFactory() {
return connectionFactory().redisConnectionFactory();
}
}
And this is how we use the template
#Component
#RequiredArgsConstructor
public final class RequestUtil {
private final StringRedisTemplate myRedisTemplate;
public String cacheId(String id, String value) {
myRedisTemplate.opsForValue().set( id, value );
}
}
These are our spring dependencies:
<properties>
<spring-boot-version>2.1.4.RELEASE</spring-boot-version>
</properties>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
<version>${spring-boot-version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<version>${spring-boot-version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<version>${spring-boot-version}</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
<version>${spring-boot-version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-cloud-connectors</artifactId>
<version>${spring-boot-version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
<version>${spring-boot-version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
<version>${spring-boot-version}</version>
</dependency>
We are quite confused on our side, since we didn't do any specific configuration on our side. It looks for us like there is something wrong with the spring configuration on the cloud. Are we doing something wrong? Do we need to configure something differently? Is this a bug?
This is what I had to do
I will see if I can find a more elegant way
#Bean(destroyMethod = "shutdown")
public DefaultClientResources lettuceClientResources() {
return DefaultClientResources.create();
}
#SuppressWarnings("unused")
#Bean
public RedisConnectionFactory redisConnectionFactory(DefaultClientResources dependency) {
return connectionFactory().redisConnectionFactory("redis-pcf-service");
}
In the end the issue disappeared on our side, because we changed the redis client from lettuce to jedis.
We had the problem with lettuce that we would lose the connection to our redis service on our cloud infrastructure. But since there was an update to the redis service at same time as we changed the client, we don't really know if it was related to lettuce.
Maybe there also just something wrong in the auto-configuration in conjunction with the redis service on our cloud instructure, which is based on cloudfoundry

How to get http_ metrics from Spring-Boot Api

I have list of metrics from Spring Boot Api but it doesn't contains http_* metrics
my pom.xml contains
<prometheus.version>0.1.0</prometheus.version>
....
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>${prometheus.version}</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_servlet</artifactId>
<version>${prometheus.version}</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>${prometheus.version}</version>
</dependency>
I made DefaultExports.initialize() and set bean
#Bean
SpringBootMetricsCollector springBootMetricsCollector(Collection<PublicMetrics> publicMetrics) {
SpringBootMetricsCollector springBootMetricsCollector = new SpringBootMetricsCollector(publicMetrics);
springBootMetricsCollector.register();
return springBootMetricsCollector;
}
then i get list of metrics from spring boot api but it does not contain htpp_ * metrics
like http_request_duration_microseconds
how can i get it?

ConnectionFactory get destroyed before camel

I'm using spring-boot with camel and ActiveMQ.
I'm using activemq autoconfiguration via #EnableJms annotation.
But creating my own ActiveMQComponent to enable "transacted(true)" on all queues.
#Bean(name = "activemq")
#ConditionalOnClass(ActiveMQComponent.class)
public ActiveMQComponent activeMQComponent(ConnectionFactory connectionFactory) {
ActiveMQComponent activeMQComponent = new ActiveMQComponent();
activeMQComponent.setConnectionFactory(connectionFactory);
activeMQComponent.setTransacted(true);
activeMQComponent.setTransactionManager(jmsTransactionManager(connectionFactory));
return activeMQComponent;
}
It works well but when I try to gracefully shutdown the application.
The PooledConnectionFactory get destroyed before the camel graceful shutdown happens.
Leading to a tons of error and the route unable to correctly stops.
Like 20 times this error :
2017-05-04 18:21:59.748 WARN 12188 --- [er[test.queue]] o.a.activemq.jms.pool.PooledSession : Caught exception trying rollback() when putting session back into the pool, will invalidate. javax.jms.IllegalStateException: The Session is closed
Followed by:
2017-05-04 18:21:59.748 INFO 12188 --- [ Thread-18] o.a.camel.spring.SpringCamelContext : Apache Camel 2.18.3 (CamelContext: route) is shutting down
Then later :
2017-05-04 18:21:59.766 INFO 12188 --- [ - ShutdownTask] o.a.camel.impl.DefaultShutdownStrategy : Waiting as there are still 1 inflight and pending exchanges to complete, timeout in 300 seconds. Inflights per route: [test2 = 1]
Anyone can help me configuring spring-boot camel activemq all together with graceful shutdown ?
Thanks
Update :
Here is a sample of my pom.xml:
<properties>
<!-- Spring -->
<spring-boot.version>1.4.3.RELEASE</spring-boot.version>
<!-- Camel -->
<camel-spring-boot.version>2.18.3</camel-spring-boot.version>
</properties>
....
<!-- Camel BOM -->
<dependency>
<!-- Import dependency management from Spring Boot -->
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>${spring-boot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring-boot-dependencies</artifactId>
<version>${camel-spring-boot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
...
<!-- Spring -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-actuator</artifactId>
</dependency>
<!-- ActiveMQ -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-activemq</artifactId>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-camel</artifactId>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-pool</artifactId>
</dependency>
<!-- Camel -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring-boot-starter</artifactId>
</dependency>
Update 2:
After further investigation and the creation of a new project adding every modification one by one I have isolated the problem.
The shutdown works correcly until I add a specific endpoint :
#EndpointInject(uri = "direct:aaa")
private Endpoint errorHandling;
Using :
private String errorHandling = "direct:aaa";
Doesn't produce the bug.
It seems like using #EndpointInject is making the activemq close first
Update 3 :
Found that SpringCamelContext is not implementing ApplicationListener and thus its method "onApplicationEvent" its not called handling the "shutdownEager" of camel.
Important thing is to use Camel Spring Boot Starter.
http://camel.apache.org/spring-boot.html
How to enable Camel auto-configuration in my Spring Boot application?
Just drop camel-spring-boot jar into your classpath:
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring-boot</artifactId>
<version>${camel.version}</version> <!-- use the same version as your Camel core version -->
</dependency>
camel-spring-boot jar comes with the spring.factories file, so as soon as you add that dependency into your classpath, Spring Boot will automatically auto-configure the Camel for you. Yay! That was fast ;) .
Auto-configured Camel context
The most important piece of functionality provided by the Camel auto-configuration is the CamelContext instance.
Camel auto-configuration creates SpringCamelContext for your and take care of the proper initialization and shutdown of that context.
Created Camel context is also registered in the Spring application context (under camelContext bean name), so you can access it just as the any other Spring bean.
#Configuration
public class MyAppConfig {
#Autowired
CamelContext camelContext;
#Bean
MyService myService() {
return new DefaultMyService(camelContext);
}
}
Apparently since https://issues.apache.org/jira/browse/CAMEL-2607
the SpringCamelContext doesn't implement ApplicationListener interface anymore.
Since I'm using spring-boot autoconfiguration, I am not using CamelContextFactoryBean which is adding the listener.
Has a temporary fix, I created a component which listen to ApplicationEvent and dispatch them to the SpringCamelContext method :
public class SpringCamelContextFix implements ApplicationListener<ApplicationEvent> {
private SpringCamelContext camelContext;
public SpringCamelContextFix(SpringCamelContext camelContext) {
this.camelContext = camelContext;
}
#Override
public void onApplicationEvent(ApplicationEvent event) {
camelContext.onApplicationEvent(event);
}
}
I had this same problem running unit/integration tests with Spring Boot, ActiveMQ or A-MQ, and Camel (version 2.18.1.redhat-000012). Apparently, when Spring Boot shuts down, the JMS thread pool is closed before the Camel context is shutdown, which is the wrong order. #John D provided a code fix in a Camel users mailing list thread which is similar to what he provided in this thread. Here is the version of John D's code that worked for me:
#Component
public class SpringCamelContextFix implements
ApplicationListener<ApplicationEvent> {
#Inject
private SpringCamelContext camelContext;
public SpringCamelContextFix(SpringCamelContext camelContext) {
this.camelContext = camelContext;
}
#Override
public void onApplicationEvent(ApplicationEvent event) {
camelContext.onApplicationEvent(event);
}
}

Resources