Spring boot not registering on promethes end point - spring-boot

I am trying to configure Prometheus and Grafana with spring boot.
#Configuration
#EnableSpringBootMetricsCollector
public class MetricsConfiguration {
/**
* Register common tags application instead of job. This application tag is
* needed for Grafana dashboard.
*
* #return registry with registered tags.
*/
#Value("${spring.application.name}")
private String applicationName;
#Value("${spring.profiles.active}")
private String environment;
#Bean
public MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() {
return registry -> {
registry.config().commonTags("application", applicationName, "environment", environment)
.meterFilter(getDefualtConfig());
};
}
private MeterFilter getDefualtConfig() {
return new MeterFilter() {
#Override
public DistributionStatisticConfig configure(Meter.Id id, DistributionStatisticConfig config) {
return DistributionStatisticConfig.builder().percentilesHistogram(true).percentiles(0.95, 0.99).build()
.merge(config);
}
};
}
}
while running the application I am able to see traing on localhost:8080/prometheus url.
but same I am not able to see on localhost:9090/metrics url which is Prometheus URL.
I have added the configuration in prometheus.yml and restarted the Prometheus.
- job_name: 'my-api'
scrape_interval: 10s
metrics_path: '/prometheus'
target_groups:
- targets: ['localhost:8080']

After spending 2 hours found the solution,
we were using basic auth for all health points also
The issue was that I was not setting up basic auth in my proemtheus.yml
- job_name: 'my-api'
scrape_interval: 10s
metrics_path: '/prometheus'
target_groups:
- targets: ['localhost:8080']
basic_auth:
username: test
password: test

Related

Spring Boot custom Kubernetes readiness probe

I want to implement custom logic to determine readiness for my pod, and I went over this: https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.endpoints.kubernetes-probes.external-state and they mention an example property:
management.endpoint.health.group.readiness.include=readinessState,customCheck
Question is - how do I override customCheck?
In my case I want to use HTTP probes, so the yaml looks like:
readinessProbe:
initialDelaySeconds: 10
periodSeconds: 10
httpGet:
path: /actuator/health
port: 12345
So then again - where and how should I apply logic that would determine when the app is ready (just like the link above, i'd like to rely on an external service in order for it to be ready)
customCheck is a key for your custom HealthIndicator. The key for a given HealthIndicator is the name of the bean without the HealthIndicator suffix
You can read:
https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.endpoints.health.writing-custom-health-indicators
You are defining readinessProbe, so probably hiting /actuator/health/readiness is a better choice.
public class CustomCheckHealthIndicator extends AvailabilityStateHealthIndicator {
private final YourService yourService;
public CustomCheckHealthIndicator(ApplicationAvailability availability, YourService yourService) {
super(availability, ReadinessState.class, (statusMappings) -> {
statusMappings.add(ReadinessState.ACCEPTING_TRAFFIC, Status.UP);
statusMappings.add(ReadinessState.REFUSING_TRAFFIC, Status.OUT_OF_SERVICE);
});
this.yourService = yourService;
}
#Override
protected AvailabilityState getState(ApplicationAvailability applicationAvailability) {
if (yourService.isInitCompleted()) {
return ReadinessState.ACCEPTING_TRAFFIC;
} else {
return ReadinessState.REFUSING_TRAFFIC;
}
}
}

Set Prefix to Spring Micrometer Merics using Statsd and Datadog

I'm trying to Implement Custom Metrics Integration for my App. Using the following setup
// DogStatsd Metrics Integration with MicroMeter
implementation group: 'io.micrometer', name: 'micrometer-registry-statsd', version: '1.7.2'
Custom Spring Configuration Added for the application
#Configuration
public class MetricConfiguration {
private final MeterRegistry meterRegistry;
#Value("${management.metrics.export.statsd.prefix}")
private String prefix;
#Autowired
public MetricConfiguration(MeterRegistry meterRegistry) {
this.meterRegistry = meterRegistry;
}
#Bean
public MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() {
return registry -> registry.config().meterFilter(new MeterFilter() {
#Override
public Meter.Id map(Meter.Id id) {
if (!id.getName().startsWith(prefix)) {
return id.withName(prefix + "." + id.getName());
} else {
return id;
}
}
});
}
#Bean
public TimedAspect timedAspect() {
return new TimedAspect(meterRegistry);
}
}
YAML Configuration for Metrics
management:
metrics:
enable:
jvm: false
process: false
tomcat: false
system: false
logback: false
distribution:
slo:
http.server.requests: 50ms
percentiles-histogram:
http.server.requests: true
percentiles:
http.server.requests: 0.99
export:
statsd:
enabled: false
flavor: datadog
host: ${DD_AGENT_HOST}
port: 8125
prefix: ${spring.application.name}
endpoints:
enabled-by-default: true
web:
exposure:
include: "*"
endpoint:
metrics:
enabled: true
health:
enabled: true
show-components: "always"
show-details: "always"
Trying to set the prefix to all the custom metrics, but after setting the prefix the excluded metrics are breaking are started showing in the /actuator/metrics response.
The response looks like below:
{
names: [
"my-service.http.server.requests",
"my-service.jvm.buffer.count",
"my-service.jvm.buffer.memory.used",
"my-service.jvm.buffer.total.capacity",
"my-service.jvm.classes.loaded",
"my-service.jvm.classes.unloaded",
"my-service.jvm.gc.live.data.size",
"my-service.jvm.gc.max.data.size",
"my-service.jvm.gc.memory.allocated",
"my-service.logback.events",
"my-service.process.cpu.usage",
"my-service.process.files.max",
"my-service.process.files.open",
"my-service.process.start.time",
"my-service.process.uptime",
"my-service.system.cpu.count",
"my-service.system.cpu.usage",
"my-service.system.load.average.1m",
"my-service.tomcat.sessions.active.current",
"my-service.tomcat.sessions.active.max",
"my-service.tomcat.sessions.alive.max",
"my-service.tomcat.sessions.created",
"my-service.tomcat.sessions.expired",
"my-service.tomcat.sessions.rejected"
]
}

Spring data cassandra - error while opening new channel

I have a problem with Cassandra's connection with spring-data. When Cassandra is running locally I have no problem with connecting, however when I ran my spring-boot app in k8s with external Cassandra I am stuck on WARN:
2020-07-24 10:26:32.398 WARN 6 --- [ s0-admin-0] c.d.o.d.internal.core.pool.ChannelPool : [s0|/127.0.0.1:9042] Error while opening new channel (ConnectionInitException: [s0|connecting...] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.7.2, CLIENT_ID=9679ee85-ff39-45b6-8573-62a8d827ec9e}): failed to send request (java.nio.channels.ClosedChannelException))
I don't understand why in the log I have [s0|/127.0.0.1:9042] instead of the IP of my contact points.
Spring configuration:
spring:
data:
cassandra:
keyspace-name: event_store
local-datacenter: datacenter1
contact-points: host1:9042,host2:9042
Also this WARN is not causing that spring-boot won't start however if I do query in service I have error:
{ error: "Query; CQL [com.datastax.oss.driver.internal.core.cql.DefaultSimpleStatement#9463dccc]; No node was available to execute the query; nested exception is com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was available to execute the query" }
Option 1: test your yml file like this. (Have you tried with ip address?)
data:
cassandra:
keyspace-name: event_store
local-datacenter: datacenter1
port:9042
contact-points: host1,host2
username: cassandra
password: cassandra
Option 2: Create new properties on your yml and than a configuration class
cassandra:
database:
keyspace-name: event_store
contact-points: host1, host2
port: 9042
username: cassandra
password: cassandra
#Configuration
public class CassandraConfig extends AbstractCassandraConfiguration {
#Value("${cassandra.database.keyspace-name}")
private String keySpace;
#Value("${cassandra.database.contact-points}")
private String contactPoints;
#Value("${cassandra.database.port}")
private int port;
#Value("${cassandra.database.username}")
private String userName;
#Value("${cassandra.database.password}")
private String password;
#Override
protected String getKeyspaceName() {
return keySpace;
}
#Bean
public CassandraMappingContext cassandraMapping() throws ClassNotFoundException {
CassandraMappingContext context = new CassandraMappingContext();
context.setUserTypeResolver(new SimpleUserTypeResolver(cluster().getObject(), keySpace));
return context;
}
#Bean
public CassandraClusterFactoryBean cluster() {
CassandraClusterFactoryBean cluster = super.cluster();
cluster.setUsername(userName);
cluster.setPassword(password);
cluster.setContactPoints(contactPoints);
cluster.setPort(port);
return cluster;
}
#Override
protected boolean getMetricsEnabled() {
return false;
}
}

Netflix Ribbon throws No instances available for MY-MICROSERVICE exception

My application uses Eureka and Ribbon. I'm trying to get two microservices to talk to each other. Below is my method of concern.
#Autowired #LoadBalanced
private RestTemplate client;
#Autowired
private DiscoveryClient dClient;
public String getServices() {
List<String> services = dClient.getServices();
List<ServiceInstance> serviceInstances = new ArrayList<>();
List<String> serviceHosts = new ArrayList<>();
for(String service : services) {
serviceInstances.addAll(dClient.getInstances(service));
}
for(ServiceInstance service : serviceInstances) {
serviceHosts.add(service.getHost());
}
//throws No instances available exception here
try {
System.out.println(this.client.getForObject("http://MY-MICROSERVICE/rest/hello", String.class, new HashMap<String, String>()));
}
catch(Exception e) {
e.printStackTrace();
}
return serviceHosts.toString();
}
The method returns an array of two hostnames(IP). So DiscoveryClient is able to see instances of the two services registered with Eureka. But RestTemplate or more precisely Ribbon throws IllegalStateExcpetion: No instances available exception.
DynamicServerListLoadBalancer for client MY-MICROSERVICE initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=MY-MICROSERVICE,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList#23edc38f
java.lang.IllegalStateException: No instances available for MY-MICROSERVICE
at org.springframework.cloud.netflix.ribbon.RibbonLoadBalancerClient.execute(RibbonLoadBalancerClient.java:119)
at org.springframework.cloud.netflix.ribbon.RibbonLoadBalancerClient.execute(RibbonLoadBalancerClient.java:99)
at org.springframework.cloud.client.loadbalancer.LoadBalancerInterceptor.intercept(LoadBalancerInterceptor.java:58)
Even the Eureka dashboard shows two services registered. I feel the problem is specifically with Ribbon. Here's my config file.
spring.application.name="my-microservice"
logging.level.org.springframework.boot.autoconfigure.logging=INFO
spring.devtools.restart.enabled=true
spring.devtools.add-properties=true
server.ribbon.eureka.enabled=true
eureka.client.serviceUrl.defaultZone = http://localhost:8761/eureka/
The other microservice also has the same configs except for a different name. What's the problem here?
Solved. I was using application.yml with Eureka-server and application.properties with the client. Once I converted everything to yml, all works fine.
spring:
application:
name: "my-microservice"
devtools:
restart:
enabled: true
add-properties: true
logging:
level:
org.springframework.boot.autoconfigure.logging: INFO
eureka:
client:
serviceUrl:
defaultZone: "http://localhost:8761/eureka/"
This is the yml file for both apps which only differ by the application name.

Spring Cloud Gateway API - Context-path on routes not working

I have setup context-path in application.yml
server:
port: 4177
max-http-header-size: 65536
tomcat.accesslog:
enabled: true
servlet:
context-path: /gb-integration
And I have configured some routes
#Bean
public RouteLocator routeLocator(RouteLocatorBuilder builder) {
final String sbl = "http://localhost:4178";
return builder.routes()
//gb-sbl-rest
.route("sbl", r -> r
.path("/sbl/**")
.filters(f -> f.rewritePath("/sbl/(?<segment>.*)", "/gb-sbl/${segment}"))
.uri(sbl)).build();
}
I want the API gateway to be reached using localhost:4177/gb-integration/sbl/**
However it is only working on localhost:4177/sbl/**
It seems my context-path is ignored.
Any ideas how I can get my context-path to work on all my routes?
You probably already figuered it out by your self, but here is what is working for me:
After reading the Spring Cloud documentation and having tryied many things on my own, I have eventually opted for a route by route configuration. In your case, it would look something like this:
.path("/gb-integration/sbl/**")
and repeat the same pattern for every route.
.path("/gb-integration/abc/**")
...
.path("/gb-integration/def/**")
You can actually see this in spring cloud documentation.
The spring clould documentation seems to be in progress. Hopefully, we shall find a better solution.
Detailing on #sendon1982 answer
If your service is exposed at localhost:8080/color/red and you want it to be accessible from gateway as localhost:9090/gateway/color/red, In the Path param of predicates, prepend the /gateway, and add StripPrefix as 1 in filters, which basically translates to
take the requested path which matches Path, strip/remove out the prefix paths till the number mentioned and route using given uri and the stripped path
my-app-gateway: /gateway
spring:
cloud:
gateway:
routes:
- id: color-service
uri: http://localhost:8080
predicates:
- Path=${my-app-gateway}/color/**
filters:
- StripPrefix=1
Using yaml file like this
spring:
cloud:
gateway:
routes:
- id: property-search-service-route
uri: http://localhost:4178
predicates:
- Path=/gb-integration/sbl/**
fixed :
application.yaml:
gateway:
discovery:
locator:
enabled: true
lower-case-service-id: true
filters:
# 去掉 /ierp/[serviceId] 进行转发
- StripPath=2
predicates:
- name: Path
# 路由匹配 /ierp/[serviceId]
# org.springframework.cloud.gateway.discovery.DiscoveryClientRouteDefinitionLocator#getRouteDefinitions
args[pattern]: "'/ierp/'+serviceId+'/**'"
filter:
#Component
public class StripPathGatewayFilterFactory extends
AbstractGatewayFilterFactory<StripPathGatewayFilterFactory.Config> {
/**
* Parts key.
*/
public static final String PARTS_KEY = "parts";
public StripPathGatewayFilterFactory() {
super(StripPathGatewayFilterFactory.Config.class);
}
#Override
public List<String> shortcutFieldOrder() {
return Arrays.asList(PARTS_KEY);
}
#Override
public GatewayFilter apply(Config config) {
return (exchange, chain) -> {
ServerHttpRequest request = exchange.getRequest();
ServerWebExchangeUtils.addOriginalRequestUrl(exchange, request.getURI());
String path = request.getURI().getRawPath();
String[] originalParts = StringUtils.tokenizeToStringArray(path, "/");
// all new paths start with /
StringBuilder newPath = new StringBuilder("/");
for (int i = 0; i < originalParts.length; i++) {
if (i >= config.getParts()) {
// only append slash if this is the second part or greater
if (newPath.length() > 1) {
newPath.append('/');
}
newPath.append(originalParts[i]);
}
}
if (newPath.length() > 1 && path.endsWith("/")) {
newPath.append('/');
}
ServerHttpRequest newRequest = request.mutate().path(newPath.toString()).contextPath(null).build();
exchange.getAttributes().put(ServerWebExchangeUtils.GATEWAY_REQUEST_URL_ATTR, newRequest.getURI());
return chain.filter(exchange.mutate().request(newRequest).build());
};
}
public static class Config {
private int parts;
public int getParts() {
return parts;
}
public void setParts(int parts) {
this.parts = parts;
}
}
}

Resources