Sending System Metrics to Graphite with Spring-Boot - spring-boot

Spring-Boot actuator exposes many useful metrics at /metrics such as uptime, memory usage, GC count.
Only a subset of these are sent to Graphite when using the Dropwizard Metrics integration. In specific, only the counters and gauges
Is there any way to get these other metrics to be published to graphite?
The documentation suggests that it should be possible:
Users of the Dropwizard ‘Metrics’ library will find that Spring Boot metrics are automatically published to com.codahale.metrics.MetricRegistry

System Metrics created by Spring boot are not reported automatically because MetricsRegistry does not know anything about those Metrics.
You should register those metrics manually when your application boots up.
#Autowired
private SystemPublicMetrics systemPublicMetrics;
private void registerSystemMetrics(MetricRegistry metricRegistry) {
systemPublicMetrics.metrics().forEach(m -> {
Gauge<Long> metricGauge = () -> m.getValue().longValue();
metricRegistry.register(m.getName(), metricGauge);
});
}
I have defined Gauge, not all the system metrics should be added as gauge. e.g. the Counter should be used to capture count values.
If you don't want to use Spring boot. Use can include metrics-jvm out of the box to capture JVM level metrics.

Here's a solution that does update DropWizard metrics on Spring metrics change. It also does that without turning #EnableScheduling on:
#EnableMetrics
#Configuration
public class ConsoleMetricsConfig extends MetricsConfigurerAdapter {
#Autowired
private SystemPublicMetrics systemPublicMetrics;
#Override
public void configureReporters(MetricRegistry metricRegistry) {
metricRegistry.register("jvm.memory", new MemoryUsageGaugeSet());
metricRegistry.register("jvm.thread-states", new ThreadStatesGaugeSet());
metricRegistry.register("jvm.garbage-collector", new GarbageCollectorMetricSet());
metricRegistry.register("spring.boot", (MetricSet) () -> {
final Map<String, Metric> gauges = new HashMap<String, Metric>();
for (final org.springframework.boot.actuate.metrics.Metric<?> springMetric :
systemPublicMetrics.metrics()) {
gauges.put(springMetric.getName(), (Gauge<Object>) () -> {
return systemPublicMetrics.metrics().stream()
.filter(m -> StringUtils.equals(m.getName(), springMetric.getName()))
.map(m -> m.getValue())
.findFirst()
.orElse(null);
});
}
return Collections.unmodifiableMap(gauges);
});
registerReporter(ConsoleReporter
.forRegistry(metricRegistry)
.convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS)
.build())
.start(intervalSecs, TimeUnit.SECONDS);
}
}
It uses the com.ryantenney.metrics library for enabling additional Spring annotations support and DropWizard reporters:
<dependency>
<groupId>com.ryantenney.metrics</groupId>
<artifactId>metrics-spring</artifactId>
<version>3.1.3</version>
</dependency>
But it is actually not necessary in this particular case.

Related

Spring boot Reactive caching

In my application I am using spring webflux and I am using webclient to retrieve details from some 3rd party API. Now, I want to store the first time webClient response in some in memory cache so that for 2nd time I can have those response directly from the cache.
I am trying to use Spring boot in memory caching mechanism and also "caffine". But none is working as expected.
application.yml:
spring:
cache:
cache-names: employee
caffiene:
spec: maximumSize=200, expireAfterAccess=5m
EmployeeApplication.java:
#SpringBootApplication
#EnableCaching
public class EmployeeApplication{
public static void main(String[] args){
}
}
EmployeeController.java:
It has a rest endpoint employee/all which fetch all employee from the 3rd party Api.
EmployeeService.java:
#Service
#Slf4j
public class EmployeeService{
#Autowired
private WebClient webClient;
#Autowired
private CacheManager cacheManager;
#Cacheable("employee")
public Mono<List<Employee>> getAllEmployee(){
log.info("inside employee service {}");
return webClient.get()
.uri("/employees/")
.retrieve()
.bodyToMono(Employee.class);
}
}
Although I have configured the cache name , 2nd time when I hit the url it is calling the service method. What cache mechanism need to be used to cache Mono response? Please suggest.
There are several options to cache reactive publishers.
Use reactive cache API to cache Mono for the defined duration
employeeService.getAllEmployee()
.cache(Duration.ofMinutes(60))
.flatMap(employees -> {
// process data
})
Use external cache with Caffeine.
Caffeine supports async cache based on CompletableFuture that could be easily adapted to Reactive API.
AsyncLoadingCache<String, List<Employee>> cache = Caffeine.newBuilder()
.buildAsync((tenant, executor) ->
employeeService.getAllEmployee(tenant).toFuture()
);
Mono<List<Employee>> getEmployee(String tenant) {
return Mono.fromCompletionStage(clientCache.get(tenant));
}
Use external cache with Guava and CacheMono from reactor-extra. This option is more suitable if you need to cache results based on different input (e.g. multi tenant environment)
UPDATE: CacheMono has been deprecated since reactor-extra 3.4.7. Better use #2 Use external cache with Caffeine.
Here is an example for Guava but you could adapt it for CacheManager
Cache<String, List<Employee>> cache = CacheBuilder.newBuilder()
.expireAfterWrite(cacheTtl)
.build();
Mono<List<Employee>> getEmployee(String tenant) {
return CacheMono.lookup(key -> Mono.justOrEmpty(cache.getIfPresent(key)).map(Signal::next), tenant)
.onCacheMissResume(() -> employeeService.getAllEmployee(tenant))
.andWriteWith((key, signal) -> Mono.fromRunnable(() ->
Optional.ofNullable(signal.get())
.ifPresent(value -> cache.put(key, value))
)
);
}

How did Spring Cloud Sleuth add tracing information to logback log lines?

I have web application based on Spring Boot and it uses logback for logging.
I also inherit some logback defaults from spring boot using:
<include resource="org/springframework/boot/logging/logback/base.xml"/>
I want to start logging tracing information, so I added:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
Sleuth adds tracing information to log lines, but I can't find any %X or %mdc in patterns: https://github.com/spring-projects/spring-boot/blob/2.3.x/spring-boot-project/spring-boot/src/main/resources/org/springframework/boot/logging/logback/defaults.xml
How does Sleuth add tracing information into log lines?
I use spring-cloud-starter-parent Hoxton.SR9 parent which brings Spring Boot 2.3.5.RELEASE and spring-cloud-starter-sleuth 2.2.6.RELEASE
(tl;dr at the bottom)
From the question I suppose you already figured out that the traceId and spanId are placed into the MDC.
If you take a look at the log integration section of the sleuth docs you will see that the tracing info in the example is between the log level (ERROR) and the pid (97192). If you try to match this with the logback config you will see that there is nothing between the log level and the pid: ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } so how the tracing information get there could be a valid question.
If you take another look to the docs, it says this:
This log configuration was automatically setup by Sleuth. You can disable it by disabling Sleuth via spring.sleuth.enabled=false property or putting your own logging.pattern.level property.
Which still not explicitly explains the mechanism but it gives you a huge hint:
putting your own logging.pattern.level property
Based on this, you could think that there is nothing between the log level and the pid, Sleuth simply overrides the log level and places the tracing information into it. And if you search for the property that the docs mention in the code, you will found out that it is exactly what happens:
TL;DR
Sleuth overrides the log level pattern and adds tracing info into it:
map.put("logging.pattern.level", "%5p [${spring.zipkin.service.name:" + "${spring.application.name:}},%X{traceId:-},%X{spanId:-}]");
In order to bring this back to Spring Boot 3.0 where Sleuth is no longer provided. The TraceEnvironmentPostProcessor has to be copied along with having an entry in META-INF/spring.factories
Here's the code I modified slightly from the original to make it pass SonarLint.
import java.util.HashMap;
import java.util.Map;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.env.EnvironmentPostProcessor;
import org.springframework.core.env.ConfigurableEnvironment;
import org.springframework.core.env.MapPropertySource;
class TraceEnvironmentPostProcessor implements EnvironmentPostProcessor {
private static final String DEFAULT_PROPERTIES_SOURCE_NAME = "defaultProperties";
#Override
public void postProcessEnvironment(
final ConfigurableEnvironment environment, final SpringApplication application) {
final Map<String, Object> map = new HashMap<>();
final boolean sleuthEnabled =
environment.getProperty("spring.sleuth.enabled", Boolean.class, true);
final boolean sleuthDefaultLoggingPatternEnabled =
environment.getProperty(
"spring.sleuth.default-logging-pattern-enabled", Boolean.class, true);
if (sleuthEnabled && sleuthDefaultLoggingPatternEnabled) {
map.put(
"logging.pattern.level",
"%5p [${spring.zipkin.service.name:${spring.application.name:}},%X{traceId:-},%X{spanId:-}]");
String neverRefreshables =
environment.getProperty(
"spring.cloud.refresh.never-refreshable", "com.zaxxer.hikari.HikariDataSource");
map.put(
"spring.cloud.refresh.never-refreshable",
neverRefreshables
+ ",org.springframework.cloud.sleuth.instrument.jdbc.DataSourceWrapper");
}
final var propertySources = environment.getPropertySources();
if (propertySources.contains(DEFAULT_PROPERTIES_SOURCE_NAME)) {
final var source = propertySources.get(DEFAULT_PROPERTIES_SOURCE_NAME);
if (source instanceof MapPropertySource target) {
map.entrySet().stream()
.filter(e -> !(target.containsProperty(e.getKey())))
.forEach(e -> target.getSource().put(e.getKey(), e.getValue()));
}
} else {
propertySources.addLast(new MapPropertySource(DEFAULT_PROPERTIES_SOURCE_NAME, map));
}
}
}
And
org.springframework.boot.env.EnvironmentPostProcessor=\
net.trajano.swarm.logging.autoconfig.TraceEnvironmentPostProcessor

io.micrometer.core.instrument.config.MeterFilter : DENY is not working in spring boot

I want to expose all metrics on the metrics endpoint but publish some of them to a remote meter registry.
For doing so, I have a SimpleMeterRegistry for the metrics endpoint and added a MeterRegistryCustomizer for the remote meter registry(Datadog) to add some MeterFilter to avoid specific metrics using MeterFilter's DENY function. For example :
#Bean
public MeterRegistryCustomizer<StatsdMeterRegistry> meterRegistryCustomizer() {
return (registry) -> new StatsdMeterRegistry(config, Clock.SYSTEM).config().meterFilter(MeterFilter.denyNameStartsWith("jvm"));
}
However, all jvm related metrics are visible in Datadog. I tried MeterFilterReply but no use.
Please suggest how this can be achieved.
You are configuring the filter on a new StatsdMeterRegistry. When using a MeterRegistryCustomizer you need to operate on the registry that was passed in.
#Bean
public MeterRegistryCustomizer<StatsdMeterRegistry> meterRegistryCustomizer() {
return (registry) -> registry.config().meterFilter(MeterFilter.denyNameStartsWith("jvm"));
}
Since the customizer will be used against all registries, you also would need to add an if statement to only filter against the registry you want filtered.
#Bean
public MeterRegistryCustomizer<StatsdMeterRegistry> meterRegistryCustomizer() {
return (registry) -> {
if(registry instanceof StatsdMeterRegistry) {
registry.config().meterFilter(MeterFilter.denyNameStartsWith("jvm"));
}
}
}

Spring Boot auto-configured metrics not arriving to Librato

I am using Spring Boot with auto-configure enabled (#EnableAutoConfiguration) and trying to send my Spring MVC metrics to Librato. Right now only my own created metrics are arriving to Librato but auto-configured metrics (CPU, file descriptors, etc) are not sent to my reporter.
If I access a metric endpoint I can see the info generated there, for instance http://localhost:8081/actuator/metrics/system.cpu.count
I based my code on this post for ConsoleReporter. so I have this:
public static MeterRegistry libratoRegistry() {
MetricRegistry dropwizardRegistry = new MetricRegistry();
String libratoApiAccount = "xx";
String libratoApiKey = "yy";
String libratoPrefix = "zz";
LibratoReporter reporter = Librato
.reporter(dropwizardRegistry, libratoApiAccount, libratoApiKey)
.setPrefix(libratoPrefix)
.build();
reporter.start(60, TimeUnit.SECONDS);
DropwizardConfig dropwizardConfig = new DropwizardConfig() {
#Override
public String prefix() {
return "myprefix";
}
#Override
public String get(String key) {
return null;
}
};
return new DropwizardMeterRegistry(dropwizardConfig, dropwizardRegistry, HierarchicalNameMapper.DEFAULT, Clock.SYSTEM) {
#Override
protected Double nullGaugeValue() {
return null;
}
};
}
and at my main function I added Metrics.addRegistry(SpringReporter.libratoRegistry());
For the Librato library I am using in my compile("com.librato.metrics:metrics-librato:5.1.2") build.gradle. Documentation here. I used this library before without any problem.
If I use the ConsoleReporter as in this post the same thing happens, only my own created metrics are printed to the console.
Any thoughts on what am I doing wrong? or what am I missing?
Also, I enabled debug mode to see the "CONDITIONS EVALUATION REPORT" printed in the console but not sure what to look for in there.
Try to make your MeterRegistry for Librato reporter as a Spring #Bean and let me know whether it works.
UPDATED:
I tested with ConsoleReporter you mentioned and confirmed it's working with a sample. Note that the sample is on the branch console-reporter, not the master branch. See the sample for details.

Spring Boot Actuator: have a customized status as plain text?

I'm trying to integrate Spring Boot Actuator with my companies existing infrastructure. To do this I need to be able to customize the status message. For instance if the app is up and running correctly I need to return a 200 and a plain text body of "HAPPY" from the health actuator endpoint.
Is such customization currently possible? Since the Status class is final I can't extend it, but I think that would work.
Spring Boot uses a HealthAggregator to aggregate all of the statuses from the individual health indicators into a single health for the entire application. You can plug in a custom aggregator that delegates to Boot's default aggregator, OrderedHealthAggregator, and then maps UP to HAPPY:
#Bean
public HealthAggregator healthAggregator() {
return new HappyHealthAggregator(new OrderedHealthAggregator());
}
static class HappyHealthAggregator implements HealthAggregator {
private final HealthAggregator delegate;
HappyHealthAggregator(HealthAggregator delegate) {
this.delegate = delegate;
}
#Override
public Health aggregate(Map<String, Health> healths) {
Health result = this.delegate.aggregate(healths);
if (result.getStatus() == Status.UP) {
return new Health.Builder(new Status("HAPPY"), result.getDetails())
.build();
}
return result;
}
}
If you want to take complete control over the format of the response, then you'll need to write your own MVC endpoint implementation. You could use the existing HealthMvcEndpointclass in Spring Boot as a super class and override its invoke method.

Resources