How do export metrics using OpenTelemetry and Spring Cloud Sleuth - metrics

My team is trying to get Spring Cloud sleuth to work with the Opentelemetry api. We observe that the spans, attributes (tags) and events are exported just fine to the OTEL collector.
The metrics we add are not exported with the spans (or separately) which would be our expectation.
We have the following dependencies in our project:
implementation platform('org.springframework.cloud:spring-cloud-sleuth-otel-dependencies:1.1.1')
implementation('io.opentelemetry:opentelemetry-api')
implementation('org.springframework.cloud:spring-cloud-sleuth-api')
...
implementation 'org.springframework.cloud:spring-cloud-sleuth-otel-autoconfigure'
implementation('org.springframework.cloud:spring-cloud-starter-sleuth') {
exclude group: 'org.springframework.cloud', module: 'spring-cloud-sleuth-brave'
}
implementation 'io.opentelemetry:opentelemetry-exporter-otlp:1.22.0'
The code that adds metrics is as follow:
#Autowired
private final OpenTelemetry openTelemetry;
//...
//make two attempts - one at both api's. The first is just using a NOOP meter provider unfortunately.
DoubleCounter doubleCounter = GlobalOpenTelemetry.getMeter("io.opentelemetry.example.metrics")
.counterBuilder("calculated_used_space")
.setDescription("Counts disk space used by file extension.")
.setUnit("MB")
.ofDoubles()
.build();
doubleCounter.add(2.0);
//this is using a SDK meter provider which should export metrics with the span.
DoubleCounter build = openTelemetry.getMeter("com.jysk.some-app.some-metric")
.counterBuilder("some-counter")
.setDescription("some-description")
.setUnit("pcs.")
.ofDoubles()
.build();
build.add(4.0f);
Any insight in how we get Spring Cloud Sleuth to export metrics to a configured collector. We are using the Opentelemetry Collector.

Related

How to configure opentelemetry using depedencies in spring boot microservices?

Add dependency of opentelemetry jar in all the docker files
Enable promethues for all spring microservices
Enable/disable opentelemetry
I wanted to implement opentelemetry-javaagent dependency in my spring boot cloud project in every microservices. here they have used jar files and show the metrics on SigNoz but I wanted to use the dependency for opentelemetry.
The problem is I am getting dependency but they are not working. No metrics are being projected in the SigNoz dashboard.
If you don't want to use the agent, you may use the direct dependency in your application code.
https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-api.
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api</artifactId>
</dependency>
And configure the SDK -
Resource resource = Resource.getDefault()
.merge(Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, "logical-service-name")));
SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder().build()).build())
.setResource(resource)
.build();
SdkMeterProvider sdkMeterProvider = SdkMeterProvider.builder()
.registerMetricReader(PeriodicMetricReader.builder(OtlpGrpcMetricExporter.builder().build()).build())
.setResource(resource)
.build();
OpenTelemetry openTelemetry = OpenTelemetrySdk.builder()
.setTracerProvider(sdkTracerProvider)
.setMeterProvider(sdkMeterProvider)
.setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
.buildAndRegisterGlobal();
Then acquire an instance of the tracer -
import io.opentelemetry.api;
//...
Tracer tracer =
openTelemetry.getTracer("instrumentation-library-name", "1.0.0");
And create spans -
Span span = tracer.spanBuilder("my span").startSpan();
// Make the span the current span
try (Scope ss = span.makeCurrent()) {
// In this scope, the span is the current/active span
} finally {
span.end();
}
Taken from here -
https://opentelemetry.io/docs/instrumentation/java/manual/

Jaeger log warning messages saying no sender factories available

I am trying to set up Jaeger to collect traces from a spring boot application. When my app starts up, I am getting this warning message
warn io.jaegertracing.internal.senders.SenderResolver - No sender factories available. Using NoopSender, meaning that data will not be sent anywhere!
I use this method to get the jaeger tracer
#Bean
Tracer jaegerTracer(#Value(defaulTraceName) String service)
{
SamplerConfiguration samplerConfig = SamplerConfiguration.fromEnv().withType("const").withParam(1);
ReporterConfiguration reporterConfig = ReporterConfiguration.fromEnv().withLogSpans(true);
Configuration config = new Configuration(service).withSampler(samplerConfig).withReporter(reporterConfig);
return config.getTracer();
}
I have manually instrumented the code, but no traces show up in the jaeger UI. I have been stuck on this problem for a few days now and would appreciate any help given!
In my pom file, I have dependencies on jaeger-core and opentracing-api
Solved by adding dependency in pom file on jaeger-thrift.

Spring Boot 2 integrate Brave MySQL-Integration into Zipkin

I am trying to integrate the Brave MySql Instrumentation into my Spring Boot 2.x service to automatically let its interceptor enrich my traces with spans concerning MySql-Queries.
The current Gradle-Dependencies are the following
compile 'io.zipkin.zipkin2:zipkin:2.4.5'
compile('io.zipkin.reporter2:zipkin-sender-okhttp3:2.3.1')
compile('io.zipkin.brave:brave-instrumentation-mysql:4.14.3')
compile('org.springframework.cloud:spring-cloud-starter-zipkin:2.0.0.M5')
I already configured Sleuth successfully to send traces concerning HTTP-Request to my Zipkin-Server and now I wanted to add some spans for each MySql-Query the service does.
The TracingConfiguration it this:
#Configuration
public class TracingConfiguration {
/** Configuration for how to send spans to Zipkin */
#Bean
Sender sender() {
return OkHttpSender.create("https://myzipkinserver.com/api/v2/spans");
}
/** Configuration for how to buffer spans into messages for Zipkin */
#Bean AsyncReporter<Span> spanReporter() {
return AsyncReporter.create(sender());
}
#Bean Tracing tracing(Reporter<Span> spanListener) {
return Tracing.newBuilder()
.spanReporter(spanReporter())
.build();
}
}
The Query-Interceptor works properly, but my problem now is that the spans are not added to the existing trace but each are added to a new one.
I guess its because of the creation of a new sender/reporter in the configuration, but I have not been able to reuse the existing one created by the Spring Boot Autoconfiguration.
That would moreover remove the necessity to redundantly define the Zipkin-Url (because it is already defined for Zipkin in my application.yml).
I already tried autowiring the Zipkin-Reporter to my Bean, but all I got is a SpanReporter - but the Brave-Tracer-Builder requries a Reporter<Span>
Do you have any advice for me how to properly wire things up?
Please use latest snapshots. Sleuth in latest snapshots uses brave internally so integration will be extremely simple.

Spring Boot 2.0.0.M6: Show all metrics with one request

With Spring Boot 2.0.0.M6 and the new actuator metrics endpoint, when I request
GET /application/metrics
the only the names of the metrics are shown
{
"names" : [ "data.source.active.connections", "jvm.buffer.memory.used", "jvm.memory.used", "jvm.buffer.count", "logback.events", "process.uptime", "jvm.memory.committed", "data.source.max.connections", "http.server.requests", "system.load.average.1m", "jvm.buffer.total.capacity", "jvm.memory.max", "process.start.time", "cpu", "data.source.min.connections" ]
}
Clearly I can access a specific metric using
GET /application/metrics/jvm.memory.used
But is there a way to see all metrics with one request?
That's how the metrics endpoint behaves in the Spring Boot 2.0.0M* releases. There are only two read operations defined in the endpoint class:
ListNamesResponse listNames()
Resolves to GET /application/metrics
MetricResponse metric(#Selector String requiredMetricName,
#Nullable List<String> tag)
Resolves to GET /application/metrics/jvm.memory.used
Metrics support has changed quite dramatically in 2.x (now backed by Micrometer) and the Spring Boot 2.x upgrade guide is lacking any details on metrics at the moment but it's a work in progress, so presumably more details will arive as Spring Boot 2.0 gets closer to a GA release.
I suspect the move from hierarchical metrics to dimensional metrics resulted in the maintainers deeming the 1.x (hierarchical) metrics display to be no longer viable/suitable.
Spring boot 2 has removed this functionality by default, but if this is requirement of your application then this custom implementation will serve your purpose: https://github.com/csankhala/spring-metrics-grabber
Add dependency in you application from local repository
compile("org.springframework.boot:spring-metrics-grabber:1.0.0-SNAPSHOT");
compile("org.springframework.boot:spring-boot-starter-actuator");
Add MetricxEndpoint.class to #SpringBootApplication scan path
import org.springframework.metricx.controller.MetricxEndpoint;
#SpringBootApplication(scanBasePackageClasses = { MetricxEndpoint.class, YourSpringBootApplication.class })
public class YourSpringBootApplication {
public static void main(String[] args) {
new SpringApplication(YourSpringBootApplication.class).run(args);
}
}
All Metrics will be published on '/metricx' endpoint with name and value both
Pattern search is also supported e.g. '/metricx/jvm.*'

Metrics don't show up on the /prometheus endpoint

I have two services, ping and pong where ping sends requests to pong. This metric shows up on the /metrics endpoint for the ping service:
gauge.servo.hystrix.hystrixcommand.http://pong.pongclient#hello().90
but it doesn't appear on the /prometheus endpoint. Other metrics appear on this endpoint, but not the servo metrics with information about Feign/Hystrix http requests.
How do I get those metrics to appear on the /prometheus endpoint?
I have the following dependencies on my build.gradle
compile 'org.springframework.boot:spring-boot-starter-web'
compile 'org.springframework.boot:spring-boot-starter-actuator'
compile 'org.springframework.cloud:spring-cloud-starter-eureka'
compile 'org.springframework.cloud:spring-cloud-starter-hystrix'
compile 'org.springframework.cloud:spring-cloud-starter-feign'
compile 'org.springframework.retry:spring-retry'
compile "io.micrometer:micrometer-core:${micrometerVersion}"
compile "io.micrometer:micrometer-spring-legacy:${micrometerVersion}"
compile "io.micrometer:micrometer-registry-prometheus:${micrometerVersion}"
with the following versions
springCloudVersion = 'Dalston.SR4'
micrometerVersion = '1.0.0-rc.4'
The code can be found here https://github.com/fiunchinho/spring-resiliency
You need to manually add the plugin for Hystrix:
HystrixPlugins.getInstance().registerMetricsPublisher(new MicrometerMetricsPublisher(Metrics.globalRegistry));
You could add it in a #PostConstruct in a configuration.
I've created https://github.com/micrometer-metrics/micrometer/issues/237 to address the shortcoming in the future.
checketts' answer did not work for me and threw a java.lang.IllegalStateException: Another strategy was already registered. on startup, but adding the HystrixMetricsBinder bean, which does more or less the same internally, did the trick.
#Configuration
public class MetricsConfig {
#Bean
HystrixMetricsBinder registerHystrixMetricsBinder() {
return new HystrixMetricsBinder();
}
}
taken from https://stackoverflow.com/a/52740957/60518
You need to instrument and configure the spring boot services to monitor with prometheus as follows:
You need to include a dependency in the gradle.build file
You need to implement Metric endpoint
You need to add the standard exporters from Prometheus JVM
For more implementation details on how to do it see the examples here and also here

Resources