Spring Boot, Sleuth, OTEL, and Honeycomb - spring-boot

Question

I have a scenario where I have Spring Boot integrated with OTEL and shipping to Honeycomb.io. I am trying to add an environment tag to each trace. I have created a class:
#Component
public class EnvironmentSpanProcessor implements SpanProcessor {
#Value("${ENVIRONMENT")
private String environment;
Queue<SpanData> spans = new LinkedBlockingQueue<>(50);
#Override
public void onStart(Context context, ReadWriteSpan readWriteSpan) {
readWriteSpan.setAttribute("env", environment);
}
#Override
public boolean isStartRequired() {
return false;
}
#Override
public void onEnd(ReadableSpan readableSpan) {
this.spans.add(readableSpan.toSpanData());
}
#Override
public boolean isEndRequired() {
return true;
}
}
I have set break points in this class, and they never hit on startup, even though the bean can be seen in actuator. I have put breakpoints on:
SdkTracerProvider otelTracerProvider(SpanLimits spanLimits, ObjectProvider<List<SpanProcessor>> spanProcessors,
SpanExporterCustomizer spanExporterCustomizer, ObjectProvider<List<SpanExporter>> spanExporters,
Sampler sampler, Resource resource, SpanProcessorProvider spanProcessorProvider) {
SdkTracerProviderBuilder sdkTracerProviderBuilder = SdkTracerProvider.builder().setResource(resource)
.setSampler(sampler).setSpanLimits(spanLimits);
List<SpanProcessor> processors = spanProcessors.getIfAvailable(ArrayList::new);
processors.addAll(spanExporters.getIfAvailable(ArrayList::new).stream()
.map(e -> spanProcessorProvider.toSpanProcessor(spanExporterCustomizer.customize(e)))
.collect(Collectors.toList()));
processors.forEach(sdkTracerProviderBuilder::addSpanProcessor);
return sdkTracerProviderBuilder.build();
}
in OtelAutoConfiguration and am not seeing them firing either on startup.
My pom.xml relevant section is:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-sleuth-brave</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-sleuth-otel-autoconfigure</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-extension-trace-propagators</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty-shaded</artifactId>
<version>1.47.0</version>
</dependency>
And my configuration from application.yaml
sleuth:
enabled: true
web:
additional-skip-pattern: /readiness|/liveness
client.skip-pattern: /readiness
sampler:
probability: 1.0
rate: 100
propagation:
type: OT_TRACER
otel:
config:
trace-id-ratio-based: 1.0
log.exporter.enabled: true
exporter:
otlp:
endpoint: https://api.honeycomb.io
headers:
x-honeycomb-team: ${TELEMETRY_API_KEY}
x-honeycomb-dataset: app-telemetry
sleuth-span-filter:
enabled: true
resource:
enabled: true
I am getting traces, so it appears the system itself is working, however I cannot get my env tag added.
Preemptive thank you to #marcingrzejszczak for the help so far on my gist: https://gist.github.com/fpmoles/b880ccfdef2d2138169ed398e87ec396

Answer 1

I'm unsure why your span processor is not being picked up by Spring and being added to your list of processors being registered with the tracer provider.
An alternative way to set process consistent values, like environment, would be to set it as a resource attribute. This is more desireable because it's set once and delivered once per batch of spans sent to the configured backend (eg Honeycomb). Using a span processor adds the same attribute to every span.
This can be done in a few different ways:
If using AutoConfigure, you can set via system property or environment variable
Set directly on the resource during your otelTracerProvider method:
resource.setAttribute("environment", "${environment}");
FYI Honeycomb has OTel Java SDK & Agent distros to help simplify sending data that reduces required configuration and sets sensible defaults.