Customize spring-boot jaeger auto-configuration - spring-boot

I'm working on a POC and was able to integrate 2 microservices with JaegerUI.
Request to an endpoint in serviceA calls an endpoint in serviceB and returns a response.
I have used below dependencies:
spring.boot.version : 2.1.4.RELEASE
<groupId>io.opentracing.contrib</groupId>
<artifactId>opentracing-spring-jaeger-web-starter</artifactId>
<version>3.1.1</version>
<groupId>io.opentracing.contrib</groupId>
<artifactId>opentracing-spring-cloud-starter</artifactId>
<version>0.4.0</version>
Spring autoconfiguration takes care of everything so just added the required properties:
opentracing.jaeger.http-sender.url
opentracing.jaeger.service-name
opentracing.jaeger.enabled
opentracing.spring.cloud.async.enabled
I want to achieve the below:
I want to add application logs to the span so that they are visible in JaegerUI.
I want to add some fields to span tags so that it's easy to search in JaegerUI.
Also, I want the spanId and traceId to the application log.
Is it possible to search in JaegerUI based on spanId/traceId? If yes, how?
Based on the answer in below SO question:
How to enrich Jaeger opentracing data with the application logs (produced by slf4j) for Spring Boot?
opentracing-spring-cloud-starter dependency should automatically take care of sending app logs to span in JaegerUI.
I have a log statement like below in serviceA:
logger.info("sending request to serviceB.");
But above log is not getting captured in corresponding span and not visible in JaegerUI.
Any suggestions on how to achieve the above scenarios are appreciated!

I was studying Opentracing and Jeager and I've used this tutorial to get familiar with the basic possibilities:
https://github.com/yurishkuro/opentracing-tutorial/tree/master/java
If you take a look in the case 1 (Hello World), it explains how to "Annotate the Trace with Tags and Logs".
That would answer your questions 1, 2 and 3, as with that you can add all the info that you would like within spans and logs.
Here is a snippet from the repository (but I'd recommend checking there, as it has a more detailed explanation):
Span span = tracer.buildSpan("say-hello").start();
span.setTag("hello-to", helloTo);
In this case helloTo is a variable containing a name, to whom the app will say hello. It would create a span tag called hello-to with the value that is coming from the execution.
Below we have an example for the logs case, where the whole helloStr message is added to the logs:
// this goes inside the sayHello method
String helloStr = String.format("Hello, %s!", helloTo);
span.log(ImmutableMap.of("event", "string-format", "value", helloStr));
System.out.println(helloStr);
span.log(ImmutableMap.of("event", "println"));
Regarding the last question, that would be easier, you can use the Jaeger UI to search for the trace you would like, there is a field for that on the top left corner:

There you go.
I want to add application logs to the span so that they are visible in JaegerUI.
Span span = tracer.buildSpan("my-span-name").start();
span.setTag("my-tag-name", "my-tag-value");
There are various overloaded methods as follows
Span setTag(String key, String value);
Span setTag(String key, boolean value);
Span setTag(String key, Number value);
I want to add some fields to span tags so that it's easy to search in JaegerUI.
Jaeger API provides log method to log multiple fields that needs to be added to a map, the method signature is as follows,
Span log(Map<String, ?> fields);
eg:
span.log(
ImmutableMap.Builder<String, Object>()
.put("event", "soft error")
.put("type", "cache timeout")
.put("waited.millis", 1500)
.build()
);
Also, I want the spanId and traceId to the application log.
spanId and traceId are stored in JaegerSpanContext class, which can be obtained from context() method of Span class.
JaegerSpanContext spanContext = (JaegerSpanContext)sprintSpan.context();
long spanId = spanContext.getSpanId();
long traceId = spanContext.getTraceId();
Is it possible to search in JaegerUI based on spanId/traceId? If yes, how?
There is a search box in the navigation bar of Jaeger UI where you can search traces by trace ID.

Related

Can I store sensitive data in a Vert.x context in a Quarkus application?

I am looking for a place to store some request scoped attributes such as user id using a Quarkus request filter. I later want to retrieve these attributes in a Log handler and put them in the MDC logging context.
Is Vertx.currentContext() the right place to put such request attributes? Or can the properties I set on this context be read by other requests?
If this is not the right place to store such data, where would be the right place?
Yes ... and no :-D
Vertx.currentContext() can provide two type of objects:
root context shared between all the concurrent processing executed on this event loop (so do NOT share data)
duplicated contexts, which are local to the processing and its continuation (you can share in these)
In Quarkus 2.7.2, we have done a lot of work to improve our support of duplicated context. While before, they were only used for HTTP, they are now used for gRPC and #ConsumeEvent. Support for Kafka and AMQP is coming in Quarkus 2.8.
Also, in Quarkus 2.7.2, we introduced two new features that could be useful:
you cannot store data in a root context. We detect that for you and throw an UnsupportedOperationException. The reason is safety.
we introduced a new utility class ( io.smallrye.common.vertx.ContextLocals to access the context locals.
Here is a simple example:
AtomicInteger counter = new AtomicInteger();
public Uni<String> invoke() {
Context context = Vertx.currentContext();
ContextLocals.put("message", "hello");
ContextLocals.put("id", counter.incrementAndGet());
return invokeRemoteService()
// Switch back to our duplicated context:
.emitOn(runnable -> context.runOnContext(runnable))
.map(res -> {
// Can still access the context local data
String msg = ContextLocals.<String>get("message").orElseThrow();
Integer id = ContextLocals.<Integer>get("id").orElseThrow();
return "%s - %s - %d".formatted(res, msg, id);
});
}

Adding Custom "trace id with Alpha numeric values and spiting it out in application Log "

I am using Sleuth 2.1.3.
I want to add a custom "trace ID" as "correlation id" with alpha numeric value and want to spit in logs with spanid and parent id.
If i use below implementation for creating new custom trace id. does it get printed in logs ?
I tried below implementation but does not see any custom trace in log
https://github.com/openzipkin/zipkin-aws/blob/release-0.11.2/brave-propagation-aws/src/main/java/brave/propagation/aws/AWSPropagation.java
Tracing.newBuilder().propagationFactory(
ExtraFieldPropagation.newFactoryBuilder(B3Propagation.FACTORY)
.addField("x-vcap-request-id")
.addPrefixedFields("x-baggage-", Arrays.asList("country-code", "user-id"))
.build()
);
I tried with above code from https://cloud.spring.io/spring-cloud-sleuth/reference/html/#propagation but didnt see any custom trace id in log
You've passed in the B3Propagation.FACTORY as the implementation of the propagation factory so you're explicitly stating that you want the default B3 headers. You've said that you want some other field that is alphanumeric to be also propagated. Then in a log parsing tool you can define that you want to use your custom field as the trace id, but it doesn't mean that the deafult X-B3-TraceId field will be changed. If you want to use your custom field as trace id that Sleuth understands, you need to change the logging format and implement a different propagation factory bean.
One of the way which worked for me is
using ExtraFieldPropagation
and adding those keys in sleuth properties under propagation-keys
and whitelisted-keys
sample code
' #Autowired Tracer tracer;
Span currentSpan = tracer.nextSpan().start();
ExtraFieldPropagation.set(
"customkey", "customvalue");
sleuth:
log:
slf4j:
whitelisted-mdc-key : customkey
propagation:
tag:
enabled: true
propagation-keys : customkey '

How to add trace id to each logs in go micro service

I wanted to add trace id to logging done for each request to the micro service.I want this in similar as for springboot application we can set trace id in MDC and fetch it and use it while logging.
I have done some research and I found that MDC equivalent in go lang is context. So, I have set the trace id in my context. Now the problem is where ever I have to log with trace id ,I need to pass context to that function which is very ugly way. I am looking for a better solution for this problem.
func HandlerFunction(f gin.HandlerFunc) gin.HandlerFunc{
    return func(cxt *gin.Context) {
reqraceId := cxt.Request.Header.Get("trace-id")
        requid , _ := uuid.NewRandom()
        if reqTraceId == "" {
            c.Request.Header.Set("trace-id", requid.String())
        }
        f(c)
    }
}
It might be worth reading up on context.Context particularly this article which has a section that says:
At Google, we require that Go programmers pass a Context parameter as the first argument to every function on the call path between incoming and outgoing requests.
TL;DR - it's fine to pass the context, but what's the best way?
There's two main patterns
Ask the context to give you a logger
Give the logger the context
Context can be used to store values:
context.WithValue(ctx, someKey, someValue)
This means we can either do:
somepackage.Log(ctx).Info("hello world")
// or
sompackage.Info(ctx, "hello world")
The implementation of these two sample APIs could interact with the context to retrieve the values required with out needing to worry about the extra information that would have been in MDC at any of the logging call sites.
From my side I found that using the default log package we could set a prefix as log.SetPrefix(traceId), doing so, the log will print the trace id as the prefix in the actual and sub-functions/structs.
import (
"log"
"github.com/google/uuid"
)
func (hdl *HTTPHandler) example() {
var traceId string = uuid.NewString()
log.SetPrefix(traceId + " - ")
log.SetFlags(log.LstdFlags)
// ...
// ...
log.Println("......")
}
This issue can also be solved using a dependency injection container.
We can implement "request-scoped" injections, and as a result, for each request, we will recreate all dependency tree that uses request-scoped dependency(logger, error reporter, clients which send requests to another service with context propagation).
But as I understood using dependency injection containers is not a best practice in go and not an "idiomatic" way.
Also, this approach can have some performance and memory issues since we will recreate objects for each request.

How to log MDC with Spring Sleuth?

I have a Spring boot + sleuth based application. All works as expected. I have for now logs like this:
2017-05-04 17:55:52.226 INFO [alert,692d0eeca479e216,c3c8b680dc29ad02,false] 17292 --- [cTaskExecutor-1] c.k.a.b.s.alert.impl.AlertServiceImpl : Alert state to process: xxx
Now, I want to add custom MDC to my log like the contract reference for example. I want to have logs like this:
2017-05-04 17:55:52.226 INFO [alert,692d0eeca479e216,c3c8b680dc29ad02,false] [CONTRACT_REF] 17292 --- [cTaskExecutor-1] c.k.a.b.s.alert.impl.AlertServiceImpl : Alert state to process: xxx
I tried various things with no success:
Use the Spring Sleuth Tracer to add a tag;
Add logging.pattern.level=%5p %mdc to my application.properties file with MDC.put(xxx, xxx)
How can I add custom MDC/tags to my log?
For versions before 2.x, You have to create your own implementation of a SpanLogger. The easiest way will be to extend the Slf4jSpanLogger and provide your own code to add / update and remove the entries from MDC context. Then you can change your logging pattern and that way your logs will contain what they need.
I was able to add data to the MDC fairly easily by doing MDC.put("yourCoolKey", "your cool value") (see MDC.put JavaDoc).
Once you put the value into the MDC, you can use the sequence %X{yourCoolKey} in your logging pattern (in my case, the value of logging.pattern.console) to print the string "your cool value" as part of each log statement.
Optionally, you can specify a default value in the pattern string by adding :-<defaultValue> after the key, such as %X{yourCoolKey:-N/A}, which will print the string "N/A" whenever the MDC does not have an entry for "yourCoolKey". The default, if not specified, is a blank string ("")

Aws integration spring: Extend Visibility Timeout

Is it possible to extend the visibility time out of a message that is in flight.
See:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html.
Section: Changing a Message's Visibility Timeout.
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQSClient.html#changeMessageVisibility-com.amazonaws.services.sqs.model.ChangeMessageVisibilityRequest-
In summary I want to be able to extend the first set visibility timeout for a given message that is in flight.
Example if 15secs have passed I then want to extend the timeout by another 20secs. Better example in java docs above.
From my understanding in the links above you can do this on the amazon side.
Below are my current settings;
SqsMessageDrivenChannelAdapter adapter =
new SqsMessageDrivenChannelAdapter(queue);
adapter.setMessageDeletionPolicy(SqsMessageDeletionPolicy.ON_SUCCESS);
adapter.setMaxNumberOfMessages(1);
adapter.setSendTimeout(2000);
adapter.setVisibilityTimeout(200);
adapter.setWaitTimeOut(20);
Is it possible to extend this timeout?
Spring Cloud AWS supports this starting with Version 2.0. Injecting a Visiblity parameter in your SQS listener method does the trick:
#SqsListener(value = "my-sqs-queue")
void onMessageReceived(#Payload String payload, Visibility visibility) {
...
var extension = visibility.extend(20);
...
}
Note, that extend will work asynchronously and will return a Future. So if you want to be sure further down the processing, that the visibility of the message is really extended at the AWS side of things, either block on the Future using extension.get() or query the Future with extension.isDone()
OK. Looks like I see your point.
We can change visibility for particular message using API:
AmazonSQS.changeMessageVisibility(String queueUrl, String receiptHandle, Integer visibilityTimeout)
For this purpose in downstream flow you have to get access to (inject) AmazonSQS bean and extract special headers from the Message:
#Autowired
AmazonSQS amazonSqs;
#Autowired
ResourceIdResolver resourceIdResolver;
...
MessageHeaders headers = message.getHeaders();
DestinationResolver destinationResolver = new DynamicQueueUrlDestinationResolver(this.amazonSqs, this.resourceIdResolver);
String queueUrl = destinationResolver.resolveDestination(headers.get(AwsHeaders.QUEUE));
String receiptHandle = headers.get(AwsHeaders.RECEIPT_HANDLE);
amazonSqs.changeMessageVisibility(queueUrl, receiptHandle, YOUR_DESIRED_VISIBILITY_TIMEOUT);
But eh, I agree that we should provide something on the matter as out-of-the-box feature. That may be even something similar to QueueMessageAcknowledgment as a new header. Or even just one more changeMessageVisibility method to this one.
Please, raise a GH issue for Spring Cloud AWS project on the matter with link to this SO topic.

Resources