In issue comment msample asked:
We are looking at using OpenTracing and my searching for the on-wire/protocol for SpanContext led me here. I was surprised to not find this part solidly defined as it seems critical to the broad adoption of OpenTracing.
OpenTracing does not specify on-wire formats, neither for in-band messages like the span context passed via RPC messages, nor for the out of band messages like those used to send tracing spans out of application to the tracing backend. The reason for this is that such standardization is not necessary as long as the target architecture is using the OpenTracing libraries from the same tracing system (like Jaeger). OpenTracing API is first of all an instrumentation API for distributed systems. The wire formats are the implementation details of tracing systems implementing the API.
The wire format specification like https://github.com/TraceContext/tracecontext-spec is useful if one wants to pass trace info between services that are using different tracing backends, e.g. requests starting with services instrumented by Jaeger and then going onto services using StackDriver. But such integration has a whole set of different issues that need to be resolved, such as the sampling approach. There was no reason why OpenTracing API should be blocked by these separate standards.
Related
I wrote some micro-services using Quarkus that communicate via Artemis. Now I want to add OpenTelemetry for tracing purpose.
What I already tried is to call service B from service A using HTTP/REST. Here the trace id from service A is automatically added to the header of the HTTP request and used in service B. So this works fine. In Jaeger I can see the correlation.
But how can this be achieved using Artemis as messaging system? Do I have to (manually) add the trace id from service A into the message and read it in service B to setup somehow the context (don't know whether this is possible)? Or is there possibly an automatism like for HTTP requests?
I would appreciate any assistance.
I have to mention at this point that I have little experience with tracing so far.
There is no quarkus, quarkiverse extension or smallrye lib that provides integration with Artemis and OpenTelemetry, yet.
Also, OpenTelemetry massaging spec is being worked at the moment, because the correct way to correlate sent, received messages and services is under definition at the OTel spec level.
However, I had exactly the same problem as you and did a manual instrumentation that you can use as inspiration: quarkus-observability-demo-activemq
It will correlate the sent service as parent of receiving end.
I'm going to implement observability across my projects which are working beside each other in a Microservices environment.
I've used MassTransit and I'm going to track the communication between projects that happen using MassTransit. Perhaps using tools like Jaeger to monitor these communications.
Is there any tools or approach in MassTransit that supports it?
MassTransit starts an Activity when it consumes a message, as well as publishes a message. It also propagates the activity id across service boundaries by adding it to the message headers.
There's an article about MassTransit diagnostics in the docs.
You can add the tracing option you want. For example, OpenTelemetry for .NET supports using diagnostic listeners for building traces as described in the docs. You can then use the Jaeger exporter for OpenTelemetry.
You can also use the OpenTracing contribution library for MassTransit. I have an example for my talk at NDC Oslo 2020, which has complete instrumentation for MassTransit with Prometheus and OpenTracing (with Jaeger). The Prometheus integration library is now a part of the MassTransit main repository.
I am investigating solution to implement microservice Saga pattern in platform hosted in K8S in GCP.
There are 2 options: Eventulate Tram and Axon. However, these frameworks seem not to support message broker managed by cloud provider such as google-cloud-Pubsub whereas I do not want to deploy either Kafka or RabbitMQ to K8S since GCP support PubSub already.
So is there any way to integrate either Eventulate or Axon to use google cloud PubSub?
Thanks
Uncertain about Eventuate's angle on this, but Axon works with extensions as message brokers other than Axon Server. Throughout Axon's lifecycle (read: last 10 years), some of these have been provided, but none are currently used for all types of messages defined by Axon Framework. So, you wouldn't be able to use Kafka for sending commands in Axon for example.
Reasoning for this? Commands, events and queries have different routing requirements which should be reflected by using the right tool for the job.
To be a bit more specific on Axon's side, the following extensions can be used for distributing your messages:
AMQP -> for Events
Kafka -> for Events
JGroups -> for Commands
Spring Cloud Discovery -> for Commands
As you can tell, there currently is no Pub/Sub extension out there to allow you to distribute your messages. Added on top of that, my gut would tell me if it was available, then it would likely only be used for Event messages due to Pub/Sub's intent when it comes to being a message broker.
Luckily this actually makes it rather straightforward to create just such a extension yourself. Going into all the details to build this would be a little much, so I would recommend to have a look at Axon's AMQP extension first when it comes to achieving this. Hints on the matter are that for publication, you should add a component to handle Axon's events and publish them on Pub/Sub. For handling events, you are required to build a StreamableMessageSource or SubscribableMessageSource. These interfaces are used respectively by the TrackingEventProcessor and SubscribingEventProcessor, which in turn are the component in charge of dealing with the technical aspect of handling events.
By the way, if you would be building such an extension and you need a hand, it would be best to request this at AxonIQ forum, which you can find here.
Last note, and rather important I'd say, is the argument that such a connector would not be able to deal with all types of messages. If you would require a more full fledged Axon application to run in a distributed fashion, I would highly recommend to give Axon Server a try prior to building your own solution from the ground up.
The case for event-driven microservices such as Spring Cloud Stream is their asynchronous nature, which I do agree it makes them more scalable
But I have an issue regarding how to code it in a way where I don't lose certain key features that I have access to using synchronous services
In a servlet-based MS, I make full use of servlet context variables and servlet-based Spring autowiring functions
For e.g., I leverage heavily on HTTP headers to carry metadata between microservices without having to impact the payload. But in Spring Cloud Stream using Kafka, Kafka doesn't support message headers of any kind! I lose that immediately if I use SCS. Putting them into the payload causes all sort of changes in my model classes if I define the attributes clearly. Yes, I can use a simple Hashmap to simulate the HTTP header object but it really seems like reinventing the wheel to me.
On the auto-wiring side: I maintain an audit log record per request, which I implement by declaring a request-scoped Hashmap bean and autowiring it into any methods in the Servlet's call stack that needs to append data to the audit log. Basically it's just a global variable to hold some data within a single request. But in SCS, again, I lose that cos bean scopes that leverage on servlets are not available.
So far, there seems to be a lot of trade-offs that I have to make just to make Spring Cloud Stream work for me.
I thought about an alternative approach where I use SCS just to create an entry point but the Source method would just get the event, use a Processor to construct a HTTP request and send the request along to a HTTP endpoint. But, why go through all that trouble then?
Hoping that some more experienced devs would be able to shed some light on how they leverage on SCS.
#feicipet Thanks for the detailed question. let me try to address some of your concerns in the order you have listed them:
+1
+1
I am not sure why you are referring to it as servlet-based instead of Spring-based? Those are features provided by Spring, but read on. . .
Spring Cloud Stream doesn't use Kafka, the end user does while Spring Cloud Stream provides Kafka binder allowing Spring Cloud Stream to integrate with Kafka. Further more, while Kafka indeed did not support headers prior to version 0.11, Spring Cloud Stream always supported and will continue support headers even with Kafka pre-0.11, embedding them in the Message and then extracting them in the consumer side into the proper Message headers completely transparent to the end user. In other words one would assume that Kafka did support headers by simply using Spring Cloud Stream. With Kafka 0.11+ headers are supported natively and we have adjusted to that with the same level of transparency.
So, you don't need to put anything in the payload. Just create an appropriate Message<payload, headers> and SCSt will take care of the rest regardless of the broker (Kafka, Rabbit, Foo etc.).
Yes you do simply due to the fact that as you eluded earlier SCSt promotes an asynchronous and stateless architecture. However, I do not agree that what you are trying to accomplish is un-accomplishable. Rather it is accomplishable the way you are describing, but there are other way to maintain context and I would be more then glad to discuss it as a separate topic.
I would not call them trade-offs, rather difference in the architecture, that has its benefits, but it is a not one-size-fits-all architecture and therefore its viability should be discussed within the context of a concrete use case.
+1. You don't have to separate it as Source and Processor. You can simply create a custom Source app with exposed REST endpoint and custom processing logic. However we are currently working on enhancements i the framework to ensure that you could do the same with the existing starter apps.
Obviously we have touched on many points here and some of them would probably need to be debated further, but I hope this clears up some of your concerns.
Cheers
Using gRPC-Go is certainly a good way to build resilient, performant and scalable distributed systems. What I don't quite get is how to actually debug services written with it. Also, is HTTP/2 the only wire protocol supported?
Not sure if you've found a solution yet...but depending on how many services you have in your Application, you can use a distributed tracing system to record calls between services. Some of these systems include:
Zipkin (http://zipkin.io/)
Dapper (https://research.google.com/pubs/pub36356.html)
Jaeger (https://uber.github.io/jaeger/)
If you use the opentracing project (http://opentracing.io/), you can abstract your tracing code in your client and server from the code that transmits information to the Tracing System. For example, you can instrument your client and servers with Trace statements from OpenTracing, and then you can switch out your trace implementation for a Zipkin or Jaeger Tracer that pushes traces in the correct format.
There are opentracing bindings for gRPC reay to use. https://github.com/grpc-ecosystem/grpc-opentracing