I'm going to implement observability across my projects which are working beside each other in a Microservices environment.
I've used MassTransit and I'm going to track the communication between projects that happen using MassTransit. Perhaps using tools like Jaeger to monitor these communications.
Is there any tools or approach in MassTransit that supports it?
MassTransit starts an Activity when it consumes a message, as well as publishes a message. It also propagates the activity id across service boundaries by adding it to the message headers.
There's an article about MassTransit diagnostics in the docs.
You can add the tracing option you want. For example, OpenTelemetry for .NET supports using diagnostic listeners for building traces as described in the docs. You can then use the Jaeger exporter for OpenTelemetry.
You can also use the OpenTracing contribution library for MassTransit. I have an example for my talk at NDC Oslo 2020, which has complete instrumentation for MassTransit with Prometheus and OpenTracing (with Jaeger). The Prometheus integration library is now a part of the MassTransit main repository.
Related
I wrote some micro-services using Quarkus that communicate via Artemis. Now I want to add OpenTelemetry for tracing purpose.
What I already tried is to call service B from service A using HTTP/REST. Here the trace id from service A is automatically added to the header of the HTTP request and used in service B. So this works fine. In Jaeger I can see the correlation.
But how can this be achieved using Artemis as messaging system? Do I have to (manually) add the trace id from service A into the message and read it in service B to setup somehow the context (don't know whether this is possible)? Or is there possibly an automatism like for HTTP requests?
I would appreciate any assistance.
I have to mention at this point that I have little experience with tracing so far.
There is no quarkus, quarkiverse extension or smallrye lib that provides integration with Artemis and OpenTelemetry, yet.
Also, OpenTelemetry massaging spec is being worked at the moment, because the correct way to correlate sent, received messages and services is under definition at the OTel spec level.
However, I had exactly the same problem as you and did a manual instrumentation that you can use as inspiration: quarkus-observability-demo-activemq
It will correlate the sent service as parent of receiving end.
I am investigating solution to implement microservice Saga pattern in platform hosted in K8S in GCP.
There are 2 options: Eventulate Tram and Axon. However, these frameworks seem not to support message broker managed by cloud provider such as google-cloud-Pubsub whereas I do not want to deploy either Kafka or RabbitMQ to K8S since GCP support PubSub already.
So is there any way to integrate either Eventulate or Axon to use google cloud PubSub?
Thanks
Uncertain about Eventuate's angle on this, but Axon works with extensions as message brokers other than Axon Server. Throughout Axon's lifecycle (read: last 10 years), some of these have been provided, but none are currently used for all types of messages defined by Axon Framework. So, you wouldn't be able to use Kafka for sending commands in Axon for example.
Reasoning for this? Commands, events and queries have different routing requirements which should be reflected by using the right tool for the job.
To be a bit more specific on Axon's side, the following extensions can be used for distributing your messages:
AMQP -> for Events
Kafka -> for Events
JGroups -> for Commands
Spring Cloud Discovery -> for Commands
As you can tell, there currently is no Pub/Sub extension out there to allow you to distribute your messages. Added on top of that, my gut would tell me if it was available, then it would likely only be used for Event messages due to Pub/Sub's intent when it comes to being a message broker.
Luckily this actually makes it rather straightforward to create just such a extension yourself. Going into all the details to build this would be a little much, so I would recommend to have a look at Axon's AMQP extension first when it comes to achieving this. Hints on the matter are that for publication, you should add a component to handle Axon's events and publish them on Pub/Sub. For handling events, you are required to build a StreamableMessageSource or SubscribableMessageSource. These interfaces are used respectively by the TrackingEventProcessor and SubscribingEventProcessor, which in turn are the component in charge of dealing with the technical aspect of handling events.
By the way, if you would be building such an extension and you need a hand, it would be best to request this at AxonIQ forum, which you can find here.
Last note, and rather important I'd say, is the argument that such a connector would not be able to deal with all types of messages. If you would require a more full fledged Axon application to run in a distributed fashion, I would highly recommend to give Axon Server a try prior to building your own solution from the ground up.
I have a small stock-market application with Spring boot and if any product updated I want to serve an updated product to the clients in realtime
does it make sense to use message queues like RabbitMQ and Sse(Server Sent Events) for this, or is there a more sensible solution?
Solution
Publish your updated data to some channel
Your clients should subscribe to that channel to get updated feed in real-time.
Tools
Use in-house setup for RabbitMQ, ActiveMQ, Kafka or other open-source tools and implement WebSocket (For Front end applications)
Use commercial service like Google Cloud PubSub
Readymade and fully packaged solution with supported SDK for backend and frontend, https://www.pubnub.com/.
For this you can use either of
Spring Integration
Web Sockets
JMS
Spring Integration is an implementation of Enterprise Integration Patterns and is ideal for asynchronous processing data at realtime.
However, looking at your scope, it is only about publisher-subscriber pattern. Hence can be solved with JMS.
With JMS the subscribers/consumers can register/de-register dynamically. Also it provides ways to have fall-backs and tracking.
In issue comment msample asked:
We are looking at using OpenTracing and my searching for the on-wire/protocol for SpanContext led me here. I was surprised to not find this part solidly defined as it seems critical to the broad adoption of OpenTracing.
OpenTracing does not specify on-wire formats, neither for in-band messages like the span context passed via RPC messages, nor for the out of band messages like those used to send tracing spans out of application to the tracing backend. The reason for this is that such standardization is not necessary as long as the target architecture is using the OpenTracing libraries from the same tracing system (like Jaeger). OpenTracing API is first of all an instrumentation API for distributed systems. The wire formats are the implementation details of tracing systems implementing the API.
The wire format specification like https://github.com/TraceContext/tracecontext-spec is useful if one wants to pass trace info between services that are using different tracing backends, e.g. requests starting with services instrumented by Jaeger and then going onto services using StackDriver. But such integration has a whole set of different issues that need to be resolved, such as the sampling approach. There was no reason why OpenTracing API should be blocked by these separate standards.
Using gRPC-Go is certainly a good way to build resilient, performant and scalable distributed systems. What I don't quite get is how to actually debug services written with it. Also, is HTTP/2 the only wire protocol supported?
Not sure if you've found a solution yet...but depending on how many services you have in your Application, you can use a distributed tracing system to record calls between services. Some of these systems include:
Zipkin (http://zipkin.io/)
Dapper (https://research.google.com/pubs/pub36356.html)
Jaeger (https://uber.github.io/jaeger/)
If you use the opentracing project (http://opentracing.io/), you can abstract your tracing code in your client and server from the code that transmits information to the Tracing System. For example, you can instrument your client and servers with Trace statements from OpenTracing, and then you can switch out your trace implementation for a Zipkin or Jaeger Tracer that pushes traces in the correct format.
There are opentracing bindings for gRPC reay to use. https://github.com/grpc-ecosystem/grpc-opentracing