Using gRPC-Go is certainly a good way to build resilient, performant and scalable distributed systems. What I don't quite get is how to actually debug services written with it. Also, is HTTP/2 the only wire protocol supported?
Not sure if you've found a solution yet...but depending on how many services you have in your Application, you can use a distributed tracing system to record calls between services. Some of these systems include:
Zipkin (http://zipkin.io/)
Dapper (https://research.google.com/pubs/pub36356.html)
Jaeger (https://uber.github.io/jaeger/)
If you use the opentracing project (http://opentracing.io/), you can abstract your tracing code in your client and server from the code that transmits information to the Tracing System. For example, you can instrument your client and servers with Trace statements from OpenTracing, and then you can switch out your trace implementation for a Zipkin or Jaeger Tracer that pushes traces in the correct format.
There are opentracing bindings for gRPC reay to use. https://github.com/grpc-ecosystem/grpc-opentracing
Related
I wrote some micro-services using Quarkus that communicate via Artemis. Now I want to add OpenTelemetry for tracing purpose.
What I already tried is to call service B from service A using HTTP/REST. Here the trace id from service A is automatically added to the header of the HTTP request and used in service B. So this works fine. In Jaeger I can see the correlation.
But how can this be achieved using Artemis as messaging system? Do I have to (manually) add the trace id from service A into the message and read it in service B to setup somehow the context (don't know whether this is possible)? Or is there possibly an automatism like for HTTP requests?
I would appreciate any assistance.
I have to mention at this point that I have little experience with tracing so far.
There is no quarkus, quarkiverse extension or smallrye lib that provides integration with Artemis and OpenTelemetry, yet.
Also, OpenTelemetry massaging spec is being worked at the moment, because the correct way to correlate sent, received messages and services is under definition at the OTel spec level.
However, I had exactly the same problem as you and did a manual instrumentation that you can use as inspiration: quarkus-observability-demo-activemq
It will correlate the sent service as parent of receiving end.
I'm going to implement observability across my projects which are working beside each other in a Microservices environment.
I've used MassTransit and I'm going to track the communication between projects that happen using MassTransit. Perhaps using tools like Jaeger to monitor these communications.
Is there any tools or approach in MassTransit that supports it?
MassTransit starts an Activity when it consumes a message, as well as publishes a message. It also propagates the activity id across service boundaries by adding it to the message headers.
There's an article about MassTransit diagnostics in the docs.
You can add the tracing option you want. For example, OpenTelemetry for .NET supports using diagnostic listeners for building traces as described in the docs. You can then use the Jaeger exporter for OpenTelemetry.
You can also use the OpenTracing contribution library for MassTransit. I have an example for my talk at NDC Oslo 2020, which has complete instrumentation for MassTransit with Prometheus and OpenTracing (with Jaeger). The Prometheus integration library is now a part of the MassTransit main repository.
I have created a microservice based architecture using Spring Boot and deployed the application on Kubernetes/Istio platform.
The different microservices communicate with each other using either JMS (ActiveMQ) or REST API.
I am getting the tracing of REST communication on Istio's Jaeger but the JMS based communication is missing in Jaeger.
I am using ElasticSearch to store my application logs.
Is it possible to use the same ElasticSearch as a backend(DB) of Jaeger?
If yes then I will store tracing specific logs in ElasticSearch and query them on Jaeger UI.
I believe you can reuse Elasticsearch for multiple purposes - each would use a different set of indices, so separation is good.
from: https://www.jaegertracing.io/docs/1.11/deployment/ :
Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends
Tying the networking all together, a docker-compose example:
How to configure Jaeger with elasticsearch?
While this isn't exactly what you asked, it sounds like what you're trying to achieve is seeing tracing for your JMS calls in Jaegar. If that is the case, you could use an OpenTracing tracing solution for JMS or ActiveMQ to report tracing data directly to Jaegar. Here's one potential solution I found with a quick google. There may be others.
https://github.com/opentracing-contrib/java-jms
Recently I am working with along with IOT department, right our project is on discussion and creating core architecture of an application. client specification is we must use MQTT protocol to communicate between device and java application (eclipse paho client).
its a web application based on spring boot and microservice architecture. but I an not able to find any good solution for API gateways that provide MQTT support.
I found zuul is good but do we have any alternative like kong..
MQTT is a TCP stream based protocol, so API Gateways that operate on the HTTP / Layer 7 are not going to fit the bill.
There are extensions to commercial API Gateways available, such as the Axway MQTT Proxy described here.
While not an API Gateway, Confluent also have a MQTT proxy that allows simple integration with Kafka, however if you have already written an application that implements the backend then Kafka is going to require some re-architecting.
The other options are really going for a simple TCP stream reverse proxy like nginx or HAProxy.
If I was asked to build something like this, I'd go straight to Kafka. It and MQTT are a very neat architectural fit and also operate very well together but it really depends on your requirements.
In issue comment msample asked:
We are looking at using OpenTracing and my searching for the on-wire/protocol for SpanContext led me here. I was surprised to not find this part solidly defined as it seems critical to the broad adoption of OpenTracing.
OpenTracing does not specify on-wire formats, neither for in-band messages like the span context passed via RPC messages, nor for the out of band messages like those used to send tracing spans out of application to the tracing backend. The reason for this is that such standardization is not necessary as long as the target architecture is using the OpenTracing libraries from the same tracing system (like Jaeger). OpenTracing API is first of all an instrumentation API for distributed systems. The wire formats are the implementation details of tracing systems implementing the API.
The wire format specification like https://github.com/TraceContext/tracecontext-spec is useful if one wants to pass trace info between services that are using different tracing backends, e.g. requests starting with services instrumented by Jaeger and then going onto services using StackDriver. But such integration has a whole set of different issues that need to be resolved, such as the sampling approach. There was no reason why OpenTracing API should be blocked by these separate standards.