I have 5 web applications which I developed using Springboot(A,B,C,D and E).
Below are the flow of 2 calls
First Flow:
A's Controllers --> A's Service --> B's Controller --> B'Service --> C's Controller --> C's Service --> C's Dao --> DB
SecondFlow:
A's Controllers --> A's Service --> D's Controller --> D'Service --> B's Controller --> B'Service --> C's Controller --> C's Service --> C's Dao --> DB
Once fetch/push data from/into the DB then the corresponding methods are returning some value. For each and every method logging the status (input details and returning status). I am able to see logs in each service separately. But I want to see complete one request-response(A's controller request to A's controller response) cycle logs in one file.
How Can I achieve it?
This is a ver bad idea, but let's take a step back and look at the problem instead of guessing a solution.
You have multiple applications that collaborate together to execute (distributed) transactions. You need to trace those interactions to see your dataflow. This is very useful for many reasons so it's correct that you care about it. It is also correct to collect all you log entries in a single sink, even if it won't be a file because it is not well suited to manage prodcution workloads. A typical scenario that many organization implements is the following
Each application send logs to files or standard output
For each node of you infrastructure there is an agent that reads that streams, does some basic conversion (eg. translates log entries in a common format) and send data to a certain sink
The sink is a database, the best technology option is a DBMS without strict requirements about data schema (you are storing everything in a single huge table after all) and transactional properites (if the data are logs, you are fine with an optimistic concurrency control). You also want some tool that is more good at reads than writes and have good performance in complex searches to drill down a large amount of structured data
a Dashboard to read logs, make searches and even create dashboard with synthetic stats about events
BONUS: use a buffer to manage load spikes
There are precise tools to do the job and they are
logstash/beats/fluentd
Elasticsearch....what else? ;)
Kibana, the favourite Elasticsearch client
BONUS: rabbimq/kafka/otherMessageBroker or Redis
but you still miss a step
Suppose you call a REST API, something simple like a POST /users/:userId/cart
API Gateway receives your request with a JWT
API Gateway calls Authentication-service to validate and decode the JWT
API Gateway calls Authorization-service to check if the client as right to perform the request
API Gateway calls User-service to find :userId
User-Service calls Cart-service to add the product on :userId cart
Cart-Service calls Notification-Service to decide whether is needed to send a notification for the completed task
Notification-Service calls Push-Gateway to invoke an external push notification service
....and back
to not get lost in this labyrinth you NEED just one thing: the correlation ID
Correlation IDs attach a unique ID to all interaction beetween these microservices (headers in HTTP calls or AMQP messages, for instance) and your custom log library (because you've already built a custom logging library and shared it among all the teams) capture this ID to include it in every log entry wrote in the context of the single request processed from each of those microservices. You can even add that correlation ID in the client response, catch it if the respone carry out an error code and perform a query on your logs DB to find all the entries with the given correlation ID. If the system clock works, they will be retrieved in the correct time order and you will be able to reconstruct the dataflow
Distributed Systems make everything more complicate and add a lot of overhead on things that we ever done before, but if you put in your pocket the right tools to manage the complexity, you can see the benefits
you can implement central logging system like:
The ELK stack (Elastic Search, Logstash and Kibana) for Centralized Logging.
Related
Problem:
Suppose there are two services A and B. Service A makes an API call to service B.
After a while service A falls down or to be lost due to network errors.
How another services will guess that an outbound call from service A is lost / never happen? I need some another concurrent app that will automatically react (run emergency code) if service A outbound CALL is lost.
What are cutting-edge solutions exist?
My thoughts, for example:
service A registers a call event in some middleware (event info, "running" status, timestamp, etc).
If this call is not completed after N seconds, some "call timeout" event in the middleware automatically starts the emergency code.
If the call is completed at the proper time service A marks the call status as "completed" in the same middleware and the emergency code will not be run.
P.S. I'm on Java stack.
Thanks!
I recommend to look into patterns such as Retry, Timeout, Circuit Breaker, Fallback and Healthcheck. Or you can also look into the Bulkhead pattern if concurrent calls and fault isolation are your concern.
There are many resources where these well-known patterns are explained, for instance:
https://www.infoworld.com/article/3310946/how-to-build-resilient-microservices.html
https://blog.codecentric.de/en/2019/06/resilience-design-patterns-retry-fallback-timeout-circuit-breaker/
I don't know which technology stack you are on but usually there is already some functionality for these concerns provided already that you can incorporate into your solution. There are libraries that already take care of this resilience functionality and you can, for instance, set it up so that your custom code is executed when some events such as failed retries, timeouts, activated circuit breakers, etc. occur.
E.g. for the Java stack Hystrix is widely used, for .Net you can look into Polly .Net to make use of retry, timeout, circuit breaker, bulkhead or fallback functionality.
Concerning health checks you can look into Actuator for Java and .Net core already provides a health check middleware that more or less provides that functionality out-of-the box.
But before using any libraries I suggest to first get familiar with the purpose and concepts of the listed patterns to choose and integrate those that best fit your use cases and major concerns.
Update
We have to differentiate between two well-known problems here:
1.) How can service A robustly handle temporary outages of service B (or the network connection between service A and B which comes down to the same problem)?
To address the related problems the above mentioned patterns will help.
2.) How to make sure that the request that should be sent to service B will not get lost if service A itself goes down?
To address this kind of problem there are different options at hand.
2a.) The component that performed the request to service A (which than triggers service B) also applies the resilience patterns mentioned and will retry its request until service A successfully answers that it has performed its tasks (which also includes the successful request to service B).
There can also be several instances of each service and some kind of load balancer in front of these instances which will distribute and direct the requests to an available instance (based on regular performed healthchecks) of the specific service. Or you can use a service registry (see https://microservices.io/patterns/service-registry.html).
You can of course chain several API calls after another but this can lead to cascading failures. So I would rather go with an asynchronous communication approach as described in the next option.
2b.) Let's consider that it is of utmost importance that some instance of service A will reliably perform the request to service B.
You can use message queues in this case as follows:
Let's say you have a queue where jobs to be performed by service A are collected.
Then you have several instances of service A running (see horizontal scaling) where each instance will consume the same queue.
You will use message locking features by the message queue service which makes sure that as soon one instance of service A reads a message from the queue the other instances won't see it. If service A was able to complete it's job (i.e. call service B, save some state in service A's persistence and whatever other tasks you need to be included for a succesfull procesing) it will delete the message from the queue afterwards so no other instance of service A will also process the same message.
If service A goes down during the processing the queue service will automatically unlock the message for you and another instance A (or the same instance after it has restarted) of service A will try to read the message (i.e. the job) from the queue and try to perform all the tasks (call service B, etc.)
You can combine several queues e.g. also to send a message to service B asynchronously instead of directly performing some kind of API call to it.
The catch is, that the queue service is some highly available and redundant service which will already make sure that no message is getting lost once published to a queue.
Of course you also could handle jobs to be performed in your own database of service A but consider that when service A receives a request there is always a chance that it goes down before it can save that status of the job to it's persistent storage for later processing. Queue services already address that problem for you if chosen thoughtfully and used correctly.
For instance, if look into Kafka as messaging service you can look into this stack overflow answer which relates to the problem solution when using this specific technology: https://stackoverflow.com/a/44589842/7730554
There is many way to solve your problem.
I guess you are talk about 2 topics Design Pattern in Microservices and Cicruit Breaker
https://dzone.com/articles/design-patterns-for-microservices
To solve your problem, Normally I put a message queue between services and use Service Discovery to detect which service is live and If your service die or orverload then use Cicruit Breaker methods
I've been aware of event sourcing, CQRS, DDD and micro services for a little while and I'm now at that point where I want to try and start implementing stuff and giving something a go.
I've been looking into the technical side of CQRS and I understand the DDD concepts in there. How both the write side handles commands from the UI and publishes events from it, and how the read side handles events and creates projections on them.
The difficulty I'm having is the communication & a handling events from service-to-service (both from a write to read service and between micro services).
So I want to focus on eventstore (this one: https://eventstore.com/ to be less ambiguous). This is what I want to use as I understand it is a perfect for event sourcing and the simple nature of storing the events means I can use this for a message bus as well.
So my issue falls into two questions:
Between the write and the read, in order for the read side to receive/fetch the events created from the write side, am i right in thinking something like a catch up subscription can be used to subscribe to a stream to receive any events written to it or do i use something like polling to fetch events from a given point?
Between micro services, I am having an even harder time... So when looking at CQRS tutorials/talks etc... they always seem to talk with an example of an isolated service which receives commands from the UI/API. This is fine. I understand the write side will have an API attached to it so the user can interact with it to perform commands. E.g. create a customer. However... say if I have two micro services, e.g. a order micro service and an shipping micro service, how does the shipping micro service get the events published from the order micro service. Specifically, how does those customer events, translate to commands for the shipping service.
So let's take a simple example of: - Command created from the order's API to place an order. - A OrderPlacedEvent is published to the event store. How does the shipping service listen and react to this is it need to then DispatchOrder and create ain turn an OrderDispatchedEvent.
Does the write side of the shipping microservice then need to poll or also have a catch up subscription to the order stream? If so how does an event get translated to an command using DDD approach?
something like a catch up subscription can be used to subscribe to a stream to receive any events written to it
Yes, using catch-up subscriptions is the right way of doing it. You need to keep the stream position of your subscription persisted somewhere as well.
Here you can find some sample code that works. I am not posting the whole snippet since it is too long.
The projection service startup flow is:
Load the checkpoint (first time ever it would be the stream start)
Subscribe to the stream from that checkpoint
The runtime flow will then be:
The subscription will then call the function you provide when it receives an event. There's some plumbing there to do, like if you subscribe to $all, you need to filter out system events (it will be easier in the next version of Event Store)
Project the event
Store the new checkpoint
If you make your projections idempotent, you can store the checkpoint from time to time and save some IO.
how does the shipping micro service get the events published from the order micro service
When you build a brand new system and you have a small team working on all the components, you can make a shortcut and subscribe to domain events from another service, as you'd do with projections. Within the integration context (between the boxes), ordering should not be important so you can use persistent subscriptions so you won't need to think about checkpoints. Event Store will do it for you.
Be aware that it introduces tight coupling on the domain event schema of the originating service. Your contexts will have the Partnership relationship or the downstream service will be a Conformist.
When you move forward with your system, you might decide to decouple those contexts properly. So, you introduce a stable event API for the service that publishes events for others to consume. The same subscription that you used for integration can now instead take care of translating domain (internal) events to integration (external) events. The consuming context would then use the stable API and the domain model of the upstream service will be free in iterating on their domain model, as soon as they keep the conversion up-to-date.
It won't be necessary to use Event Store for the downstream context, they could just as well use a message broker. Integration events usually don't need to be persisted due to their transient nature.
We are running a webinar series about Event Sourcing at Event Store, check our web site to get on-demand access to previous webinars and you might find interesting to join future ones.
The difficulty I'm having is the communication & a handling events from service-to-service (both from a write to read service and between micro services).
The difficulty is not your fault - the DDD literature is really weak when it comes to discussing the plumbing.
Greg Young discusses some of the issues of subscription in the latter part of his Polygot Data talk.
Eventide Project has documentation that does a decent job of explaining the principles behind how the plumbing fits things together.
Between micro services, I am having an even harder time...
The basic idea: your message store is fundamentally a database; when the host of your microservice wakes up, it queries the message store for messages after some checkpoint, and then feeds them to your domain logic (updating its own local copy of the checkpoint as needed).
So the host pulls a document with events in it from the store, and transforms that document into a stream of handle(Event) commands that ultimately get passed to your domain component.
Put another way, you build a host that polls the database for information, parses the response, and then passes the parsed data to the domain model, and writes its own checkpoints.
I am using OpenCensus in Go to push tracing data to Stackdriver for calls involving a chain of 2 or more micro services and I noticed that I get many traces which contain spans only for certain services but not the entire end to end call.
At the moment I attribute this to the fact that not all calls are traced (only a certain sample) and each service decides whether to trace its current span or not.
Is this the way it is intended to work? Is there any way to make sure when a trace is sampled, it is done so by all services in the call chain?
Architecturally I will say when you are developing your microservices make sure your API Gateway creates a unique id like GUID, which gets propagated through all the microservices & similarly you make sure that you have log aggregator which will be collecting logs from all the services & finally you are getting nice tracebility of request.
Currently learning spring integration, I want to retrieve information from a MySQL database to use inside an int:service-activator, or an int:splitter .
Unfortunately, it would seem that most examples and documentation is based around the idea of using an int-jdbc:inbound-channel-adapter, which in itself requires a poller. I don't want to poll a database, but rather retrieve specific data based on the payload of an existing message originating from an int:gateway. This data would then be used to further modify the payload, or assist in how the message is split.
I tried using int-jdbc:outbound-gateway, as the description states:
... jdbc.JdbcOutboundGateway' for updating a database in response to a message on the request channel, and/or for retrieving data from the database ...
This implies that it can be used for retrieval of data only and not just updates, but as I implement it, there's a complaint that at least one update statement is required:
And so I'm currently sitting with a faulty prototype that initially looks like so:
The circled piece being the non-functioning int-jdbc:outbound-gateway.
My end goal is to, based on the payload coming from the incomingGateway (in the picture above), retrieve some information from a MySQL database, and use that data to split the message in the analyzerSplitter, or to perhaps modify the payload using an int:service-activator. This should then all be linked up to a int-jdbc:message-store which I believe could assist with performance. I do not wish to poll the database on a regular basis, and I do not wish to update anything in the database.
By testing using the polling int-jdbc:inbound-channel-adapter, I am confident that my datasource bean is set up correctly and the query can execute.
How would I go about correctly setting up such behaviour in spring integration?
If you want to proceed with the flow after updating the database, you can simply use a JdbcTemplate in a method invoked by a service activator, or, if it's the end of the flow, use an outbound channel adapter.
The outbound channel adapter is the inverse of the inbound: its role is to handle a message and use it to execute a SQL query. By default, the message payload and headers are available as input parameters to the query, as the following example shows:
...
Suppose I have service which does the following:
Receives input notification
Processes input notification which means:
some computing
storing in DB
some computring
generating it's own notification
Sends its own notification to multiple clients
What is the best practice in this case, should I granularly trace each operation like computing, storing in db etc with separate span or leave that for metrics (i.e. prometheus) and create single span for the whole notification processing?
It's somewhat up to you as to the granularity that's appropriate for your application, and also the volume of tracing data you're expecting to generate. An application handling a few requests per minute is going to have different needs than one handling 1000s of requests per second.
That said, I recommend creating spans when control flow enters or leaves your application (such as when your application starts processing a request or message from an external system, and when your application calls out to an external dependency, such as HTTP requests, sending notifications, or writing/reading from the database), and using logs/tags for everything that's internal to your application.