I'm experimenting with microservices, event sourcing and CQRS. However, I'm a little bit confused about how I go from issuing a command to performing a query to return the new state, specifically with regard to interactions with a web API gateway.
As an example, the simple application I am attempting to write (which probably doesn't actually need any of these; it is just something to aid my learning) creates a random-graph and then performs some long-running calculations on the graph. I've modelled this as two separate services: the GraphService and the ComputationService. The imagined process flow is as follows:
User requests new random graph.
API gateway constructs CreateGraph command and sends it to the
graph service.
GraphService command handler creates a graph and publishes a
GraphCreated event.
GraphService event handler subscribes to topic for graph events,
processes GraphCreated event and stores graph in persistent read
storage.
Client somehow gets the newly created graph.
ComputationService event handler subscribes to topic for graph
events, processes GraphCreated event and begins potentially
long-running computation, e.g. calculate diameter.
ComputationService publishes DiameterComputed event.
GraphService event handler subscribes to topic for computation
events, processed DiameterComputed event and updates the graph in
persistent read storage.
Client somehow gets updated - easier than getting the new graph, since already have an ID and can poll for changes / websockets / SSE, etc.
That seems relatively simple. However, my confusion lies in how to go about informing the API gateway, and thus the web client, of the new graph (as highlighted in bold above). In a typical CRUD process, the result of the POST request to create a new graph would be to return the URL of the new resource, for instance. However, with CQRS, commands should return nothing or an exception.
How do I pass information back to the client of the service (in this case the API gateway) about the ID of the new graph so that it can perform a query to get the representation of the new resource and send it to the user? Or at least get an ID so that the web client can ask the API gateway, etc?
As I see it at the moment, after sending a command, everyone is just left hanging. There needs to be some sort of subscription model that can be interrogated for the status of the graph creation. I considered having the API gateway generate a request ID which gets embedded with the CreateGraph command, but this then couples the service to the API.
I'm obviously missing something, but have no idea what. None of the examples I've looked at or discussions I've read address this issue and assume that the ID of whatever resource is already known. I couldn't find any discussions here addressing this issue, but if I've just missed them, please point me there rather than duplicating questions. Any pointers would be hugely welcomed.
How do I pass information back to the client of the service (in this case the API gateway) about the ID of the new graph so that it can perform a query to get the representation of the new resource and send it to the user? Or at least get an ID so that the web client can ask the API gateway, etc?
By listening for the echo.
The basic idea behind at least once delivery is that I'm going to send you a message, and keep sending it over and over until I receive a message that proves you've received at least one copy of my message.
Therefore, my protocol looks something like
Establish a mail box where I can collect messages
Encode into the message instructions for delivering to my mailbox
Send the message to you
Check my mailbox
if the answer is there, I'm done
otherwise, I send you another copy of the message
The mail box could be implemented any number of ways -- it could be a callback; it could be a promise, it could be a correlation identifier. You could have the signal dispatched by the command handler, when it gets acknowledgement of the write by the book of record, or by the "read model" when the new resource is available.
Related
I am prettty new to microservices, and I am trying to figure out how to set a micro-service architecture in which my publisher that emits an event, can receive a response with data from the consumer within the publisher?
From what i have read about message-brokers and message-queues, it seems like it's one-way communication. The producer emits an event (or rather, sends a message) which is handled by the message broker, and then the consumer consumes that event and performs some action.
This allows for decoupled code, which is part of what im looking for, but i dont understand if the consumer is able to return any data to the caller.
Say for example I have a microservice that communicates with an external API to fetch data. I want to be able to send a message or emit an event from my front-facing server, which then calls the service that fetches data, parses the data, and then returns that data back to my servver1 (front-facing server)
Is there a way to make message brokers or queues bidirectional? Or is it only useable in one direction. I keep reading message brokers allow services to communicate with each other, but I only find examples in which data flow goes one way.
Even reading rabbitMQ documentation hasn't really made it very clear to me how i could do this
In general, when talking about messaging, it's one-way.
When you send a letter to someone you're not opening up a mind-meld so that they telepathically communicate their response to you.
Instead, you include a return address (or some other means of contacting you).
So to map a request-response interaction when communicating with explicit messaging (e.g. via a message queue), the solution is the same: you include some directions which the recipient can/will interpret as "send a response here". That could, for instance be, "publish a message on this queue with this correlation ID".
Your publisher then, after sending this message, subscribes to the queue it's designated and waits for a message with the expected correlation ID.
Needless to say, this is fairly elaborate: you are, in some sense, reimplementing a decent portion of a session protocol like TCP on top of a datagram protocol like IP (albeit in this case, we may have some stronger reliability guarantees than we'd get from IP). It's worth noting that this sort of request-response interaction intrinsically couples the two parties (we can't really say "sender and receiver": each is the other's audience), so we're basically putting in some effort to decouple the two sides and then some more effort to recouple them.
With that in mind, if the actual business use case calls for a request-response interaction like this, consider implementing it with an actual request-response protocol (e.g. REST over HTTP or gRPC...) and accept that you have this coupling.
Alternatively, if you really want to pursue loose coupling, go for broke and embrace the asynchronicity at the heart of the universe (maybe that way lies true enlightenment?). Have your publisher return success with that correlation ID as soon as its sent its message. Meanwhile, have a different service be tracking the state of those correlation IDs and exposing a query interface (CQRS, hooray!). Your client can then check at any time whether the thing it wanted succeeded, even if its connection to your publisher gets interrupted.
Queues are the wrong level of abstraction for request-reply. You can build an application out of them, but it would be nontrivial to support and operate.
The solution is to use an orchestration system like temporal.io or AWS Step Functions. These services out of the box provide state management, asynchronous communication, and automatic recovery in case of various types of failures.
Let's consider the flow below:
API client calls [POST] /api/v1/invitation/:InvitationId/confirm
Confirm the invitation within a SAGA
Eventually raise an InvitationConfirmed event to indicate success
We are some troubles finding a good place to validate the "event" we pass to the SAGA. For instance, we want to make sure that:
- The specified InvitationId exists
- The corresponding invitation is not expired or already processed
We tried a couple of things:
Fire a command:
Fire a command RequestInvitationConfirmation
Handle synchronously this command and return an error if the command is not valid OR otherwise raise the InvitationConfirmationRequested event.
The rest of the flow is the same
CONS:
- Requires us to follow a "request/response" pattern (synchronous within the HTTP request lifetime)
Raise an event:
Raise an event InvitationConfirmationRequested
Within the SAGA, query the Invitation service and perform the validations. If the command is not valid, we publish an event InvitationConfirmationFailed
(...)
CONS:
- As far as I understand SAGA should be used to orchestrate the flow. Here we are introducing the concept of "validation". I'm not sure it's the recommended approach.
Validation is a very common concept. How would you handle it in a distributed fully asynchronous system?
Important point in the design of this system is: "Who is the client of this API?".
If this client is an internal Service or Application that's one thing (as in a distributed app, microservices etc.).
If the API is used by third party client's, that's another thing.
Short answer
If the API is used internally between Services, sending a command with invalid Id in the system is a fault, so it should be logged and examined by the system developers. Also cases like these should be accounted for by having a manual way of fixing them (by some administrative backend). Log these kinds of stuff and notify developers.
If the API is used from third party apps, then it matters how responsibilities are separated between the API and the other part of the system that it uses. Make the API responsible for validation and don't send commands with invalid id's. Treat command with invalid ID's like fault, as in the first case. In this case if you use asynchronous flow, you will need a way to communicate with the third party app to notify it. You can use something like WebHooks.
For the second part of the validations check these series of blog posts and the original paper.
Long answer
If you search around you will see a lot of discussions on errors and validations, so here's my take on that.
Since we do separation of other parts of our systems, it's seems natural to separate the types of error that we have. You can check this paper on that topic.
Let's define some error types.
Domain Errors
Application Errors
Technical Errors (database connections lost etc.)
Because we have different types of errors, the validation should be performed from different parts of our systems.
Also the communication of these errors can be accomplished by different mechanisms depending on:
the requester of the operation and the receiver
the communication channel used
the communication type: synchronous or asynchronous
Now the validations that you have are:
Validate that an Invitation with the specified Id exists
Validate that the Invitation has not expired
Validate that the Invitation is not already processed (accepted, rejected etc.)
How this is handled will depend on how we separate the responsibilities in our application. Let's use the DesignByContract principle and define clear rules what each layer (Domain, Application etc.) should expect from the other ones.
Let's define a rule that a Command containing an InvitationId that doesn't correspond to an existing Invitation should not be created and dispatched.
NOTE the terminology used here can vary vastly depending of what type of architecture is used on the project (Layered Architecture, Hexagonal etc.)
This forces the CommandCreator to validate that an Invitation exists with the specified Id before dispatching the command.
In the case with the API, the RouteHandler (App controller etc.) that will accept the request will have to:
perform this validation himself
delegate to someone else to do the validation
Let's further define that this is part of our ApplicationLayer (or module, components etc. doesn't matter how it's called, so I'll use Layer) and make this an ApplicationError. From here we can do it in many different ways.
One way is to have a DispatchConfirmInvitationCommandApplicationService that will ask the DomainLayer if an Invitation with the requested Id exists and raise an error (throw exception for example) if it doesn't. This error will be handled by the RouteHandler and will be send back to the requester.
You can use both a sync and async communication. If it's async you will need to create a mechanism for that. You can refer to EnterpriseIntegrationPatterns for more information on this.
The main point here is: It's not part of the Domain
From here on, everyone else in our system should consider that the invitation with the specified Id in the ConfirmInvitationCommand exists. If it doesn't, it's treated like a fault in the system and should be checked by developers and/or administrators. There should be a manual way (an administrative backend) to cancel such invalid commands, so this must be taken into account when developing the system, bu treated like a fault in the system.
The other two validations are part of the Domain.
So let's say you have a
Invitation aggregate
InvitationConfirmationSaga
Let's make them these aggregates communicate with messages. Let's define these types of messages:
RequestConfirmInvitation
InvitationExpired
InvitationAlreadyProcessed
Here's the basic flow:
ConfirmInvitationCommand starts a InvitationConfirmationSaga
InvitationConfirmationSaga send RequestConfirmInvitation message to Invitation
And then:
If the Invitation is expired it sends InvitationExpired message to InvitationConfirmationSaga
If the Invitation is processed it sends InvitationAlreadyProcessed message to InvitationConfirmationSaga
If the Invitation is not expired it, it's accepted and it sends InvitationAccepted message to InvitationConfirmationSaga
Then:
InvitationConfirmationSaga will receive these messages and raise events accordingly.
This way you keep the domain logic in the Domain, in this case the Invitation Aggregate.
You have a command ConfirmInvitation containing InvitationId. You send it to your Invitation domain from InvaitationAppService. Your Invitation domain should look like this
...
public void ConfirmInvitation()
{
if (this.Status == InvitationStatus.Confirmed)
throw new InvalidInvitationException("Requested invitation has already been confirmed");
//check more business logic here
this.Status = InvitationStatus.Confirmed;
Publish(new InviationConfirmedEvent(...));
}
...
Your InvitationAppService should have something like below:
...
public void ConfirmInvitation(Guid invitationId)
{
// rehydrate your domain from eventstore
var invitation = repo.GetById<Invitation>(invitationId);
if (invitation == null)
throw new InvalidInvitationException("Invalid Invitation requested");
invitation.ConfirmInvitation(new ConfirmInvitation(...));
}
You don't need to introduce a new event InvitationConfirmationRequested. DDD is an approach in which your domain/business validation should reside inside domains. Don't try to fit other patterns or technologies in your domain. Validating your domain inside saga(which is used to orchestrate distribute transactions across the services) might create complexities and chaos
Let's say we have two services A and B. B has a relation to A so it needs to know about the existing entities of A.
Service A publishes events every time an entity is created or updated. Service B subscribes to the events published by A and therefore knows about the entities existing in service A.
Problem: The client (UI or other micro services) creates a new entity 'a' and right away creates a new entity 'b' with a reference to 'a'. This is done without much delay so what happens if service B did not receive/handle the event from B before getting the create request with a reference to 'b'?
How should this be handled?
Service B must fail and the client should handle this and possibly do retry.
Service B accepts the entity and over time expect the relation to be fulfilled when the expected event is received. Service B provides a state for the entity that ensures it cannot be trusted before the relation have been verified.
It is poor design that the client can/has to do these two calls in the same transaction. The design should be different. How?
Other ways?
I know that event platforms like Kafka ensures very fast event transmittance but there will always be a delay and since this is an asynchronous process there will be kind of a race condition.
What you're asking about falls under the general category of bridging the gap between Eventual Consistency and good User Experience which is a well-documented challenge with a distributed architecture. You have to choose between availability and consistency; typically you cannot have both.
Your example raises the question as to whether service boundaries are appropriate. It's a common mistake to define microservice boundaries around Entities, but that's an anti-pattern. Microservice boundaries should be consistent with domain boundaries related to the business use case, not how entities are modeled within those boundaries. Here's a good article that discusses decomposition, but the TL;DR; is:
Microservices should be verbs, not nouns.
So, for example, you could have a CreateNewBusinessThing microservice that handles this specific case. But, for now, we'll assume you have good and valid reasons to have the services divided as they are.
The "right" solution in your case depends on the needs of the consuming service/application. If the consumer is an application or User Interface of some sort, responsiveness is required and that becomes your overriding need. If the consumer is another microservice, it may well be that it cares more about getting good "finalized" data rather than being responsive.
In either of those cases, one good option is a facade (aka gateway) service that lives between your client and the highly-dependent services. This service can receive and persist the request, then respond however you'd like. It can give the consumer a 200 - OK response with an endpoint to call back to check status of the request - very responsive. Or, it could receive a URL to use as a webhook when the response is completed from both back-end services, so it could notify the client directly. Or it could publish events of its own (it likely should). Essentially, you can tailor the facade service to provide to as many consumers as needed in the way each consumer wants to talk.
There are other options too. You can look into Task-Based UI, the Saga pattern, or even just Faking It.
I think you would like to leverage the flexibility of a broker and the confirmation of a synchronous call . Both of them can be achieved by this
https://www.rabbitmq.com/tutorials/tutorial-six-dotnet.html
I am designing and developing a microservice platform based on the specifications of http://microservices.io/
The entire framework integrates through socket thus removing the overhead of multiple HTTP requests (like most REST APIs).
A service registry host receives the registry of multiple microservice hosts, each microservice is responsible for a domain of the business. Another host we call a router (or API gateway) is responsible for exposing the microservices for consumption by third parties.
We will use the structure of Sagas (in choreography style) to distribute the requisitions, so we have some doubts:
Should a microservice issue the event in any process manager or should it be passed directly to the next microservice responsible for the chain of events? (the same logic applies to rollback)
Who should know how to build the Saga chain of events? The first microservice that receives a certain work or the router?
If an event needs to pass a very large volume of data to the next Saga event, how is this done in terms of the request structure? Is it divided into multiple Sagas for example (as a result pagination type)?
I think the main point is that in this router and microservice structure, who is responsible for building the Sagas and propagating their events.
The article Patterns for Microservices — Sync vs. Async does a great job defining many of the terms used here and has animated gifs demonstrating sync vs. async and orchestrated vs. choreographed as well as hybrid setups.
I know the OP answered his own question for his use case, but I want to try and address the questions raised a bit more generally in lieu of the linked article.
Should a microservice issue the event in any process manager or should it be passed directly to the next microservice responsible for the chain of events?
To use a more general term, a process manager is an orchestrator. A concrete implementation of this may involve a stateful actor that orchestrates a workflow, keeping track of the progress in some way. Since a saga is workflow itself (composed of both forward and compensating actions), it would be the job of the process manager to keep track of the state the saga until completion (success or failure). This typically involves the actor sending synchronous* calls to services waiting for some result before going to the next step. Parallel operations can of course be introduced and what not, but the point is that this actor dictates the progression of the saga.
This is fundamentally different from the choreography model. With this model there is no central actor keeping track of the state of a saga, but rather the saga progresses implicitly via the events that each step emits. Arguably, this is a more pure case of an event-driven model since there is no coordination.
That said, the challenge with this model is observing the state at any given point in time. With the orchestration model above, in theory, each actor could be queried for the state of the saga. In this choreographed model, we don't have this luxury, so in practice a correlation ID is added to every message corresponding to (in this case) a saga. If the messages are queryable in some way (the event bus supports it or through some other storage means), then the messages corresponding to a saga could be queried and the saga state could be reconstructed.. (effectively an event sourced modeled).
Who should know how to build the Saga chain of events? The first microservice that receives a certain work or the router?
This is an interesting question by itself and one that I have been thinking about quite a lot. The easiest and default answer would be.. hard code the saga plans and map them to the incoming message types. E.g. message A triggers plan X, message B triggers plan Y, etc.
However, I have been thinking about what a control plane might look like that manages these plans and provides the mechanism for pushing changes dynamically to message handlers and/or orchestrators dynamically. The two specific use cases in mind are changes in authorization policies or dynamically adding new steps to a plan.
If an event needs to pass a very large volume of data to the next Saga event, how is this done in terms of the request structure? Is it divided into multiple Sagas for example (as a result pagination type)?
The way I have approached this is to include references to the large data if these are objects such as a file or something. For data that are inherently streams themselves, a parallel channel could be referenced that a consumer could read from once it receives the message. I think the important distinction here is to decouple thinking about the messages driving the workflow from where the data is physically materialized which depends on the data representation.
For microservices, every microservice should be responsible for its domain business.
Should a microservice issue the event in any process manager or should it be passed directly to the next microservice responsible for the chain of events? (the same logic applies to rollback)
All events are not passed to the next microservice, but are published, then all microservices interested in the events should subscribe to them.
If there is rollback, you should consider orchestration.
Who should know how to build the Saga chain of events? The first microservice that receives a certain work or the router?
The microservice who publish the event will certainly know how to build it. There are no chain of events, because every microservice interested in the event will subscribe it separately.
If an event needs to pass a very large volume of data to the next Saga event, how is this done in terms of the request structure? Is it divided into multiple Sagas for example (as a result pagination type)?
Only publish the data others may be interested, not all. In most cases, the data are not large, and message queue can handle them efficiently
I was trying to understanding ES+CQRS and tech stack can be used.
As per my understanding flow should be as below.
UI sends a request to Controller(HTTP Adapter)
Controller calls application service by passing Request Object as parameter.
Application Service creates Command from Request Object passed from controller.
Application Service pass this Command to Message Consumer.
Message Consumer publish Command to message broker(RabbitMQ)
Two Subscriber will be listening for above command
a. One subscriber will generate Aggregate from eventStore using command
and will apply command than generated event will be stored in event store.
b. Another subscriber will be at VIEW end,that will populate data in view database/cache.
Kindly suggest my understanding is correct.
Kindly suggest my understanding is correct
I think you've gotten a bit tangled in your middleware.
As a rule, CQRS means that the writes happen to one data model, and reads in another. So the views aren't watching commands, they are watching the book of record.
So in the subscriber that actually processes the command, the command handler will load the current state from the book of record into memory, update the copy in memory according to the domain model, and then replace the state in the book of record with the updated version.
Having update the book of record, we can now trigger a refresh of the data model that backs the view; no business logic is run here, this is purely a transform of the data from the model we use for writes to the model we use for reads.
When we add event sourcing, this pattern is the same -- the distinction is that the data model we use for writes is a history of events.
How atomicity is achieved in writing data in event store and writing data in VIEW Model?
It's not -- we don't try to make those two actions atomic.
how do we handle if event is stored in EventStrore but System got crashed before we send event in Message Queue
The key idea is to realize that we typically build new views by reading events out of the event store; not by reading the events out of the message queue. The events in the queue just tell us that an update is available. In the absence of events appearing in the message queue, we can still poll the event store watching for updates.
Therefore, if the event store is unreachable, you just leave the stale copy of the view in place, and wait for the system to recover.
If the event store is reachable, but the message queue isn't, then you update the view (if necessary) on some predetermined schedule.
This is where the eventual consistency part comes in. Given a successful write into the event store, we are promising that the effects of that write will be visible in a finite amount of time.