Greg Young, author of Event Store, defined a while ago the correlation ID to reference the root cause and the causation ID to reference the direct cause of a message.
Therefore you should be able to find the one message to which another message is a direct response through the other message's causation ID. And you should be able to find all messages within the same conversation by them having the same causation ID.
If this pattern is applied, the second message in a conversation should have both the correlation ID and the causation ID set to the first message.
In Commanded, I would have expected the first event to have both correlation ID and causation ID to point to the command which caused the event to occur (also visually described here).
Confusingly in Commanded's own event store, it seems, that the first event after a command has differing correlation and causation IDs. So it seems Commanded doesn't apply Greg Young's definition.
My question would be, what exactly is the definition for these IDs in Commanded? Was it a conscious decision not to follow Greg Young's definition, and if so, why?
Commanded has implemented correlation and causation ids as described in the guides, copied below.
By default commands will not be persisted, therefore you cannot follow the chain of cause and affect from command > event > command unless you also persist incoming commands. You can use Commanded audit middleware to store dispatched commands to any Ecto supported database. Doing so will allow you to tie commands and their resultant events together. The correlation id can also be used to associated all related messages, assuming you are copying them when dispatching commands. There's an example of doing so given below.
In Greg's Event Store I think the causation id is copied from event to event, ignoring the intermediate command, since his event store is only storing the events.
Correlation and causation ids
To assist with monitoring and debugging your deployed application it is useful to track the causation and correlation ids for your commands and events.
causation_id - the UUID of the command causing an event, or the event causing a command dispatch.
correlation_id - a UUID used to correlate related commands/events.
You can set causation and correlation ids when dispatching a command:
:ok = ExampleRouter.dispatch(command, causation_id: UUID.uuid4(), correlation_id: UUID.uuid4())
When dispatching a command in an event handler, you should copy these values from the event your are processing:
defmodule ExampleHandler do
use Commanded.Event.Handler, name: "ExampleHandler"
def handle(%AnEvent{..}, metadata) do
%{event_id: causation_id, correlation_id: correlation_id} = metadata
ExampleRouter.dispatch(%ExampleCommand{..},
causation_id: causation_id,
correlation_id: correlation_id
)
end
end
Commands dispatched by a process manager will be automatically assigned the appropriate causation and correlation ids from the source domain event.
Related
I often have to use the event model in an application based on Windows messages to notify the main thread of the application about some of the results of other threads. This approach allows you to get rid of blocking between threads.
The documentation says that system message IDs must be unique. But I did not find such a condition for application messages. At the same time, very often on the Internet you can find the opinion that within the application, message identifiers must be unique.
I see no point in being unique within the application. When sending a message, we specify a specific window handle that should process the message. I can use the same message ID in different windows, but it is the window handle that determines the recipient of the message. And it works.
Are there any obvious reasons to keep track of the uniqueness of message IDs in the application?
There is no requirement that all message IDs be unique within an application. In fact, we are using multiply assigned message IDs all the time: WM_USER-based messages are used by window classes to implement class-specific behavior. The most prominent example are the Common Controls that implement control-specific behavior using the [WM_USER..WM_APP) range of message IDs.
If you are calling RegisterClassEx you opt in to using WM_USER-based messages. You are free to reuse any value used by a different window class without risking a collision. It's the combination of the receiving window's class and the message ID that controls the behavior.
If you don't have a receiver (e.g. when calling PostThreadMessage) you would need to make sure that you can uniquely identify a message (and its payload). The easiest way in this case is to use unique message IDs in the WM_APP range.
My team is considering if we can use mass transit as a primary solution for sagas in RabbitMq (vs NServiceBus). I admit that our experience which solution like masstransit and nserviceBus are minimal and we have started to introduce messaging into our system. So I sorry if my question will be simple or even stupid.
However, when I reviewed the mass transit documentation I noticed that I am not sure if that is possible to solve one of our cases.
The case looks like:
One of our components will produce up to 100 messages which will be "sent" to queue. These messages are a result of a single operation in a system. All of the messages will have the same Correlated Id and our internal publication id (same too).
1) is it possible to define a single instance saga (by correlated id) which will wait until it receives all messages from a queue and then process them as a single batch?
2) otherwise, is there any solution to ensure all of the sent messages was processed? (Consistency batch?) I assume that correlated Id will serve as a way to found an existing saga instance (singleton). In the ideal case, I would like to complete an instance of a saga When the system will process every message which belongs to a single group (to one publication)
I look at CompositeEvent too but I do not sure if I could use it to "ensure" that every message was processed and then I would let to complete saga for specific correlated Id.
Can you explain how could it be achieved? And into what mechanism I should look at in order to correlated id a lot of messages with the same id to the single saga and then complete if all of msg will be consumed?
Thank you in advance for any response
What you describe is how correlation by id works. It is like that out of the box.
So, in short - when you configure correlation for your messages correctly, all messages with the same correlation id will be handled by the same saga instance.
Concerning the second question - unless you publish a separate event that would inform the saga about how messages it should expect, how would it know that? You can definitely schedule a long timeout, attempting and assuming that within the timeout all the messages will be received by the saga, but it's not reliable.
Composite events won't help here since they are for messages with different types to be handled as one when all of them arrive and it doesn't count for the number of messages of each type. It just waits for one message of each type.
The ability to receive a series of messages and then operate on them in a batch is a common case, so much so that there is a sample showing how to do just that:
Batch Sample
Each saga instance has a unique correlation identifier, and as long as those messages can be correlated to that single instance, MassTransit will manage the concurrency (either optimistic or pessimistic, and depending upon the saga storage engine).
I'd suggest reviewing the state machine in the sample, and seeing how that compares to your scenario.
As to my understanding, in event sourcing, events are recorded. However that would also mean a state changed first happened and thereafter we record the event. For example, assuming:
A Client sends a command to a server to "Create user".
The server validates the command and creates user i.e. stores new
user in a database.
The server then logs/stores a Created User event. i.e event
sourcing.
Created User event is propagated to subscribers
In the scenario above, how do we handle scenarios where step (2) succeeded but step (3) failed due to say network failures, database offline etc? The whole system would be in an indeterminate state now that there was a new user created but the event was never logged. How do we mitigate these types of failures? Or are the steps that I've listed above not the way to do event sourcing?
Thanks!
This is not what happens exactly in Event sourcing, not even in plain CQRS.
In Event sourcing, after the command is validated, the domain events are generated by the source (the Aggregate in DDD) and then they are appended to the Event store in the first step. After that the subscribers (read models, projections, Sagas, external systems) receive and process the new domain events.
In CQRS, after the domain events are generated, they are applied to the Aggregate and then the Aggregate's state and the new events are persisted in the same local transaction, as the first step. Only after that the subscribers receive the events.
So you see? Your situation cannot happen: steps 2 and 3 are persisted atomically, they succeed or fail together.
I was trying to understanding ES+CQRS and tech stack can be used.
As per my understanding flow should be as below.
UI sends a request to Controller(HTTP Adapter)
Controller calls application service by passing Request Object as parameter.
Application Service creates Command from Request Object passed from controller.
Application Service pass this Command to Message Consumer.
Message Consumer publish Command to message broker(RabbitMQ)
Two Subscriber will be listening for above command
a. One subscriber will generate Aggregate from eventStore using command
and will apply command than generated event will be stored in event store.
b. Another subscriber will be at VIEW end,that will populate data in view database/cache.
Kindly suggest my understanding is correct.
Kindly suggest my understanding is correct
I think you've gotten a bit tangled in your middleware.
As a rule, CQRS means that the writes happen to one data model, and reads in another. So the views aren't watching commands, they are watching the book of record.
So in the subscriber that actually processes the command, the command handler will load the current state from the book of record into memory, update the copy in memory according to the domain model, and then replace the state in the book of record with the updated version.
Having update the book of record, we can now trigger a refresh of the data model that backs the view; no business logic is run here, this is purely a transform of the data from the model we use for writes to the model we use for reads.
When we add event sourcing, this pattern is the same -- the distinction is that the data model we use for writes is a history of events.
How atomicity is achieved in writing data in event store and writing data in VIEW Model?
It's not -- we don't try to make those two actions atomic.
how do we handle if event is stored in EventStrore but System got crashed before we send event in Message Queue
The key idea is to realize that we typically build new views by reading events out of the event store; not by reading the events out of the message queue. The events in the queue just tell us that an update is available. In the absence of events appearing in the message queue, we can still poll the event store watching for updates.
Therefore, if the event store is unreachable, you just leave the stale copy of the view in place, and wait for the system to recover.
If the event store is reachable, but the message queue isn't, then you update the view (if necessary) on some predetermined schedule.
This is where the eventual consistency part comes in. Given a successful write into the event store, we are promising that the effects of that write will be visible in a finite amount of time.
This should be easy to follow, but after some reading I still can find an answer.
So, say that the user needs to change his mobile number, to accomplished that, we might have a command as: ChangedUserMobileNumber
holding the new number. The domain responsible for handling the command will perform the change in the aggregate and publish an event: UserMobilePhoneChanged
There is a subscriber for that event in another domain, which also holds the user mobile number in its aggregate but according to our software architect, events can not old any data so what we end up is rather stupid to say the least:
The Domain 1, receives the command to update the mobile number, the number is updated and one event is published, also, because the event cannot hold data, the command handler in the Domain 1 issues yet another command which is sent to Domain 2. The subscriber of that event lives in Domain 2 too, we then have a Saga to handle both the event and the command.
In terms of implementation we are using NServiceBus, so we have this saga to handle these message and in it we have this line of code, where the entity.IsMobilePhoneUpdated field stored in a saga entity is changed when the event is handeled.
bool isReady = (entity.IsMobilePhoneUpdated && entity.MobilePhoneNumber != null);
Effectively the Saga is started by both the command and the event raised in the Domain 1, and until this condition is met, the saga is kept alive.
If it was up to me, I would be sending the mobile number in the event itself, I just want to get a few other opinions on this.
Thanks
I'm not sure how a UserMobilePhoneChanged event could be useful in any way unless it contained the new phone number. User asks to change a number, the event shoots out that it has. Should be very simple indeed. Why does your architect say that events shouldn't contain any information?
In the first event based system i've designed events also had no data. I also did enforce that rule. At the time that sounded like a clever decision. After a while i realised that it was dumb, and i was making a lot of workarounds because of it. Also this caused a lot of querying form the event subscribers, even for trivial data. I had no problem changing this "rule" after i realised i'm doing it wrong.
Events should have all the data required to make them meaningful. Also they should only have the data that makes sense for that event. ( No point in having the user address in a ChangePhoneNumber message )
If your architect imposes such a restriction, it's not going to be easy to develop a CQRS system. How are the read models updated? Since the events have no data then you either query something to get the data ( the write side ? ) of find some way of sending a command to the read model ( then what's the point of publishing events? ). To fix your problem you should try to have a professional discussion with this architect, preferably including other tech heads and without offending anybody try to get him to relax this constraint.
On argument you could use is Event Sourcing. Event Sourcing is complementary to CQRS and would not make sense without events that have data. Even more when using event sourcing, the only data you have is the data stored in the events. Even if you don't actually implement event sourcing you can use it's existence as a reason for events to have data.
There is little point in finding a technical solution to a people problem.