New to Masstransit here and I wanted to correlate my message with an int value (let's call it OrderId of type INT. I will use the same order management example from Masstransit).
I wanted to see if anyone was able to use an int to correlate events and messages in Masstransit.
I used this code in my events (both the events)
x.CorrelateById<int>(state => state.OrderId, context => context.Message.OrderId)
This compiles fine but then throws and exception
NotImplementedByDesignException. Masstransit doc says : "Redis only supports event correlation by CorrelationId, it does not support queries. If a saga uses expression-based correlation, a NotImplementedByDesignException will be thrown."
I am using CorrelateById() so not sure why I am seeing this exception. I don't see any query here (or is it this func that returns the OrderId? The correlationId has a similar expression although it takes only one argument of type: Func<ConsumeContext<TMessage>, Guid> selector)
All I want is to be able to correlate message and event by an Int prop coming to my first event of the state machine. (That I have no control over by the way).
I feel creating another GUID just for Masstransit and link it to that OrderId is not the best option here.
My other question: is there any out of the box way in MassTransit to get the all event data for a specific CorrelationId (event sourcing style). I saw there was a MassTransit.EventStoreIntegration repo on github but since we can get the last version of the state instance I thought there would be a way to see all state instances/ or messages being persisted then pulled.
Sure I can see that in the log but I would like to see only the state changes that were pushed.
Thanks
Correlation by anything other than the Guid CorrelationId property is a query. And, as you've found out, Redis doesn't support queries. If you need to correlate events using a type other than a Guid, Redis is not an option.
As per #Chris answer, and because I wanted to use an INT property and still use redis, I used Guid CorrelationId that was generated from that INT value with padded zeros.
CorrelationId = new Guid(OrderIdIntValue.ToString().PadLeft(32, '0'))
I am aware of the Guid vs int discussion in distributed systems.
For my case this works with no issues since that Int value is generated by a unique source of truth (that I consume and don't have control over).
Related
Below is a sample POC developed in ASP.net Core 6.0 API that uses MassTransit and RabbitMQ to simulate a simple publish/subscribe using MassTransit consumer. However when the code is executed it results in creation of 2 Exchanges and 1 Queue in RabbitMQ.
Program.cs
builder.Services.AddMassTransit(msConfig =>
{
msConfig.AddConsumers(Assembly.GetEntryAssembly());
msConfig.UsingRabbitMq((hostcontext, cfg) =>
{
cfg.Host("localhost", 5700, "/", h =>
{
h.Username("XXXXXXXXXXX");
h.Password("XXXXXXXXXXX");
});
cfg.ConfigureEndpoints(hostcontext);
});
});
OrderConsumer.cs
public class OrderConsumer : IConsumer<OrderDetails>
{
readonly ILogger<OrderConsumer> _logger;
public OrderConsumer(ILogger<OrderConsumer> logger)
{
_logger = logger;
}
public Task Consume(ConsumeContext<OrderDetails> context)
{
_logger.LogInformation("Message picked by OrderConsumer. OrderId : {OrderId}", context.Message.OrderId);
return Task.CompletedTask;
}
}
Model
public class OrderDetails
{
public int OrderId { get; set; }
public string OrderName { get; set; }
public int Quantity { get; set; }
}
Controller
readonly IPublishEndpoint _publishEndpoint;
[HttpPost("PostOrder")]
public async Task<ActionResult> PostOrder(OrderDetails orderDetails)
{
await _publishEndpoint.Publish<OrderDetails>(orderDetails);
return Ok();
}
Output from Asp.Net
As highlighted 2 Exchanges are created Sample:OrderDetails and Order.
However, the Sample:OrderDetails is bound to Order (Exchange)
And the Order (Exchange) routes to "Order" queue.
So, the question is regarding the 2 Exchanges that got created where I am not sure if that's per design or its a mistake on the code that led to both getting created and if its per design, why the need for 2 exchange.
I was pondering the same question when I first started playing with MassTransit, and in the end came to understand it as follows:
You are routing two types of messages via MassTransit, events and commands. Events are multicast to potentially multiple consumers, commands to a single consumer. Every consumer has their own input queue to which messages are being routed via exchanges.
For every message type, MassTransit by default creates one fanout exchange based on the message type and one fanout exchange and one queue for every consumer of this message.
This makes absolute sense for events, as you are publishing events using the event type (with no idea who or if anyone at all will consume it), so in your case, you publish to the OrderDetails exchange. MassTransit has to make sure that all consumers of this event are bound to this exchange. In this case, you have one consumer, OrderConsumer. MassTransit by default generates the name of the consumer exchange based on the type name of this consumer, removing the Consumer suffix. The actual input queue for this consumer is bound to this exchange.
So you get something like this:
EventTypeExchange => ConsumerExchange => ConsumerQueue
or in your case:
Sample:OrderDetails (based on the type Sample.OrderDetails) => Order (based on the type OrderConsumer) => Order (again based on the OrderConsumer type)
For commands this is a bit less obvious, because a command can only ever be consumed by one consumer. In fact you can actually tell MassTransit not to create the exchanges based on the command type. However, what you would then have to do is route commands not based on the command type, but on the command handler type, which is really not a good approach as now you would have to know - when sending a command - what the type name of the handler is. This would introduce coupling that you really do not want. Thus, I think it's best to keep the exchanges based on the command type and route to them, based on the command type.
As Chriss (author of MassTransit) mentions in the MassTransit RabbitMQ deep dive video (YouTube), this setup also allows you to potentially do interesting stuff like siphon off messages to another queue for monitoring/auditing/debugging, just by creating a new queue and binding it to the existing fanout exchange.
All the above is based on me playing with the framework, so it's possible I got some of this wrong, but it does make sense to me at least. RabbitMQ is extremely flexible with its routing options, so Chriss could've chosen a different approach (e. g. Brighter, a "competing" library uses RabbitMQ differently to achieve the same result) but this one has merit as well.
MassTransit also - unlike some other frameworks like NServiceBus or Brighter - doesn't really technically distinguish or care about the semantic difference between these two, e. g. you can just as well send or publish a command just as you can an event.
I'm bothering with situation when I want to emit query update via queryUpdateEmitter but in different module (microservice). I have application built upon microservices and both are connected to the same Axon Server. First service creates subscriptionQuery, and sends some commands. After a while (through few commands and events) second service handles some event, and emits update for firstly subscribed query. Unfortunately it seems like this emit doesn't get to subscriber. Queries are exactly the same and sits in the same packages.
Subscription:
#GetMapping("/refresh")
public Mono<MovieDTO> refreshMovies() {
commandGateway.send(
new CreateRefreshMoviesCommand(UUID.randomUUID().toString()));
SubscriptionQueryResult<MovieDTO, MovieDTO> refreshedMoviesSubscription =
queryGateway.subscriptionQuery(
new GetRefreshedMoviesQuery(),
ResponseTypes.instanceOf(MovieDTO.class),
ResponseTypes.instanceOf(MovieDTO.class)
);
return refreshedMoviesSubscription.updates().next();
}
Emitter:
#EventHandler
public void handle(DataRefreshedEvent event) {
log.info("[event-handler] Handling {}, movieId={}",
event.getClass().getSimpleName(),
event.getMovieId());
queryUpdateEmitter.emit(GetRefreshedMoviesQuery.class, query -> true,
Arrays.asList(
MovieDTO.builder().aggregateId("as").build(),
MovieDTO.builder().aggregateId("be").build()));
}
This situation is even possible in the newest version of Axon? Similar configuration but within one service is working as expected.
#Edit
I have found a workardound for this situation:
Second service instead of emitting query via queryUpdateEmitter, publishes event with list of movies
First service handles this event and then emits update via queryUpdateEmitter
But still I'd like to know if there is a way to do this using queries only, because it seems natural to me (commandGateways/eventGateways works as expected, queryUpdateEmitter is the exception).
This follows from the implementation of the QueryUpdateEmitter (regardless of using Axon Server yes/no).
The QueryUpdateEmitter stores a set of update handlers, referencing the issued subscription queries. It however only maintains the issued subscription queries handled by the given JVM (as the QueryUpdateEmitter implementation is not distributed).
It's intent is to be paired in the component (typically a Query Model "projector") which answers queries about a given model, updates the model and emits those updates.
Hence, placing the QueryUpdateEmitter operations in a different (micro)service as where the query is handled will not work.
I'm wondering if I'm doing something wrong, I expected MassTransit would automatically register ReceiveEndpoints in the EndpointConvention.
Sample code:
services.AddMassTransit(x =>
{
x.AddServiceBusMessageScheduler();
x.AddConsumersFromNamespaceContaining<MyNamespace.MyRequestConsumer>();
x.UsingAzureServiceBus((context, cfg) =>
{
// Load the connection string from the configuration.
cfg.Host(context.GetRequiredService<IConfiguration>().GetValue<string>("ServiceBus:ConnectionString"));
cfg.UseServiceBusMessageScheduler();
// Without this line I'm getting an error complaining about no endpoint convention for x could be found.
EndpointConvention.Map<MyRequest>(new Uri("queue:queue-name"));
cfg.ReceiveEndpoint("queue-name", e =>
{
e.MaxConcurrentCalls = 1;
e.ConfigureConsumer<MyRequestConsumer>(context);
});
cfg.ConfigureEndpoints(context);
});
});
I thought this line EndpointConvention.Map<MyRequest>(new Uri("queue:queue-name")); wouldn't be necessary to allow sending to the bus without specifing the queue name, or am I missing something?
await bus.Send<MyRequest>(new { ...});
The EndpointConvention is a convenience method that allows the use of Send without specifying the endpoint address. There is nothing in MassTransit that will automatically configured this because, frankly, I don't use it. And I don't think anyone else should either. That stated, people do use it for whatever reason.
First, think about the ramifications - if every message type was registered as an endpoint convention, what about messages that are published and consumed on multiple endpoints? That wouldn't work.
So, if you want to route messages by message type, MassTransit has a feature for that. It's called Publish and it works great.
But wait, it's a command, and commands should be Sent.
That is true, however, if you are in control of the application and you know that there is only one consumer in your code base that consumes the KickTheTiresAndLightTheFires message contract, publish is as good as send and you don't need to know the address!
No, seriously dude, I want to use Send!
Okay, fine, here are the details. When using ConfigureEndpoints(), MassTransit uses the IEndpointNameFormatter to generate the receive endpoint queue names based upon the types registered via AddConsumer, AddSagaStateMachine, etc. and that same interface can be used to register your own endpoint conventions if you want to use Send without specifying a destination address.
You are, of course, coupling the knowledge of your consumer and message types, but that's your call. You're already dealing with magic (by using Send without an explicit destination) so why not right?
string queueName = formatter.Consumer<T>()
Use that string for the message types in that consumer as a $"queue:{queueName}" address and register it on the EndpointConvention.
Or, you know, just use Publish.
I have a simple route like this
from("file:data/inbox?noop=true").transform().body().to("file:data/outbox").bean(UpdateInventory.class);
from("direct:update").to("file:data/anotherbox").to("direct:newupdate");
from("direct:newupdate").to("file:data/newbox");
And the output i am expecting is
-file://data/inbox
--transform[simple{body}] 15 ms
--file:data/outbox 5
---bean[com.classico.sample.UpdateInventory#3a469fea 19
---file:data/anotherbox 6
----direct:newupdate 5
-----file:data/newbox 4
I tried using a EventNotifier and when the ExchangeCompletedEvent is received i fetched the message History.But since the second exchange is completed first message history is showing up in the reverse order of invocation.Is it possible to store all the messgae histories in a collection and print them in reverse order or Is there any event that is suitable for this.?
if (event instanceof ExchangeCompletedEvent) {
ExchangeCompletedEvent exchangeCompletedEvent = (ExchangeCompletedEvent) event;
Exchange exchange = exchangeCompletedEvent.getExchange();
String routeId = exchange.getFromRouteId();
List<MessageHistory> list = exchange.getProperty(Exchange.MESSAGE_HISTORY, List.class);
for (MessageHistory history : list) {
String id = history.getNode().getId();
String label = URISupport.sanitizeUri(history.getNode().getLabel());
log.info(String.format(MESSAGE_HISTORY_OUTPUT, routeId, id, label, history.getElapsed()));
}
}
You can use JMX to get all that details for each processor.
There is also a dumpRouteStatsAsXml operation on each route / camelContext that can output a xml file of the route(s) with all performance stats.
We use this in the hawtio web console to list this kind of information.
http://hawt.io/
Also the Camel Karaf / Jolokia Commands uses this as well
https://github.com/apache/camel/blob/master/platforms/commands/commands-core/src/main/java/org/apache/camel/commands/ContextInfoCommand.java#L175
And in next release of Camel you can also easier get the various processor mbeans, from CamelContext if you know their id, using
https://github.com/apache/camel/blob/master/camel-core/src/main/java/org/apache/camel/CamelContext.java#L545
Then you can use the getters on the mbean to get the performance stats.
The event notifer which was suggested is also great, but the events are on a higher level, although you get an event for sending to an endpoint, such as to some external system, which often is enough to capture details about. For low level details as asked here, then you need to use the JMX stats.
Ohh I forgot to tell about the message history EIP which also has a trace of how the message was routed with time taken stats as well.
http://camel.apache.org/message-history.html
That is maybe also just what you need, then you can get that information from the exchange as shown on that link.
I would suggest using the camel EventNotifier. You can find documentation on how to use it here:
http://camel.apache.org/eventnotifier-to-log-details-about-all-sent-exchanges.html
I'd like to understand some details of the relations between command handlers, aggregates, the repository and the event store in CQRS-based systems.
What I've understood so far:
Command handlers receive commands from the bus. They are responsible for loading the appropriate aggregate from the repository and call the domain logic on the aggregate. Once finished, they remove the command from the bus.
An aggregate provides behavior and an internal state. State is never public. The only way to change state is by using the behavior. The methods that model this behavior create events from the command's properties, and apply these events to the aggregate, which in turn call an event handlers that sets the internal state accordingly.
The repository simply allows loading aggregates on a given ID, and adding new aggregates. Basically, the repository connects the domain to the event store.
The event store, last but not least, is responsible for storing events to a database (or whatever storage is used), and reloading these events as a so-called event stream.
So far, so good.
Now there are some issues that I did not yet get:
If a command handler is to call behavior on a yet existing aggregate, everything is quite easy. The command handler gets a reference to the repository, calls its loadById method and the aggregate is returned. But what does the command handler do when there is no aggregate yet, but one should be created? From my understanding the aggregate should later-on be rebuilt using the events. This means that creation of the aggregate is done in reply to a fooCreated event. But to be able to store any event (including the fooCreated one), I need an aggregate. So this looks to me like a chicken-and-egg problem: I can not create the aggregate without the event, but the only component that should create events is the aggregate. So basically it comes down to: How do I create new aggregates, who does what?
When an aggregate triggers an event, an internal event handler responses to it (typically by being called via an apply method) and changes the aggregate's state. How is this event handed over to the repository? Who originates the "please send the new events to the repository / event store" action? The aggregate itself? The repository by watching the aggregate? Someone else who is subscribed to the internal events? ...?
Last but not least I have a problem understanding the concept of an event stream correctly: In my imagination, it's simply something like an ordered list of events. What's of importance is that it's "ordered". Is this right?
The following is based on my own experience and my experiments with various frameworks like Lokad.CQRS, NCQRS, etc. I'm sure there are multiple ways to handle this. I'll post what makes most sense to me.
1. Aggregate Creation:
Every time a command handler needs an aggregate, it uses a repository. The repository retrieves the respective list of events from the event store and calls an overloaded constructor, injecting the events
var stream = eventStore.LoadStream(id)
var User = new User(stream)
If the aggregate didn't exist before, the stream will be empty and the newly created object will be in it's original state. You might want to make sure that in this state only a few commands are allowed to bring the aggregate to life, e.g. User.Create().
2. Storage of new Events
Command handling happens inside a Unit of Work. During command execution every resulting event will be added to a list inside the aggregate (User.Changes). Once execution is finished, the changes will be appended to the event store. In the example below this happens in the following line:
store.AppendToStream(cmd.UserId, stream.Version, user.Changes)
3. Order of Events
Just imagine what would happen, if two subsequent CustomerMoved events are replayed in the wrong order.
An Example
I'll try to illustrate the with a piece of pseudo-code (I deliberately left repository concerns inside the command handler to show what would happen behind the scenes):
Application Service:
UserCommandHandler
Handle(CreateUser cmd)
stream = store.LoadStream(cmd.UserId)
user = new User(stream.Events)
user.Create(cmd.UserName, ...)
store.AppendToStream(cmd.UserId, stream.Version, user.Changes)
Handle(BlockUser cmd)
stream = store.LoadStream(cmd.UserId)
user = new User(stream.Events)
user.Block(string reason)
store.AppendToStream(cmd.UserId, stream.Version, user.Changes)
Aggregate:
User
created = false
blocked = false
Changes = new List<Event>
ctor(eventStream)
isNewEvent = false
foreach (event in eventStream)
this.Apply(event, isNewEvent)
Create(userName, ...)
if (this.created) throw "User already exists"
isNewEvent = true
this.Apply(new UserCreated(...), isNewEvent)
Block(reason)
if (!this.created) throw "No such user"
if (this.blocked) throw "User is already blocked"
isNewEvent = true
this.Apply(new UserBlocked(...), isNewEvent)
Apply(userCreatedEvent, isNewEvent)
this.created = true
if (isNewEvent) this.Changes.Add(userCreatedEvent)
Apply(userBlockedEvent, isNewEvent)
this.blocked = true
if (isNewEvent) this.Changes.Add(userBlockedEvent)
Update:
As a side note: Yves' answer reminded me of an interesting article by Udi Dahan from a couple of years ago:
Don’t Create Aggregate Roots
A small variation on Dennis excellent answer:
When dealing with "creational" use cases (i.e. that should spin off new aggregates), try to find another aggregate or factory you can move that responsibility to. This does not conflict with having a ctor that takes events to hydrate (or any other mechanism to rehydrate for that matter). Sometimes the factory is just a static method (good for "context"/"intent" capturing), sometimes it's an instance method of another aggregate (good place for "data" inheritance), sometimes it's an explicit factory object (good place for "complex" creation logic).
I like to provide an explicit GetChanges() method on my aggregate that returns the internal list as an array. If my aggregate is to stay in memory beyond one execution, I also add an AcceptChanges() method to indicate the internal list should be cleared (typically called after things were flushed to the event store). You can use either a pull (GetChanges/Changes) or push (think .net event or IObservable) based model here. Much depends on the transactional semantics, tech, needs, etc ...
Your eventstream is a linked list. Each revision (event/changeset) pointing to the previous one (a.k.a. the parent). Your eventstream is a sequence of events/changes that happened to a specific aggregate. The order is only to be guaranteed within the aggregate boundary.
I almost agree with yves-reynhout and dennis-traub but I want to show you how I do this. I want to strip my aggregates of the responsibility to apply the events on themselves or to re-hydrate themselves; otherwise there is a lot of code duplication: every aggregate constructor will look the same:
UserAggregate:
ctor(eventStream)
foreach (event in eventStream)
this.Apply(event)
OrderAggregate:
ctor(eventStream)
foreach (event in eventStream)
this.Apply(event)
ProfileAggregate:
ctor(eventStream)
foreach (event in eventStream)
this.Apply(event)
Those responsibilities could be left to the command dispatcher. The command is handled directly by the aggregate.
Command dispatcher class
dispatchCommand(command) method:
newEvents = ConcurentProofFunctionCaller.executeFunctionUntilSucceeds(tryToDispatchCommand)
EventDispatcher.dispatchEvents(newEvents)
tryToDispatchCommand(command) method:
aggregateClass = CommandSubscriber.getAggregateClassForCommand(command)
aggregate = AggregateRepository.loadAggregate(aggregateClass, command.getAggregateId())
newEvents = CommandApplier.applyCommandOnAggregate(aggregate, command)
AggregateRepository.saveAggregate(command.getAggregateId(), aggregate, newEvents)
ConcurentProofFunctionCaller class
executeFunctionUntilSucceeds(pureFunction) method:
do this n times
try
call result=pureFunction()
return result
catch(ConcurentWriteException)
continue
throw TooManyRetries
AggregateRepository class
loadAggregate(aggregateClass, aggregateId) method:
aggregate = new aggregateClass
priorEvents = EventStore.loadEvents()
this.applyEventsOnAggregate(aggregate, priorEvents)
saveAggregate(aggregateId, aggregate, newEvents)
this.applyEventsOnAggregate(aggregate, newEvents)
EventStore.saveEventsForAggregate(aggregateId, newEvents, priorEvents.version)
SomeAggregate class
handleCommand1(command1) method:
return new SomeEvent or throw someException BUT don't change state!
applySomeEvent(SomeEvent) method:
changeStateSomehow() and not throw any exception and don't return anything!
Keep in mind that this is pseudo code projected from a PHP application; the real code should have things injected and other responsibilities refactored out in other classes. The ideea is to keep aggregates as clean as possible and avoid code duplication.
Some important aspects about aggregates:
command handlers should not change state; they yield events or
throw exceptions
event applies should not throw any exception and should not return anything; they only change internal state
An open-source PHP implementation of this could be found here.