We have UserCreated event that gets published from UserManagement.Api. I have two other Apis, Payments.Api and Notification.Api that should react to that event.
In both Apis I have public class UserCreatedConsumer : IConsumer<UserCreated> (so different namespaces) but only one queue (on SQS) gets created for both consumers.
What is the best way to deal with this situation?
You didn't share your configuration, but if you're using:
x.AddConsumer<UserCreatedConsumer>();
As part of your MassTransit configuration, you can specify an InstanceId for that consumer to generate a unique endpoint address.
x.AddConsumer<UserCreatedConsumer>()
.Endpoint(x => x.InstanceId = "unique-value");
Every separate service (not an instance of the same service) needs to have a different queue name of the receiving endpoint, as described in the docs:
cfg.ReceiveEndpoint("queue-name-per-service-type", e =>
{
// rest of the configuration
});
It's also mentioned in the common mistakes article.
Related
We have 5 topics and we want to have a service that scales for example to 5 instances of the same app.
This would mean that i would want to dynamically (via for example Redis locking or similar mechanism) determine which instance should listen to what topic.
I know that we could have 1 topic that has 5 partitions - and each node in the same consumer group would pick up a partition. Also if we have a separately deployed service we can set the topic via properties.
The issue is that those two are not suitable for our situation and we want to see if it is possible to do that via what i explained above.
#PostConstruct
private void postConstruct() {
// Do logic via redis locking or something do determine topic
dynamicallyDeterminedVariable = // SOME LOGIC
}
#KafkaListener(topics = "{dynamicallyDeterminedVariable")
void listener(String data) {
LOG.info(data);
}
Yes, you can use SpEL for the topic name.
#{#someOtherBean.whichTopicToUse()}.
Below is a sample POC developed in ASP.net Core 6.0 API that uses MassTransit and RabbitMQ to simulate a simple publish/subscribe using MassTransit consumer. However when the code is executed it results in creation of 2 Exchanges and 1 Queue in RabbitMQ.
Program.cs
builder.Services.AddMassTransit(msConfig =>
{
msConfig.AddConsumers(Assembly.GetEntryAssembly());
msConfig.UsingRabbitMq((hostcontext, cfg) =>
{
cfg.Host("localhost", 5700, "/", h =>
{
h.Username("XXXXXXXXXXX");
h.Password("XXXXXXXXXXX");
});
cfg.ConfigureEndpoints(hostcontext);
});
});
OrderConsumer.cs
public class OrderConsumer : IConsumer<OrderDetails>
{
readonly ILogger<OrderConsumer> _logger;
public OrderConsumer(ILogger<OrderConsumer> logger)
{
_logger = logger;
}
public Task Consume(ConsumeContext<OrderDetails> context)
{
_logger.LogInformation("Message picked by OrderConsumer. OrderId : {OrderId}", context.Message.OrderId);
return Task.CompletedTask;
}
}
Model
public class OrderDetails
{
public int OrderId { get; set; }
public string OrderName { get; set; }
public int Quantity { get; set; }
}
Controller
readonly IPublishEndpoint _publishEndpoint;
[HttpPost("PostOrder")]
public async Task<ActionResult> PostOrder(OrderDetails orderDetails)
{
await _publishEndpoint.Publish<OrderDetails>(orderDetails);
return Ok();
}
Output from Asp.Net
As highlighted 2 Exchanges are created Sample:OrderDetails and Order.
However, the Sample:OrderDetails is bound to Order (Exchange)
And the Order (Exchange) routes to "Order" queue.
So, the question is regarding the 2 Exchanges that got created where I am not sure if that's per design or its a mistake on the code that led to both getting created and if its per design, why the need for 2 exchange.
I was pondering the same question when I first started playing with MassTransit, and in the end came to understand it as follows:
You are routing two types of messages via MassTransit, events and commands. Events are multicast to potentially multiple consumers, commands to a single consumer. Every consumer has their own input queue to which messages are being routed via exchanges.
For every message type, MassTransit by default creates one fanout exchange based on the message type and one fanout exchange and one queue for every consumer of this message.
This makes absolute sense for events, as you are publishing events using the event type (with no idea who or if anyone at all will consume it), so in your case, you publish to the OrderDetails exchange. MassTransit has to make sure that all consumers of this event are bound to this exchange. In this case, you have one consumer, OrderConsumer. MassTransit by default generates the name of the consumer exchange based on the type name of this consumer, removing the Consumer suffix. The actual input queue for this consumer is bound to this exchange.
So you get something like this:
EventTypeExchange => ConsumerExchange => ConsumerQueue
or in your case:
Sample:OrderDetails (based on the type Sample.OrderDetails) => Order (based on the type OrderConsumer) => Order (again based on the OrderConsumer type)
For commands this is a bit less obvious, because a command can only ever be consumed by one consumer. In fact you can actually tell MassTransit not to create the exchanges based on the command type. However, what you would then have to do is route commands not based on the command type, but on the command handler type, which is really not a good approach as now you would have to know - when sending a command - what the type name of the handler is. This would introduce coupling that you really do not want. Thus, I think it's best to keep the exchanges based on the command type and route to them, based on the command type.
As Chriss (author of MassTransit) mentions in the MassTransit RabbitMQ deep dive video (YouTube), this setup also allows you to potentially do interesting stuff like siphon off messages to another queue for monitoring/auditing/debugging, just by creating a new queue and binding it to the existing fanout exchange.
All the above is based on me playing with the framework, so it's possible I got some of this wrong, but it does make sense to me at least. RabbitMQ is extremely flexible with its routing options, so Chriss could've chosen a different approach (e. g. Brighter, a "competing" library uses RabbitMQ differently to achieve the same result) but this one has merit as well.
MassTransit also - unlike some other frameworks like NServiceBus or Brighter - doesn't really technically distinguish or care about the semantic difference between these two, e. g. you can just as well send or publish a command just as you can an event.
Sometimes, you want to say, "I have this message, who can handle it?"
In nestjs a client proxy is bounded directly to a single microservice.
So, as an example, let say that I have the following micro-services:
CleaningService, FixingService.
Both of the above can handle the message car, but only CleaningService can handle the message glass.
So, I want to have something like:
this.generalProxy.emit('car', {id: 2});
In this case, I want 2 different microservices to handle the car: CleaningService and FixingService.
in this case:
this.generalProxy.emit('glass', {id: 5});
I want only CleaningService to handle it.
How is that possible? how can I create clientProxy that is not bonded directly to a specific microservice.
The underlying transport layer matters because despite the fact that there is an abstraction in front of the different transports each underlying one has completely different characteristics and capabilities. The type of messaging pattern you're talking about is simple to accomplish with RabbitMQ because it has the notion of exchanges, queues, publisher, subscribers etc while a TCP based microservice requires a connection from one service to another. Likewise, the Redis transport layer uses simple channels without the necessary underlying implementation to be able to support some messages being fanned out to multiple subscribers and some going directly to specific subscribers.
This might not be the most popular opinion but I've been using NestJS professionally for over 3 years and I can definitely say that the official microservices packages are not sufficient for most actual production applications. They work great as a proof of concept but quickly fall apart because of exactly these types of issues.
Luckily, NestJS provides great building blocks and primitives in the form of the Module and DI system to allow for much more feature rich plugins to be built. I created one specifically for RabbitMQ to be able to support the exact type of scenario you are describing.
I highly recommend that since you're using RabbitMQ already that you check out #golevelup/nestjs-rabbitmq which can easily support what you want to accomplish using native RMQ concepts like Exchanges and Routing Keys. (Disclaimer: I am the author). It also allows you to manage as many exchanges and queues as you like (instead of being forced to try to push all things through a single queue) and has native support for multiple messaging patterns including PubSub and RPC.
You simply decorate your methods that you want to act as microservice message handlers with the appropriate metadata and messaging will just work as expected. For example:
#Injectable()
export class CleaningService {
#RabbitSubscribe({
exchange: 'app',
routingKey: 'cars',
queue: 'cleaning-cars',
})
public async cleanCar(msg: {}) {
console.log(`Received message: ${JSON.stringify(msg)}`);
}
#RabbitSubscribe({
exchange: 'app',
routingKey: 'glass',
queue: 'cleaning-glass',
})
public async cleanGlass(msg: {}) {
console.log(`Received message: ${JSON.stringify(msg)}`);
}
}
#Injectable()
export class FixingService {
#RabbitSubscribe({
exchange: 'app',
routingKey: 'cars',
queue: 'fixing-cars',
})
public async fixCar(msg: {}) {
console.log(`Received message: ${JSON.stringify(msg)}`);
}
}
With this setup both the cleaning service and the fixing service will receive the car message to their individual handlers (since they use the same routing key) and only the cleaning service will receive the glass message
Publishing message is simple. You just include the exchange and routing key and the right handlers will receive it based on their configuration:
amqpConnection.publish('app', 'cars', { year: 2020, make: 'toyota' });
I'm wondering if I'm doing something wrong, I expected MassTransit would automatically register ReceiveEndpoints in the EndpointConvention.
Sample code:
services.AddMassTransit(x =>
{
x.AddServiceBusMessageScheduler();
x.AddConsumersFromNamespaceContaining<MyNamespace.MyRequestConsumer>();
x.UsingAzureServiceBus((context, cfg) =>
{
// Load the connection string from the configuration.
cfg.Host(context.GetRequiredService<IConfiguration>().GetValue<string>("ServiceBus:ConnectionString"));
cfg.UseServiceBusMessageScheduler();
// Without this line I'm getting an error complaining about no endpoint convention for x could be found.
EndpointConvention.Map<MyRequest>(new Uri("queue:queue-name"));
cfg.ReceiveEndpoint("queue-name", e =>
{
e.MaxConcurrentCalls = 1;
e.ConfigureConsumer<MyRequestConsumer>(context);
});
cfg.ConfigureEndpoints(context);
});
});
I thought this line EndpointConvention.Map<MyRequest>(new Uri("queue:queue-name")); wouldn't be necessary to allow sending to the bus without specifing the queue name, or am I missing something?
await bus.Send<MyRequest>(new { ...});
The EndpointConvention is a convenience method that allows the use of Send without specifying the endpoint address. There is nothing in MassTransit that will automatically configured this because, frankly, I don't use it. And I don't think anyone else should either. That stated, people do use it for whatever reason.
First, think about the ramifications - if every message type was registered as an endpoint convention, what about messages that are published and consumed on multiple endpoints? That wouldn't work.
So, if you want to route messages by message type, MassTransit has a feature for that. It's called Publish and it works great.
But wait, it's a command, and commands should be Sent.
That is true, however, if you are in control of the application and you know that there is only one consumer in your code base that consumes the KickTheTiresAndLightTheFires message contract, publish is as good as send and you don't need to know the address!
No, seriously dude, I want to use Send!
Okay, fine, here are the details. When using ConfigureEndpoints(), MassTransit uses the IEndpointNameFormatter to generate the receive endpoint queue names based upon the types registered via AddConsumer, AddSagaStateMachine, etc. and that same interface can be used to register your own endpoint conventions if you want to use Send without specifying a destination address.
You are, of course, coupling the knowledge of your consumer and message types, but that's your call. You're already dealing with magic (by using Send without an explicit destination) so why not right?
string queueName = formatter.Consumer<T>()
Use that string for the message types in that consumer as a $"queue:{queueName}" address and register it on the EndpointConvention.
Or, you know, just use Publish.
We have the following use case:
We have two busses (internal and external). The idea is that our own services use the internal bus and all third-party services use the external bus. We have a created a service that acts as a message router (aptly named MessageRouter).
When a message is published on the internal bus, the MessageRouter can pick up that message and place it on the external bus, and vice-versa. Through configuration we can tell which messages are allowed to pass from internal to external of from external to internal. This in itself works fine.
We have grouped our messages, we have Events, Commands and Requests. Each service has three ReceiveEnpoints, one for each message type. This means that all events are queued on the Events queue etc. Each message to which a service 'subscribes' gets it's own consumer.
var queues = new Dictionary<string, IEnumerable<Type>>
{
// ReSharper disable once PossibleMultipleEnumeration
{ "Events", consumerTypes.Where(ct => ct.GetGenericArguments().Any(ga => typeof(IEvent).IsAssignableFrom(ga))) },
// ReSharper disable once PossibleMultipleEnumeration
{ "Commands", consumerTypes.Where(ct => ct.GetGenericArguments().Any(ga => typeof(ICommand).IsAssignableFrom(ga))) },
// ReSharper disable once PossibleMultipleEnumeration
{ "Queries", consumerTypes.Where(ct => ct.GetGenericArguments().Any(ga => typeof(IQuery).IsAssignableFrom(ga))) }
};
foreach (var queue in queues)
{
config.ReceiveEndpoint(GetConsumerQueueName(queue.Key), cfg =>
{
foreach (var consumerType in queue.Value)
{
cfg.Consumer(consumerType, consumerContainer.Resolve);
}
configurator?.Invoke((T)cfg);
});
}
Where config is a IBusFactoryConfigurator. This code called when the service starts.
What we would like to be able to do in our MessageRouter is to 'dynamically' add and, more importantly, remove consumers from the ReceiveEndpoint.
So far, we haven't had any luck. We have tried to add the Consumer through the use of the ConnectConsumer method on the BusControl instance. This gives us a ConnectHandle which has a Disconnect method. However, when using this approach, messages are not picked up by our consumers. Looking at the handle shows us that it is a MultipleConnectHandle, however, the handle has no 'internal' handles.
Is there any way to use the Consumer method to register the different consumers and to get their ConnectHandles, So that we can Disconnect the consumer if needed?
As stated, ideally we would like to be able to dynamically add and remove consumers to a ReceiveEndpoint.
You can't add/remove consumers on a receive endpoint while the bus is started, that isn't supported in any way, shape, or form.
You can, however, connect new receive endpoints on separate queues with one or more consumers. In your case above, it seems like you are not getting your consumers registered before connecting the receive endpoint, which is why you aren't seeing anything in the handle collection.