I know that to enable encryption at rest for SNS Topics I need to add topic attribute like this one
cfg.Publish<SomeEvent>(p =>
{
p.TopicAttributes.Add(QueueAttributeName.KmsMasterKeyId, "<key arn>");
});
But the problem I have is that we are using very simple configuration for aspnetcore with UsingAmazonSQS and cfg.ConfigureEndpoints(context) doing all those configurations automatically so there is no call to cfg.Publish. This is very important as we have many messages to configure. I believe the same thing for queue attributes can be achieved by registering custom implementation of IConfigureReceiveEndpoint but I can't find equivalent for topics.
How can I keep using the automatic method and add this encryption topic attribute for all topics?
There isn't currently a way to apply topic attributes to all topics, but it isn't an unreasonable request for an addition. Surely someone can build it as a pull request adding to the general publish topology for Amazon SQS.
Update
I added the ability to specify attributes on the root PublishTopology which should be applied to all topics.
Used in code (from the comment below):
configure.UsingAmazonSqs((context, cfg) =>
{
cfg.PublishTopology.TopicAttributes
.Add(QueueAttributeName.KmsMasterKeyId, encryptionKeyArn);
cfg.ConfigureEndpoints(context);
});
Related
We have UserCreated event that gets published from UserManagement.Api. I have two other Apis, Payments.Api and Notification.Api that should react to that event.
In both Apis I have public class UserCreatedConsumer : IConsumer<UserCreated> (so different namespaces) but only one queue (on SQS) gets created for both consumers.
What is the best way to deal with this situation?
You didn't share your configuration, but if you're using:
x.AddConsumer<UserCreatedConsumer>();
As part of your MassTransit configuration, you can specify an InstanceId for that consumer to generate a unique endpoint address.
x.AddConsumer<UserCreatedConsumer>()
.Endpoint(x => x.InstanceId = "unique-value");
Every separate service (not an instance of the same service) needs to have a different queue name of the receiving endpoint, as described in the docs:
cfg.ReceiveEndpoint("queue-name-per-service-type", e =>
{
// rest of the configuration
});
It's also mentioned in the common mistakes article.
Sometimes, you want to say, "I have this message, who can handle it?"
In nestjs a client proxy is bounded directly to a single microservice.
So, as an example, let say that I have the following micro-services:
CleaningService, FixingService.
Both of the above can handle the message car, but only CleaningService can handle the message glass.
So, I want to have something like:
this.generalProxy.emit('car', {id: 2});
In this case, I want 2 different microservices to handle the car: CleaningService and FixingService.
in this case:
this.generalProxy.emit('glass', {id: 5});
I want only CleaningService to handle it.
How is that possible? how can I create clientProxy that is not bonded directly to a specific microservice.
The underlying transport layer matters because despite the fact that there is an abstraction in front of the different transports each underlying one has completely different characteristics and capabilities. The type of messaging pattern you're talking about is simple to accomplish with RabbitMQ because it has the notion of exchanges, queues, publisher, subscribers etc while a TCP based microservice requires a connection from one service to another. Likewise, the Redis transport layer uses simple channels without the necessary underlying implementation to be able to support some messages being fanned out to multiple subscribers and some going directly to specific subscribers.
This might not be the most popular opinion but I've been using NestJS professionally for over 3 years and I can definitely say that the official microservices packages are not sufficient for most actual production applications. They work great as a proof of concept but quickly fall apart because of exactly these types of issues.
Luckily, NestJS provides great building blocks and primitives in the form of the Module and DI system to allow for much more feature rich plugins to be built. I created one specifically for RabbitMQ to be able to support the exact type of scenario you are describing.
I highly recommend that since you're using RabbitMQ already that you check out #golevelup/nestjs-rabbitmq which can easily support what you want to accomplish using native RMQ concepts like Exchanges and Routing Keys. (Disclaimer: I am the author). It also allows you to manage as many exchanges and queues as you like (instead of being forced to try to push all things through a single queue) and has native support for multiple messaging patterns including PubSub and RPC.
You simply decorate your methods that you want to act as microservice message handlers with the appropriate metadata and messaging will just work as expected. For example:
#Injectable()
export class CleaningService {
#RabbitSubscribe({
exchange: 'app',
routingKey: 'cars',
queue: 'cleaning-cars',
})
public async cleanCar(msg: {}) {
console.log(`Received message: ${JSON.stringify(msg)}`);
}
#RabbitSubscribe({
exchange: 'app',
routingKey: 'glass',
queue: 'cleaning-glass',
})
public async cleanGlass(msg: {}) {
console.log(`Received message: ${JSON.stringify(msg)}`);
}
}
#Injectable()
export class FixingService {
#RabbitSubscribe({
exchange: 'app',
routingKey: 'cars',
queue: 'fixing-cars',
})
public async fixCar(msg: {}) {
console.log(`Received message: ${JSON.stringify(msg)}`);
}
}
With this setup both the cleaning service and the fixing service will receive the car message to their individual handlers (since they use the same routing key) and only the cleaning service will receive the glass message
Publishing message is simple. You just include the exchange and routing key and the right handlers will receive it based on their configuration:
amqpConnection.publish('app', 'cars', { year: 2020, make: 'toyota' });
I'm wondering if I'm doing something wrong, I expected MassTransit would automatically register ReceiveEndpoints in the EndpointConvention.
Sample code:
services.AddMassTransit(x =>
{
x.AddServiceBusMessageScheduler();
x.AddConsumersFromNamespaceContaining<MyNamespace.MyRequestConsumer>();
x.UsingAzureServiceBus((context, cfg) =>
{
// Load the connection string from the configuration.
cfg.Host(context.GetRequiredService<IConfiguration>().GetValue<string>("ServiceBus:ConnectionString"));
cfg.UseServiceBusMessageScheduler();
// Without this line I'm getting an error complaining about no endpoint convention for x could be found.
EndpointConvention.Map<MyRequest>(new Uri("queue:queue-name"));
cfg.ReceiveEndpoint("queue-name", e =>
{
e.MaxConcurrentCalls = 1;
e.ConfigureConsumer<MyRequestConsumer>(context);
});
cfg.ConfigureEndpoints(context);
});
});
I thought this line EndpointConvention.Map<MyRequest>(new Uri("queue:queue-name")); wouldn't be necessary to allow sending to the bus without specifing the queue name, or am I missing something?
await bus.Send<MyRequest>(new { ...});
The EndpointConvention is a convenience method that allows the use of Send without specifying the endpoint address. There is nothing in MassTransit that will automatically configured this because, frankly, I don't use it. And I don't think anyone else should either. That stated, people do use it for whatever reason.
First, think about the ramifications - if every message type was registered as an endpoint convention, what about messages that are published and consumed on multiple endpoints? That wouldn't work.
So, if you want to route messages by message type, MassTransit has a feature for that. It's called Publish and it works great.
But wait, it's a command, and commands should be Sent.
That is true, however, if you are in control of the application and you know that there is only one consumer in your code base that consumes the KickTheTiresAndLightTheFires message contract, publish is as good as send and you don't need to know the address!
No, seriously dude, I want to use Send!
Okay, fine, here are the details. When using ConfigureEndpoints(), MassTransit uses the IEndpointNameFormatter to generate the receive endpoint queue names based upon the types registered via AddConsumer, AddSagaStateMachine, etc. and that same interface can be used to register your own endpoint conventions if you want to use Send without specifying a destination address.
You are, of course, coupling the knowledge of your consumer and message types, but that's your call. You're already dealing with magic (by using Send without an explicit destination) so why not right?
string queueName = formatter.Consumer<T>()
Use that string for the message types in that consumer as a $"queue:{queueName}" address and register it on the EndpointConvention.
Or, you know, just use Publish.
I've been searching to find out how to perform a subscription to key space notifications on Redis using ServiceStack.Redis library for removal of Key.
Checking available tests on the git-hub and other websites I've found IRedisSubscription can be used for subscribing to specific Redis key events, For set operation it is working absolutely fine but when it comes to Delete operation the action is not invoked.
Is it possible to take advantage of this Redis feature using ServiceStack.Redis and get event on delete operation too?
In the configuration file I have added this line:
notify-keyspace-events KEAg
I am using the following code.
var channels = new[] { "__keyevent#0__:set" , "__keyevent#0__:del" };
using (var redisConsumer = new RedisClient("localhost:6379"))
using (var subscription = redisConsumer.CreateSubscription()) {
subscription.OnMessage = onKeyChange;
subscription.SubscribeToChannelsMatching(channels );
}
From the surface, it looks like what you got should work.
Try setting notify-keyspace-events to AKE, the g is redundant, as noted in Notifications Config:
A Alias for g$lshztxe, so that the "AKE" string means all the
events.
Try using SubscribeToChannels instead of SubscribeToChannelsMatching. The latter is for pattern subscription.
You can test how many subscribers you have with the PUBSUB NUMSUB __keyevent#0__:del command from redis-cli.
Try testing your events are being triggered with SUBSCRIBE __keyevent#0__:del from redis-cli. This will help you determine if the problem is on redis-server or the app code.
Please update the question with results if you can't get it to work after trying the above.
Is there a way to send events from the server to all or some clients without using collections.
I want to send events with some custom data to clients. While meteor is very good in doing this with collections, in this case the added complexity and storage its not needed.
On the server there is no need for Mongo storage or local collections.
The client only needs to be alerted that it received an event from the server and act accordingly to the data.
I know this is fairly easy with sockjs but its very difficult to access sockjs from the server.
Meteor.Error does something similar to this.
The package is now deprecated and do not work for versions >0.9
You can use the following package which is originally aim to broadcast messages from clients-server-clients
http://arunoda.github.io/meteor-streams/
No collection, no mongodb behind, usage is as follow (not tested):
stream = new Meteor.Stream('streamName'); // defined on client and server side
if(Meteor.isClient) {
stream.on("channelName", function(message) {
console.log("message:"+message);
});
}
if(Meteor.isServer) {
setInterval(function() {
stream.emit("channelName", 'This is my message!');
}, 1000);
}
You should use Collections.
The "added complexity and storage" isn't a factor if all you do is create a collection, add a single property to it and update that.
Collections are just a shape for data communication between server and client, and they happen to build on mongo, which is really nice if you want to use them like a database. But at their most basic, they're just a way of saying "I want to store some information known as X", which hooks into the publish/subscribe architecture that you should want to take advantage of.
In the future, other databases will be exposed in addition to Mongo. I could see there being a smart package at some stage that strips Collections down to their most basic functionality like you're proposing. Maybe you could write it!
I feel for #Rui and the fact of using a Collection just to send a message feel cumbersome.
At the same time, once you have several of such message to send around is convenient to have a Collection named something like settings or similar where you keep these.
Best package I have found is Streamy. It allows you to send to everybody, or just one specific user
https://github.com/YuukanOO/streamy
meteor add yuukan:streamy
Send message to everybody:
Streamy.broadcast('ddpEvent', { data: 'something happened for all' });
Listen for message on client:
// Attach an handler for a specific message
Streamy.on('ddpEvent', function(d, s) {
console.log(d.data);
});
Send message to one user (by id)
var socket = Streamy.socketsForUsers(["nJyQvECmkBSXDZEN2"])._sockets[0]
Streamy.emit('ddpEvent', { data: 'something happened for you' }, socket);