I've discovered no Masstransit configuration that allows a service bus Topic to be created with Duplicate Detection enabled.
You can do it with Queues simply enough. But for Topics it seems a bit of a mystery.
Does anybody have a working sample?
Perhaps it is not possible.
I've been trying to use the IServiceBusBusFactoryConfigurator provided by the Bus.Factory.CreateUsingAzureServiceBus method.
I'd thought that some use of IServiceBusBusFactoryConfigurator.Publish method and IServiceBusBusFactoryConfigurator.SubscriptionEndpoint method would accomplish the task, but after a myriad of trials I've come up with no solution.
To configure your message type topic with duplicate detection, you must configure the publish topology in both the producer and the consumer (it only needs to be configured once per bus instance, but if your producer is a separate bus instance, it would also need the configuration). The topic must also not already exist as it would not be updated once created in Azure.
To configure the publish topology:
namespace DupeDetection
{
public interface DupeCommand
{
string Value { get; }
}
}
var busControl = Bus.Factory.CreateUsingAzureServiceBus(cfg =>
{
cfg.Publish<DupeCommand>(x => x.EnableDuplicateDetection(TimeSpan.FromMinutes(10)));
cfg.ReceiveEndpoint("dupe", e =>
{
e.Consumer<DupeConsumer>();
});
}
The consumer is normal (no special settings required).
class DupeConsumer :
IConsumer<DupeCommand>
{
public Task Consume(ConsumeContext<DupeCommand> context)
{
return Task.CompletedTask;
}
}
I've added a unit test to verify this behavior, and can confirm that when two messages with the same MessageId are published back-to-back, only a single message is delivered to the consumer.
Test log output:
10:53:15.641-D Create send transport: sb://masstransit-build.servicebus.windows.net/MassTransit.Azure.ServiceBus.Core.Tests.DupeDetection/DupeCommand
10:53:15.784-D Topic: MassTransit.Azure.ServiceBus.Core.Tests.DupeDetection/DupeCommand (dupe detect)
10:53:16.375-D SEND sb://masstransit-build.servicebus.windows.net/MassTransit.Azure.ServiceBus.Core.Tests.DupeDetection/DupeCommand dc3a0000-ebb8-e450-949c-08d8e8939c7f MassTransit.Azure.ServiceBus.Core.Tests.DupeDetection.DupeCommand
10:53:16.435-D SEND sb://masstransit-build.servicebus.windows.net/MassTransit.Azure.ServiceBus.Core.Tests.DupeDetection/DupeCommand dc3a0000-ebb8-e450-949c-08d8e8939c7f MassTransit.Azure.ServiceBus.Core.Tests.DupeDetection.DupeCommand
10:53:16.469-D RECEIVE sb://masstransit-build.servicebus.windows.net/MassTransit.Azure.ServiceBus.Core.Tests/input_queue dc3a0000-ebb8-e450-949c-08d8e8939c7f MassTransit.Azure.ServiceBus.Core.Tests.DupeDetection.DupeCommand MassTransit.IConsumer<MassTransit.Azure.ServiceBus.Core.Tests.DupeDetection.DupeCommand>(00:00:00.0017972)
You can see the (dupe detect) attribute shown on the topic declaration.
Here is the solution I finally found. It does not rely on trying any of the ReceiveEndpoint or SubscriptionEndpoint configuration methods which never seemed to give me what I wanted.
IBusControl bus = Bus.Factory.CreateUsingAzureServiceBus(cfg =>
{
cfg.Publish<MembershipNotifications.MembershipSignupMessage>(configure =>
{
configure.EnableDuplicateDetection(_DuplicateDetectionWindow);
configure.AutoDeleteOnIdle = _AutoDeleteOnIdle;
configure.DefaultMessageTimeToLive = _MessageTimeToLive;
});
}
await bus.Publish(new MessageTest());
Related
Is there a way to register Consumer like the service below:
services.AddTransient < IMyService > (provider => {
return new MyServiceImplementation(2);
});
with AddConsumer<T>() method?
What I need is a custom implementation of Consumer factory because it will be injected with a different instance of one of its dependencies depending on the configuration.
MassTransit registers the consumer added via AddConsumer as shown below:
collection.AddScoped<T>();
You're welcome to create your own register after configuring MassTransit, which should replace the one registered by MassTransit. In your example above, it could be something like:
services.AddScoped<TConsumer>(provider =>
{
var options = provider.GetService<SomeOptions>();
if (options.UseFirst)
return new TConsumer(provider.GetRequiredService<Impl1>()
return new TConsumer(provider.GetRequiredService<Impl2>()
});
You get the picture, right?
I'm switching some code which uses MassTransit (v7.2.2 on .NET 5) to use a more declarative format (and away from multiple calls to ReceiveEndpoint()) and ideally to using ConsumerDefinitions for the configuration (though not part of this example for simplicity), along with some Dependency Injection with Quartz.NET (yanked from this example, though it running 3.3.3), in doing so I find now that my Consumers are not consuming, despite messages being sent and examples referenced. Take the following standing up of the MassTransit service:
var hostBuilder = Host.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) =>
{
services.AddMassTransit(mt =>
{
mt.AddBus(provider => Bus.Factory.CreateUsingInMemory(cfg =>
{
//cfg.AutoStart = true; //No change when on
cfg.UseInMemoryOutbox();
cfg.ConfigureEndpoints(provider);
}));
mt.AddConsumer<TheMessageConsumer>();
services.AddMediator(cfg =>
{
cfg.AddConsumer<TheMessageConsumer>();
});
});
services.AddMassTransitHostedService();
});
var host = hostBuilder.Build();
var busControl = host.Services.GetService<IBusControl>();
busControl.Start(); //Just in case
var message = new TheMessage() { Message = $"<Message-{DateTime.Now.ToLongTimeString()}>" };
Console.WriteLine($"Sending: {message.Message}");
await busControl.Publish(message);
host.Run();
For note, the breaking out of the message sending here is to simplify my repro, as in my full code base, it's being sent by a Quartz fired job.
For this example, the message & receiver are also quite simple:
public class TheMessage
{
public string Message { get; set; }
}
public class TheMessageConsumer : IConsumer<TheMessage>
{
public Task Consume(ConsumeContext<TheMessage> context)
{
Console.WriteLine($"Message received: {context.Message.Message}");
return Task.CompletedTask;
}
}
The bus is started, either in the case I explicitly start it, the AutoStart flag is set, or the MassTransitHostedService does it, yet the message doesn't get received. Ditto if I have the full example with Quartz firing off a job with messages much later.
Can someone suggest what I am missing?
The code you posted is seriously a hodgepodge of snippets, none of which make any sense when used together. For example, AddBus is deprecated, and mediator has no business being in that project at all.
I'd suggest using one of the MassTransit Templates to create a new project from scratch (you may need to up the NuGet versions to 7.2.2).
Watch this video which explains the templates and how to use them.
With your comment, and updated question, you really only need the following:
var hostBuilder = Host.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) =>
{
services.AddMassTransit(mt =>
{
mt.AddConsumer<TheMessageConsumer>();
mt.UsingInMemory((context,cfg) =>
{
cfg.UseInMemoryOutbox();
cfg.ConfigureEndpoints(context);
}));
});
services.AddMassTransitHostedService();
});
var host = hostBuilder.Build();
await host.RunAsync();
If you want to test it, and send messages in the same process, simply add a BackgroundService (after AddMassTransitHostedService) to publish your messages.
You shouldn't publish until after the bus has been started.
I'm creating few microservices using nestjs.
For instance I have x, y & z services all interconnected by grpc but I want service x to send updates to a webapp on a particular entity change so I have considered server-sent-events [open to any other better solution].
Following the nestjs documentation, they have a function running at n interval for sse route, seems to be resource exhaustive. Is there a way to actually sent events when there's a update.
Lets say I have another api call in the same service that is triggered by a button click on another webapp, how do I trigger the event to fire only when the button is clicked and not continuously keep sending events. Also if you know any idiomatic way to achieve this which getting hacky would be appreciated, want it to be last resort.
[BONUS Question]
I also considered MQTT to send events. But I get a feeling that it isn't possible for a single service to have MQTT and gRPC. I'm skeptical of using MQTT because of its latency and how it will affect internal message passing. If I could limit to external clients it would be great (i.e, x service to use gRPC for internal connections and MQTT for webapp just need one route to be exposed by mqtt).
(PS I'm new to microservices so please be comprehensive about your solutions :p)
Thanks in advance for reading till end!
You can. The important thing is that in NestJS SSE is implemented with Observables, so as long as you have an observable you can add to, you can use it to send back SSE events. The easiest way to work with this is with Subjects. I used to have an example of this somewhere, but generally, it would look something like this
#Controller()
export class SseController {
constructor(private readonly sseService: SseService) {}
#SSE()
doTheSse() {
return this.sseService.sendEvents();
}
}
#Injectable()
export class SseService {
private events = new Subject();
addEvent(event) {
this.events.next(event);
}
sendEvents() {
return this.events.asObservable();
}
}
#Injectable()
export class ButtonTriggeredService {
constructor(private readonly sseService: SseService) {}
buttonClickedOrSomething() {
this.sseService.addEvent(buttonClickedEvent);
}
}
Pardon the pseudo-code nature of the above, but in general it does show how you can use Subjects to create observables for SSE events. So long as the #SSE() endpoint returns an observable with the proper shape, you're golden.
There is a better way to handle events with SSE of NestJS:
Please see this repo with code example:
https://github.com/ningacoding/nest-sse-bug/tree/main/src
Where basically you have a service:
import {Injectable} from '#nestjs/common';
import {fromEvent} from "rxjs";
import {EventEmitter} from "events";
#Injectable()
export class EventsService {
private readonly emitter = new EventEmitter();
subscribe(channel: string) {
return fromEvent(this.emitter, channel);
}
emit(channel: string, data?: object) {
this.emitter.emit(channel, {data});
}
}
Obviously, channel can be any string, as recommendation use path style.
For example: "events/for/<user_id>" and users subscribed to that channel will receive only the events for that channel and only when are fired ;) - Fully compatible with #UseGuards, etc. :)
Additional note: Don't inject any service inside EventsService, because of a known bug.
#Sse('sse-endpoint')
sse(): Observable<any> {
//data have to strem
const arr = ['d1','d2', 'd3'];
return new Observable((subscriber) => {
while(arr.len){
subscriber.next(arr.pop()); // data have to return in every chunk
}
if(arr.len == 0) subscriber.complete(); // complete the subscription
});
}
Yes, this is possible, instead of using interval, we can use event emitter.
Whenever the event is emitted, we can send back the response to the client.
So I've been trying to use Spring Cloud Contract to test RabbitListener.
So far I have found out that by defining "triggeredBy" in contract, the generated test will call the method provided there and so we will need to provide the actual implementation of what that method do in the TestBase.
Another thing is "outputMessage", where we can verify whether the method call before have correctly resulting on some message body sent to certain exchange.
Source: documentation and sample
My question is, is there any way to produce the input message from the contract, instead of triggering own custom method?
Perhaps something similar like Spring Integration or Spring Cloud Stream example in the documentation:
Contract.make {
name("Book Success")
label("book_success")
input {
messageFrom 'input.exchange.and.maybe.route'
messageHeaders {
header('contentType': 'application/json')
header('otherMessageHeader': '1')
}
messageBody ([
bookData: someData
])
}
outputMessage {
sentTo 'output.exchange.and.maybe.route'
headers {
header('contentType': 'application/json')
header('otherMessageHeader': '2')
}
body([
bookResult: true
])
}
}
I couldn't find any examples in their sample project that show how to do this.
Having used spring cloud contract to document and test rest api services, if possible I would like to stay consistent by defining both the input and expected output in contract files for event based services.
Never mind, actually its already supported.
For unknown reason the documentation in "Stub Runner Spring AMQP" does not list the scenario like others previous sample.
Here is how I make it works:
Contract.make {
name("Amqp Contract")
label("amqp_contract")
input {
messageFrom 'my.exchange'
messageHeaders {
header('contentType': 'text/plain')
header('amqp_receivedRoutingKey' : 'my.routing.key')
}
messageBody(file('request.json'))
}
outputMessage {
sentTo 'your.exchange'
headers {
header('contentType': 'text/plain')
header('amqp_receivedRoutingKey' : 'your.routing.key')
}
body(file('response.json'))
}
}
This will create a test that will call your listener based on "my.exchange" and "my.routing.key" triggering the handler method.
It will then capture the message and routing key on your RabbitTemplate call to "your.exchange".
verify(this.rabbitTemplate, atLeastOnce()).send(eq(destination), routingKeyCaptor.capture(),
messageCaptor.capture(), any(CorrelationData.class));
Both message and routing key then will be asserted.
I'm trying to understand the proper way to use Windows.Foundation.Diagnostics.LoggingChannel. In particular I'd like to understand the purpose behind the Level property and when is this property set.
As described in the MSDN documentation of LoggingChannel, the Level property is read-only. So how can I set the level that a channel accepts messages at?
Currently what I have designed as a logger for my app is something like below:
public class Logger
{
public LoggingLevel LoggerLoggingLevel { get; set; }
private LoggingSession _session;
private LoggingChannel _channel;
public Logger()
{
_channel = new LoggingChannel("MyChannel");
_session = new LoggingSession("MySession");
_session.AddLoggingChannel(_channel);
}
public void LogMessage(string msg, LoggingLevel level)
{
if (level >= LoggerLoggingLevel)
{
_channel.LogMessage(msg, level);
}
}
.
.
.
}
// The consumer of the Logger class will instantiate an instance of it,
// sets the LoggerLoggingLevel, and then starts logging messages at various levels.
// At any point, the consumer can change LoggerLoggingLevel to start accepting
// messages at different levels.
IS this the right approach or is there a better way (for example by somehow setting the level of _channel and then passing the message & level to the channel, letting the channel decide whether it should filter out the message or accept and log it)?
LoggingChannel.Level tells you "somebody has expressed interest in receiving messages from you that are of severity 'Level' or higher". This property will be set automatically by the runtime when somebody subscribes to events from your LoggingChannel instance. (Within your app, you can subscribe to your app's events using the LoggingSession class; outside of your app, you can record your app's events using a tool like tracelog or xperf.)
In simple scenarios, you don't need to worry about the value of LoggingChannel.Level. The LoggingChannel.LogMessage function already checks the value of LoggingChannel.Level. It also checks the value of LoggingChannel.Enabled, which tells you whether anybody is subscribed to your events at any level. (Note that the value of LoggingChannel.Level is UNDEFINED and MEANINGLESS unless LoggingChannel.Enabled is true.) In normal use, you don't need to worry about LoggingChannel.Enabled or LoggingChannel.Level -- just call LogMessage and let LoggingChannel check the levels for you.
LoggingChannel exposes the Enabled and Level properties to support a more complex scenario where it is expensive to gather the data you are about to log. In this case, you would probably like to skip gathering the data if nobody is listening for your event. You would then write code like this:
if (channel.Enabled && channel.Level <= eventLevel)
{
string expensiveData = GatherExpensiveData();
channel.LogMessage(expensiveData, eventLevel);
}
Note that the Windows 10 version of LoggingChannel added a bunch of new methods to make life a bit easier. If your program will run on Windows 10 or later, you can use the IsEnabled method instead of separate checks for Enabled and Level:
if (channel.IsEnabled(eventLevel))
{
string expensiveData = GatherExpensiveData();
channel.LogMessage(expensiveData, eventLevel);
}
A bunch of other stuff was also added to LoggingChannel for Windows 10. You can now log complex events (strongly-typed fields) instead of just strings, you can define keywords and opcodes (look up ETW documentation for more information), and you can basically have your LoggingChannel act like a first-class ETW citizen.