Masstransit channels - masstransit

I configured MassTransit on my .NET core application as follows:
public void ConfigureServices(IServiceCollection services)
{
[...]
// producer
services.AddMassTransit(x =>
{
x.AddBus(provider => Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Host(new Uri(_configuration["RabbitMQ:URI"] + _configuration["RabbitMQ:VirtualHost"]), $"ENG {_configuration["SiteID"]} Producer", h =>
{
h.Username(_configuration["RabbitMQ:UserName"]);
h.Password(_configuration["RabbitMQ:Password"]);
});
cfg.Publish<NormUpdate>(x =>
{
x.Durable = true;
x.AutoDelete = false;
x.ExchangeType = "fanout"; // default, allows any valid exchange type
});
cfg.ConfigurePublish(x => x.UseExecute(x =>
{
x.Headers.Set("SiteID", _configuration["SiteID"]);
}));
}));
});
services.AddMassTransit<ISecondBus>(x =>
{
x.AddConsumer<NormConsumer>();
x.AddBus(context => Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.UseMessageRetry(r => r.Intervals(100, 200, 500, 800, 1000));
cfg.Host(new Uri(_configuration["RabbitMQ:URI"] + _configuration["RabbitMQ:VirtualHost"]), $"ENG {_configuration["SiteID"]} Consumer", h =>
{
h.Username(_configuration["RabbitMQ:UserName"]);
h.Password(_configuration["RabbitMQ:Password"]);
});
cfg.ReceiveEndpoint($"norm-queue-{_configuration["SiteID"]}", e =>
{
e.Durable = true;
e.AutoDelete = false;
e.Consumer<NormConsumer>(context);
e.UseConcurrencyLimit(1);
e.ExchangeType = "fanout";
e.PrefetchCount = 1;
});
}));
});
services.AddOptions<MassTransitHostOptions>().Configure(options =>
{
options.WaitUntilStarted = false;
options.StopTimeout = TimeSpan.FromSeconds(30);
});
[...]
}
public interface ISecondBus : IBus
{
}
I noticed that when connections are created, the consumer connection has 2 channels. Channel (1) with no attached consumers, channel (2) with one consumer.
I expected to have only one channel on receiver.
Is this a normal behavior or am I doing something wrong?

You should have three channels:
One for the first bus you configured
One for the second bus you configured (MultiBus configures completely separate bus instances, nothing is shared)
One for the receive endpoint on the second bus you configured.
1 + 1 + 1 = 3
Q.E.D.
Yes, I was in Math club back in primary school.

Related

Create custom exchange-to-exchange binding using MassTransit

I have microservice-based system which works with documents. Service publishes DocflowErrorMq, ImportedDocflowMq events, and other services are subscribed to these events. Critical service DocflowRegistry should process messages quickly, so we have to introduce multiple consumers. On the other hand message order shouldn't be broken and competing consumer doesn't suite. Consistent hash exchange distributes messages by routing key equals to document id, messages related to one document goes to one queue. So, we have simple manual scaling. I can't create binding between MqModels.Docflows:ImportedDocflowMq and docflow-process-dr exchanges (marked red on Diagram). Is it possible to create it with MassTransit?
DocflowRegistry service config:
services.AddMassTransit(x =>
{
x.AddConsumer<DocflowSendingErrorTestConsumer>();
x.AddConsumer<DocflowImportTestConsumer>();
x.UsingRabbitMq((context, cfg) =>
{
var virtualHost = configuration["RabbitMq:Settings:VirtualHost"] ?? "/";
cfg.Host(configuration["RabbitMqHost"], virtualHost, h =>
{
h.Username(configuration["RabbitMqUserName"]);
h.Password(configuration["RabbitMqPassword"]);
});
cfg.ReceiveEndpoint("docflow.process-1.docflowregistry", e =>
{
e.ConfigureConsumer<DocflowSendingErrorTestConsumer>(context);
e.ConfigureConsumer<DocflowImportTestConsumer>(context);
e.Bind("docflow-process-dr", x =>
{
x.Durable = true;
x.AutoDelete = false;
x.ExchangeType = "x-consistent-hash";
x.RoutingKey = "1";
});
e.ConfigureConsumeTopology = false;
e.SingleActiveConsumer = true;
});
cfg.ReceiveEndpoint("docflow.process-2.docflowregistry", e =>
{
e.ConfigureConsumer<DocflowSendingErrorTestConsumer>(context);
e.ConfigureConsumer<DocflowImportTestConsumer>(context);
e.Bind("docflow-process-dr", x =>
{
x.Durable = true;
x.AutoDelete = false;
x.ExchangeType = "x-consistent-hash";
x.RoutingKey = "1";
});
e.ConfigureConsumeTopology = false;
e.ConcurrentMessageLimit = 1;
e.SingleActiveConsumer = true;
});
});
});
Config of TodoList service:
services.AddMassTransit(x =>
{
x.AddConsumer<DocflowSendingErrorTestConsumer>();
x.AddConsumer<DocflowImportTestConsumer>();
x.UsingRabbitMq((context, cfg) =>
{
var virtualHost = configuration["RabbitMq:Settings:VirtualHost"] ?? "/";
cfg.Host(configuration["RabbitMqHost"], virtualHost, h =>
{
h.Username(configuration["RabbitMqUserName"]);
h.Password(configuration["RabbitMqPassword"]);
});
cfg.ReceiveEndpoint("docflow-process-todolist", e =>
{
e.ConfigureConsumer<DocflowSendingErrorTestConsumer>(context);
e.ConfigureConsumer<DocflowImportTestConsumer>(context);
e.SingleActiveConsumer = true;
});
});
});
Publish code:
var endPoint = await _massTransitBus.GetPublishSendEndpoint<DocflowErrorMq>();
var docflowGuid = Guid.NewGuid();
await endPoint.Send(new DocflowErrorMq
{
DocflowId = docflowGuid,
AbonentId = Guid.NewGuid()
},
context =>
{
context.SetRoutingKey(docflowGuid.ToString());
});
Create an interface, DocflowProcessDr, and make each of those message contracts published implement it. Then, you can configure the publish topology for that interface in the bus:
cfg.Message<DocflowProcessDr>(x => x.SetEntityName("docflow-process-dr"));
cfg.Publish<DocflowProcessDr>(x =>
{
x.ExchangeType = "x-consistent-hash";
});
Since MassTransit will create a polymorphic topology on the broker, you'll have an exchange-to-exchange binding between the published type and the interface.
Then, just publish the message:
var docflowGuid = Guid.NewGuid();
var endPoint = await _massTransitBus.Publish<DocflowErrorMq>(new DocflowErrorMq
{
DocflowId = docflowGuid,
AbonentId = Guid.NewGuid()
},
context =>
{
context.SetRoutingKey(docflowGuid.ToString());
});
Calling GetPublishSendEndpoint<T>() is weird, don't encourage it.

Why do I have the following error (MassTransit)

The error is -
ConfigurationException: A receive endpoint with the same key was already added: Events
I have appsettings.Development.json with
"EventsBusOptions": {
"HostUri": "rabbitmq://rabbitmq.test.com/gate",
"UserName": "xxx",
"Password": "xxxxxx",
"QueueName": "events", //<<< if is change queue name some different string e.g. "events1" - NO error
"PrefetchCount": 16,
"UseConcurrencyLimit": 15
}
and Startup.cs (with MultiBus Configuration)
services.AddMassTransit<IEventsBus>(x =>
{
x.UsingRabbitMq((context, cfg) =>
{
var _options = context.GetRequiredService<IOptions<EventsBusOptions>>().Value;
cfg.Host(new Uri(_options.HostUri), h =>
{
h.Username(_options.UserName);
h.Password(_options.Password);
});
cfg.ReceiveEndpoint(_options.QueueName, ep =>
{
ep.Consumer<EventsConsumer>(context);
ep.PrefetchCount = _options.PrefetchCount ?? 15;
ep.UseConcurrencyLimit(_options.UseConcurrencyLimit ?? 16);
});
cfg.ConfigureEndpoints(context);
});
x.AddConsumer<EventsConsumer>();
});
Why I have got the error when I use "QueueName": "events"?
Because you're using the wrong method to configure the consumer. ConfigureConsumer should be used instead of just Consumer, as shown in the updated configuration below.
services.AddMassTransit<IEventsBus>(x =>
{
x.AddConsumer<EventsConsumer>();
x.UsingRabbitMq((context, cfg) =>
{
var _options = context.GetRequiredService<IOptions<EventsBusOptions>>().Value;
cfg.Host(new Uri(_options.HostUri), h =>
{
h.Username(_options.UserName);
h.Password(_options.Password);
});
cfg.ReceiveEndpoint(_options.QueueName, ep =>
{
ep.PrefetchCount = _options.PrefetchCount ?? 16;
ep.ConcurrentMessageLimit = _options.ConcurrentMessageLimit ?? 16;
ep.ConfigureConsumer<EventsConsumer>(context);
});
cfg.ConfigureEndpoints(context);
});
});
NOTE I also fixed your concurrent message limit configuration to use the built-in limiter, instead of adding a filter.
ALSO you could leave off ConfigureEndpoints since you're manually configuring the receive endpoint for the consumer.

MassTransit is only batching 10 when more is configured

I'm trying to configure MassTransit batching, but when running it only batches 10 at a time.
hostHandler = receiveEndpointConnector.ConnectReceiveEndpoint(queueName, (context, cfg) =>
{
cfg.TrySetPrefetchCount(2000);
cfg.Batch<T>(cfg =>
{
cfg.Consumer(() => consumer);
cfg.ConcurrencyLimit = 2;
cfg.MessageLimit = 1000;
cfg.TimeLimit = TimeSpan.FromSeconds(1);
});
cfg.UseMessageRetry(r => r.Immediate(2)));
});
await hostHandler.Ready;
You could use the newer batch syntax as well, but it still needs to be specified prior to the Consumer call:
var handle = receiveEndpointConnector.ConnectReceiveEndpoint(queueName, (context, cfg) =>
{
cfg.TrySetPrefetchCount(2000);
cfg.UseMessageRetry(r => r.Immediate(2)));
cfg.ConfigureConsumer<YourConsumer>(context, cons =>
{
cons.Options<BatchOptions>(options => options
.SetMessageLimit(1000)
.SetTimeLimit(1000)
.SetConcurrencyLimit(2));
});
});
await handle.Ready;
You could also, since you're using the receive endpoint connector, configure the batch options in the consumer definition as shown in the documentation.
5 minutes after I posted the question I tried to change the order of the batch configuration, and putting the consumer as the last statement, did the trick.
hostHandler = receiveEndpointConnector.ConnectReceiveEndpoint(queueName, (context, cfg) =>
{
cfg.TrySetPrefetchCount(2000);
cfg.Batch<T>(cfg =>
{
cfg.ConcurrencyLimit = 2;
cfg.MessageLimit = 1000;
cfg.TimeLimit = TimeSpan.FromSeconds(1);
cfg.Consumer(() => consumer);
});
cfg.UseMessageRetry(r => r.Immediate(2)));
});
await hostHandler.Ready;

Cannot Register saga using Masstransit.Integration.AspNetCore

I cannot register saga using Masstransit.AspNetCore package, it results on this error. I'm using Default Container with error "No service for type 'Automatonymous.Registration.ISagaStateMachineFactory' has been registered"
e.AddSagaStateMachine<CityAvailabilityStateMachine, CityAvailabilityState>(new NullSagaStateMachineRegistrar());
e.AddBus(provider =>{
var credentials = provider.GetService<Credential>();
return Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri(credentials.Uri), h =>
{
h.Username(credentials.UserName);
h.Password(credentials.Password);
h.Heartbeat(60);
});
cfg.ReceiveEndpoint(host,{credentials.BoundedContext}-sagas", configurator =>
{
configurator.PrefetchCount = 16;
configurator.UseRetry(r => r.Interval(2, 100));
configurator.ConfigureSaga<CityAvailabilityState>(provider);
});
});

Create an Rx.Subject using Subject.create that allows onNext without subscription

When creating an Rx.Subject using Subject.create(observer, observable), the Subject is so lazy. When I try to use subject.onNext without having a subscription, it doesn't pass messages on. If I subject.subscribe() first, I can use onNext immediately after.
Let's say I have an Observer, created like so:
function createObserver(socket) {
return Observer.create(msg => {
socket.send(msg);
}, err => {
console.error(err);
}, () => {
socket.removeAllListeners();
socket.close();
});
}
Then, I create an Observable that accepts messages:
function createObservable(socket) {
return Observable.fromEvent(socket, 'message')
.map(msg => {
// Trim out unnecessary data for subscribers
delete msg.blobs;
// Deep freeze the message
Object.freeze(msg);
return msg;
})
.publish()
.refCount();
}
The subject is created using these two functions.
observer = createObserver(socket);
observable = createObservable(socket);
subject = Subject.create(observer, observable);
With this setup, I'm not able to subject.onNext immediately (even if I don't care about subscribing). Is this by design? What's a good workaround?
These are actually TCP sockets, which is why I haven't relied on the super slick websocket subjects.
The basic solution, caching nexts before subscription with ReplaySubject:
I think all you wanted to do is use a ReplaySubject as your observer.
const { Observable, Subject, ReplaySubject } = Rx;
const replay = new ReplaySubject();
const observable = Observable.create(observer => {
replay.subscribe(observer);
});
const mySubject = Subject.create(replay, observable);
mySubject.onNext(1);
mySubject.onNext(2);
mySubject.onNext(3);
mySubject.subscribe(x => console.log(x));
mySubject.onNext(4);
mySubject.onNext(5);
Results in:
1
2
3
4
5
A socket implementation (example, don't use)
... but if you're looking at doing a Socket implementation, it gets a lot more complicated. Here is a working socket implementation, but I don't recommend you use it. Rather, I'd suggest that you use one of the community supported implementations either in rxjs-dom (if you're an RxJS 4 or lower) or as part of RxJS 5, both of which I've helped work on.
function createSocketSubject(url) {
let replay = new ReplaySubject();
let socket;
const observable = Observable.create(observer => {
socket = new WebSocket(url);
socket.onmessage = (e) => {
observer.onNext(e);
};
socket.onerror = (e) => {
observer.onError(e);
};
socket.onclose = (e) => {
if (e.wasClean) {
observer.onCompleted();
} else {
observer.onError(e);
}
}
let sub;
socket.onopen = () => {
sub = replay.subscribe(x => socket.send(x));
};
return () => {
socket && socket.readyState === 1 && socket.close();
sub && sub.dispose();
}
});
return Subject.create(replay, observable);
}
const socket = createSocketSubject('ws://echo.websocket.org');
socket.onNext('one');
socket.onNext('two');
socket.subscribe(x => console.log('response: ' + x.data));
socket.onNext('three');
socket.onNext('four');
Here's the obligatory JsBin

Resources