I'm trying to configure MassTransit batching, but when running it only batches 10 at a time.
hostHandler = receiveEndpointConnector.ConnectReceiveEndpoint(queueName, (context, cfg) =>
{
cfg.TrySetPrefetchCount(2000);
cfg.Batch<T>(cfg =>
{
cfg.Consumer(() => consumer);
cfg.ConcurrencyLimit = 2;
cfg.MessageLimit = 1000;
cfg.TimeLimit = TimeSpan.FromSeconds(1);
});
cfg.UseMessageRetry(r => r.Immediate(2)));
});
await hostHandler.Ready;
You could use the newer batch syntax as well, but it still needs to be specified prior to the Consumer call:
var handle = receiveEndpointConnector.ConnectReceiveEndpoint(queueName, (context, cfg) =>
{
cfg.TrySetPrefetchCount(2000);
cfg.UseMessageRetry(r => r.Immediate(2)));
cfg.ConfigureConsumer<YourConsumer>(context, cons =>
{
cons.Options<BatchOptions>(options => options
.SetMessageLimit(1000)
.SetTimeLimit(1000)
.SetConcurrencyLimit(2));
});
});
await handle.Ready;
You could also, since you're using the receive endpoint connector, configure the batch options in the consumer definition as shown in the documentation.
5 minutes after I posted the question I tried to change the order of the batch configuration, and putting the consumer as the last statement, did the trick.
hostHandler = receiveEndpointConnector.ConnectReceiveEndpoint(queueName, (context, cfg) =>
{
cfg.TrySetPrefetchCount(2000);
cfg.Batch<T>(cfg =>
{
cfg.ConcurrencyLimit = 2;
cfg.MessageLimit = 1000;
cfg.TimeLimit = TimeSpan.FromSeconds(1);
cfg.Consumer(() => consumer);
});
cfg.UseMessageRetry(r => r.Immediate(2)));
});
await hostHandler.Ready;
Related
I'm rather new to observables and still learning and i need some help.
I have a slider with some campaigns and for each campaign i have a producer which i need get and set to the campaign.
I know that using pipe, merge and MergeMap here is probably some of the solution but i can't seem to get it to work, does anyone have an idea for me?
This is what i currently have.
fetchCampaigns() {
this.loading = true;
this.campaignService
.getCampaigns(
this.campaignAmount,
this.sorting,
this.desc
)
.subscribe(
campaignListDto => {
this.campaigns = campaignListDto;
this.campaigns.map(campaign => {
this.producerService.getProducerByName(campaign.producer_name)
.subscribe((producer) => {
campaign.producer = producer
campaign.producer_name
})
})
this.loading = false;
this.fetchProducerMinPrices();
this.fetchProducerTotalCampaigns();
})
};
Try:
fetchCampaigns() {
this.loading = true;
this.campaignService.getCampaigns(this.campaignAmount,this.sorting,this.desc).pipe(
switchMap(campaignListDto => forkJoin(campaignListDto.map(
campaign => this.producerService.getProducerByName(campaign.producer_name).pipe(
map(producer => ({ ...campaign, producer })),
),
))),
).subscribe(campaignListDtoExtended => {
this.campaigns = campaignListDtoExtended;
this.loading = false;
this.fetchProducerMinPrices();
this.fetchProducerTotalCampaigns();
});
}
I configured MassTransit on my .NET core application as follows:
public void ConfigureServices(IServiceCollection services)
{
[...]
// producer
services.AddMassTransit(x =>
{
x.AddBus(provider => Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Host(new Uri(_configuration["RabbitMQ:URI"] + _configuration["RabbitMQ:VirtualHost"]), $"ENG {_configuration["SiteID"]} Producer", h =>
{
h.Username(_configuration["RabbitMQ:UserName"]);
h.Password(_configuration["RabbitMQ:Password"]);
});
cfg.Publish<NormUpdate>(x =>
{
x.Durable = true;
x.AutoDelete = false;
x.ExchangeType = "fanout"; // default, allows any valid exchange type
});
cfg.ConfigurePublish(x => x.UseExecute(x =>
{
x.Headers.Set("SiteID", _configuration["SiteID"]);
}));
}));
});
services.AddMassTransit<ISecondBus>(x =>
{
x.AddConsumer<NormConsumer>();
x.AddBus(context => Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.UseMessageRetry(r => r.Intervals(100, 200, 500, 800, 1000));
cfg.Host(new Uri(_configuration["RabbitMQ:URI"] + _configuration["RabbitMQ:VirtualHost"]), $"ENG {_configuration["SiteID"]} Consumer", h =>
{
h.Username(_configuration["RabbitMQ:UserName"]);
h.Password(_configuration["RabbitMQ:Password"]);
});
cfg.ReceiveEndpoint($"norm-queue-{_configuration["SiteID"]}", e =>
{
e.Durable = true;
e.AutoDelete = false;
e.Consumer<NormConsumer>(context);
e.UseConcurrencyLimit(1);
e.ExchangeType = "fanout";
e.PrefetchCount = 1;
});
}));
});
services.AddOptions<MassTransitHostOptions>().Configure(options =>
{
options.WaitUntilStarted = false;
options.StopTimeout = TimeSpan.FromSeconds(30);
});
[...]
}
public interface ISecondBus : IBus
{
}
I noticed that when connections are created, the consumer connection has 2 channels. Channel (1) with no attached consumers, channel (2) with one consumer.
I expected to have only one channel on receiver.
Is this a normal behavior or am I doing something wrong?
You should have three channels:
One for the first bus you configured
One for the second bus you configured (MultiBus configures completely separate bus instances, nothing is shared)
One for the receive endpoint on the second bus you configured.
1 + 1 + 1 = 3
Q.E.D.
Yes, I was in Math club back in primary school.
I have microservice-based system which works with documents. Service publishes DocflowErrorMq, ImportedDocflowMq events, and other services are subscribed to these events. Critical service DocflowRegistry should process messages quickly, so we have to introduce multiple consumers. On the other hand message order shouldn't be broken and competing consumer doesn't suite. Consistent hash exchange distributes messages by routing key equals to document id, messages related to one document goes to one queue. So, we have simple manual scaling. I can't create binding between MqModels.Docflows:ImportedDocflowMq and docflow-process-dr exchanges (marked red on Diagram). Is it possible to create it with MassTransit?
DocflowRegistry service config:
services.AddMassTransit(x =>
{
x.AddConsumer<DocflowSendingErrorTestConsumer>();
x.AddConsumer<DocflowImportTestConsumer>();
x.UsingRabbitMq((context, cfg) =>
{
var virtualHost = configuration["RabbitMq:Settings:VirtualHost"] ?? "/";
cfg.Host(configuration["RabbitMqHost"], virtualHost, h =>
{
h.Username(configuration["RabbitMqUserName"]);
h.Password(configuration["RabbitMqPassword"]);
});
cfg.ReceiveEndpoint("docflow.process-1.docflowregistry", e =>
{
e.ConfigureConsumer<DocflowSendingErrorTestConsumer>(context);
e.ConfigureConsumer<DocflowImportTestConsumer>(context);
e.Bind("docflow-process-dr", x =>
{
x.Durable = true;
x.AutoDelete = false;
x.ExchangeType = "x-consistent-hash";
x.RoutingKey = "1";
});
e.ConfigureConsumeTopology = false;
e.SingleActiveConsumer = true;
});
cfg.ReceiveEndpoint("docflow.process-2.docflowregistry", e =>
{
e.ConfigureConsumer<DocflowSendingErrorTestConsumer>(context);
e.ConfigureConsumer<DocflowImportTestConsumer>(context);
e.Bind("docflow-process-dr", x =>
{
x.Durable = true;
x.AutoDelete = false;
x.ExchangeType = "x-consistent-hash";
x.RoutingKey = "1";
});
e.ConfigureConsumeTopology = false;
e.ConcurrentMessageLimit = 1;
e.SingleActiveConsumer = true;
});
});
});
Config of TodoList service:
services.AddMassTransit(x =>
{
x.AddConsumer<DocflowSendingErrorTestConsumer>();
x.AddConsumer<DocflowImportTestConsumer>();
x.UsingRabbitMq((context, cfg) =>
{
var virtualHost = configuration["RabbitMq:Settings:VirtualHost"] ?? "/";
cfg.Host(configuration["RabbitMqHost"], virtualHost, h =>
{
h.Username(configuration["RabbitMqUserName"]);
h.Password(configuration["RabbitMqPassword"]);
});
cfg.ReceiveEndpoint("docflow-process-todolist", e =>
{
e.ConfigureConsumer<DocflowSendingErrorTestConsumer>(context);
e.ConfigureConsumer<DocflowImportTestConsumer>(context);
e.SingleActiveConsumer = true;
});
});
});
Publish code:
var endPoint = await _massTransitBus.GetPublishSendEndpoint<DocflowErrorMq>();
var docflowGuid = Guid.NewGuid();
await endPoint.Send(new DocflowErrorMq
{
DocflowId = docflowGuid,
AbonentId = Guid.NewGuid()
},
context =>
{
context.SetRoutingKey(docflowGuid.ToString());
});
Create an interface, DocflowProcessDr, and make each of those message contracts published implement it. Then, you can configure the publish topology for that interface in the bus:
cfg.Message<DocflowProcessDr>(x => x.SetEntityName("docflow-process-dr"));
cfg.Publish<DocflowProcessDr>(x =>
{
x.ExchangeType = "x-consistent-hash";
});
Since MassTransit will create a polymorphic topology on the broker, you'll have an exchange-to-exchange binding between the published type and the interface.
Then, just publish the message:
var docflowGuid = Guid.NewGuid();
var endPoint = await _massTransitBus.Publish<DocflowErrorMq>(new DocflowErrorMq
{
DocflowId = docflowGuid,
AbonentId = Guid.NewGuid()
},
context =>
{
context.SetRoutingKey(docflowGuid.ToString());
});
Calling GetPublishSendEndpoint<T>() is weird, don't encourage it.
I perform http requests to my db and have noticed that if I send all the requests at once, some of them will get a timeout errors. I'd like to add a delay between calls so the server doesn't get overloaded. I'm trying to find the RxJS solution to this problem and don't want to add a setTimeout.
Here is what I currently do:
let observables = [];
for(let int = 0; int < 10000; int++){
observables.push(new Observable((observer) => {
db.add(doc[int], (err, result)=>{
observer.next();
observer.complete();
})
}))
}
forkJoin(observables).subscribe(
data => {
},
error => {
console.log(error);
},
() => {
db.close();
}
);
You can indeed achieve this with Rxjs quite nicely. You'll need higher order observables, which means you'll emit an observable into an observable, and the higher order observable will flatten this out for you.
The nice thing about this approach is that you can easily run X requests in // without having to manage the pool of requests yourself.
Here's the working code:
import { Observable, Subject } from "rxjs";
import { mergeAll, take, tap } from "rxjs/operators";
// this is just a mock to demonstrate how it'd behave if the API was
// taking 2s to reply for a call
const mockDbAddHtppCall = (id, cb) =>
setTimeout(() => {
cb(null, `some result for call "${id}"`);
}, 2000);
// I have no idea what your response type looks like so I'm assigning
// any but of course you should have your own type instead of this
type YourRequestType = any;
const NUMBER_OF_ITEMS_TO_FETCH = 10;
const calls$$ = new Subject<Observable<YourRequestType>>();
calls$$
.pipe(
mergeAll(3),
take(NUMBER_OF_ITEMS_TO_FETCH),
tap({ complete: () => console.log(`All calls are done`) })
)
.subscribe(console.log);
for (let id = 0; id < NUMBER_OF_ITEMS_TO_FETCH; id++) {
calls$$.next(
new Observable(observer => {
console.log(`Starting a request for ID "${id}""`);
mockDbAddHtppCall(id, (err, result) => {
if (err) {
observer.error(err);
} else {
observer.next(result);
observer.complete();
}
});
})
);
}
And a live demo on Stackblitz: https://stackblitz.com/edit/rxjs-z1x5m9
Please open the console of your browser and note that the console log showing when a call is being triggered starts straight away for 3 of them, and then wait for 1 to finish before picking up another one.
Looks like you could use an initial timer to trigger the http calls. e.g.
timer(delayTime).pipe(combineLatest(()=>sendHttpRequest()));
This would only trigger the sendHttpRequest() method after the timer observable had completed.
So with your solution. You could do the following...
observables.push(
timer(delay + int).pipe(combineLatest(new Observable((observer) => {
db.add(doc[int], (err, result)=>{
observer.next();
observer.complete();
}))
}))
Where delay could probably start off at 0 and you could increase it using the int index of your loop by some margin.
Timer docs: https://www.learnrxjs.io/learn-rxjs/operators/creation/timer
Combine latest docs: https://www.learnrxjs.io/learn-rxjs/operators/combination/combinelatest
merge with concurrent value:
mergeAll and mergeMap both allow you to define the max number of subscribed observables. mergeAll(1)/mergeMap(LAMBDA, 1) is basically concatAll()/concatMap(LAMBDA).
merge is basically just the static mergeAll
Here's how you might use that:
let observables = [...Array(10000).keys()].map(intV =>
new Observable(observer => {
db.add(doc[intV], (err, result) => {
observer.next();
observer.complete();
});
})
);
const MAX_CONCURRENT_REQUESTS = 10;
merge(...observables, MAX_CONCURRENT_REQUESTS).subscribe({
next: data => {},
error: err => console.log(err),
complete: () => db.close()
});
Of note: This doesn't batch your calls, but it should solve the problem described and it may be a bit faster than batching as well.
mergeMap with concurrent value:
Perhaps a slightly more RxJS way using range and mergeMap
const MAX_CONCURRENT_REQUESTS = 10;
range(0, 10000).pipe(
mergeMap(intV =>
new Observable(observer => {
db.add(doc[intV], (err, result) => {
observer.next();
observer.complete();
});
}),
MAX_CONCURRENT_REQUESTS
)
).subscribe({
next: data => {},
error: err => console.log(err),
complete: () => db.close()
});
How do I get multiple subscribers waiting the same promise to resolve if it is already inflight with latecomers given a new resolution?
doSomething = () => {
return new Promise((resolve) => {
setTimeout(() => resolve(Math.random(), 1000)
})
}
// how to define obs?
obs.subscribe(v => console.log(v)); // 0.39458743297857473
obs.subscribe(v => console.log(v)); // 0.39458743297857473
obs.subscribe(v => console.log(v)); // 0.39458743297857473
setTimeout(() => obs.subscribe(v => console.log(v)), 2000); // 0.9485769395265746
I'd like the observable to remain cold until the first subscriber, then go cold again after the result is streamed to all subsequent concurrent subscribers. I basically don't want any concurrent requests to the same underlying function.
You can use defer as the creation-operator and then share the stream:
doSomething = () => {
return new Promise((resolve) => {
setTimeout(() => resolve(Math.random(), 1000));
});
}
const obs = Rx.Observable
.defer(doSomething)
.share();
obs.subscribe(console.log); // resolve #1
obs.subscribe(console.log); // resolve #1
obs.subscribe(console.log); // resolve #1
setTimeout(() => obs.subscribe(console.log), 2000); // resolve #2
<script src="https://unpkg.com/rxjs/bundles/Rx.min.js"></script>