How to create system wide retry policy in MassTransit and modify/overwrite it in specific consumer? - masstransit

I would like to create system wide retry policy (retry x times every y seconds) but modify/overwrite it in specific consumer (some exception should be ignored and moved to error queue without retry)
I was thinking that something like this should work(bus configuration part):
services.AddMassTransit<T>(
massTransit =>
{
massTransit.UsingRabbitMq(
(context, cfg) =>
{
cfg.UseMessageRetry(r =>
{
r.Intervals(TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(30));
});
....
massTransit.AddConsumers(assembly);
consumer definition part:
public class DoSomethingConsumerDefinition :
ConsumerDefinition<DoSomethingConsumer>
{
protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator,
IConsumerConfigurator<DoSomethingConsumer> consumerConfigurator)
{
endpointConfigurator.UseMessageRetry(r =>
{
r.Ignore<DoSomethingSpecificException>();
r.Intervals(TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(30));
});
}
}
I can't achieve it this way. DoSomethingSpecificException is retried using global policy.
What is the correct way to achieve it?

When configuring multiple retry policies, which is what happens in this case, you'd need to ensure that exceptions you do not want retried are properly overridden. If you ignore the exception, it isn't handled by the retry filter, so you'd need to handle it with no retry:
public class DoSomethingConsumerDefinition :
ConsumerDefinition<DoSomethingConsumer>
{
protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator,
IConsumerConfigurator<DoSomethingConsumer> consumerConfigurator)
{
endpointConfigurator.UseMessageRetry(r =>
{
r.Handle<DoSomethingSpecificException>();
r.None();
});
endpointConfigurator.UseMessageRetry(r =>
{
r.Ignore<DoSomethingSpecificException>();
r.Intervals(TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(30));
});
}
}
With the None retry filter, it would signify that exception should not be retried by a retry/redelivery filter defined prior to the None filter.

Related

How do I run multiple jobs with a given IJobConsumer within a single service instance?

I want to be able to execute multiple jobs concurrently on a Job Consumer. At the moment if I run one service instance and try to execute 2 jobs concurrently, 1 job waits for the other to complete (i.e. waits for the single job slot to become available).
However if I run 2 instances by using dotnet run twice to create 2 separate processes I am able to get the desired behavior where both jobs run at the same time.
Is it possible to run 2 (or more) jobs at the same time for a given consumer inside a single process? My application requires the ability to run several jobs concurrently but I don't have the ability to deploy many instances of my application.
Checking the application log I see this line which I feel may have something to do with it:
[04:13:43 DBG] Concurrent Job Limit: 1
I tried changing the SagaPartitionCount to something other than 1 on instance.ConfigureJobServiceEndpoints to no avail. I can't seem to get the Concurrent Job Limit to change.
My configuration looks like this:
services.AddMassTransit(x =>
{
x.AddDelayedMessageScheduler();
x.SetKebabCaseEndpointNameFormatter();
// registering the job consumer
x.AddConsumer<DeploymentConsumer>(typeof(DeploymentConsumerDefinition));
x.AddSagaRepository<JobSaga>()
.EntityFrameworkRepository(r =>
{
r.ExistingDbContext<JobServiceSagaDbContext>();
r.LockStatementProvider = new SqlServerLockStatementProvider();
});
// add other saga repositories here for JobTypeSaga and JobAttemptSaga here as well
x.UsingRabbitMq((context, cfg) =>
{
var rmq = configuration.GetSection("RabbitMq").Get<RabbitMq>();
cfg.Host(rmq.Host, rmq.Port, rmq.VirtualHost, h =>
{
h.Username(rmq.Username);
h.Password(rmq.Password);
});
cfg.UseDelayedMessageScheduler();
var options = new ServiceInstanceOptions()
.SetEndpointNameFormatter(context.GetService<IEndpointNameFormatter>() ?? KebabCaseEndpointNameFormatter.Instance);
cfg.ServiceInstance(options, instance =>
{
instance.ConfigureJobServiceEndpoints(js =>
{
js.SagaPartitionCount = 1;
js.FinalizeCompleted = true;
js.ConfigureSagaRepositories(context);
});
instance.ConfigureEndpoints(context);
});
});
}
Where DeploymentConsumerDefinition looks like
public class DeploymentConsumerDefinition : ConsumerDefinition<DeploymentConsumer>
{
protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator,
IConsumerConfigurator<DeploymentConsumer> consumerConfigurator)
{
consumerConfigurator.Options<JobOptions<DeploymentConsumer>>(options =>
{
options.SetJobTimeout(TimeSpan.FromMinutes(20));
options.SetConcurrentJobLimit(10);
options.SetRetry(r =>
{
r.Ignore<InvalidOperationException>();
r.Interval(5, TimeSpan.FromSeconds(10));
});
});
}
}
Your definition should specify the job consumer message type, not the job consumer type:
public class DeploymentConsumerDefinition : ConsumerDefinition<DeploymentConsumer>
{
protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator,
IConsumerConfigurator<DeploymentConsumer> consumerConfigurator)
{
// MESSAGE TYPE NOT CONSUMER TYPE
consumerConfigurator.Options<JobOptions<DeploymentCommand>>(options =>
{
options.SetJobTimeout(TimeSpan.FromMinutes(20));
options.SetConcurrentJobLimit(10);
options.SetRetry(r =>
{
r.Ignore<InvalidOperationException>();
r.Interval(5, TimeSpan.FromSeconds(10));
});
});
}
}

MassTransit endpoint name is ignored in ConsumerDefinition

The EndpointName property in a ConsumerDefinition file seems to be ignored by MassTransit. I know the ConsumerDefinition is being used because the retry logic works. How do I get different commands to go to a different queue? It seems that I can get them all to go through one central queue but I don't think this is best practice for commands.
Here is my app configuration that executes on startup when creating the MassTransit bus.
Bus.Factory.CreateUsingAzureServiceBus(cfg =>
{
cfg.Host(_config.ServiceBusUri, host => {
host.SharedAccessSignature(s =>
{
s.KeyName = _config.KeyName;
s.SharedAccessKey = _config.SharedAccessKey;
s.TokenTimeToLive = TimeSpan.FromDays(1);
s.TokenScope = TokenScope.Namespace;
});
});
cfg.ReceiveEndpoint("publish", ec =>
{
// this is done to register all consumers in the assembly and to use their definition files
ec.ConfigureConsumers(provider);
});
And my handler definition in the consumer (an azure worker service)
public class CreateAccessPointCommandHandlerDef : ConsumerDefinition<CreateAccessPointCommandHandler>
{
public CreateAccessPointCommandHandlerDef()
{
EndpointName = "specific";
ConcurrentMessageLimit = 4;
}
protected override void ConfigureConsumer(
IReceiveEndpointConfigurator endpointConfigurator,
IConsumerConfigurator<CreateAccessPointCommandHandler> consumerConfigurator
)
{
endpointConfigurator.UseMessageRetry(r =>
{
r.Immediate(2);
});
}
}
In my app that is sending the message I have to configure it to send to the "publish" queue, not "specific".
EndpointConvention.Map<CreateAccessPointsCommand>(new Uri($"queue:specific")); // does not work
EndpointConvention.Map<CreateAccessPointsCommand>(new Uri($"queue:publish")); // this does work
Because you are configuring the receive endpoint yourself, and giving it the name publish, that's the receive endpoint.
To configure the endpoints using the definitions, use:
cfg.ConfigureEndpoints(provider);
This will use the definitions that were registered in the container to configure the receive endpoints, using the consumer endpoint name defined.
This is also explained in the documentation.

What is the correct way to start a MassTransit bus when using the CreateRequestClient method?

I'm using the following code to send a request/response message between two different processes.
This is the process that "sends" the request:
// configure host
var hostUri = new Uri(configuration["RabbitMQ:Host"]);
services.AddSingleton(provider => Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(hostUri, h =>
{
h.Username(configuration["RabbitMQ:Username"]);
h.Password(configuration["RabbitMQ:Password"]);
});
}));
// add request client
services.AddScoped(provider => provider.GetRequiredService<IBus>().CreateRequestClient<QueryUserInRole, QueryUserInRoleResult>(new Uri(hostUri, "focus-authorization"), TimeSpan.FromSeconds(5)));
// add dependencies
services.AddSingleton<IPublishEndpoint>(provider => provider.GetRequiredService<IBusControl>());
services.AddSingleton<ISendEndpointProvider>(provider => provider.GetRequiredService<IBusControl>());
services.AddSingleton<IBus>(provider => provider.GetRequiredService<IBusControl>());
// add the service class so that the runtime can automatically handle the start and stop of our bus
services.AddSingleton<Microsoft.Extensions.Hosting.IHostedService, BusService>();
Here's the implementation of the BusService:
public class BusService : Microsoft.Extensions.Hosting.IHostedService
{
private readonly IBusControl _busControl;
public BusService(IBusControl busControl)
{
_busControl = busControl;
}
public Task StartAsync(CancellationToken cancellationToken)
{
return _busControl.StartAsync(cancellationToken);
}
public Task StopAsync(CancellationToken cancellationToken)
{
return _busControl.StopAsync(cancellationToken);
}
}
The problem is that when the CreateRequestClient code runs, the bus has not started up yet. Thus the response is never returned from the consumer.
If I replace the host configuration with the following code, I get the desired behavior:
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(hostUri, h =>
{
h.Username(configuration["RabbitMQ:Username"]);
h.Password(configuration["RabbitMQ:Password"]);
});
});
bus.Start();
services.AddSingleton(bus);
For some reason, the BusService(IHostedService) executes AFTER the AddScoped delegates.
My question is: what is the correct way to start up the bus before using the CreateRequestClient method? Or is the latter approach to starting up the bus sufficient?

How/where do I register the IDbPerTenantConnectionStringResolver

Trying to run core api host. i have this in the EntityFrameworkModule
public override void PreInitialize()
{
Configuration.MultiTenancy.IsEnabled = true;`
// CONNECTION STRING RESOLVER
Configuration.ReplaceService<IConnectionStringResolver, DbPerTenantConnectionStringResolver>(DependencyLifeStyle.Transient);
Configuration.DefaultNameOrConnectionString = MyConsts.DefaultConnectionStringName;
if (!SkipDbContextRegistration)
{
//DEFAULT
Configuration.Modules.AbpEfCore().AddDbContext<MyContext>(options =>
{
if (options.ExistingConnection != null)
MyContextConfigurer.Configure(options.DbContextOptions, options.ExistingConnection);
else
MyContextConfigurer.Configure(options.DbContextOptions, options.ConnectionString);
});
}
}
ERROR
Mvc.ExceptionHandling.AbpExceptionFilter - Can't create component 'Ccre.EntityFrameworkCore.AbpZeroDbMigrator' as it has dependencies to be satisfied.
'Ccre.EntityFrameworkCore.AbpZeroDbMigrator' is waiting for the following dependencies:
- Service 'Abp.MultiTenancy.IDbPerTenantConnectionStringResolver' which was not registered.
How/where do I register the IDbPerTenantConnectionStringResolver?
I have this line in the PreInitialize of the Migrator.MigratorModule
Configuration.ReplaceService<IConnectionStringResolver, DbPerTenantConnectionStringResolver>(DependencyLifeStyle.Transient);
as well as in the EntityFrameworkModule

Recurring message Masstransit rabbitmq

I am trying to implement a heartbeat feature for my application hence was trying to implement recurring message feature from masstransit rabbitmq. I was trying to implement it on the sample given on masstransit's website. Here's all of the code.
namespace MasstransitBasicSample
{
using System;
using System.Threading.Tasks;
using MassTransit;
using MassTransit.Scheduling;
class Program
{
static void Main(string[] args)
{
var bus = Bus.Factory.CreateUsingRabbitMq(sbc =>
{
var host = sbc.Host(new Uri("rabbitmq://localhost"), h =>
{
h.Username("guest");
h.Password("guest");
});
sbc.UseMessageScheduler(new Uri("rabbitmq://localhost/quartz"));
sbc.ReceiveEndpoint(host, "test_queue", ep =>
{
ep.Handler<YourMessage>(context =>
{
return Console.Out.WriteLineAsync($"Received: {context.Message.Text}");
});
ep.Handler<PollExternalSystem>(context =>
{
return Console.Out.WriteLineAsync($"Received: {context.Message}");
});
});
});
bus.Start();
SetRecurring(bus);
Console.WriteLine("Press any key to exit");
Console.ReadKey();
bus.Stop();
}
private static async Task SetRecurring(IBusControl bus)
{
var schedulerEndpoint = await bus.GetSendEndpoint(new Uri("rabbitmq://localhost/quartz"));
var scheduledRecurringMessage = await schedulerEndpoint.ScheduleRecurringSend(
new Uri("rabbitmq://localhost/test_queue"), new PollExternalSystemSchedule(), new PollExternalSystem());
}
}
public class YourMessage { public string Text { get; set; } }
public class PollExternalSystemSchedule : DefaultRecurringSchedule
{
public PollExternalSystemSchedule()
{
CronExpression = "* * * * *"; // this means every minute
}
}
public class PollExternalSystem { }
}
I have created a queue called quartz in my rabbitmq queue.
When i run the application it sends a message to the quartz queue and that message just stays there , it does not go to the test queue.
I was also expecting another message to be sent to the quartz queue after a minute based on the cron expression, that also does not happen.
Is my setup wrong?
Any help would be much appreciated.
You need to run the scheduling service that listens on rabbitmq://localhost/quartz, where your messages are being sent.
The documentation page says:
There is a standalone MassTransit service, MassTransit.QuartzService,
which can be installed and used on servers for this purpose. It is
configured via the App.config file and is a good example of how to
build a standalone MassTransit service.
Alternatively, you can host Quartz scheduling in the same process by using in-memory scheduling, described here, by configuring it like this:
var busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
cfg.UseInMemoryScheduler();
});

Resources