The main objective is to guarantee the reliability of the message delivery of our integration events in case of failures with the Application or DB, and in this scenario the new Transactional Outbox seems to be the most indicated.
With the MT Transactional Outbox, the writing (publishing) of our integration events happens atomically in the Commit of our Command Handlers.
Our solution is built as microservices that use one DB per tenant. However, the applications are shared and the DB (connection string) selection is built into the request scope.
The Outbox MT tables (OutboxMessage, In/OutboxState) are created in each tenant DB and with that we have the atomic writing of the operation along the triggering of integration events.
The problem is in the delivery to the Bus. How to make the MT background services (DeliveryService and InboxCleanupService) reach the outbox in the tenant DB, since there is no request to identify the tenant in advance?
It wouldn't be a problem if the application also runs in an environment for each tenant with their respective configuration / connection string, but that's not the case.
An alternative would be to create a third application dedicated only to running delivery/cleanup services for all tenants in separate tasks.
But to use MT delivery, we would have to dynamically create a DbContext instance (with only the Mappings for MT) for each tenant at startup. I don't know if this is the best way and it also doesn't seem possible to use the MassTransit configuration API this way.
Does anyone have an advice, approach or guide that could help us?
We were having exactly the same issue as you are describing, and we've ended up with the solution you are essentially describing yourself, a separate service that does nothing but outbox delivery and inbox cleanup.
For every MassTransit endpoint, we disable both outbox delivery and inbox cleanup
mt.AddEntityFrameworkOutbox<ContextWithOutboxTables>(outbox =>
{
var outboxConfig = outbox.UseSqlServer();
outboxConfig.DisableInboxCleanupService();
outboxConfig.UseBusOutbox(busOutbox =>
{
busOutbox.DisableDeliveryService();
});
}
and the separate service is configured similar to the below (with some code replaced/removed)
builder.ConfigureServices(s =>
{
foreach (var connectionString in distinctTenantConnectionStrings)
{
IServiceCollection outboxServices = new ServiceCollection();
outboxServices.AddDbContext<ContextWithOutboxTables>(options =>
{
options.UseSqlServer(connectionString);
});
var outboxContainer = outboxServices.BuildServiceProvider();
s.AddSingleton<IHostedService>(container =>
{
return new BusOutboxDeliveryService<ContextWithOutboxTables>(
container.GetRequiredService<IBusControl>(),
new OptionsWrapper<OutboxDeliveryServiceOptions>(new OutboxDeliveryServiceOptions()
{
// tweak your options here
}),
new OptionsWrapper<EntityFrameworkOutboxOptions>(new EntityFrameworkOutboxOptions()
{
LockStatementProvider = new SqlServerLockStatementProvider(),
// tweak your options here
}),
logger,
outboxContainer);
});
s.AddSingleton<IHostedService>(container =>
new InboxCleanupService<ContextWithOutboxTables>(new OptionsWrapper<InboxCleanupServiceOptions>(
new InboxCleanupServiceOptions()
{
// tweak your options here
}),
logger,
outboxContainer)
);
}
});
It is necessary to register the hosted service using the AddSingleton<IHostedService> override instead of AddHostedService because MS dependency injection eliminates duplicate service implementations under the covers, which is not the behavior you want, you ARE actually registering the same hosted service multiple times, with different configuration.
Creating a separate container via
IServiceCollection outboxServices = new ServiceCollection();
is necessary due to MT resolving the data context from the container, and you need it to provide a differently configured data context (connection string) for each instance of the hosted service.
Related
I have earlier achieved this .net 3.1. But it couldn't be possible with .Net 6 because of startup.cs removed.
I have registered a few services,
builder.Services.AddControllers();
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
var appSettings = builder.Configuration.GetSection("AppSettings").Get<AppSettings>();
builder.Services.AddScoped<IEncryption, Encryption>();
//Here I need to get the IEncryption Service, and call the method in this service to encrypt/decrypt the connection string to pass to DBContext Service.
builder.Services.AddDbContext<CatalogDbContext>(options => options.UseNpgsql(
appSettings.ConnectionString));
var app = builder.Build();
Earlier in .NET 3.1, I used BuildServicProvider() to get the Encryption service, and call the methods in that service to do the required logic then got the proper connection string I wanted that would be passed to the DBContext service on the next line.
Now, .NET 6/7 is forced to use the services only after app = builder.Build(); so, I can't register the DBCOntext after the build() method.
How can I solve this case? Any recommended approach to do this in .NET 6/7?
You still can useStartup.cs in .net 6
var builder = WebApplication.CreateBuilder(args);
var startup = new Startup(builder.Configuration);
startup.ConfigureServices(builder.Services); // calling ConfigureServices method
var app = builder.Build();
startup.Configure(app, builder.Environment); // calling Configure method
And then you can use ConfigureServices and Configure methods to register your services before building.
You didn't need to use BuildServiceProvider in .NET Core 3.1 either. AddDbContext has an overload that provides access to an IServiceProvider instance :
builder.Services.AddDbContext<CatalogDbContext>((services,options) =>{
var myOwnDecrypter=services.GetRequiredService<IMyOwnDecrypter>();
var cns=myOwnDecrypter.Decrypt(appSettings.ConnectionString,key);
options.UseNpgsql(cns);
});
or, if you use the ASP.NET Core Data Protection package :
builder.Services.AddDataProtection();
...
builder.Services.AddDbContext<CatalogDbContext>((services,options) =>{
var protector = services.GetDataProtector("Contoso.Example.v2");
var cns=protector.Unprotect(appSettings.ConnectionString);
options.UseNpgsql(cns);
});
or, if IConfiguration.GetConnectionString is used :
builder.Services.AddDataProtection();
...
builder.Services.AddDbContext<CatalogDbContext>((services,options) =>{
var conn_string=services.GetService<IConfiguration>()
.GetConnectionString("MyConnectionString");
var protector = services.GetDataProtector("Contoso.Example.v2");
var cns=protector.Unprotect(conn_string);
options.UseNpgsql(cns);
});
That said, it's the configuration provider's job to decrypt encrypted settings, not the service/context's. ASP.NET Core's configuration allows using multiple different configuration sources in the same host, not just a single settings file. There's nothing special about appsettings.json. That's just the default settings file name.
You can add another settings file with sensitive contents with AddJsonSettings. That file could use the file system's encryption, eg NTFS Encryption, to ensure it's only readable by the web app account
You can read settings from a key management service, like Hashicorp, Azure Key Vault, Amazon Key Management etc.
You can create your own provider that decrypts its input. The answers to this SO questino show how to do this and one of them inherits from JsonConfigurationProvider directly.
Important Caveat: In general, my suggestion below is a bad practice
Do not call BuildServiceProvider
Why is bad? Calling BuildServiceProvider from application code results in more than one copy of singleton services being created which might result in incorrect application behavior.
Justification: I think it is safe to call BuildServiceProvider as long as you haven't registered any singletons before calling it. Admittedly not ideal, but it should work.
You can still callBuildServiceProvider() in .Net6:
builder.Services.AddScoped<IEncryption, Encryption>();
// create service provider
var provider = builder.Services.BuildServiceProvider();
var encryption = scope.ServiceProvider.GetService<IEncryptionService>();
// use service here
or alternatively
builder.Services.AddScoped<IEncryption, Encryption>();
var provider = builder.Services.BuildServiceProvider();
using (var scope = provider.CreateScope()) {
var encryption = scope.ServiceProvider.GetService<IEncryptionService>();
// use service here
}
Alternative:
You can still use the classic startup structure in .Net6/7. We upgraded our .Net3.1 projects to .Net6 without having to rewrite/restructure the Startup()
I am using Hangfire to execute recurring jobs in my web API and I use System.Web.HttpContext.Current.Server.MapPath in the "RoutineMethod" function.
But the problem is it throws object null exception.
I searched the problem and found that recurring jobs don't use http threads, they use background threads.
Now to resolve this problem I need to call my internal (endpoint) using httpclient.
But to do that I need to give URL of the Web API (to generate URI). So is there any other way to call internal function using httpclient.
My current code:
public static async Task<IHttpActionResult> RoutineTask()
{
//It was like this before.
//await new DemoController().RoutineMethod();
//await new DemoController().RoutineMethod2();
//I am planning to do this.
using (var client = new HttpClient())
{
//But I need to give URI which I don't think is a good idea.
var uri = new Uri("http://localhost/DemoApp/api/DemoController/RoutineMethod");
await client.GetAsync(uri);
}
return new DemoController().Ok();
}
The short answer is no. The HttpClient, as its name implies, requires an http connection.
There are no smell issues with storing service connection information in a configuration file.
However, to expand on Nkosi's comment, it appears that since your code can create an instance of DemoController, that it must have a reference to that controller's project or even be in the same project. I would extract the interesting code into a library or service that all areas needing the information can reference.
I've read here that is should be possible to host tightly coupled ActorTypes within the same service but I can't seem to find any documentation on exactly how to do it.
I thought it might I need to create my own instance of the ActorService and pass the context into it but I don't seen to be able to find the right API's from the document.
Does anyone have an example they could share ?
Sort of but not really. You can have multiple actor types in the same application. It looks like they're in the same service in Visual Studio, but they're actually deployed as separate services. Bear with me for a minute if you will..
So you can register multiple actors like this:
internal static class Program
{
private static void Main()
{
ActorRuntime.RegisterActorAsync<Actor1>().GetAwaiter().GetResult();
ActorRuntime.RegisterActorAsync<Actor2>().GetAwaiter().GetResult();
Thread.Sleep(Timeout.Infinite);
}
}
Great, I have multiple actor types. This works and you can just do that.
But you want know how it works! Well, that's just a simplified version of this:
internal static class Program
{
private static void Main()
{
ActorRuntime.RegisterActorAsync<Actor1>(
(context, actorType) => new ActorService(context, actorType, () => new Actor1())).GetAwaiter().GetResult();
ActorRuntime.RegisterActorAsync<Actor2>(
(context, actorType) => new ActorService(context, actorType, () => new Actor2())).GetAwaiter().GetResult();
Thread.Sleep(Timeout.Infinite);
}
}
This is more telling of what's actually happening, because you see here I now have two services. So what's going on?
The secret is in the ActorRuntime. It does a little more work than ServiceRuntime (where you register Reliable Services normally). The Actor framework build process does some magic on your behalf to configure a service type and a default service instance inside your application for each actor type. You can see this in your ApplicationManifest.xml, where the build tools sets up default services for you:
<DefaultServices>
<Service Name="Actor1ActorService" GeneratedIdRef="3262c188-3eee-44c5-9d1e-d2c2a2685f89|Persisted">
<StatefulService ServiceTypeName="Actor1ActorServiceType" TargetReplicaSetSize="[Actor1ActorService_TargetReplicaSetSize]" MinReplicaSetSize="[Actor1ActorService_MinReplicaSetSize]">
<UniformInt64Partition PartitionCount="[Actor1ActorService_PartitionCount]" LowKey="-9223372036854775808" HighKey="9223372036854775807" />
</StatefulService>
</Service>
<Service Name="Actor2ActorService" GeneratedIdRef="1bc66d2c-0479-4bb2-a9aa-3254030506f1|Persisted">
<StatefulService ServiceTypeName="Actor2ActorServiceType" TargetReplicaSetSize="[Actor2ActorService_TargetReplicaSetSize]" MinReplicaSetSize="[Actor2ActorService_MinReplicaSetSize]">
<UniformInt64Partition PartitionCount="[Actor2ActorService_PartitionCount]" LowKey="-9223372036854775808" HighKey="9223372036854775807" />
</StatefulService>
</Service>
As an example, if I take the two actor types I have defined above and deploy that application, here is the result:
These are actually separate service instances in the application, and each is of a different service type, all of which is automatically generated for you:
And of course, because they're different service instances, they'll be distribute across the cluster as you would normally expect:
I'll go update that doc.
We're using RabbitMQ + StompJS (w/ SockJS & Spring Websocket as middleware, FWIW) to facilitate broadcasting messages over websockets. Everything is working great, except no matter what we try StompJS creates the Queues as non-auto-delete, meaning we end up with TONS of queues.
We're working around it right now with a policy that cleans out inactive queues after several hours, but we'd rather just have auto-delete queues that terminate after all clients disconnect.
We've attempted setting headers auto_delete, auto-delete, autoDelete and every other possible incantation we can find.
If we stop an inspect the frames before they're transmitted (at the lowest possible level in the depths of StompJS's source) we can see those headers being present. However, they don't seem to be making it to RabbitMQ (or it just doesn't look at them on the "SUBSCRIPTION" command??) and creates them as non-auto-delete.
Interestingly, if we create the queue manually beforehand as auto-delete, the StompJS registration calls error out because the requested SUBSCRIBE expected non-auto-delete. This suggests that it's StompJS (or SockJS) that explicitly state non-auto-delete, but we've poured over the source and ruled that out.
So, the million dollar question: how can we have auto-delete queues with StompJS? Please, pretty please, and thanks in advance :)
Example registration
function reg(dest, callback, headers){
stomp.subscribe(dest, callback, headers);
}
function cb(payload){
console.log(JSON.parse(payload.body));
}
reg('/queue/foobar', cb, {});
Setup details
RabbitMQ 3.5.2 and StompJS 2.3.3
** Note **
If I subscribe directly to the exchange (with destinations like /exchange/foo or /topic/foo) the exchange will be defined as auto-delete. It's only queues that aren't auto-delete.
I'm using StompJS/RabbitMQ in production and I'm not seeing this issue. I can't say for sure what your problem is, but I can detail my setup in the hope you might spot some differences that may help.
I'm running against Rabbit MQ 3.0.1.
I'm using SockJS 0.3.4, I seem to recall having some issues using a more recent release from GitHub, but unfortunately I didn't take notes so I'm not sure what the issue was.
I'm using StompJS 2.3.4
For reasons I won't go into here - I've disabled the WebSockets transport, by whitelisting all the other transports.
Here's some simplified code showing how I connect:
var socket = new SockJS(config.stompUrl, null, { protocols_whitelist: ['xdr-streaming', 'xhr-streaming', 'iframe-eventsource', 'iframe-htmlfile', 'xdr-polling', 'xhr-polling', 'iframe-xhr-polling', 'jsonp-polling'] });
var client = Stomp.over(socket);
client.debug = function () { };
client.heartbeat.outgoing = 0;
client.heartbeat.incoming = 0;
client.connect(config.rabbitUsername, config.rabbitPassword, function () {
onConnected();
}, function () {
reconnect(d);
}, '/');
And here's how I disconnect:
// close the socket first, otherwise STOMP throws an error on disconnect
socket.close();
client.disconnect(function () {
isConnected = false;
});
And here's how I subscribe (this happens inside my onConnected function):
client.subscribe('/topic/{routing-key}', function (x) {
var message = JSON.parse(x.body);
// do stuff with message
});
My first recommendation would be to try the specific versions of the client libs I've listed. I had some issues getting these to play nicely - and these versions work for me.
It is possible with RabbitMQ 3.6.0+ by setting auto-delete in subscribe headers to true. Please see https://www.rabbitmq.com/stomp.html#queue-parameters for details.
Ok so I have an area where people can create OpenHouse listings for a period of 14 days then the listing is expired and should be deleted, my question is; is there some type of way to have the Database delete listings automatically? This is the simple code I have below it works but it is not optimal
protected void Application_Start()
{
openhouse mydate= new openhouse();
if (mydate.expired > DateTime.Now)
{
db.openhouses.Remove(mydate);
}
}
That as you can tell is in my Global.asax but the issue is that listings only get deleted if I Compile the application I am thinking there must be a more efficient way to do this than me compiling every day just for that, any suggestions would be very appreciated ...
The database itself won't provide any timers or scheduled tasks for you (at least SQLLite won't). You have to program a task that cleans up expired listings, for that, you have several options:
Using a task scheduled within your own ASP.net application
Using a task (a separate application) scheduled and managed by your hosting provider. If you are on the cloud, your cloud provider most likely has the capability to do this for you.
If you want to go with the first option, take a look at Quartz. I have a WorkerRole on Azure than uses Quartz to do things like what you need here. It's fairly simple to setup, here is an example:
// Initialize Quartz
ISchedulerFactory schedulerFactory = new StdSchedulerFactory();
_scheduler = schedulerFactory.GetScheduler();
_scheduler.JobFactory = new NinjectJobFactory(_kernel);
DateTimeOffset morning = DateBuilder.NewDateInTimeZone(TimeZoneInfo.FindSystemTimeZoneById("Eastern Standard Time"))
.AtHourMinuteAndSecond(5, 30, 0)
.Build();
// Schedule task for email being sent to people who have not signed up after their trial expiry
IJobDetail userFeedbackJobDetail = JobBuilder.Create<LostUserFeedbackRequestJob>().WithIdentity("User Feedback Emails").Build();
ITrigger dailyTriggerForUserFeedback = TriggerBuilder.Create()
.WithIdentity("Daily Trigger (User Feedback)")
.StartAt(morning)
.WithSimpleSchedule(schedule => schedule.WithInterval(TimeSpan.FromDays(1)).RepeatForever())
.ForJob(userFeedbackJobDetail)
.Build();
_scheduler.ScheduleJob(userFeedbackJobDetail, dailyTriggerForUserFeedback);
// Finally, start the scheduler
_scheduler.Start();
This sends an email everyday asking for feedback from users who have not elected to use the services of one of my projects after their trial period has finished. The job is run everyday at 5:30 a.m.