Saga handling undeliverable messages - masstransit

From documentation here:
If the configuration of an endpoint changes, or if a message is
mistakenly sent to an endpoint, it is possible that a message type is
received that does not have any connected consumers. If this occurs,
the message is moved to a _skipped queue (prefixed by the original
queue name). The original message content is retained, and additional
headers are added to indicate the host which moved the message.
Should I expect the same behavior for a saga when some message tried to be delivered to the saga but saga instance is not created yet?
In other words, if want to handle message in saga but that message is not a trigger for saga creation. I would expect to see that messages in _skipped queue, but in reality I don't see them either in _skipped or _error queues. Also I can see that the massage successfully was delivered to the saga queue and successfully was consumed somehow without any warnings or errors. Who has consumed the message?
UPDATED
State machine:
public class TestState : SagaStateMachineInstance, IVersionedSaga
{
public Guid CorrelationId { get; set; }
public Guid Id { get; set; }
public string CurrentState { get; set; }
public int Version { get; set; }
public DateTime TimeStart { get; set; }
public int Progress { get; set; }
}
public class TestStateMachine : MassTransitStateMachine<TestState>
{
public TestStateMachine()
{
InstanceState(x => x.CurrentState);
Event(() => ProcessingStarted, x => x.CorrelateById(context => context.Message.Id));
Event(() => ProgressUpdated, x => x.CorrelateById(context => context.Message.Id));
Event(() => ProcessingFinished, x => x.CorrelateById(context => context.Message.Id));
Initially(
When(ProcessingStarted)
.Then(context =>
{
context.Instance.TimeStart = DateTime.Now;
})
.TransitionTo(Processing)
);
During(Processing,
When(ProgressUpdated)
.Then(context =>
{
context.Instance.Progress = context.Data.Progress;
}),
When(ProcessingFinished)
.TransitionTo(Processed)
.ThenAsync(async context =>
{
await context.Raise(AllDone);
})
);
During(Processed,
When(AllDone)
.Then(context =>
{
Log.Information($"Saga {context.Instance.Id} finished");
})
.Finalize());
SetCompletedWhenFinalized();
}
public State Processing { get; set; }
public State Processed { get; set; }
public Event AllDone { get; set; }
public Event<ProcessingStarted> ProcessingStarted { get; set; }
public Event<ProgressUpdated> ProgressUpdated { get; set; }
public Event<ProcessingFinished> ProcessingFinished { get; set; }
}
public interface ProcessingStarted
{
Guid Id { get; }
}
public interface ProgressUpdated
{
Guid Id { get; }
int Progress { get; }
}
public interface ProcessingFinished
{
Guid Id { get; }
}
Configuration
var repository = new MongoDbSagaRepository<TestState>(Environment.ExpandEnvironmentVariables(configuration["MongoDb:ConnectionString"]), "sagas");
var services = new ServiceCollection();
services.AddSingleton(context => Bus.Factory.CreateUsingRabbitMq(x =>
{
IRabbitMqHost host = x.Host(new Uri(Environment.ExpandEnvironmentVariables(configuration["MassTransit:ConnectionString"])), h => { });
x.ReceiveEndpoint(host, "receiver_saga_queue", e =>
{
e.StateMachineSaga(new TestStateMachine(), repository);
});
x.UseRetry(r =>
{
r.Incremental(10, TimeSpan.FromMilliseconds(10), TimeSpan.FromMilliseconds(50));
r.Handle<MongoDbConcurrencyException>();
r.Handle<UnhandledEventException>();
});
x.UseSerilog();
}));
var container = services.BuildServiceProvider();
var busControl = container.GetRequiredService<IBusControl>();
busControl.Start();
Later, when I publish ProgressUpdated event
busControl.Publish<ProgressUpdated>(new
{
Id = id,
Progress = 50
});
I would expect it will raise UnhandledEventException and move the massage to the _error queue but in reality I can see that message appeared in the queue, consumed somehow and I don't see it in either _error or _skipped queues. MongoDB Saga storage also doesn't have any new saga instances created.
What am I doing wrong? I need to configure saga the way that if it get ProgressUpdated event when the saga instance is not created it would be retried later when the instance is ready to accept it.
SOLUTION
Reconfigure the event ProgressUpdated and ProcessingFinished as
Event(() => ProgressUpdated, x =>
{
x.CorrelateById(context => context.Message.Id);
x.OnMissingInstance(m => m.Fault());
});
Event(() => ProcessingFinished, x =>
{
x.CorrelateById(context => context.Message.Id);
x.OnMissingInstance(m => m.Fault());
});
and catching SagaException in UseRetry later
r.Handle<SagaException>();
did the work!
Thanks #Alexey Zimarev for sharing the knowledge!

If a message should be handled by a saga but there is no matching instance, the OnMissingInstance callback is called.
Reconfigure ProgressUpdated and ProcessingFinished:
Event(() => ProgressUpdated, x =>
{
x.CorrelateById(context => context.Message.Id);
x.OnMissingInstance(m => m.Fault());
});
Event(() => ProcessingFinished, x =>
{
x.CorrelateById(context => context.Message.Id);
x.OnMissingInstance(m => m.Fault());
});
and catch SagaException in the retry filter:
r.Handle<SagaException>();

Related

MassTransit/Quartz.NET schedule works for InMemory but not in persistent mode

When using InMemoryMessageScheduler with Quartz, the ScheduledMessage actually gets scheduled and the "Message" is published at the defined time.
The problem is when the Quartz with persisting option is used, the message is persisted into the database, the ScheduledMessage is consumed by the ScheduleMessageConsumer BUT the "Message" is never published at the defined time.
I've used Sample-GettingStarted and I've added the next changes:
QuartzConfig:
public class QuartzConfig : Dictionary<string, string>
{
public QuartzConfig(string connectionString)
{
this["quartz.scheduler.instanceName"] = "MassTransit-Scheduler";
this["quartz.scheduler.instanceId"] = "AUTO";
this["quartz.serializer.type"] = "json";
this["quartz.jobStore.type"] = "Quartz.Impl.AdoJobStore.JobStoreTX, Quartz";
this["quartz.jobStore.driverDelegateType"] = "Quartz.Impl.AdoJobStore.StdAdoDelegate, Quartz";
this["quartz.jobStore.tablePrefix"] = "QRTZ_";
this["quartz.jobStore.dataSource"] = "myDS";
this["quartz.dataSource.myDS.provider"] = "Npgsql";
this["quartz.dataSource.myDS.connectionString"] = connectionString;
this["quartz.jobStore.useProperties"] = "true";
}
public NameValueCollection ToNameValueCollection()
{
return this.Aggregate(new NameValueCollection(), (seed, current) =>
{
seed.Add(current.Key, current.Value);
return seed;
});
}
}
Configuration:
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) =>
{
services.AddMassTransit(x =>
{
x.AddConsumer<MessageConsumer>();
var scheduler = CreateScheduler();
x.UsingRabbitMq((context,cfg) =>
{
cfg.ReceiveEndpoint("quartz", endpoint =>
{
endpoint.Consumer(() => new ScheduleMessageConsumer(scheduler));
endpoint.Consumer(() => new CancelScheduledMessageConsumer(scheduler));
cfg.UseMessageScheduler(endpoint.InputAddress);
});
cfg.ConfigureEndpoints(context);
});
});
services.AddMassTransitHostedService();
services.AddHostedService<Worker>();
});
static IScheduler CreateScheduler()
{
var dbConnectionString = "Host=localhost;Database=scheduler;Port=5432;Password=pass;User ID=user;Pooling=true;MaxPoolSize=200;Enlist=true";
var quartzConfig = new QuartzConfig(dbConnectionString)
.ToNameValueCollection();
ISchedulerFactory schedulerFactory = new StdSchedulerFactory(quartzConfig);
return schedulerFactory.GetScheduler().GetAwaiter().GetResult();
}
Worker.cs:
public class Worker : BackgroundService
{
readonly IBus _bus;
public Worker(IBus bus)
{
_bus = bus;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
await _bus.CreateMessageScheduler().SchedulePublish(DateTime.UtcNow + TimeSpan.FromSeconds(5), new Message { Text = $"I really hope this is scheduled {DateTime.Now}" });
await Task.Delay(10000, stoppingToken);
}
}
}
Any ideas?
The Quartz Integration package connects a bus observer to handle the start/stop of Quartz.NET, as shown in the source. Unfortunately the documentation isn't great on how to do it.

MongoDB search nested objects in an array

I am using MongoDB to store all the events in my Eventbrite clone. So I have a collection called events then the objects in this collection consists of their name and and array of users that have rsvp to the event. I can query for any events that the current user has created but unable to figure out how to query both events the user has created and rsvp to.
Here is the compiled query that I am using to try to get all the users events.
events.find({"$and":[{"user_id":"5d335704802df000076bad97"},{"user_id":{"$ne":null}}],"$or":[{"checkins.user_id":"5d335704802df000076bad97"}]},{"typeMap":{"root":"array","document":"array"}})
I am using the Laravel MongoDB plugin to query my data in php it looks like this
$user->events()->orWhere(function ($query) use ($user){
return $query->where('checkins.user_id',new ObjectID($user->id));
})->get()
The event object looks something like this:
{
"name": "test",
"user_id": "1"
"rsvp": [
{"user_id": "12"}
]
}
An user can rsvp to other event that are not their own.
you need an $or filter and $elemMatch to get events that belong to a given user or events they've rsvp'd to.
db.events.find({
"$or": [
{
"user_id": "5d33e732e1ea9d0d6834ef3d"
},
{
"rsvp": {
"$elemMatch": {
"user_id": "5d33e732e1ea9d0d6834ef3d"
}
}
}
]
})
unfortunately i can't help you with laravel version of the query. in case it helps, below is the c# code that generated the above mongo query.
using MongoDB.Entities;
using System.Linq;
namespace StackOverflow
{
public class Program
{
public class user : Entity
{
public string name { get; set; }
}
public class Event : Entity
{
public string name { get; set; }
public string user_id { get; set; }
public rsvp[] rsvp { get; set; }
}
public class rsvp
{
public string user_id { get; set; }
}
private static void Main(string[] args)
{
new DB("test");
var mike = new user { name = "mike" };
var dave = new user { name = "dave" };
mike.Save();
dave.Save();
(new[] {
new Event
{
name = "mike's event",
user_id = mike.ID,
rsvp = new[]
{
new rsvp { user_id = dave.ID }
}
},
new Event
{
name = "dave's event",
user_id = dave.ID,
rsvp = new[]
{
new rsvp { user_id = mike.ID }
}
}
}).Save();
var result = DB.Find<Event>()
.Many(e =>
e.user_id == mike.ID ||
e.rsvp.Any(r => r.user_id == mike.ID));
}
}
}

ReactiveUI Xamarin iOS routing

I started experimenting with Reactive UI in Xamarin iOS and i'm confused to how I should handle routing.
Let's take the typical 'LoginViewModel' which has something like:
public class LoginViewModel : ReactiveObject
{
private readonly ReactiveCommand loginCommand;
public ReactiveCommand LoginCommand => this.loginCommand;
string username;
public string Username
{
get { return username; }
set { this.RaiseAndSetIfChanged(ref username, value); }
}
string password;
public string Password
{
get { return password; }
set { this.RaiseAndSetIfChanged(ref password, value); }
}
public LoginViewModel()
{
var canLogin = this.WhenAnyValue(
x => x.Username,
x => x.Password,
(u, p) => !string.IsNullOrEmpty(u) && !string.IsNullOrEmpty(p)
);
this.loginCommand = ReactiveCommand.CreateFromTask(async () =>
{
//Simulate login
await Task.Delay(2000);
return true;
}, canLogin);
}
}
And the ViewDidLoad (Controller):
this.WhenActivated(d =>
{
this.Bind(this.ViewModel, x => x.Username, x => x.Username.Text).DisposeWith(d);
this.Bind(this.ViewModel, x => x.Password, x => x.Password.Text).DisposeWith(d);
this.BindCommand(this.ViewModel, x => x.LoginCommand, x => x.LoginButton).DisposeWith(d);
});
This effectively binds the values of those UITextFields and Login button enable states + OnTouchUpInside
Now, in de documentation you can find the following about vm-routing: Android, iOS Native: Very difficult to make work
So what would be my options here ?
Expose a DidLogIn property (bool) and listen (in the view) on that with:
this.WhenAnyValue(x => x.ViewModel.DidLogIn)
.ObserveOn(RxApp.MainThreadScheduler)
.Subscribe(() => {
//Routing logic here
});
Are there other ways to handle view-routing (not vm-routing), I can find very little information about this
ReactiveUI's routing on Xamarin Forms is very easy, Xamarin Android/iOS is different history, but you can try ReactiveUI's interactions here is an example:
public class LoginViewModel : ReactiveObject
{
private readonly Interaction<Unit, Unit> _navigate;
public Interaction<Unit, Unit> Navigate => Navigate;
public ReactiveCommand<Unit,bool> LoginCommand { get; set; }
public LoginViewModel()
{
var canLogin = this.WhenAnyValue(
x => x.Username,
x => x.Password,
(u, p) => !string.IsNullOrEmpty(u) && !string.IsNullOrEmpty(p));
_navigate = new Interaction<Unit, Unit>();
LoginCommand = ReactiveCommand.CreateFromTask<Unit, bool>(async _ =>
{
/*Logic here*/
return true;
}, canLogin);
LoginCommand.Subscribe(async result =>
{
if (result)//this logic gets executed on your view by registering a handler :D
await await _navigate.Handle(Unit.Default) ;
else
{}
});
}
}
So in your view
this.WhenActivated(disposables =>
{
//bindings...
//Register a handler:
ViewModel.Navigate.RegisterHandler(async interaction =>
{
await NavigationLogic();
interaction.SetOutput(Unit.Default);
}).DisposeWith(disposables);
});
The example is not perfect, but is one way to do it.
I hope this helps you, you can find more on interactions in: https://reactiveui.net/docs/handbook/interactions/
Also there is an example in Xamarin Android + ReactiveUI: https://github.com/bl8/ReactivePhoneword
Regards

NEST ElasticSearch Guid problems

I have got some problems to filter my queries on field Guid. Here a sample of my code. What did I miss?
public class myObject
{
public Guid Id {get;set}
public String Field1 { get; set; }
public String Field2 { get; set; }
public String Fieldn { get; set; }
public ReadingRightsEnum ReadingRights { get; set; }
public Guid UserId { get; set; }
}
// Index fct example
public void IndexMyObject(obj)
{
var result = await myClient.IndexAsync(obj, i => i
.Index("myIndexName")
.Type("myObject")
.Id(obj.Id.ToString())
.Refresh());
}
// Index fct example
public void SearchOnlyInMyUserObject(string userQuery, Guid userId)
{
var searchResult = await myClient.SearchAsync<myObject>(body =>
body.Query(q =>
q.QueryString(qs => qs.MinimumShouldMatchPercentage(100).Query(userQuery))
&& q.Term(i => i.UserId, userId))
.Fields(f => f.Id)
.Size(200));
}
// Index fct example with filter
public void SearchOnlyInMyUserObject(string userQuery, Guid userId)
{
var filters = new List<FilterContainer>
{
new FilterDescriptor<myObject>().Bool(b => b.Must(m => m.Term(i => i.UserId, userId)));
};
var searchResult = await myClient.SearchAsync<myObject>(body =>
body
.Filter(f => f.And(filters.ToArray()))
.Query(q =>
q.QueryString(qs => qs.MinimumShouldMatchPercentage(100).Query(userQuery)))
.Fields(f => f.Id)
.Size(200));
}
Both functions work fine if I filter on others parameters but return nothing when I filter on Guid. Should a convert my Guid to string when I index my object?
If i do http://xxxxxxx:9200/_search?q=userId:e4aec7b4-c400-4660-a09e-a0ce064f612e it's work fine.
Any ideas?
Thanks by advance
Edit 06/12 here a sample of myindex:
myIndexName":{
"mappings":{
"myObject":{
"properties":{
"readingrights":{
"type":"integer"
},
"id":{
"type":"string"
},
"field1":{
"type":"string"
},
"field2":{
"type":"string"
},
"userId":{
"type":"string"
}
}
}
}
}
GUID field is tricky in Elastic. If you use analyze function of elastic client it will show you how it breaks up a GUID.
AnalyzeRequest obj = new AnalyzeRequest(_index, item);
_client.Analyze(obj);
When you create an entity, You need to define guid as not be analyzed.
[String(Index = FieldIndexOption.NotAnalyzed)]
public Guid TextAssetId
I didn't fine for now why Guid got problems... but I fine a poor alternative way for now:
Instead of :
q.QueryString(qs => qs.MinimumShouldMatchPercentage(98).Query(userQuery))
&& q.Term(i => i.UserId, userId)
I do a double QueryString:
q.QueryString(qs => qs.MinimumShouldMatchPercentage(98).Query(userQuery))
&& q.QueryString(qs => qs.MinimumShouldMatchPercentage(100).Query(" \"" + userId+ "\""))

Kendo DropDownList not populating

Kendo drop down is empty for some reason and I am not sure, below is all my code
#(Html.Kendo().DropDownList()
.Name("parties")
.HtmlAttributes(new { style = "width: 250px" })
.DataTextField("Name")
.DataValueField("PartyId")
.DataSource(source =>
{
source.Read(read =>
{
read.Action("GetParties", "Concept");
});
})
)
Controller Call
public JsonResult GetParties([DataSourceRequest] DataSourceRequest request)
{
var parties = MiscAdapter.GetParties().Select(x => new PartyModel
{
Name = x.PartyName,
PartyId = x.PartyId
});
return Json(parties.ToDataSourceResult(request), JsonRequestBehavior.AllowGet);
}
Model
public class PartyModel
{
public int PartyId { get; set; }
public string Name { get; set; }
}
Data returned according to the F12 tools
{"Data":[{"PartyId":1,"Name":"New Democratic Party"},{"PartyId":2,"Name":"Saskatchewan Party"},{"PartyId":3,"Name":"Liberal"},{"PartyId":4,"Name":"Green"},{"PartyId":5,"Name":"Independant"}],"Total":5,"AggregateResults":null,"Errors":null}
The dropdown does not show anything in there even though i cant see anything with the code or returned data.
Please try with the below code snippet. The method you have used its used for grid data binding.
public JsonResult GetParties()
{
List<PartyModel> models = new List<PartyModel>();
models.Add(new PartyModel() { Name = "Name1", PartyId = 1 });
models.Add(new PartyModel() { Name = "Name2", PartyId = 2 });
return Json(models, JsonRequestBehavior.AllowGet);
}
Let me know if any concern.

Resources