How to connect an endpoint(exchange-exchange-queue) in masstransit, accumulate data in the queue, and then, after some time, connect a consumer to this endpoint?
I wanted to do something like:
Task.Run(async () =>
{
for (var i = 0;; i++)
{
await _bus.Publish(new Event(i), stoppingToken);
await Task.Delay(1_000, stoppingToken);
}
});
// a command comes to connect the consumer
var endpoint = _bus.ConnectReceiveEndpoint();
await endpoint.Ready;
// I prepare the consumer, as soon as it is ready, I connect it,
then I read the data that has accumulated during the preparation
(my consumer needs to load the state before reading the data,
and also cannot skip the data during its preparation)
endpoint.ReceiveEndpoint.ConnectConsumer(() =>
_serviceProvider.GetRequiredService<EventConsumer>());
but this code will not create an exchange-exchange relationship, so the queue will be empty
If the exchange bindings (wired to the receive endpoint, which ultimately is a queue) do not exist when messages are published, they are discarded by RabbitMQ.
You would need to connect the receive endpoint in advance, so that the messages end up in the queue.
Related
In MassTransit Send and RequestClient will be mapped to exchange or queue, That will be handled by LoadBalanced Consumer.
But for Publish Message, It will be consumed by all the instances that are running and waiting for the Message.
So, In StateMachine, Consumer has to publish the Events, That will make if more than once StateMachine Instance running it will be Picked by both StateMachine and Process will be duplicated? This is what happening at my work. So, We end up running Single StateMachine Instance.
await context.Publish(new
{
context.Message.OrderId,
context.Message.Timestamp,
context.Message.CustomerNumber,
context.Message.PaymentCardNumber,
context.Message.Notes
});
This publishes the events to Saga, if Saga is running in LoadBalancer. Both Instance will be receiving the SameEvent. And Start Processing the Event and changing the Next State.
If this is the Case, How to solve this. Only one StateMachine Should Pick the published message at once.
We end up running Single StateMachine Instance. So, the Published message wont be picked by both instance and will endup haivng duplicate process.
The Current Implmentation:
Have a REST Api - That receives the request to Start the Initial State.
var sendToUri =
new Uri(
$"rabbitMq://{_rabbitMqConfig.Host}/{_rabbitMqConfig.VirtualHost}-{_rabbitMqConfig.WfSagaQueue}");
var endPoint = await bus.GetSendEndpoint(sendToUri);
var req = wfRequest;
await endPoint.Send<IWfExecRequest>(req);
In the StateMachine :
services.AddMassTransit(x =>
{
x.AddConsumer<WfExecRequestConsumer>();
x.AddConsumer<WfTaskCompletedConsumer>();
x.UsingRabbitMq((context, cfg) =>
{
var wfTaskExecHandler = context.GetRequiredService<IWfTaskExecHandler>();
var wfManagementClient = context.GetRequiredService<IWfManagementClient>();
var wfSagaStateMachine = new MsrAutomationStateMachine(wfTaskExecHandler, wfManagementClient);
cfg.Host(HostCredets);
cfg.ReceiveEndpoint(queueName: "msr-automation-wf-exec-request", configureEndpoint: e =>
{
e.PrefetchCount = 1;
e.ConfigureConsumer<WfExecRequestConsumer>(context);
e.StateMachineSaga(wfSagaStateMachine, repo);
});
cfg.ReceiveEndpoint(queueName: "WfTaskCompleted", configureEndpoint: e =>
{
e.PrefetchCount = 1;
e.ConfigureConsumer<WfTaskCompletedConsumer>(context);
});
});
});
This StateMachine Receives , WfExecRequest (Inital Event), TaskCompleted and TaskFaulted (From Muliple Consumer Saga/Consumer) - This was done at Consumer Side as Context.Publish.
So, What I see if we Run more than one Instance of the same StateMachine the TaskCompled Message getting Consumed by both Instances.
Thanks Again.
First, clearly something is wrong with your configuration. If the saga state machine is running on a single queue (receive endpoint, regardless of how many instances of your service are running) it will automatically load balance on that single queue across all running instances.
If you are running multiple instances of the saga state machine on different queues, well yeah, you're basically doing it wrong.
Second, I'm not sure what "LoadBalancer" is but typically something that unnecessary when using a message broker. If "LoadBalancer" is something for your HTTP/API endpoints, that's fine, but the broker and the queue are the scale out points in a message-based system.
If you had posted actual code, or shared some explicit details that would help as it is now it's entirely based on supposition.
I'm running a MassTransit configuration with AmazonSQS. In my program I start by creating a receiveenpoint with the queue "input-queue1", I subscribe this SQS queue to an SNS topic named "topic1" and associate a consumer to this receiveendpoint that does some standard printing of the messages it receives. After starting the bus i want to subscribe the already created queue "input-queue1" to another SNS topic, named "topic2", but I couldn't find a way of doing this after starting the Bus (It's important to me that i can do this after the Bus is started). Is there a way of doing this and i'm just missing something, or is it not possible at all? (I tried with the commented portion of the code but it didn't work)
class Program
{
static async Task Main(string[] args)
{
var bus = Bus.Factory.CreateUsingAmazonSqs(x =>
{
x.Host(Constants.Region, h =>
{
h.AccessKey(Constants.AccesskeyId);
h.SecretKey(Constants.SecretAccessKey);
});
x.ReceiveEndpoint("input-queue1", e =>
{
e.Subscribe("topic1", callback => { });
e.Consumer(() => new Handler());
});
});
bus.StartAsync().Wait();
/*var handle = bus.ConnectReceiveEndpoint("input-queue1", e => {
e.Subscribe("topic2", callback => { });
});
var ready = await handle.Ready;*/
Console.WriteLine("Listening to messages...");
Console.WriteLine("Press enter to quit");
Console.ReadLine();
}
}
You can't change the topology of a receive endpoint once it has been created. This means that no new topic subscriptions can be created, and existing subscriptions cannot be removed.
If you need to change the configuration of the receive endpoint, you would need to do it yourself by using the SNS API to add the subscription yourself. I would question why you would want to do this though. If the consumer isn't able to consume the message forwarded to the queue, it would be moved to the skipped queue.
When a subscriber of a topic throws an exception non-handled message lands in {subscribername}_error queue.
Given the example:
const string subsriberName = "AnotherSubscriber";
cfg.SubscriptionEndpoint<AnotherThingHappened>(host, subsriberName, configurator =>
{
configurator.Handler<AnotherThingHappened>(context =>
{
Console.WriteLine(context.Message.AnotherThingType);
if (Random.NextDouble() < 0.1)
{
throw new Exception("Oups, I failed :(");
}
return Task.CompletedTask;
});
});
It created "AnotherSubscriber" subscription on topic ObjectCreatedA. But when it fails the message goes to the queue anothersubscriber_error. It makes it harder to diagnose, monitor and replay messages. Because from ASB perspective this is just an ordinary queue.
How do I route failures to the DLQ of topic ObjectCratedA/AnotherSubscriber instead of **_error one?
Thanks in advance.
This is now possible as of MassTransit 6.2, see the related GitHub issue.
Your configuration will now need to look something like:
cfg.SubscriptionEndpoint(
"my-subscription",
"my-topic",
e =>
{
e.ConfigureConsumer<MyConsumer>(provider);
// Send failures to built-in Azure Service Bus Dead Letter queue
e.ConfigureDeadLetterQueueDeadLetterTransport();
e.ConfigureDeadLetterQueueErrorTransport();
});
I am working with Spring WebSocket and Stomp.
Note: the broker is ActiveMQ
I have two #Schedule methods that sends messages to Queue and Topic respectively
For a subscription for a Queue, the code is as follows:
$('#ws_connect').click(function(){
console.log('Connect clicked');
var socket = new SockJS('/project-app/ws/notification');
stompClient = Stomp.over(socket);
stompClient.connect({}, function (frame) {
console.log('Connected: ' + frame);
$('#notification').append('<p>Connected</p>');
stompClient.subscribe('/queue/somedestination', function (notification) {
... append the content to the html page, it works
});
});
});
If there is no user the Queue is getting message by message. Once the user arrives and do the connection, automatically it gets all the messages enqueued and the Queue pass to be empty. If a new message arrives it appears automatically, it because the user remains connected yet. If the user disconnects and later do the connection, it can see again all the messages enqueued. Until here all is Ok
As follows for a subscription for a Topic
$('#ws_connect').click(function(){
console.log('Connect clicked');
var socket = new SockJS('/project-app/ws/notification');
stompClient = Stomp.over(socket);
stompClient.connect({}, function (frame) {
console.log('Connected: ' + frame);
$('#notification').append('<p>Connected</p>');
stompClient.subscribe('/topic/somedestination', function (notification) {
... append the content to the html page, it works
});
});
});
Practically the js code is the same than the Queue version. Just the destination is different.
Here the problem is that if exists messages enqueued in the Topic and if the user does the connection the messages do not appear automatically. Of course meanwhile the user remains connected he can see each new message, same case if multiple users are connected to the same Topic all can see the same new message.
But again for the first user to connect to the Topic destination. He is not able to get the messages enqueued from that Topic when he does the connection.
Is it the normal behaviour?
How can be retrieved the messages enqueued?
Perhaps a suggested approach to handle this scenario?
In some way the "worst" scenario would be create multiple Queues for each potential user. But just curious if through Topic is possible get this requeriment
The Topic destination does not store messages when there are no subscribers so the code is working as expected. The only time a Topic would retain messages for subscribers is if the subscriber had created a durable topic subscription and then gone offline at which point the broker would store any message sent to the Topic with the persistent flag enabled. The caveat here being that any message sent to that Topic before the subscription is made would be dropped.
See the documentation for more help.
I have a saga setup in a fork/join configuration.
Events defined on the saga
FileMetadataMsg
FileReadyMsg
SomeOtherMsg
Process starts off when a file comes in on a separate listener.
Publishes SagaStart(correlationId)
Publishes FileSavedToMsg(correlationId, fileLoc)
Publishes FileMetadataMsg(correlationId, metadata)
Publishes FileReadyMsg(correlationId, fileLoc)
Downstream endpoint of does some work on the file
Consumer<FileSavedToMsg>
Publishes SomeOtherMsg(GotTheFileMsg.correlationId, data)
I am getting a FileSavedToMsg in the saga_skipped queue. I can only assume it's due to having a correlationId on the FileSavedToMsg because the saga itself is not using FileSavedToMsg in its state machine and does not have an Event<FileSavedToMsg>.
If this is the reason why...should I be passing the correlationId along in a field other than the CorrelationId, so the saga doesn't see it? I need it somewhere so I can tag SomeOtherMsg with it.
Here is how the saga endpoint is defined
return Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
cfg.ReceiveEndpoint(host, "study_saga", epCfg =>
{
epCfg.StateMachineSaga(machine, repository);
});
});
Here is how the worker endpoint is defined
return Bus.Factory.CreateUsingRabbitMq(x =>
{
var host = x.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
x.ReceiveEndpoint(host, "study_3d_volume_worker", c =>
{
c.PrefetchCount = 1;
c.Instance(_studyCreatedMsgConsumer);
});
});
These are running on the same machine, but in seperate Console/Topshelf applications.
If you are getting messages on a queue that are not consumed by a consumer on that receive endpoint, it might be that you either previously were consuming that message type and removed it from the consumer (or saga, in your case) or you were using the queue from some other purpose and it consumed that message type.
Either way, if you go into the RabbitMQ management console and look for the queue, you can expand the Bindings chevron, click to go to the exchange of the same name (that's a standard MassTransit convention), and then expand the bindings of the exchange to see which message types (the exchanges named like .NET type names) are bound to that exchange.
If you see one that is not consumed by the endpoint, that's the culprit. You can Unbind it using the UI, after which messages published will no longer be sent to the queue.