I have a long running PDF generation as part of a SAGA, so I read about the Turnout and I am trying to use it. However I have a complex consumer - several calls to DB, dependency on 3rd party service for generating the PDF and etc. I have troubles finding a good example how exactly I should use the Turnout endpoint.
My current usage is:
c.TurnoutEndpoint<PdfGenerationCommand>(host, "pdf-generation-TURNOUT", e =>
{
e.SuperviseInterval = TimeSpan.FromSeconds(30);
e.SetJobFactory(async context =>
{
var consumer = container.GetInstance<PdfGenerationCommandConsumer>();
var sw = new Stopwatch();
sw.Start();
Console.WriteLine($"{DateTime.Now} - [PdfGenerationCommand] Job started!");
await consumer.Consume(context.Command).ConfigureAwait(false);
sw.Stop();
Console.WriteLine($"{DateTime.Now} - [PdfGenerationCommand] Job finished in {sw.ElapsedMilliseconds}!");
});
});
However, the messages end up in the _skipped queue. I am doing something wrong and I cannot see it
Related
In MassTransit Send and RequestClient will be mapped to exchange or queue, That will be handled by LoadBalanced Consumer.
But for Publish Message, It will be consumed by all the instances that are running and waiting for the Message.
So, In StateMachine, Consumer has to publish the Events, That will make if more than once StateMachine Instance running it will be Picked by both StateMachine and Process will be duplicated? This is what happening at my work. So, We end up running Single StateMachine Instance.
await context.Publish(new
{
context.Message.OrderId,
context.Message.Timestamp,
context.Message.CustomerNumber,
context.Message.PaymentCardNumber,
context.Message.Notes
});
This publishes the events to Saga, if Saga is running in LoadBalancer. Both Instance will be receiving the SameEvent. And Start Processing the Event and changing the Next State.
If this is the Case, How to solve this. Only one StateMachine Should Pick the published message at once.
We end up running Single StateMachine Instance. So, the Published message wont be picked by both instance and will endup haivng duplicate process.
The Current Implmentation:
Have a REST Api - That receives the request to Start the Initial State.
var sendToUri =
new Uri(
$"rabbitMq://{_rabbitMqConfig.Host}/{_rabbitMqConfig.VirtualHost}-{_rabbitMqConfig.WfSagaQueue}");
var endPoint = await bus.GetSendEndpoint(sendToUri);
var req = wfRequest;
await endPoint.Send<IWfExecRequest>(req);
In the StateMachine :
services.AddMassTransit(x =>
{
x.AddConsumer<WfExecRequestConsumer>();
x.AddConsumer<WfTaskCompletedConsumer>();
x.UsingRabbitMq((context, cfg) =>
{
var wfTaskExecHandler = context.GetRequiredService<IWfTaskExecHandler>();
var wfManagementClient = context.GetRequiredService<IWfManagementClient>();
var wfSagaStateMachine = new MsrAutomationStateMachine(wfTaskExecHandler, wfManagementClient);
cfg.Host(HostCredets);
cfg.ReceiveEndpoint(queueName: "msr-automation-wf-exec-request", configureEndpoint: e =>
{
e.PrefetchCount = 1;
e.ConfigureConsumer<WfExecRequestConsumer>(context);
e.StateMachineSaga(wfSagaStateMachine, repo);
});
cfg.ReceiveEndpoint(queueName: "WfTaskCompleted", configureEndpoint: e =>
{
e.PrefetchCount = 1;
e.ConfigureConsumer<WfTaskCompletedConsumer>(context);
});
});
});
This StateMachine Receives , WfExecRequest (Inital Event), TaskCompleted and TaskFaulted (From Muliple Consumer Saga/Consumer) - This was done at Consumer Side as Context.Publish.
So, What I see if we Run more than one Instance of the same StateMachine the TaskCompled Message getting Consumed by both Instances.
Thanks Again.
First, clearly something is wrong with your configuration. If the saga state machine is running on a single queue (receive endpoint, regardless of how many instances of your service are running) it will automatically load balance on that single queue across all running instances.
If you are running multiple instances of the saga state machine on different queues, well yeah, you're basically doing it wrong.
Second, I'm not sure what "LoadBalancer" is but typically something that unnecessary when using a message broker. If "LoadBalancer" is something for your HTTP/API endpoints, that's fine, but the broker and the queue are the scale out points in a message-based system.
If you had posted actual code, or shared some explicit details that would help as it is now it's entirely based on supposition.
I'm running a MassTransit configuration with AmazonSQS. In my program I start by creating a receiveenpoint with the queue "input-queue1", I subscribe this SQS queue to an SNS topic named "topic1" and associate a consumer to this receiveendpoint that does some standard printing of the messages it receives. After starting the bus i want to subscribe the already created queue "input-queue1" to another SNS topic, named "topic2", but I couldn't find a way of doing this after starting the Bus (It's important to me that i can do this after the Bus is started). Is there a way of doing this and i'm just missing something, or is it not possible at all? (I tried with the commented portion of the code but it didn't work)
class Program
{
static async Task Main(string[] args)
{
var bus = Bus.Factory.CreateUsingAmazonSqs(x =>
{
x.Host(Constants.Region, h =>
{
h.AccessKey(Constants.AccesskeyId);
h.SecretKey(Constants.SecretAccessKey);
});
x.ReceiveEndpoint("input-queue1", e =>
{
e.Subscribe("topic1", callback => { });
e.Consumer(() => new Handler());
});
});
bus.StartAsync().Wait();
/*var handle = bus.ConnectReceiveEndpoint("input-queue1", e => {
e.Subscribe("topic2", callback => { });
});
var ready = await handle.Ready;*/
Console.WriteLine("Listening to messages...");
Console.WriteLine("Press enter to quit");
Console.ReadLine();
}
}
You can't change the topology of a receive endpoint once it has been created. This means that no new topic subscriptions can be created, and existing subscriptions cannot be removed.
If you need to change the configuration of the receive endpoint, you would need to do it yourself by using the SNS API to add the subscription yourself. I would question why you would want to do this though. If the consumer isn't able to consume the message forwarded to the queue, it would be moved to the skipped queue.
I've been trying to come up with a demo of a website that uses MassTransit with RabbitMQ to post messages to a service running on Service Fabric as a Stateful service.
Everything was going fine, my client would post a message:
IBusControl bus = BusConfigurator.ConfigureBus();
Uri sendToUri = new Uri($"{RabbitMqConstants.RabbitMqUri}" + $"{RabbitMqConstants.PeopleServiceQueue}");
ISendEndpoint endPoint = await bus.GetSendEndpoint(sendToUri);
await endPoint.Send<ICompanyRequest>(new {CompanyId = id });
My consumer in my service fabric service was defined like:
IBusControl busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
IRabbitMqHost host = cfg.Host(new Uri(RabbitMqConstants.RabbitMqUri), h =>
{
h.Username(RabbitMqConstants.UserName);
h.Password(RabbitMqConstants.Password);
});
cfg.ReceiveEndpoint(host, RabbitMqConstants.PeopleServiceQueue, e =>
{
e.Consumer<PersonInformationConsumer>();
});
});
busControl.Start();
This does allow me to consume the message in my class and I can process it fine. The problem comes when we want to use IReliableDictonary or IReliableQueue or anything that needs to reference the context that is run from the RunAsync function in the service fabric service.
So my question is, how can I configure (is it possible) MassTransit to work within a Stateful Service Fabric Service which knowledge of the service context itself?
Many thanks in advance.
Mike
Update
Ok, I've made some progress on this, if I point the register routines to my message consumer class (eg):
ServiceRuntime.RegisterServiceAsync("ServiceType", context => new PersonInformationConsumer(context)).GetAwaiter().GetResult();
ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(PersonInformationConsumer).Name);
Then in my consumer class for my messages I can do the following:
internal sealed class PersonInformationConsumer : StatefulService, IConsumer<ICompanyRequest>
{
private static StatefulServiceContext _currentContext;
#region Constructors
public PersonInformationConsumer(StatefulServiceContext serviceContext) : base(serviceContext)
{
_currentContext = serviceContext;
}
public PersonInformationConsumer() : base(_currentContext)
{
}
I can now successfully call the service message:
ServiceEventSource.Current.ServiceMessage(this.Context, "Message has been consumed, request Id: {0}", context.Message.CompanyId);
The problem I have now is trying to store something on the IReliableDictionary, doing this causes as "Object reference not set to an instance of an object" error :( ... any ideas would be appreciated (although may not read until new year now!)
public async Task Consume(ConsumeContext<ICompanyRequest> context)
{
ServiceEventSource.Current.ServiceMessage(this.Context, "Message has been consumed, request Id: {0}", context.Message.CompanyId);
using (ITransaction tx = StateManager.CreateTransaction())
{
try
{
var myDictionary = await StateManager.GetOrAddAsync<IReliableDictionary<string, long>>("myDictionary");
This is causing the error.... HELP! :)
You'll need to do a bit more to get MassTransit and stateful services working together, there's a few issues to concern yourself here.
Only the master within a stateful partition (n masters within n partitions) will be able to write/update to the stateful service, all replicas will throw exceptions when trying to write back any state. So you'll need to deal with this issue, on the surface it sounds easy until you take in to consideration the master can move around the cluster due to re-balancing the cluster, the default for general service fabric applications is to just turn off the processing on the replicas and only run the work on the master. This is all done by the RunAsync method (try it out, run 3 stateful services with something noddy in the RunAsync method, then terminate the master).
There is also partitioning of your data to consider, due to stateful services scale with partitions, you'll need to create a way to distributing data to separate endpoint on your service bus, maybe have a separate queue that only listens to a given partition range? Say you have a UserCreated message, you might split this on country UK goes to partition 1, US goes to partition 2 etc...
If you just want to get something basic up and running, I'd limit it to one partition and just try putting your bus creation within the the RunAsync and shutdown the bus once a cancelation is requested on the cancelation token.
protected override async Task RunAsync(CancellationToken cancellationToken)
{
var busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
IRabbitMqHost host = cfg.Host(new Uri(RabbitMqConstants.RabbitMqUri), h =>
{
h.Username(RabbitMqConstants.UserName);
h.Password(RabbitMqConstants.Password);
});
cfg.ReceiveEndpoint(host, RabbitMqConstants.PeopleServiceQueue, e =>
{
// Pass in the stateful service context
e.Consumer(c => new PersonInformationConsumer(Context));
});
});
busControl.Start();
while (true)
{
if(cancellationToken.CancellationRequested)
{
//Service Fabric wants us to stop
busControl.Stop();
cancellationToken.ThrowIfCancellationRequested();
}
await Task.Delay(TimeSpan.FromSeconds(1));
}
}
I have an application that I wrote in .NET that can monitor multiple RabbitMq queues using a single consumer.
For example:
using (IConnection connection = factory.CreateConnection())
{
using (IModel channel = connection.CreateModel())
{
var _consumer = new QueueingBasicConsumer(channel);
string[] list = new string[] { "Queue1", "Queue2", "Queue3", "Queue4" };
_consumer = new QueueingBasicConsumer(channel);
foreach (string currQueueName in list)
{
channel.QueueDeclare(currQueueName, false, false, false, null);
channel.BasicConsume(currQueueName, true, _consumer);
}
while (true)
{
var ea = (BasicDeliverEventArgs)_consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
ProcessMessage(message);
}
}
}
Basically, I just want to be able to distribute work across multiple queues, but only have a single application consuming them all (or multiple applications can be deployed and perform the same function).
I'm trying to spread work out across queues so that consumers are taking jobs equally across the queues.
Is this possible using Bunny, or the native Ruby driver?
I guess I just needed to spend a bit more time on this.
I'm adding a brief answer just in case this helps anyone else (or if someone wants to provide a cleaner solution :P)
client = Bunny.new
client.start
channel = client.create_channel
queues = %w(testing1 testing2 testing3)
queues.each do |name|
channel.queue(name).subscribe(timeout: 2) do |info, props, body|
puts "[#{name}] Got #{body}"
end
end
loop do
sleep 1
end
This effectively does what I was describing in the OP.
I'm looking to increase the performance of a high-throughput producer that I'm writing against ActiveMQ, and according to this useAsyncSend will:
Forces the use of Async Sends which adds a massive performance boost;
but means that the send() method will return immediately whether the
message has been sent or not which could lead to message loss.
However I can't see it making any difference to my simple test case.
Using this very basic application:
const string QueueName = "....";
const string Uri = "....";
static readonly Stopwatch TotalRuntime = new Stopwatch();
static void Main(string[] args)
{
TotalRuntime.Start();
SendMessage();
Console.ReadLine();
}
static void SendMessage()
{
var session = CreateSession();
var destination = session.GetQueue(QueueName);
var producer = session.CreateProducer(destination);
Console.WriteLine("Ready to send 700 messages");
Console.ReadLine();
var body = new byte[600*1024];
Parallel.For(0, 700, i => SendMessage(producer, i, body, session));
}
static void SendMessage(IMessageProducer producer, int i, byte[] body, ISession session)
{
var message = session.CreateBytesMessage(body);
var sw = new Stopwatch();
sw.Start();
producer.Send(message);
sw.Stop();
Console.WriteLine("Running for {0}ms: Sent message {1} blocked for {2}ms",
TotalRuntime.ElapsedMilliseconds,
i,
sw.ElapsedMilliseconds);
}
static ISession CreateSession()
{
var connectionFactory = new ConnectionFactory(Uri)
{
AsyncSend = true,
CopyMessageOnSend = false
};
var connection = connectionFactory.CreateConnection();
connection.Start();
var session = connection.CreateSession(AcknowledgementMode.AutoAcknowledge);
return session;
}
I get the following output:
Ready to send 700 messages
Running for 2430ms: Sent message 696 blocked for 12ms
Running for 4275ms: Sent message 348 blocked for 1858ms
Running for 5106ms: Sent message 609 blocked for 2689ms
Running for 5924ms: Sent message 1 blocked for 2535ms
Running for 6749ms: Sent message 88 blocked for 1860ms
Running for 7537ms: Sent message 610 blocked for 2429ms
Running for 8340ms: Sent message 175 blocked for 2451ms
Running for 9163ms: Sent message 89 blocked for 2413ms
.....
Which shows that each message takes about 800ms to send and the call to session.Send() blocks for about two and a half seconds. Even though the documentation says that
"send() method will return immediately"
Also these number are basically the same if I either change the parallel for to a normal for loop or change the AsyncSend = true to AlwaysSyncSend = true so I don't believe that the async switch is working at all...
Can anyone see what I'm missing here to make the send asynchronous?
After further testing:
According to ANTS performance profiler that vast majority of the runtime is being spent waiting for synchronization. It appears that the issue is that the various transport classes block internally through monitors. In particular I seem to get hung up on the MutexTransport's OneWay method which only allows one thread to access it at a time.
It looks as though the call to Send will block until the previous message has completed, this explains why my output shows that the first message blocked for 12ms, while the next took 1858ms. I can have multiple transports by implementing a connection-per-message pattern which improves matters and makes the message sends work in parallel, but greatly increases the time to send a single message, and uses up so many resources that it doesn't seem like the right solution.
I've retested all of this with 1.5.6 and haven't seen any difference.
As always the best thing to do is update to the latest version (1.5.6 at the time of this writing). A send can block if the broker has producer flow control enabled and you've reached a queue size limit although with async send this shouldn't happen unless you are sending with a producerWindowSize set. One good way to get help is to create a test case and submit it via a Jira issue to the NMS.ActiveMQ site so that we can look into it using your test code. There have been many fixes since 1.5.1 so I'd recommend giving that new version a try as it could already be a non-issue.