We are using Masstransit 8.0.2 with RabbitMQ(3.8.1 Erlang 22.1.5) and .Net6 . The message is being published from a TCPClient application. The message publishing time is increasing gradually and taking upto 30 minutes in publishing a single message. All the messages are being published in a background TCP receiver service. The data rate is fast as 60-70 messages are received at TCP client and being published to Rabbit using Masstransit asynchornously.
DataMessage consumer is deployed on 3 different servers with total 5 consumer having consumer utilization of 100% (RabbitMQ Portal).
All consumers have prefetch count of 100(we have tried with 1,5,16,50,100 and prefetch count having same results)
The message published and acknowledging rate on this queue is on avg 10/s.
(RabbitMQ server configuration 4 core cpu, 16 gb ram,Disk I/O 6400)
We have tried multiple configurations by playing with prefetch count and increasing and decreasing the consumers. We have also viewed the server cpu/memory/network utilization which all are under 50% on average
Bus is being starting in application Startup .
public static void AddServiceBus(this IServiceCollection services, IConfiguration configuration, int prefetchCount = 0, params Type[] consumers)
{
services.AddMassTransit(x =>
{
if (consumers != null && consumers.Any())
{
x.AddConsumers(consumers);
}
x.UsingRabbitMq((context, configurator) =>
{
var rabbitMqSettings = configuration.GetSection(nameof(RabbitMqConfiguration)).Get<RabbitMqConfiguration>();
configurator.Host(rabbitMqSettings.Host, d =>
{
d.Username(rabbitMqSettings.Username);
d.Password(rabbitMqSettings.Password);
});
configurator.ConfigureEndpoints(context);
configurator.UseRetry(b =>
{
b.Immediate(3);
});
if (prefetchCount > 0)
configurator.PrefetchCount = prefetchCount;
});
x.Configure<MassTransitHostOptions>(options =>
{
options.WaitUntilStarted = true;
options.StartTimeout = TimeSpan.FromSeconds(30);
options.StopTimeout = TimeSpan.FromMinutes(1);
});
});
}
private async Task<ErrorCode> PublishDataAsync(BaseData Data, string _messageGuid)
{
try
{
using var scope = _serviceProviderFactory.CreateScope();
var publishEndpoint = scope.ServiceProvider.GetRequiredService<IPublishEndpoint>();
var DataMessage = new DataMessage(Data);
await _publishEndpoint.Publish(DataMessage);
_logger.LogInformation("{messgeguid} Data Published to MassTransit", _messageGuid);
return ErrorCodes.SUCCESS;
}
catch (Exception e)
{
_logger.LogError(e, $"Message could not be published. {JsonConvert.SerializeObject(Data)}");
}
}
We are trying to increasing our message publishing rate. Our publishing message size is on avg 2kb.
There is a benchmark built into MassTransit for verifying the throughput of your message broker.
MassTransit is easily capable of over 10,000 messages a second on a decent RabbitMQ broker instance, even 5,000 using CloudAMQP from a desktop and home internet connection.
If you're experiencing latency issues, they're related to your network or broker, and not a limitation of MassTransit.
Your MassTransit version is also significantly out of date, but that won't affect the performance.
We have upgraded the server and it solved the issue of slow publish rate. Until anything further comes this is the resolution for us.
Any pointers to find relevance to RabbitMq performance with respect to available queues.
Related
When a subscriber of a topic throws an exception non-handled message lands in {subscribername}_error queue.
Given the example:
const string subsriberName = "AnotherSubscriber";
cfg.SubscriptionEndpoint<AnotherThingHappened>(host, subsriberName, configurator =>
{
configurator.Handler<AnotherThingHappened>(context =>
{
Console.WriteLine(context.Message.AnotherThingType);
if (Random.NextDouble() < 0.1)
{
throw new Exception("Oups, I failed :(");
}
return Task.CompletedTask;
});
});
It created "AnotherSubscriber" subscription on topic ObjectCreatedA. But when it fails the message goes to the queue anothersubscriber_error. It makes it harder to diagnose, monitor and replay messages. Because from ASB perspective this is just an ordinary queue.
How do I route failures to the DLQ of topic ObjectCratedA/AnotherSubscriber instead of **_error one?
Thanks in advance.
This is now possible as of MassTransit 6.2, see the related GitHub issue.
Your configuration will now need to look something like:
cfg.SubscriptionEndpoint(
"my-subscription",
"my-topic",
e =>
{
e.ConfigureConsumer<MyConsumer>(provider);
// Send failures to built-in Azure Service Bus Dead Letter queue
e.ConfigureDeadLetterQueueDeadLetterTransport();
e.ConfigureDeadLetterQueueErrorTransport();
});
I am trying to implement publish/subscribe architecture using Web API and Rabbit MQ message broker.
I have two projects in my solution: Publisher and Subscriber.
Publishing is implementing successfully but I cannot find place in my
subscriber project to read published message from the queue.
Both of my projects are .Net Core ASP WEB API
Thanks in advance
Register rabbitMq as HostedService using the AddSingleton method in ConfigureServices Method. IHostedService internally calls ApplicationGetStarted event. So the rabbit starts listening there
public void ConfigureServices(IServiceCollection services)
{
services.AddMassTransit(x =>
{
x.UsingRabbitMq();
});
// OPTIONAL, but can be used to configure the bus options
services.AddOptions<MassTransitHostOptions>()
.Configure(options =>
{
// if specified, waits until the bus is started before
// returning from IHostedService.StartAsync
// default is false
options.WaitUntilStarted = true;
// if specified, limits the wait time when starting the bus
options.StartTimeout = TimeSpan.FromSeconds(10);
// if specified, limits the wait time when stopping the bus
options.StopTimeout = TimeSpan.FromSeconds(30);
});
}
}
I've been trying to come up with a demo of a website that uses MassTransit with RabbitMQ to post messages to a service running on Service Fabric as a Stateful service.
Everything was going fine, my client would post a message:
IBusControl bus = BusConfigurator.ConfigureBus();
Uri sendToUri = new Uri($"{RabbitMqConstants.RabbitMqUri}" + $"{RabbitMqConstants.PeopleServiceQueue}");
ISendEndpoint endPoint = await bus.GetSendEndpoint(sendToUri);
await endPoint.Send<ICompanyRequest>(new {CompanyId = id });
My consumer in my service fabric service was defined like:
IBusControl busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
IRabbitMqHost host = cfg.Host(new Uri(RabbitMqConstants.RabbitMqUri), h =>
{
h.Username(RabbitMqConstants.UserName);
h.Password(RabbitMqConstants.Password);
});
cfg.ReceiveEndpoint(host, RabbitMqConstants.PeopleServiceQueue, e =>
{
e.Consumer<PersonInformationConsumer>();
});
});
busControl.Start();
This does allow me to consume the message in my class and I can process it fine. The problem comes when we want to use IReliableDictonary or IReliableQueue or anything that needs to reference the context that is run from the RunAsync function in the service fabric service.
So my question is, how can I configure (is it possible) MassTransit to work within a Stateful Service Fabric Service which knowledge of the service context itself?
Many thanks in advance.
Mike
Update
Ok, I've made some progress on this, if I point the register routines to my message consumer class (eg):
ServiceRuntime.RegisterServiceAsync("ServiceType", context => new PersonInformationConsumer(context)).GetAwaiter().GetResult();
ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(PersonInformationConsumer).Name);
Then in my consumer class for my messages I can do the following:
internal sealed class PersonInformationConsumer : StatefulService, IConsumer<ICompanyRequest>
{
private static StatefulServiceContext _currentContext;
#region Constructors
public PersonInformationConsumer(StatefulServiceContext serviceContext) : base(serviceContext)
{
_currentContext = serviceContext;
}
public PersonInformationConsumer() : base(_currentContext)
{
}
I can now successfully call the service message:
ServiceEventSource.Current.ServiceMessage(this.Context, "Message has been consumed, request Id: {0}", context.Message.CompanyId);
The problem I have now is trying to store something on the IReliableDictionary, doing this causes as "Object reference not set to an instance of an object" error :( ... any ideas would be appreciated (although may not read until new year now!)
public async Task Consume(ConsumeContext<ICompanyRequest> context)
{
ServiceEventSource.Current.ServiceMessage(this.Context, "Message has been consumed, request Id: {0}", context.Message.CompanyId);
using (ITransaction tx = StateManager.CreateTransaction())
{
try
{
var myDictionary = await StateManager.GetOrAddAsync<IReliableDictionary<string, long>>("myDictionary");
This is causing the error.... HELP! :)
You'll need to do a bit more to get MassTransit and stateful services working together, there's a few issues to concern yourself here.
Only the master within a stateful partition (n masters within n partitions) will be able to write/update to the stateful service, all replicas will throw exceptions when trying to write back any state. So you'll need to deal with this issue, on the surface it sounds easy until you take in to consideration the master can move around the cluster due to re-balancing the cluster, the default for general service fabric applications is to just turn off the processing on the replicas and only run the work on the master. This is all done by the RunAsync method (try it out, run 3 stateful services with something noddy in the RunAsync method, then terminate the master).
There is also partitioning of your data to consider, due to stateful services scale with partitions, you'll need to create a way to distributing data to separate endpoint on your service bus, maybe have a separate queue that only listens to a given partition range? Say you have a UserCreated message, you might split this on country UK goes to partition 1, US goes to partition 2 etc...
If you just want to get something basic up and running, I'd limit it to one partition and just try putting your bus creation within the the RunAsync and shutdown the bus once a cancelation is requested on the cancelation token.
protected override async Task RunAsync(CancellationToken cancellationToken)
{
var busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
IRabbitMqHost host = cfg.Host(new Uri(RabbitMqConstants.RabbitMqUri), h =>
{
h.Username(RabbitMqConstants.UserName);
h.Password(RabbitMqConstants.Password);
});
cfg.ReceiveEndpoint(host, RabbitMqConstants.PeopleServiceQueue, e =>
{
// Pass in the stateful service context
e.Consumer(c => new PersonInformationConsumer(Context));
});
});
busControl.Start();
while (true)
{
if(cancellationToken.CancellationRequested)
{
//Service Fabric wants us to stop
busControl.Stop();
cancellationToken.ThrowIfCancellationRequested();
}
await Task.Delay(TimeSpan.FromSeconds(1));
}
}
I have some Java code that reads messages from an ActiveMQ queue. The code uses a JmsTemplate from Spring and I use the "browseSelected" method to retrieve any messages from the queue that have a timestamp in their header older than 7 days (by creating the appropriate criteria as part of the messageSelector parameter).
myJmsTemplate.browseSelected(myQueue, myCriteria, new BrowserCallback<Integer>() {
#Override
public Integer doInJms(Session s, QueueBrowser qb) throws JMSException {
#SuppressWarnings("unchecked")
final Enumeration<Message> e = qb.getEnumeration();
int count = 0;
while (e.hasMoreElements()) {
final Message m = e.nextElement();
final TextMessage tm = (TextMessage) MyClass.this.jmsQueueTemplate.receiveSelected(
MyClass.this.myQueue, "JMSMessageID = '" + m.getJMSMessageID() + "'");
myMessages.add(tm);
count++;
}
return count;
}
});
The BrowserCallback's "doInJms" method adds the messages which match the criteria to a list ("myMessages") which subsequently get processed further.
The issue is that I'm finding the code will only process 400 messages each time it runs, even though there are several thousand messages which match the criteria specified.
When I previously used another queueing technology with this code (IBM MQ), it would process all records which met the criteria.
I'm wondering whether I'm experiencing an issue with ActiveMQ's prefetch limit: http://activemq.apache.org/what-is-the-prefetch-limit-for.html
Versions: ActiveMQ 5.10.1 and Spring 3.2.2.
Thanks in advance for any assistance.
The broker will only return up to 400 message by default as configured by the maxBrowsePageSize option in the destination policies. You can increase that value but must use caution as the messages are paged into memory and as such can lead you into an OOM situation.
You must always remember that a message broker is not a database, using it as one will generally end in tears.
I'm looking to increase the performance of a high-throughput producer that I'm writing against ActiveMQ, and according to this useAsyncSend will:
Forces the use of Async Sends which adds a massive performance boost;
but means that the send() method will return immediately whether the
message has been sent or not which could lead to message loss.
However I can't see it making any difference to my simple test case.
Using this very basic application:
const string QueueName = "....";
const string Uri = "....";
static readonly Stopwatch TotalRuntime = new Stopwatch();
static void Main(string[] args)
{
TotalRuntime.Start();
SendMessage();
Console.ReadLine();
}
static void SendMessage()
{
var session = CreateSession();
var destination = session.GetQueue(QueueName);
var producer = session.CreateProducer(destination);
Console.WriteLine("Ready to send 700 messages");
Console.ReadLine();
var body = new byte[600*1024];
Parallel.For(0, 700, i => SendMessage(producer, i, body, session));
}
static void SendMessage(IMessageProducer producer, int i, byte[] body, ISession session)
{
var message = session.CreateBytesMessage(body);
var sw = new Stopwatch();
sw.Start();
producer.Send(message);
sw.Stop();
Console.WriteLine("Running for {0}ms: Sent message {1} blocked for {2}ms",
TotalRuntime.ElapsedMilliseconds,
i,
sw.ElapsedMilliseconds);
}
static ISession CreateSession()
{
var connectionFactory = new ConnectionFactory(Uri)
{
AsyncSend = true,
CopyMessageOnSend = false
};
var connection = connectionFactory.CreateConnection();
connection.Start();
var session = connection.CreateSession(AcknowledgementMode.AutoAcknowledge);
return session;
}
I get the following output:
Ready to send 700 messages
Running for 2430ms: Sent message 696 blocked for 12ms
Running for 4275ms: Sent message 348 blocked for 1858ms
Running for 5106ms: Sent message 609 blocked for 2689ms
Running for 5924ms: Sent message 1 blocked for 2535ms
Running for 6749ms: Sent message 88 blocked for 1860ms
Running for 7537ms: Sent message 610 blocked for 2429ms
Running for 8340ms: Sent message 175 blocked for 2451ms
Running for 9163ms: Sent message 89 blocked for 2413ms
.....
Which shows that each message takes about 800ms to send and the call to session.Send() blocks for about two and a half seconds. Even though the documentation says that
"send() method will return immediately"
Also these number are basically the same if I either change the parallel for to a normal for loop or change the AsyncSend = true to AlwaysSyncSend = true so I don't believe that the async switch is working at all...
Can anyone see what I'm missing here to make the send asynchronous?
After further testing:
According to ANTS performance profiler that vast majority of the runtime is being spent waiting for synchronization. It appears that the issue is that the various transport classes block internally through monitors. In particular I seem to get hung up on the MutexTransport's OneWay method which only allows one thread to access it at a time.
It looks as though the call to Send will block until the previous message has completed, this explains why my output shows that the first message blocked for 12ms, while the next took 1858ms. I can have multiple transports by implementing a connection-per-message pattern which improves matters and makes the message sends work in parallel, but greatly increases the time to send a single message, and uses up so many resources that it doesn't seem like the right solution.
I've retested all of this with 1.5.6 and haven't seen any difference.
As always the best thing to do is update to the latest version (1.5.6 at the time of this writing). A send can block if the broker has producer flow control enabled and you've reached a queue size limit although with async send this shouldn't happen unless you are sending with a producerWindowSize set. One good way to get help is to create a test case and submit it via a Jira issue to the NMS.ActiveMQ site so that we can look into it using your test code. There have been many fixes since 1.5.1 so I'd recommend giving that new version a try as it could already be a non-issue.