I want to design a messaging system with MassTransit according to below image. When I used Masstransit and ReceiveEndpoint configuration, an exchange and a queue created per each action, which increases the number of exchange and queue. And with the increase of processes, it causes confusion for me. Is there a solution to solve this problem and design a messaging system according to below image?
I expect to be able to design as shown above
MassTransit supports the configuration of DIRECT exchanges, using routing keys to route message beyond the default type-based routing. There is a sample that gives some guidance, and likely a few questions and answers here as well as GitHub discussions.
Related
all new to Masstransit and are currently evaluating it for a larger project and
wondering if anyone could help get a better understanding of the following challenges:
"Single consumer of events in a loadbalanced environment"
In production our services will be runinng multiple instances for scalability and failvover and be
part of a larger ecosystem of microservices. The overall architecture is based on Microsofts eshoponcontainers
reference implementation where different microservices are communicating with each other via "integrationevents".
When publishing a IntegrationEvent to other services which I assume should be done as described in Masstransit Producers / Publish
, https://masstransit-project.com/usage/producers.html#publish, how can I assure that only ONE instance in a specific microservice are processing
the event but of course that the event reaches ALL microsystem that depends on the event? When we have done similar solutions based on Azure Functions
this requirement has been solved by using the "Singelton" attribute (https://learn.microsoft.com/en-us/azure/app-service/webjobs-sdk-how-to#singleton-attribute).
Azure Service Bus
Reading the documentation my impression is that Masstransit is very RabbitMQ centric.
Since we will be on the Azure Service bus when moving to production is there any limitations or features
not available on that "transport"?
Regards Niclas
For the first question, it's a normal publish-subscribe with competing consumers. It works like this out of the box, there's nothing that needs to be done to achieve this.
When running multiple instances of the same service
Use the same queue name for each instance
Messages from the queue will be load balanced across all instances (the competing consumer pattern)
It's from the RMQ Guidances, but it's like this for all transports.
Concerning the Azure Service Bus transport, it works as expected and has a lot of production users. It's properly documented as well.
I'd say for both of your questions the answer is "it just works".
We are designing a reporting system using microservice architecture. All the services are supposed to be subscribers to the event bus and they communicate by raising events. We also decided to expose each of our services using REST api. Now the question is , is it a good idea to create our services as web api [RESTful] applications which are also subscribers to the event bus? so basically there are 2 ponits of entry to each service - api and events. I have a feeling that we should separate out these 2 as these are 2 different concerns. Any ideas?
Since Microservices architecture are Un-opinionated software design. So you may get different answers on this questions.
Yes, REST and Event based are two different things but sometime both combined gives design to achieve better flexibility.
Answering to your concerns, I don't see any harm if REST APIs also subscribe to a queue as long as you can maintain both of them i.e changes to message does not have any impact of APIs and you have proper fallback and Eventual consistency mechanism in place. you can check discussion . There are already few project which tried it such as nakadi and ponte.
So It all depends on your service's communication behaviour to choose between REST APIs and Event-Based design Or Both.
What you do is based on your requirement you can choose REST APIs where you see synchronous behaviour between services
and go with Event based design where you find services needs asynchronous behaviour, there is no harm combining both also.
Ideally for inter-process communication protocol it is better to go with messaging and for client-service REST APIs are best fitted.
Check the Communication style in microservices.io
REST based Architecture
Advantage
Request/Response is easy and best fitted when you need synchronous environments.
Simpler system since there in no intermediate broker
Promotes orchestration i.e Service can take action based on response of other service.
Drawback
Services needs to discover locations of service instances.
One to one Mapping between services.
Rest used HTTP which is general purpose protocol built on top of TCP/IP which adds enormous amount of overhead when using it to pass messages.
Event Driven Architecture
Advantage
Event-driven architectures are appealing to API developers because they function very well in asynchronous environments.
Loose coupling since it decouples services as on a event of once service multiple services can take action based on application requirement. it is easy to plug-in any new consumer to producer.
Improved availability since the message broker buffers messages until the consumer is able to process them.
Drawback
Additional complexity of message broker, which must be highly available
Debugging an event request is not that easy.
We give support to an organization that uses IBM MQ (v9.0) to communicate to an external partner. It's a simple layout, a single local queue manager connected to an external one trough one single channel. Various applications use different queues to communicate trough this QM.
Last week, after solving a support ticket, my boss came to the conclusion that having one single channel to funnel all the messages from the different queues (and applications that use such queues) is not good.
I have to write a report detailing what criteria should we use to separate the message flows into different channels, and what benefits and inconveniences would this entail.
I'm fairly new to IBM MQ, and I'm having trouble finding information on this topic. So far I've read the official documentation and other bibliography and i have a superficial idea about it, but i cant find concrete information about this topic.
Is separating message flows into different channels a good practice? What would be a good criteria to use?
Here is a good write up on channels and it discusses reasons for parallel channels. You may be able to use this in your report. MQ Channels
If you have applications with large messages that are of the fire-and-forget pattern it is common to put these on a different path than the real time apps that are of the request-response pattern.
Whether or not this configuration change will lessen the chance of problem tickets depends largely on what led to the issue. Good luck.
I have an architectural problem.
I know what represents a (fanout, direct, topic, headers) exchange, bindings, queues and almost everything about message architectures. I have the following problem and I need some pieces of advice.
I would like to implement notification logic to my application, where each user will receive in real-time notification only intended for him. ( Actually, I don't want mention what are my UI and BE languages/frameworks, because of an additional level of abstraction) The UI will make a connection to RabbitMQ with WebSocket, SockJS and STOMP. My UI will be only a consumer, the BE is the writer - that one, which will add some Messages to RabbitMQ.
It's perfectly clear to me if I have a direct Exchange with routing key, which uniquely to identify the specific user (for example: my-routing-to-empoyee-with-id-1) and N-number of Queues for each user. This is too heavy to me (I don't know actually whether is normal situation to have so much queues).
Is there any solution, where I can use only one Queue and the message to be delivered only to user for who is intended ?
I know a solution, where I can have a topic exchange and to have one writer and many subscribers, but on this way, I can filter the message only on clients level, which is not so secure. :(
Actually, I found a very interesting article, which describes me a problem that I have pretty well.
What I want is called Selective Consumers and for Enterprise Integration Patterns this is an anti-pattern and we should not use it.
For more details, all who want can read this article: https://derickbailey.com/2015/07/22/airport-baggage-claims-selective-consumers-and-rabbitmq-anti-patterns/
I was browsing for an open source messaging software and after some good bit of research I came across these three products. I've taken these out for a preliminary test drive, having had them handle messages for queues and topics, and from what I've read all three of these products are good picks for an Open Source messaging solution for most companies. What I was wondering was what are the advantages that these products may have over one another? What I'm particularly interested in is messaging throughput, including persistent messaging throughput, security, scalability, reliability, support, routing capabilities, administrative options such as metrics and monitoring, and generally just how well each program runs in a large business environment.
Check out http://queues.io/
From their site:
The goal is to create a quality list of queues with a collection of articles, blog posts, slides, and videos about them. After reading the linked articles, you should have a good idea about: the pros and cons of each queue, a basic understanding of how the queue works, and what each queue is trying to achieve. Basically, you should have all the information you need to decide which queue will best fit your needs.
'messaging' covers a lot of options - and there must be at least a dozen different types of technologies that could be the right answer - having built many production messaging environments, using a variety of technologies/approaches, having a better understanding your requirements would help.
are you needing subject-based subscriptions? do you need multicast delivery? do you need dynamic subscribers/listeners? would your listeners be requerying for best sources even after finding an acceptable publisher/feed?
do you need guaranteed delivery? delivery confirmation? is you publisher storing any undelivered messages, or do you need the messaging system to do that for you automagically? how often does your feed data go stale - e.g. email-ish alerts can be store-and-forward but real-time pricing data is only valid for a short interval (and then probably needs to go away rather than cause confusion)
how volatile is your network topology? are your subscribers (or publishers) expecting to live at a fixed address? or are they mobile devices? could they appear to you over more complex internetwork topologies requiring registration and possibly imposing routing restrictions? if so any idea the frequency of these topology changes?
do you only need a java interface? are any of your subscribers to be integrated into windows components (like feeds into excel)?
if you're only interested in experience comparing the similar products you named then perhaps you have already thought through these topics.
as to products, in my experience Tibco is still the leader in throughput and scalability, especially in a real-time environment. ibm MQ would be next, especially in a store-and-forward architecture. with both of those products you get a level of support on which you can justify betting a fundamental part of your business systems. there's a reason both of those have been around for a couple of decades.
another often overlooked option is Tuxedo - it provides not only messaging but a proven transactional capability that remains unparalleled. Oracle continue to be committed to this product and, again, the level of support available is second to none.
i love open sourced solutions and am always glad to find production quality software for free - but if you are creating a fundamental part of your business infrastructure then an active community still might not indicate whether a particular voluntary project is the best bet.
my 2c worth. hope it helps.
First, I am no expert in this, but maybe I can give you some thought hints.
ActiveMQ and Qpid are both under the Apache umbrella and are message queues. But Qpid is an implementation of the AMQP specification.
AMQP is a protocol specification, on the wire level, so messages can be exchanged with other AMQP message queues (e.g RabbitMQ).
ActiveMQ and HornetQ are queues that you can use with a JMS API. The Java Message Service is a specification on an API level.
But you have the option to access Qpid via a JMS API, too.
I think performance is a secondary thought. To have an active community is more important.
http://x-aeon.com/wp/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/
Benchmark includes some performance numbers for you to decide, with both persistent and transient results.