We have few micro services created in .net core 1.0, we are following CQRS pattern and we are also using swagger which list all api's, we have a requirement were in we need to implement Message Bus(not decided yet might be AWS), this message bus will orchestrate UI operations, that spans across multiple backend services, so I don't know how to start because I 'm new to this, I need to understand message bus, queues, publishing events to message bus so could you please help me to understand?
Also any pointers to tutorial videos and so on with explanation would be helpful.
You would read this topic from Maira Wenzel at Microsoft. It helped me a lot on my beginning days.
An event bus allows publish/subscribe-style communication between microservices without requiring the components to explicitly be aware of each other
In a microservice architecture, Services should not communicate with each other in a synchronous way, instead, all services need to communicate via Message-Brocker. There are lots of messages brokers like RabbitMq, Kafka and so on which you can use them.
Using a message broker directly needs to manage all of the details related to message brokers' behaviors and need lots of code to do that. So here Service-Bus comes to the story.
The service bus is an intermediate between each service and message broker to encapsulate all details and configurations in the Message broker. So microservice blocks (each bounded context in DDD) could communicate with each other simply and in a high-level manner.
The point is in the service bus you just get rid of message broker complexities. So you don't need to worry about low-level concepts like the type of exchanges, queues, and so on in message broker.
I just have tried to give you high-level information, But in fact, Message Bus could do lots of other things.
Related
all new to Masstransit and are currently evaluating it for a larger project and
wondering if anyone could help get a better understanding of the following challenges:
"Single consumer of events in a loadbalanced environment"
In production our services will be runinng multiple instances for scalability and failvover and be
part of a larger ecosystem of microservices. The overall architecture is based on Microsofts eshoponcontainers
reference implementation where different microservices are communicating with each other via "integrationevents".
When publishing a IntegrationEvent to other services which I assume should be done as described in Masstransit Producers / Publish
, https://masstransit-project.com/usage/producers.html#publish, how can I assure that only ONE instance in a specific microservice are processing
the event but of course that the event reaches ALL microsystem that depends on the event? When we have done similar solutions based on Azure Functions
this requirement has been solved by using the "Singelton" attribute (https://learn.microsoft.com/en-us/azure/app-service/webjobs-sdk-how-to#singleton-attribute).
Azure Service Bus
Reading the documentation my impression is that Masstransit is very RabbitMQ centric.
Since we will be on the Azure Service bus when moving to production is there any limitations or features
not available on that "transport"?
Regards Niclas
For the first question, it's a normal publish-subscribe with competing consumers. It works like this out of the box, there's nothing that needs to be done to achieve this.
When running multiple instances of the same service
Use the same queue name for each instance
Messages from the queue will be load balanced across all instances (the competing consumer pattern)
It's from the RMQ Guidances, but it's like this for all transports.
Concerning the Azure Service Bus transport, it works as expected and has a lot of production users. It's properly documented as well.
I'd say for both of your questions the answer is "it just works".
What I have?
A lot of different microservices managing by different teams. All microservices persist data in Aerospike database.
What I want to achieve?
I'm building new microservice that relies on data handled by another services. I want to listen the changes in entities, but unfortunately that microservices don't put anything in message queue, they have only usual REST APIs, so I cant just subscribe to events.
The idea is to listen a transaction log(event log/commit log/WAL) of database. This approach is also using in different Event Sourcing systems, but I cant found any Aerospike API that would stream this log. So the question - does Aerospike provide any similar functionality, may be with different name?
Aerospike, in its enterprise edition, has a feature called change notification framework which may fit your requirements. It informs an external agent about all the write operations. This is built over the XDR functionality which is meant for replicating across data centers using a digestlog.
If you are not planning for enterprise, you should reconsider having your own message queue in front of Aerospike.
We are designing a reporting system using microservice architecture. All the services are supposed to be subscribers to the event bus and they communicate by raising events. We also decided to expose each of our services using REST api. Now the question is , is it a good idea to create our services as web api [RESTful] applications which are also subscribers to the event bus? so basically there are 2 ponits of entry to each service - api and events. I have a feeling that we should separate out these 2 as these are 2 different concerns. Any ideas?
Since Microservices architecture are Un-opinionated software design. So you may get different answers on this questions.
Yes, REST and Event based are two different things but sometime both combined gives design to achieve better flexibility.
Answering to your concerns, I don't see any harm if REST APIs also subscribe to a queue as long as you can maintain both of them i.e changes to message does not have any impact of APIs and you have proper fallback and Eventual consistency mechanism in place. you can check discussion . There are already few project which tried it such as nakadi and ponte.
So It all depends on your service's communication behaviour to choose between REST APIs and Event-Based design Or Both.
What you do is based on your requirement you can choose REST APIs where you see synchronous behaviour between services
and go with Event based design where you find services needs asynchronous behaviour, there is no harm combining both also.
Ideally for inter-process communication protocol it is better to go with messaging and for client-service REST APIs are best fitted.
Check the Communication style in microservices.io
REST based Architecture
Advantage
Request/Response is easy and best fitted when you need synchronous environments.
Simpler system since there in no intermediate broker
Promotes orchestration i.e Service can take action based on response of other service.
Drawback
Services needs to discover locations of service instances.
One to one Mapping between services.
Rest used HTTP which is general purpose protocol built on top of TCP/IP which adds enormous amount of overhead when using it to pass messages.
Event Driven Architecture
Advantage
Event-driven architectures are appealing to API developers because they function very well in asynchronous environments.
Loose coupling since it decouples services as on a event of once service multiple services can take action based on application requirement. it is easy to plug-in any new consumer to producer.
Improved availability since the message broker buffers messages until the consumer is able to process them.
Drawback
Additional complexity of message broker, which must be highly available
Debugging an event request is not that easy.
I have an architectural problem.
I know what represents a (fanout, direct, topic, headers) exchange, bindings, queues and almost everything about message architectures. I have the following problem and I need some pieces of advice.
I would like to implement notification logic to my application, where each user will receive in real-time notification only intended for him. ( Actually, I don't want mention what are my UI and BE languages/frameworks, because of an additional level of abstraction) The UI will make a connection to RabbitMQ with WebSocket, SockJS and STOMP. My UI will be only a consumer, the BE is the writer - that one, which will add some Messages to RabbitMQ.
It's perfectly clear to me if I have a direct Exchange with routing key, which uniquely to identify the specific user (for example: my-routing-to-empoyee-with-id-1) and N-number of Queues for each user. This is too heavy to me (I don't know actually whether is normal situation to have so much queues).
Is there any solution, where I can use only one Queue and the message to be delivered only to user for who is intended ?
I know a solution, where I can have a topic exchange and to have one writer and many subscribers, but on this way, I can filter the message only on clients level, which is not so secure. :(
Actually, I found a very interesting article, which describes me a problem that I have pretty well.
What I want is called Selective Consumers and for Enterprise Integration Patterns this is an anti-pattern and we should not use it.
For more details, all who want can read this article: https://derickbailey.com/2015/07/22/airport-baggage-claims-selective-consumers-and-rabbitmq-anti-patterns/
So, I have built a Netty 3.6.2 based Websockets server application. This application will have many, many users.
The idea is that clients register to listen for information on a topic, and when information flows through the server, the server sends the information to the clients. Sound straightforward so far, right?
I implemented this by building a giant map held in memory mapping the topic to the client's Channel. When the server wants to send a message about a topic too all interested clients, it loops over all channels mapped to that topic. Seem straightforward, right?
However, in some preliminary multi-user testing, I find myself realizing there is not a one-to-one mapping between channel and client. How do I specifically target sending a message to a particular client if not through the channel? I am at a loss....
There should be a one-one ratio of clients to open channels. The fact that there is not is some sort of issue that is not related to netty.
Thank you for your help.