What would be difference between Microservice Patterns Transaction Log Trailing vs Polling Publisher.
Looking at listed example (Eventuate Local) seems that both the patterns are based on trailing database logs and publishing to message broker.
I have just started to discover microservices world. I know a bit about these two hope this helps you. In order to use these two pattern, you have to already implement transactional outbox pattern. These two pattern comes after that and these two answers the question "How can I deliver the domain object messages to the relevant microservices keeping the Atomicity principal?"
Polling publisher pattern: Sender periodically queries the OUTBOX table if there is a record, publishes this record to the message broker and after sending message it deletes that record.
Transactional Log Tailing: A bit sophisticated solution, third party frameworks are used. General obligation of these frameworks is that it listens commit logs and publishes each change as a message to the message broker.
You have asked question one year ago. If you enhance your knowledge about these patterns please share them :)
The frequent DB queries may become too expansive in large systems. Also, not all NoSql DB supports according queries. NoSql dBs request a document, this can be not so efficient as expected. That’s why transaction log may be more preferable. But as for me for nasal and postgresql polling works well. More details in Kris Richardson’s book.
Related
What I have?
A lot of different microservices managing by different teams. All microservices persist data in Aerospike database.
What I want to achieve?
I'm building new microservice that relies on data handled by another services. I want to listen the changes in entities, but unfortunately that microservices don't put anything in message queue, they have only usual REST APIs, so I cant just subscribe to events.
The idea is to listen a transaction log(event log/commit log/WAL) of database. This approach is also using in different Event Sourcing systems, but I cant found any Aerospike API that would stream this log. So the question - does Aerospike provide any similar functionality, may be with different name?
Aerospike, in its enterprise edition, has a feature called change notification framework which may fit your requirements. It informs an external agent about all the write operations. This is built over the XDR functionality which is meant for replicating across data centers using a digestlog.
If you are not planning for enterprise, you should reconsider having your own message queue in front of Aerospike.
I am now trying to design database for my micro service-oriented application in a distributed way. My application is related with management of universities. I have different universities say A, B, C. Each university have separate users for using their business data. Now I am planning to design separate databases for separate universities for storing their user data. So each university has their own database for their users and additional one database for managing their application tables. If I have 2 universities, Then I have 2 user details DB and other 2 DB for application tables.
Here my confusion is that, when I am searching for database design, I only see the approach of keeping one common database for storing all users (Here for one DB for all users of all universities). So every user is mixed within one database.
If I am following separate database for each university, Is possible to support distributed DB architecture pattern and micro service oriented standard? Or Do I need to keep one DB for all users?
How can I find out which method is appropriate for microservice / Distributed database design pattern?
Actually there could be multiple solutions and not one solution is best, the best solution is the one which is appropriate for your product's requirements.
I think it would be a better idea to go with separate databases for each of your client (university) to keep the data always isolated even if somethings wrong happens. Also with time, the database could go so huge that it could cause problems to configure/manage separate backups, cleanups for individual clients etc.
Now with separate databases there comes a challenge for managing distributed transactions across databases as you don't know which part is going to fail among many. To manage that, you may have to implement message/event driven mechanism across all your micro-services and ensure consistency.
Regarding message/event mechanism, here is a simple use case scenario, suppose there are two services "A" (user-registration) and "B" (email-service)
"A" registers a user temporarily and publishes an event of sending confirmation email.
The message goes to message broker
The message is received by "B".
The confirmation email is sent to the user.
The user confirms the email to "B"
The "B" publishes event of user confirmation to the broker
"A" receives the event of confirmation and the process is completed.
The above is the best case scenario, problems still can happen in between even with broker itself.
You have to go deep into it if you think you need this.
Some links that may help.
http://how-to-implement-a-microservice-event-driven-architecture-with-spring-cloud-stre
A Guide to Transactions Across Microservices
I don't think that this is a valid design, using a database per client which is a Multi-tenant architecture practice, and database per microservice is a microservice architecture practice. you are mixing things up.
if you will use microservice architecture you better design it as Bounded contexts and each Context has its own database to achieve microservices main rule Autonomy
In my understanding when database transactions span across microservices ,we can solve this problem with using message-broker(kafka,RabbitMQ etc) by publishing events so that Subscriber Microservices can update their database by listening to these events.
In case of exception we can send event for failure ,so that Subscriber services can update their state.
Is this not sufficient? What is the problem with this approach?
why and when we need event sourcing?
Do we need really event sourcing ?
Not at all. You can have well a very well defined microservices-styled architecture without CQRS and Event Sourcing. CQRS and Event Sourcing is a solution for intra-microservice design. You can choose to implement all or some of your microservices using CQRS and Event Sourcing.
Let's see how Event Sourcing may help you. Event Sourcing is an alternative to restoring your current state of an entity using events instead of an ORM like Entity Framework or Hibernate and a SQL database. Suppose you have a microservice to store data about Books. If you use SQL, you would have controllers and end-points to Create, Update, and Delete a book and store those books in a SQL table. If you want to update that book, then in order to get the current state you would go back to SQL table and query for that book (by its id) and then your ORM would convert that table representation into a book object (object‑relational impedance mismatch problem) and then you would apply the changes and save the changed book object back into SQL table. As an alternative, you can store events for the books objects in a NoSQL database like MongoDB or maybe an event store. Now in order to update the book, first you would want to restore the current state and you can do that by getting back all the events related to this book and replaying these events to restore the current state. Your events became a source of truth and you completely avoided the bottleneck of ORM mapping and SQL joins. Events are stored as JSON documents and are usually ultra-fast.
Now, coming to CQRS - CQRS is purely a pattern for separation of concerns. You would use CQRS to bifurcate your read-side from the write-side. End-points related to write-side like create, update, and delete live in one service and end-point for read-side live in another service. The advantage you get here is independent scaling, deploying, maintenance and many more. If your application is read-intensive, then have multiple numbers of instances deployed for read-side service.
If you want to learn more, feel free to PM me. Good Luck!
I think you're confused about the microservices stuff :) They're a solution to design a scalable application, nothing to do with db transactions. Even more, a db transaction (ideally) shouldn't span across processes, nevermind microservices.
The pub-sub approach is how different micro services (works for in process models as well) communicate. Nothing to do with db transaction. Event sourcing is all about looking at the domain state as a collection of relevant changes. Very different objective compared to microservices.
We're using ES because we like to store domain events as being the 'single source of truth', microservices or not. It's all about domain model design.
Learn more about ES, DDD, CQRS and leave microservices for the future. They're too buzzwordy for now, few apps needs them, few developers can actually use them properly.
You are describing a compensation pattern as a replacement of a distributed transaction. In a microservice-oriented architecture, this is a good approach to focus on availability by utilizing eventual consistency: Instead of having one centrally coordinated, distributed transaction across services, each service will execute its sub-task without a transactional context. If something goes wrong, each service will be informed about the failure and execute some kind of (semantic) compensation of the previous action. Thus, the transactional operation is eventually undone.
As you have already stated, communication can be done via a message bus system and there is no need for Event Sourcing or CQRS, the compensation pattern does not depend on these principles.
I have an architectural problem.
I know what represents a (fanout, direct, topic, headers) exchange, bindings, queues and almost everything about message architectures. I have the following problem and I need some pieces of advice.
I would like to implement notification logic to my application, where each user will receive in real-time notification only intended for him. ( Actually, I don't want mention what are my UI and BE languages/frameworks, because of an additional level of abstraction) The UI will make a connection to RabbitMQ with WebSocket, SockJS and STOMP. My UI will be only a consumer, the BE is the writer - that one, which will add some Messages to RabbitMQ.
It's perfectly clear to me if I have a direct Exchange with routing key, which uniquely to identify the specific user (for example: my-routing-to-empoyee-with-id-1) and N-number of Queues for each user. This is too heavy to me (I don't know actually whether is normal situation to have so much queues).
Is there any solution, where I can use only one Queue and the message to be delivered only to user for who is intended ?
I know a solution, where I can have a topic exchange and to have one writer and many subscribers, but on this way, I can filter the message only on clients level, which is not so secure. :(
Actually, I found a very interesting article, which describes me a problem that I have pretty well.
What I want is called Selective Consumers and for Enterprise Integration Patterns this is an anti-pattern and we should not use it.
For more details, all who want can read this article: https://derickbailey.com/2015/07/22/airport-baggage-claims-selective-consumers-and-rabbitmq-anti-patterns/
I am a programmer working in the domain of information system integration (SOA, ESB, messages broker etc...).
At the moment I am working on a proprietary ESB which unfortunately does not implement the Scatter-Gather pattern.
Actually, I would be interested in a solution to implement this pattern asynchronously. Meaning in this example, the Best Quote is not managed in the same synchronous transaction than Quote Request (e.g. not a synchronous request/reply service).
Because we are talking about asynchronous processing, I am looking for a reliable solution. As an example, if the Vendor B fails (due to a technical issue), I do not want to send back a error Quote Request. I have to consider the whole transaction as a guaranteed one and being able to reprocess the call to Vendor B at one point in time. Then the transaction would magically continue and I would be able to send back a successful Quote Request.
I have already been able to implement this pattern in the past using a proprietary Complex Event Processing (CEP) tool. Indeed the CEP tool was able to persist the global transaction state and to correlate events coming back from Vendor A, B and C.
So I was wondering if there was an existing lightweight solution to implement this pattern in a reliable way.
Last but not least I do not search another ESB tool. I know that Apache Camel, Spring Integration, Mule or WSO2 are implementing this pattern but I was rather interested in a dedicated solution.
Thanks
Your requirements fall in two broad spaces. Event correlation/filtering is usually done with a CEP engine while the integration patterns like Scatter-Gather are to be done with an ESB. It's true that a CEP engine will facilitate some level of message transformation and integration capabilities while an ESB will also support basic filtering/correlations of events (mostly event processing rather than complex event processing), but these are not the purposes they are originally designed for.
So if it can't be implemented using one of ESB or CEP servers, your solution can include both CEP and ESB servers, each performing some specific set of tasks they are best suited for. (it's unlikely that a vendor building a combined server to perform all these stuff)
Having said that, if you are going to implement this with WSO2 products, and if you really need a single server instance, you can consider installing CEP features on top of the ESB as explained in this doc. Otherwise, you can connect the two servers using a high-performant protocol like Thrift.