We generally use SNS/SQS for inter-service communication. Now, We need to have a microservice - call it audit or events. It basically keeps track of user activity with different parts of the system. To me, it looks like events streaming in from multiple microservices to our new "events" service. I generally read "Kinesis is better suited for -Log and event data collection as a use case of Kinses instead of SNS/SQS".
The question is why. What do I not get from SNS/SQS in this case that I should consider Kinses for the event and log data collection instead of the SNS/SQS workflow?
Find the table having selection criteria for various AWS Services for Streaming and messaging
Related
Data is available in chat.* topics in Kafka.
We can just spin up multiple consumers and make the all hook into this single event for consumption.
But, not all products want to work on the complete events in the main topic. We want to branch the topic into multiple topics based on the content of the record in topic. How to perform content based routing in Kafka?
Kafka Streams are good for Java applications, but we need a multi stack solution that works across the companies for many languages?
We are looking for a event router kind of thing for Kafka. AWS event bridge does just that! But it is not with Kafka. How do you do such content based routing in Kafka?
Any suggestion?
ksqldb is the open source solution by Confluent that solves this problem.
It is built on top of Kafka Streams. It has a SQL like DSL to make following stream joins possible. Also it has an HTTP API, which allows to query it directly.
Support filtering and stream joins.
Stream-Stream Join (Window Join)
Stream-Table Join (Stream Enrichment)
Table-Table Join (Materialized View)
Declarative query language to specify how to create derived streams from existing streams and tables. (KSQL)
API to specify the query to create derived streams. (KSQL DB API)
KSQL DB now supports adding source connectors as well. This looks brilliant! Going to try it.
Here is how KSQL DB can be used:
What I have?
A lot of different microservices managing by different teams. All microservices persist data in Aerospike database.
What I want to achieve?
I'm building new microservice that relies on data handled by another services. I want to listen the changes in entities, but unfortunately that microservices don't put anything in message queue, they have only usual REST APIs, so I cant just subscribe to events.
The idea is to listen a transaction log(event log/commit log/WAL) of database. This approach is also using in different Event Sourcing systems, but I cant found any Aerospike API that would stream this log. So the question - does Aerospike provide any similar functionality, may be with different name?
Aerospike, in its enterprise edition, has a feature called change notification framework which may fit your requirements. It informs an external agent about all the write operations. This is built over the XDR functionality which is meant for replicating across data centers using a digestlog.
If you are not planning for enterprise, you should reconsider having your own message queue in front of Aerospike.
In the current plan, incoming commands are handled via Function Apps, resulting in Events being sent to an Event Hub, and then materializing the views
Someone is arguing that instead of storing events in something like table storage, and materializing views based on events and snapshots, that we should:
Just stream events to a log in Azure Monitor to have auditing
We can make changes to a domain object immediately in response to a command and use the change feed as our source of events for materialized views.
He doesn’t see the advantage of even having a materialized view. Why not just use a query? Argument is we don’t expect a lot of traffic.
He wants to fulfill the whole audit log by saving events to the azure monitor log - Just an application log. Instead, that commands should just directly modify the representation of an entity in cosmos, and we'd use the change feed from CosmosDB as our domain object events, or we would create new events off of that via subscribers to that stream.
Is this actually an advantageous approach? Can ya'll think of any reasons why we wouldn't want to do that? Seems like we'd be losing something here.
He's saying we'd no longer need to be concerned with eventual consistency, as we'd have immediate consistency.
Every reference implementation I've evaluated does NOT do it the way he's suggesting. I'm not deeply versed in the advantages/disadvantages of the event sourcing / CQRS paradigm so I'm at a loss at the moment.. Currently researching furiously
This is a conceptual issue so there's not so much a code example. However, here's some references that seem to back up the approach I'm taking..
https://medium.com/#thomasweiss_io/planet-scale-event-sourcing-with-azure-cosmos-db-48a557757c8d
https://sajeetharan.com/2019/02/03/event-sourcing-with-azure-eventhub-and-cosmosdb/
https://learn.microsoft.com/en-us/azure/architecture/patterns/event-sourcing
If your goal is only to have the audit log, state-based persistence could be a good choice. Event sourcing adds some complexity to the implementation side and unless you can identify more advantages of using it, you might not convince your team to bring this complexity to the system. There are numerous questions and answers on SO, as well as in some blog posts, about pros and cons of event sourcing, so I won't get into that discussion here.
I can warn you, though, that the second article in your list is very weak and would most probably lead you to many difficulties. The role of Event Hub there is completely unclear and it doesn't explain anything about projections and read-models (what you call "materialised views"). Only a very limited number of use-cases can live with only getting one entity by id and without being able to execute a query across multiple entities. That also probably answers your concern of having read-models at all. You will need them very soon when for the first time you will start figuring out how to get a list of entities based on some condition (query).
Using CosmosDb as the event store is completely feasible, as described in the first article if you can manage the costs involved. Just remember to set the change feed TTL to -1, otherwise, you won't be able to replay your projections when you need to.
To summarise:
Keeping the audit log can be done without event-sourcing, but you need to ensure that events are published reliably, preferably in the same transaction as the entity state update. It is often hard or impossible but you might accept the risk of your audit requirement is not strict. You can also base your audit log on the CosmosDb change feed, just collecting document changes and logging them somewhere.
Event sourcing is a powerful technique but it has both pros and cons. The most common prejudice against using event sourcing is its implementation complexity. It might not be a big issue if you have a team that is somewhat experienced in building event-sourced systems. If you don't have such a team, you might want to build a small-scale spike to get some experience.
If you don't get full buy-in from the team to use event sourcing, you will later get all the blame if anything goes wrong. And it will go wrong at some point, especially with little experience in this area.
Spend some time reading books and trying out things yourself, before going wild in production.
Don't use Event Hub for anything that it is not designed for. Event Hub is the powerful event ingestion transport with limited TTL and it should be used for that purpose.
Don't use Table Storage as the event store, unless you only read entities by id. I used it in production for such a scenario and it worked (to some extent) but you can't project read-models from there.
A simple rule of thumb is to not use products for tasks they weren't designed for.
Azure Monitor was not designed to store application domain data. Azure Monitor is designed to store telemetry data from your applications and services and provides features such as alerts and other types of integration into DevOps tools for managing the operation and health of your apps.
There is a simple reason why you were able to find articles on event sourcing using Cosmos DB and why our own docs talk about it. Because it was designed to be used this way. It is simple to set up Cosmos DB to be an append only event store for your applications and use Change Feed to fire off messages in other apps or services or, in your case, to maintain a materialized view state of domain objects within your app.
As far as my little current experience allows me to understand, one of the core concepts about "microservice" is that it relies on its own database which is independent from other microservices.
Diving into how to handle distributed transactions in a microservices system, the best strategy seems to be the Event Sourcing pattern whose core is the Event Store.
Is the event store shared between different microservices? Or there are multiple independent event stores databases for each microservice and a single common event broker?
If the first option is the solution, using CQRS I can now assume that every microservice's database is intended as query-side, while the shared event store is on the command-side. Is it a wrong assumption?
And since we are in the topic: how many retries I have to do in case of a concurrent write in a Stream using optimistic locking?
A very big big thanks in advance for every piece of advice you can give me!
Is the event store shared between different microservices? Or there are multiple independent event stores databases for each microservice and a single common event broker?
Every microservice should write to its own Event store, from their point of view. This could mean separate instances or separate partitions inside the same instance. This allows the microservices to be scaled independently.
If the first option is the solution, using CQRS I can now assume that every microservice's database is intended as query-side, while the shared event store is on the command-side. Is it a wrong assumption?
Kinda. As I wrote above each microservice should have its own Event store (or a partition inside a shared instance). A microservice should not append events to other microservice Event store.
Regarding reading events, I think that reading events should be in general permitted. Polling the Event store is the simplest (and the best in my opinion) solution to propagate changes to other microservices. It has the advantage that the remote microservice polls at the rate it can and what events it wants. This can be scaled very nice by creating Event store replicas, as much as it is needed.
There are some cases when you would want to not publish every domain event from the Event store. Some say that there are could exist internal domain events on that the other microservices should not depend. In this case you could mark the events as free (or not) for external consuming.
The cleanest solution to propagate changes in a microservice is to have live queries to whom other microservices could subscribe. It has the advantage that the projection logic does not leak to other microservice but it also has the disadvantage that the emitting microservice must define+implement those queries; you can do this when you notice that other microservices duplicate the projection logic. An example of this query is the total order price in an ecommerce application. You could have a query like this WhatIsTheTotalPriceOfTheOrder that is published every time an item is added to/removed from/updated in an Order.
And since we are in the topic: how many retries I have to do in case of a concurrent write in a Stream using optimistic locking?
As many as you need, i.e. until the write succeeds. You could have a limit of 99999, just to be detect when something is horribly wrong with the retry mechanism. In any case, the concurrent write should be retried only when a write is done at the same time on the same stream (for one Aggregate instance) and not for the entire Event store.
As a rule: in service architectures, which includes micro services, each service tracks its state in a private database.
"Private" here primarily means that no other service is permitted to write or read from it. This could mean that each service has a dedicated database server of its own, or services might share a single appliance but only have access permissions for their own piece.
Expressed another way: services communicate with each other by sharing information via the public api, not by writing messages into each others databases.
For services using event sourcing, each service would have read and write access only to its streams. If those streams happen to be stored on the same home - fine; but the correctness of the system should not depend on different services storing their events on the same appliance.
TLDR: All of these patterns apply to a single bounded context (service if you like), don't distribute domain events outside your bounded context, publish integration events onto an ESB (enterprise service bus) or something similar, as the public interface.
Ok so we have three patterns here to briefly cover individually and then together.
Microservices
CQRS
Event Sourcing
Microservices
https://learn.microsoft.com/en-us/azure/architecture/microservices/
Core objective: Isolate and decouple changes in a system to individual services, enabling independent deployment and testing without collateral impact.
This is achieved by encapsulating change behind a public API and limiting runtime dependencies between services.
CQRS
https://learn.microsoft.com/en-us/azure/architecture/patterns/cqrs
Core objective: Isolate and decouple write concerns from read concerns in a single service.
This can be achieved in a few ways, but the core idea is that the read model is a projection of the write model optimised for querying.
Event Sourcing
https://learn.microsoft.com/en-us/azure/architecture/patterns/event-sourcing
Core objective: Use the business domain rules as your data model.
This is achieved by modelling state as an append-only stream of immutable domain events and rebuilding the current aggregate state by replaying the stream from the start.
All Together
There is a lot of great content here https://learn.microsoft.com/en-us/previous-versions/msp-n-p/jj554200(v=pandp.10)
Each of these has its own complexity, trade-offs and challenges and while a fun exercise you should consider if the cost outway the benefits. All of them apply within a single service or bounded context. As soon as you start sharing a data store between services, you open yourself up to issues, as the shared data store can not be changed in isolation as it is now a public interface.
Rather try publish integration events to a shared bus as the public interface for other services and bounded contexts to consume and use to build projections of other domain contexts data.
It's a good idea to publish integration events as idempotent snapshots of the current aggregate state (upsert X, delete X), especially if your bus is not persistent. This allows you to republish integration events from a domain if needed without producing an inconsistent state between consumers.
In my understanding when database transactions span across microservices ,we can solve this problem with using message-broker(kafka,RabbitMQ etc) by publishing events so that Subscriber Microservices can update their database by listening to these events.
In case of exception we can send event for failure ,so that Subscriber services can update their state.
Is this not sufficient? What is the problem with this approach?
why and when we need event sourcing?
Do we need really event sourcing ?
Not at all. You can have well a very well defined microservices-styled architecture without CQRS and Event Sourcing. CQRS and Event Sourcing is a solution for intra-microservice design. You can choose to implement all or some of your microservices using CQRS and Event Sourcing.
Let's see how Event Sourcing may help you. Event Sourcing is an alternative to restoring your current state of an entity using events instead of an ORM like Entity Framework or Hibernate and a SQL database. Suppose you have a microservice to store data about Books. If you use SQL, you would have controllers and end-points to Create, Update, and Delete a book and store those books in a SQL table. If you want to update that book, then in order to get the current state you would go back to SQL table and query for that book (by its id) and then your ORM would convert that table representation into a book object (object‑relational impedance mismatch problem) and then you would apply the changes and save the changed book object back into SQL table. As an alternative, you can store events for the books objects in a NoSQL database like MongoDB or maybe an event store. Now in order to update the book, first you would want to restore the current state and you can do that by getting back all the events related to this book and replaying these events to restore the current state. Your events became a source of truth and you completely avoided the bottleneck of ORM mapping and SQL joins. Events are stored as JSON documents and are usually ultra-fast.
Now, coming to CQRS - CQRS is purely a pattern for separation of concerns. You would use CQRS to bifurcate your read-side from the write-side. End-points related to write-side like create, update, and delete live in one service and end-point for read-side live in another service. The advantage you get here is independent scaling, deploying, maintenance and many more. If your application is read-intensive, then have multiple numbers of instances deployed for read-side service.
If you want to learn more, feel free to PM me. Good Luck!
I think you're confused about the microservices stuff :) They're a solution to design a scalable application, nothing to do with db transactions. Even more, a db transaction (ideally) shouldn't span across processes, nevermind microservices.
The pub-sub approach is how different micro services (works for in process models as well) communicate. Nothing to do with db transaction. Event sourcing is all about looking at the domain state as a collection of relevant changes. Very different objective compared to microservices.
We're using ES because we like to store domain events as being the 'single source of truth', microservices or not. It's all about domain model design.
Learn more about ES, DDD, CQRS and leave microservices for the future. They're too buzzwordy for now, few apps needs them, few developers can actually use them properly.
You are describing a compensation pattern as a replacement of a distributed transaction. In a microservice-oriented architecture, this is a good approach to focus on availability by utilizing eventual consistency: Instead of having one centrally coordinated, distributed transaction across services, each service will execute its sub-task without a transactional context. If something goes wrong, each service will be informed about the failure and execute some kind of (semantic) compensation of the previous action. Thus, the transactional operation is eventually undone.
As you have already stated, communication can be done via a message bus system and there is no need for Event Sourcing or CQRS, the compensation pattern does not depend on these principles.