In the current plan, incoming commands are handled via Function Apps, resulting in Events being sent to an Event Hub, and then materializing the views
Someone is arguing that instead of storing events in something like table storage, and materializing views based on events and snapshots, that we should:
Just stream events to a log in Azure Monitor to have auditing
We can make changes to a domain object immediately in response to a command and use the change feed as our source of events for materialized views.
He doesn’t see the advantage of even having a materialized view. Why not just use a query? Argument is we don’t expect a lot of traffic.
He wants to fulfill the whole audit log by saving events to the azure monitor log - Just an application log. Instead, that commands should just directly modify the representation of an entity in cosmos, and we'd use the change feed from CosmosDB as our domain object events, or we would create new events off of that via subscribers to that stream.
Is this actually an advantageous approach? Can ya'll think of any reasons why we wouldn't want to do that? Seems like we'd be losing something here.
He's saying we'd no longer need to be concerned with eventual consistency, as we'd have immediate consistency.
Every reference implementation I've evaluated does NOT do it the way he's suggesting. I'm not deeply versed in the advantages/disadvantages of the event sourcing / CQRS paradigm so I'm at a loss at the moment.. Currently researching furiously
This is a conceptual issue so there's not so much a code example. However, here's some references that seem to back up the approach I'm taking..
https://medium.com/#thomasweiss_io/planet-scale-event-sourcing-with-azure-cosmos-db-48a557757c8d
https://sajeetharan.com/2019/02/03/event-sourcing-with-azure-eventhub-and-cosmosdb/
https://learn.microsoft.com/en-us/azure/architecture/patterns/event-sourcing
If your goal is only to have the audit log, state-based persistence could be a good choice. Event sourcing adds some complexity to the implementation side and unless you can identify more advantages of using it, you might not convince your team to bring this complexity to the system. There are numerous questions and answers on SO, as well as in some blog posts, about pros and cons of event sourcing, so I won't get into that discussion here.
I can warn you, though, that the second article in your list is very weak and would most probably lead you to many difficulties. The role of Event Hub there is completely unclear and it doesn't explain anything about projections and read-models (what you call "materialised views"). Only a very limited number of use-cases can live with only getting one entity by id and without being able to execute a query across multiple entities. That also probably answers your concern of having read-models at all. You will need them very soon when for the first time you will start figuring out how to get a list of entities based on some condition (query).
Using CosmosDb as the event store is completely feasible, as described in the first article if you can manage the costs involved. Just remember to set the change feed TTL to -1, otherwise, you won't be able to replay your projections when you need to.
To summarise:
Keeping the audit log can be done without event-sourcing, but you need to ensure that events are published reliably, preferably in the same transaction as the entity state update. It is often hard or impossible but you might accept the risk of your audit requirement is not strict. You can also base your audit log on the CosmosDb change feed, just collecting document changes and logging them somewhere.
Event sourcing is a powerful technique but it has both pros and cons. The most common prejudice against using event sourcing is its implementation complexity. It might not be a big issue if you have a team that is somewhat experienced in building event-sourced systems. If you don't have such a team, you might want to build a small-scale spike to get some experience.
If you don't get full buy-in from the team to use event sourcing, you will later get all the blame if anything goes wrong. And it will go wrong at some point, especially with little experience in this area.
Spend some time reading books and trying out things yourself, before going wild in production.
Don't use Event Hub for anything that it is not designed for. Event Hub is the powerful event ingestion transport with limited TTL and it should be used for that purpose.
Don't use Table Storage as the event store, unless you only read entities by id. I used it in production for such a scenario and it worked (to some extent) but you can't project read-models from there.
A simple rule of thumb is to not use products for tasks they weren't designed for.
Azure Monitor was not designed to store application domain data. Azure Monitor is designed to store telemetry data from your applications and services and provides features such as alerts and other types of integration into DevOps tools for managing the operation and health of your apps.
There is a simple reason why you were able to find articles on event sourcing using Cosmos DB and why our own docs talk about it. Because it was designed to be used this way. It is simple to set up Cosmos DB to be an append only event store for your applications and use Change Feed to fire off messages in other apps or services or, in your case, to maintain a materialized view state of domain objects within your app.
Related
I am thinking what is the best way to structure your micro-services, in the past the team I was working with used Axon Framework and PostgreSQL and each microservice had its own event store in the PostgreSQL database, then we built communication between using REST.
I am thinking that it would be smarter to have all microservices talk to the same event store as we would be able to share events faster instead of rewriting the communication lines using REST.
The questions that follows from the backstory is:
What is the best practice for having an event store
Would each service have its own? Would they share the same eventstore?
Where would I find information to inspire and gather more answers? As searching the internet for best practices and how to structure the Event Store seems like searching for a needle in a haystack.
Bear in mind, the question stated is in no way aimed at Axon Framework, but more the general idea on building scalable and good code. As the applications would work with each own event store for write model and read models.
Thank you for reading and I wish you all the best
-- Me
I'd add a slightly different notion to Tore's response, although the mainline is identical to what I'm sharing here. So, I don't aim to overrule Tore, just hoping to provide additional insight.
If the (micro)services belong to the same Bounded Context, then they're allowed to "learn about each other's language."
This language thus includes the events these applications publish and store.
Whenever there's communication required between different Bounded Contexts, you'd separate the stores, as one context shouldn't be bothered by the specifics of another context.
Hence it is beneficial to deduce what services belong to which Bounded Context since that would dictate the required separation.
Axon aims to support this by allowing multiple contexts with the Axon Server, as you can read here.
It simply allows the registration of applications to specific contexts, within which it will completely separate all message streams (so commands, events, and queries) and the Event Store.
You can also set this up from scratch yourself, of course. Tore's recommendation of Kafka is what's used quite broadly for Event Streaming needs between applications. Honestly, any broadcast type of infrastructure suits event distribution, as that's how events are typically propagated.
You want to have one EventStore per service, just as you would want to have one relation database per service for a non EventSourced system.
Sharing a database/eventstore between services creates coupling and we have all learned the hard way that this is an anti-pattern today.
If you want to use a event log to share events across services, then Kafka is a popular choice.
Important to remember that you only do event-sourcing within a service bounded context.
I am new to using graphql and we have built a backend graphql server using elixir and we are building a frontend app using react and react-relay.
My question is whether it is better to have one large subscription at the root of my query renderer instead of having loads of smaller subscriptions for individual components. I think I would prefer using lots and lots of smaller subscriptions rather than fewer (or even one) very large subscriptions but there are concerns that too many subscriptions will be very heavy. Is this valid?
TIA
There are a few things to consider here, and really, they all depend on what your definition of "very heavy" is. Note "very heavy" might mean something very different for your Elixir server implementation than it does on the client, so I will attempt to cover some directions you may want to investigate for both here.
What is your subscription transport? Websockets can be expensive and difficult to scale on both ends at a certain point, but if you can deal with unidirectional data flow (server to client only), SSE (Server-Sent Events) are a great option. See more on a breakdown between SSE and WS here. This is more a comment on your server than on your client.
From an API design perspective, I'd caution against the few (or one) large subscriptions idea. Why? Inevitably, you are going to be pushing data on the client that it never asked for; this causes unnecessary work for both client and server. Furthermore, an individual component should only be able to subscribe to data screams with data specifically designated for it. If you go the large subscription route, then you'll have to write a good deal of defensive code to filter the event stream, looking for the data you need. That shouldn't be your responsibility to micromanage, not to mention the dirty event stream on your server.
This is not necessarily to lead you down the "small subscription" route either. Ultimately, you might want to look at this hybrid approach , which articulates my opinions on the matter better than I can myself. TL;DR design the subscriptions API so that you can enjoy the tightly scoped benefits of lots of small subscriptions ("per entity," as the author titles them), but still allow you to share payloads and reuse the same handlers that your mutations do to resolve data.
Plus, if you wanted to use persisted queries the hybrid approach is going to serve you better.
In my understanding when database transactions span across microservices ,we can solve this problem with using message-broker(kafka,RabbitMQ etc) by publishing events so that Subscriber Microservices can update their database by listening to these events.
In case of exception we can send event for failure ,so that Subscriber services can update their state.
Is this not sufficient? What is the problem with this approach?
why and when we need event sourcing?
Do we need really event sourcing ?
Not at all. You can have well a very well defined microservices-styled architecture without CQRS and Event Sourcing. CQRS and Event Sourcing is a solution for intra-microservice design. You can choose to implement all or some of your microservices using CQRS and Event Sourcing.
Let's see how Event Sourcing may help you. Event Sourcing is an alternative to restoring your current state of an entity using events instead of an ORM like Entity Framework or Hibernate and a SQL database. Suppose you have a microservice to store data about Books. If you use SQL, you would have controllers and end-points to Create, Update, and Delete a book and store those books in a SQL table. If you want to update that book, then in order to get the current state you would go back to SQL table and query for that book (by its id) and then your ORM would convert that table representation into a book object (object‑relational impedance mismatch problem) and then you would apply the changes and save the changed book object back into SQL table. As an alternative, you can store events for the books objects in a NoSQL database like MongoDB or maybe an event store. Now in order to update the book, first you would want to restore the current state and you can do that by getting back all the events related to this book and replaying these events to restore the current state. Your events became a source of truth and you completely avoided the bottleneck of ORM mapping and SQL joins. Events are stored as JSON documents and are usually ultra-fast.
Now, coming to CQRS - CQRS is purely a pattern for separation of concerns. You would use CQRS to bifurcate your read-side from the write-side. End-points related to write-side like create, update, and delete live in one service and end-point for read-side live in another service. The advantage you get here is independent scaling, deploying, maintenance and many more. If your application is read-intensive, then have multiple numbers of instances deployed for read-side service.
If you want to learn more, feel free to PM me. Good Luck!
I think you're confused about the microservices stuff :) They're a solution to design a scalable application, nothing to do with db transactions. Even more, a db transaction (ideally) shouldn't span across processes, nevermind microservices.
The pub-sub approach is how different micro services (works for in process models as well) communicate. Nothing to do with db transaction. Event sourcing is all about looking at the domain state as a collection of relevant changes. Very different objective compared to microservices.
We're using ES because we like to store domain events as being the 'single source of truth', microservices or not. It's all about domain model design.
Learn more about ES, DDD, CQRS and leave microservices for the future. They're too buzzwordy for now, few apps needs them, few developers can actually use them properly.
You are describing a compensation pattern as a replacement of a distributed transaction. In a microservice-oriented architecture, this is a good approach to focus on availability by utilizing eventual consistency: Instead of having one centrally coordinated, distributed transaction across services, each service will execute its sub-task without a transactional context. If something goes wrong, each service will be informed about the failure and execute some kind of (semantic) compensation of the previous action. Thus, the transactional operation is eventually undone.
As you have already stated, communication can be done via a message bus system and there is no need for Event Sourcing or CQRS, the compensation pattern does not depend on these principles.
At my company, we're about to move to the micro services architecture. I read a lot about it, and there are tons of obscure areas where it's specific to the project built, but one area seems to get everyone to agree, microservices need to have isolated persistence or another way to say it, they need to have they own database.
Now I love the idea, that means every microservice has its own database schema, its own domain objects and is 100% independent of any other microservice data structure.
There are things I don't quite understand though.
The "Customer Service" is obviously central to the application, and we can see that basically any other microservice will need some data about the user at some point. Whether it'd be the user's credit amount, its ID, or its name.
But since other microservices can't directly read into the Customer Service database, they'll need to query this service over and over again. This is fine (I guess) for simple stuff like getting the name of current logged user, but when we need to display 60 users on a page and we can't do any SQL join, it feels like we're missing something. This is even worse when microservices depend upon tons of microservices.
So I found out that some people actually queried microservices X times a day to get data into their own microservices.
So if microservice "Search" needs data from "Product", "Customer", it'll actually query these microservices and will persist the data with its own data structure.
The question I have is should it be "Search" that queries "Product" and "Customer", or should "Product" and "Customer" send data to "Search" ?
The first option looks a bit easier to do, we only need to have this logic on one side, and that's where the data is needed. But we'll only get static freshness of data which is not very smart, but could definitely work.
The second option looks a bit more difficult but more scalable too, because we could have very fresh data when we need it, since the data changed where it's sent, it could also be more granular.
I think you correctly identified downsides to the microservices approach! And there are no elegant solutions to these specific problems. You will have to eat the additional work and architecture deterioration that this brings.
Concretely addressing your question now:
The question I have is should it be "Search" that queries "Product" and "Customer", or should "Product" and "Customer" send data to "Search" ?
You seem to be looking for a data synchronization service. You want to decide between push and pull. You are concerned about data freshness and logic duplication.
The key point here is that the source service cannot know about its consumers. This is to prevent an unwanted reverse dependency. This would break architectural isolation. Any data sync process that maintains this is fine. You can do what is most convenient.
For example, you could make the data source expose two APIs:
An API to get the whole data set. This would be called periodically by the destination (e.g. nightly). It can also be used to seed the destination at will and to fix data errors there.
A feed of changes in the source database keyed by the date and time the change occurred. The destination can now poll that change feed very frequently (e.g. every few seconds or minutes) and apply the small delta that occurred.
You can even build a realtime change feed through a publish-subscribe middleware. Many message queue softwares can do that. The source would just send out changes to the middleware.
Building all of this is conceptually simple but takes a lot of work. It also creates lots of ongoing work and increases the potential for bugs. Debugging becomes much harder. I have worked on systems like that.
I'm going to add a subjective note: Microservices are not well understood by many teams. The downsides are often ignored. You identified a few of the downsides correctly and they are nasty! Given what I read on the web I believe many teams do not realize the mess they are getting themselves into. Managing disparate data stores can be a nightmare. This is not a one-time "mess" but an ongoing one.
As an alternative I'd recommend using a common data store and building services simply as classes or projects that live in the same process. This gives you the microservices code structuring with the convenience of normal development. It also leaves a few of the upsides of microservices on the table.
your identification of the problem is correct.
But the solution to your problem will depend on use case to use case.
In your example of search service , product service and customer service should publish their events on kafka or similar messaging and search service listen to them and updates it.
In case of lets say in order service while creating an order for a customer , you want to check customer exists , then you might do it by calling the sync api of customer service , but for that also there are variour other approaches , i have answered here linking Microservices and allowing for one to be unavailable
From my perspective sync communication between services should be avoided , and there are way around for this , above link would help
You can use domain driven design philosophy to correctly break your services and their contract
Something here doesn't feel right to me here, and so I would like the community's input - perhaps I am approaching this in the wrong way....
Q: Is is appropriate to use traditional infrastructure logging frameworks (like log4net) to log business events?
When I say business events, I mean I want a global log like this:
xx:xx Customer A purchased widget B.
xx:xx Widget B was dispatched from warehouse.
xx:xx Customer B payment declined.
Most traditional infrastructure logging frameworks have event levels something like this:
FATAL
ERROR
WARN
INFO
DEBUG
An of course these messages don't fit well into that. Best description would be INFO, but of course these are important events, and INFO is of very low importance.
I would still like this as a 'log' (e.g. I don't want to have to extract this from my business objects each time I want to see it)
Seems to me I have two options:
1) Use a framework like log4net and just define a special logger for this (and live with the fact that it doesn't feel right).
2) Provide a service for performing this that doesn't rely on a traditional logging services.
I'm leaning towards 2. What has anyone else done in a similar situations?
Thanks!
What you're wanting sounds like an auditing service, not a logging service. If I'm right, your goals are to keep track of these business events for historical and maybe even reporting purposes. You can use the details in the audit to, for lack of a better phrase, place blame for events that happen in the system.
I probably wouldn't use a logging system, like log4j, for this purpose. In our system, auditing is a first class citizen as a full service.
--
HTH,
Dusty
Leave the logger for things having to do with the program, not the business. It is just a tool to help the developers.
Write your own system to log business events. If it is a business requirement to have a record, you will want something you have control over and you will need to use the logger above to keep track of how it works.
Basically, #2 in your question.
To me the idea of a Business Event is that it plays a role in some future business processing, anything from actually triggering Business Actions to simply available for analytics.
Hence, completely different QOS requirements. needs its own API.
Conceviably initially that maps down to logging, but in future could go to reliable messaging or DB.
These sound like the sorts of things that your customers might potentially want to query or report on from within your application - the obvious choice would be the database.
In particualr, in this case I'd feel like traditional logging frameworks wouldnt be suitable because when it comes to data that you might later want to access within your application logging frameworks allow you to do things that dont really make sense, for example you might be able to change where the logging is sent to based on the app.config file (which is unhelful if you try and read it from a different location).
That said, if a logging framework allows you to do exactly what you want already then there isnt any shame in just using the logging framework as your implementation and saving yourself the effort:
class TransactionLogger
{
public void Log (Message message)
{
MyLoggingFramework.Log(message.string, etc...);
}
}
In my experience business events comprise large or huge number of technical operations behind the scenes, with only certain business events being important to the business.
This creates problems when trying to use a generic logging methodology, so in general, in the systems I've worked on, both are used.
Logging for the technical aspects, and business event logging for the business events.
The business event logging, doesn't use the same technology as the technical logging, and instead logs to a custom designed history/audit table (Sometimes these are split, depending on the required detail), which is designed specifically for each application. (This keeps the auditors and users nice and happy.)
This allows easy reporting, and management of the information, while obviously expanding the scope of each specification slightly.
you could use it but you need is business activity monitoring and event processing software. Off the top of my head, IBM WebSphere Business Monitor provides this capability. It processes Common Base Event (an IBM implementation of the Web Services Distributed Management Web Event Format standard) and then takes that data and create business activity dashboards.
Check out DiALog: A Distributed Model for Capturing Provenance and Auditing Information, apart from the distributed aspect, you can use the subject-predicate-object principle to record the business events. And afterwards reconstruct certain trails.
Here is a related post - mine. Audit logging and exception management framework.