What I have?
A lot of different microservices managing by different teams. All microservices persist data in Aerospike database.
What I want to achieve?
I'm building new microservice that relies on data handled by another services. I want to listen the changes in entities, but unfortunately that microservices don't put anything in message queue, they have only usual REST APIs, so I cant just subscribe to events.
The idea is to listen a transaction log(event log/commit log/WAL) of database. This approach is also using in different Event Sourcing systems, but I cant found any Aerospike API that would stream this log. So the question - does Aerospike provide any similar functionality, may be with different name?
Aerospike, in its enterprise edition, has a feature called change notification framework which may fit your requirements. It informs an external agent about all the write operations. This is built over the XDR functionality which is meant for replicating across data centers using a digestlog.
If you are not planning for enterprise, you should reconsider having your own message queue in front of Aerospike.
Related
I am thinking what is the best way to structure your micro-services, in the past the team I was working with used Axon Framework and PostgreSQL and each microservice had its own event store in the PostgreSQL database, then we built communication between using REST.
I am thinking that it would be smarter to have all microservices talk to the same event store as we would be able to share events faster instead of rewriting the communication lines using REST.
The questions that follows from the backstory is:
What is the best practice for having an event store
Would each service have its own? Would they share the same eventstore?
Where would I find information to inspire and gather more answers? As searching the internet for best practices and how to structure the Event Store seems like searching for a needle in a haystack.
Bear in mind, the question stated is in no way aimed at Axon Framework, but more the general idea on building scalable and good code. As the applications would work with each own event store for write model and read models.
Thank you for reading and I wish you all the best
-- Me
I'd add a slightly different notion to Tore's response, although the mainline is identical to what I'm sharing here. So, I don't aim to overrule Tore, just hoping to provide additional insight.
If the (micro)services belong to the same Bounded Context, then they're allowed to "learn about each other's language."
This language thus includes the events these applications publish and store.
Whenever there's communication required between different Bounded Contexts, you'd separate the stores, as one context shouldn't be bothered by the specifics of another context.
Hence it is beneficial to deduce what services belong to which Bounded Context since that would dictate the required separation.
Axon aims to support this by allowing multiple contexts with the Axon Server, as you can read here.
It simply allows the registration of applications to specific contexts, within which it will completely separate all message streams (so commands, events, and queries) and the Event Store.
You can also set this up from scratch yourself, of course. Tore's recommendation of Kafka is what's used quite broadly for Event Streaming needs between applications. Honestly, any broadcast type of infrastructure suits event distribution, as that's how events are typically propagated.
You want to have one EventStore per service, just as you would want to have one relation database per service for a non EventSourced system.
Sharing a database/eventstore between services creates coupling and we have all learned the hard way that this is an anti-pattern today.
If you want to use a event log to share events across services, then Kafka is a popular choice.
Important to remember that you only do event-sourcing within a service bounded context.
all new to Masstransit and are currently evaluating it for a larger project and
wondering if anyone could help get a better understanding of the following challenges:
"Single consumer of events in a loadbalanced environment"
In production our services will be runinng multiple instances for scalability and failvover and be
part of a larger ecosystem of microservices. The overall architecture is based on Microsofts eshoponcontainers
reference implementation where different microservices are communicating with each other via "integrationevents".
When publishing a IntegrationEvent to other services which I assume should be done as described in Masstransit Producers / Publish
, https://masstransit-project.com/usage/producers.html#publish, how can I assure that only ONE instance in a specific microservice are processing
the event but of course that the event reaches ALL microsystem that depends on the event? When we have done similar solutions based on Azure Functions
this requirement has been solved by using the "Singelton" attribute (https://learn.microsoft.com/en-us/azure/app-service/webjobs-sdk-how-to#singleton-attribute).
Azure Service Bus
Reading the documentation my impression is that Masstransit is very RabbitMQ centric.
Since we will be on the Azure Service bus when moving to production is there any limitations or features
not available on that "transport"?
Regards Niclas
For the first question, it's a normal publish-subscribe with competing consumers. It works like this out of the box, there's nothing that needs to be done to achieve this.
When running multiple instances of the same service
Use the same queue name for each instance
Messages from the queue will be load balanced across all instances (the competing consumer pattern)
It's from the RMQ Guidances, but it's like this for all transports.
Concerning the Azure Service Bus transport, it works as expected and has a lot of production users. It's properly documented as well.
I'd say for both of your questions the answer is "it just works".
As far as my little current experience allows me to understand, one of the core concepts about "microservice" is that it relies on its own database which is independent from other microservices.
Diving into how to handle distributed transactions in a microservices system, the best strategy seems to be the Event Sourcing pattern whose core is the Event Store.
Is the event store shared between different microservices? Or there are multiple independent event stores databases for each microservice and a single common event broker?
If the first option is the solution, using CQRS I can now assume that every microservice's database is intended as query-side, while the shared event store is on the command-side. Is it a wrong assumption?
And since we are in the topic: how many retries I have to do in case of a concurrent write in a Stream using optimistic locking?
A very big big thanks in advance for every piece of advice you can give me!
Is the event store shared between different microservices? Or there are multiple independent event stores databases for each microservice and a single common event broker?
Every microservice should write to its own Event store, from their point of view. This could mean separate instances or separate partitions inside the same instance. This allows the microservices to be scaled independently.
If the first option is the solution, using CQRS I can now assume that every microservice's database is intended as query-side, while the shared event store is on the command-side. Is it a wrong assumption?
Kinda. As I wrote above each microservice should have its own Event store (or a partition inside a shared instance). A microservice should not append events to other microservice Event store.
Regarding reading events, I think that reading events should be in general permitted. Polling the Event store is the simplest (and the best in my opinion) solution to propagate changes to other microservices. It has the advantage that the remote microservice polls at the rate it can and what events it wants. This can be scaled very nice by creating Event store replicas, as much as it is needed.
There are some cases when you would want to not publish every domain event from the Event store. Some say that there are could exist internal domain events on that the other microservices should not depend. In this case you could mark the events as free (or not) for external consuming.
The cleanest solution to propagate changes in a microservice is to have live queries to whom other microservices could subscribe. It has the advantage that the projection logic does not leak to other microservice but it also has the disadvantage that the emitting microservice must define+implement those queries; you can do this when you notice that other microservices duplicate the projection logic. An example of this query is the total order price in an ecommerce application. You could have a query like this WhatIsTheTotalPriceOfTheOrder that is published every time an item is added to/removed from/updated in an Order.
And since we are in the topic: how many retries I have to do in case of a concurrent write in a Stream using optimistic locking?
As many as you need, i.e. until the write succeeds. You could have a limit of 99999, just to be detect when something is horribly wrong with the retry mechanism. In any case, the concurrent write should be retried only when a write is done at the same time on the same stream (for one Aggregate instance) and not for the entire Event store.
As a rule: in service architectures, which includes micro services, each service tracks its state in a private database.
"Private" here primarily means that no other service is permitted to write or read from it. This could mean that each service has a dedicated database server of its own, or services might share a single appliance but only have access permissions for their own piece.
Expressed another way: services communicate with each other by sharing information via the public api, not by writing messages into each others databases.
For services using event sourcing, each service would have read and write access only to its streams. If those streams happen to be stored on the same home - fine; but the correctness of the system should not depend on different services storing their events on the same appliance.
TLDR: All of these patterns apply to a single bounded context (service if you like), don't distribute domain events outside your bounded context, publish integration events onto an ESB (enterprise service bus) or something similar, as the public interface.
Ok so we have three patterns here to briefly cover individually and then together.
Microservices
CQRS
Event Sourcing
Microservices
https://learn.microsoft.com/en-us/azure/architecture/microservices/
Core objective: Isolate and decouple changes in a system to individual services, enabling independent deployment and testing without collateral impact.
This is achieved by encapsulating change behind a public API and limiting runtime dependencies between services.
CQRS
https://learn.microsoft.com/en-us/azure/architecture/patterns/cqrs
Core objective: Isolate and decouple write concerns from read concerns in a single service.
This can be achieved in a few ways, but the core idea is that the read model is a projection of the write model optimised for querying.
Event Sourcing
https://learn.microsoft.com/en-us/azure/architecture/patterns/event-sourcing
Core objective: Use the business domain rules as your data model.
This is achieved by modelling state as an append-only stream of immutable domain events and rebuilding the current aggregate state by replaying the stream from the start.
All Together
There is a lot of great content here https://learn.microsoft.com/en-us/previous-versions/msp-n-p/jj554200(v=pandp.10)
Each of these has its own complexity, trade-offs and challenges and while a fun exercise you should consider if the cost outway the benefits. All of them apply within a single service or bounded context. As soon as you start sharing a data store between services, you open yourself up to issues, as the shared data store can not be changed in isolation as it is now a public interface.
Rather try publish integration events to a shared bus as the public interface for other services and bounded contexts to consume and use to build projections of other domain contexts data.
It's a good idea to publish integration events as idempotent snapshots of the current aggregate state (upsert X, delete X), especially if your bus is not persistent. This allows you to republish integration events from a domain if needed without producing an inconsistent state between consumers.
I'm developing an application that must both handle events coming from other systems and provide a REST API. I want to split the applications into micro services and I'm trying to figure out which approach I should use. I drew attention to the Spring Cloud Netflix and the Spring Cloud Data Flow toolkit, but it's not clear to me whether they can be integrated and how.
As an example, suppose we have the following functionality in the system:
1. information about users
card of orders
product catalog
sending various notifications
obtaining information about the orders from third-party systems
processing, filtering, and transformation of order events
processing of various rules based on orders and sending notifications
sending information about user orders from third-party systems to other users using websockets (with pre-filtering)
Point 1-4 - there I see the classical micro service architecture. Framework - Spring Netflix Stack.
Point 5-9 - it's best to use an event-driven approach. Toolkit - Spring Data Flow.
The question is how to build communication between these platforms.
In particular - POPULATE ORDER DETAILS SERVICE must transform the incoming orders and save additional information (in case it needed) in the database. ORDER RULE EXECUTOR SERVICE should obtain information about the current saved rules, execute them and send notifications. WEB SOCKET SERVICE should send orders information only if a particular user has set the filters, and ORDER SAVER SERVICE should store the information about the transformed orders in the database.
1.
Communication between the micro-services within the two platforms could be using the API GATEWAY, but in this case, I have the following questions:
Does the Spring Cloud platform allow to work with micro services that way?
Performance - the number of events is very huge, which can significantly slow down the processing of events. Is it possible to use other approaches, for example, communication not through the API Gateway but through in-memory cache?
2.
Since some functionality intersects between these services, I have a question about what is "microservice" in the understanding of the Spring Cloud Stream framework. In particular, does it make sense to have separate services? Can the microservice in the Spring Cloud Stream have a REST API, work with the database and simultaneously process the events? Does such a diagram make sense and is it possible to build such a stack at the moment?
The question is which of these approaches is more correct? What did Spring Data Streams mean by "microservice"?
Given the limited information in the post, it is hard to convince on all the matters pertaining to this type of architecture, but I'll attempt to share some specifics, and point to samples. Also for the same reasons, it is hard to solve for your needs end-to-end. From the surface, it appears you're attempting to build event-driven applications and wondering whether Spring Cloud Stream (SCSt) and Spring Cloud Data Flow (SCDF) could help.
They can, yes.
The Order, User, and Catalog seem like domain objects and it would all come together to solve for a use-case. For instance, querying for a number of orders for a particular product, and group by the user. There are a few samples that articulate the data flow between the entities to solve similar problems. Here's a live code-walkthrough of event-driven systems in action. There's another example of social-graph application, too.
Though these event-driven applications can run standalone as individual services with the help of of message broker (eg: Kafka or RabbitMQ), you could of course also register them in SCDF and use them in the SCDF DSL to build a coherent data pipeline. We are expanding on more direct capabilities in SCDF for these types of use-cases, but there are ways to orchestrate them today with current abilities, too. Follow spring-cloud/spring-cloud-#2331#issuecomment-406444350 for more details.
I hope this gives an idea. Try to build something small using SCSt/SCDF, prove it out, and expand to more complex use-cases.
We are designing a reporting system using microservice architecture. All the services are supposed to be subscribers to the event bus and they communicate by raising events. We also decided to expose each of our services using REST api. Now the question is , is it a good idea to create our services as web api [RESTful] applications which are also subscribers to the event bus? so basically there are 2 ponits of entry to each service - api and events. I have a feeling that we should separate out these 2 as these are 2 different concerns. Any ideas?
Since Microservices architecture are Un-opinionated software design. So you may get different answers on this questions.
Yes, REST and Event based are two different things but sometime both combined gives design to achieve better flexibility.
Answering to your concerns, I don't see any harm if REST APIs also subscribe to a queue as long as you can maintain both of them i.e changes to message does not have any impact of APIs and you have proper fallback and Eventual consistency mechanism in place. you can check discussion . There are already few project which tried it such as nakadi and ponte.
So It all depends on your service's communication behaviour to choose between REST APIs and Event-Based design Or Both.
What you do is based on your requirement you can choose REST APIs where you see synchronous behaviour between services
and go with Event based design where you find services needs asynchronous behaviour, there is no harm combining both also.
Ideally for inter-process communication protocol it is better to go with messaging and for client-service REST APIs are best fitted.
Check the Communication style in microservices.io
REST based Architecture
Advantage
Request/Response is easy and best fitted when you need synchronous environments.
Simpler system since there in no intermediate broker
Promotes orchestration i.e Service can take action based on response of other service.
Drawback
Services needs to discover locations of service instances.
One to one Mapping between services.
Rest used HTTP which is general purpose protocol built on top of TCP/IP which adds enormous amount of overhead when using it to pass messages.
Event Driven Architecture
Advantage
Event-driven architectures are appealing to API developers because they function very well in asynchronous environments.
Loose coupling since it decouples services as on a event of once service multiple services can take action based on application requirement. it is easy to plug-in any new consumer to producer.
Improved availability since the message broker buffers messages until the consumer is able to process them.
Drawback
Additional complexity of message broker, which must be highly available
Debugging an event request is not that easy.