I am starting a project where I want to have multiple services that communicate with each other using the axon server.
I have more than one service with the following stack:
Spring Boot 2.3.0.RELEASE (with starters: Data, JPA, web, mysql)
Axon
Spring Boot Starter - 4.2.1
Each one of the services uses different schemas in the mysql server.
When I start the spring boot service with the axon framework activated, some tables for tokens, sagas, etc are created in the database schema of each application.
I have two questions
In the architecture that I am trying to build, should I have only
one database for all the ‘axon enabled’ services, so the sagas,
tokens, events, etc are only in one place?
If so, can anyone
provide an example of how to configure a custom
EntityManagerProvider to have the database of the service separated
from the database of Axon?
I assume each of your microservices models a sub-domain. Since the events do model a (sub)domain, along with aggregates, entities and value objects, I very much favor keeping the Axon-related schemas separated, most likely along with the databases/schemas corresponding to each service. I would, thus, prefer a modeling-first approach when considering such technical options.
It is what we're currently doing in our microservices ecosystem.
There is at least one more technical reason to go with the same schema (one per sub-domain, that is), both for Axon assets and application-specific assets. It was pointed out to me by my colleague Marian. If you (will) use Event Sourcing (thus reconstructing the state of an aggregate by fetching and applying all past events resulted after handling the commands) then you will, most likely, need transactions which encompass this fetching as well as the command handling code which might, in turn, trigger (through events) writes to your microservice-specific database.
Axon can require five tables, depending on your usages of Axon of course.
These are:
The Event table.
The Snapshot Event table.
The Token table.
The Saga table.
The Association Value Entry table.
When using Axon Server, tables 1 and 2 will not be created since Axon Server is the storage solution for events and snapshots.
When not using Axon Server, I would indeed suggest to have a dedicated datasource for these.
Table 3 which services the TokenStore, should be as close as possible to your Query Models. The tokens portray how far a given EventProcessor is with handling events. As these EventProcessors typically service projectors which create your query models, keeping them together is sensible from a transactional perspective.
Table 4 and 5 are both required for Sagas. The "Saga table" stores the serialized sagas, whereas the "Association Value Entry table" carries the associations values between events and sagas so that the framework can load the right sagas. I'd store these either in a dedicated database or along with the other tables of the given (micro)service.
Related
We are rewriting legacy app using microservices. Each microservice has its own DB. There are certain api calls that require to call another microservice and persist data into both DBs. How to implement distributed transaction management effectively in this case?
Since we are not migrated completely to the new micro services environment, we still writeback data to old monolith. For this when an microservice end point is called, we call monolith service from microservice api to writeback same data. How to deal with the same problem in this case as well.
Thanks in advance.
There are different distributer transaction frameworks usually included and maintained as part of heavy application servers like JBoss and WebLogic.
The standard usually used by such services is Jakarta Transactions (JTA; formerly Java Transaction API).
Tomcat and Spring don't support distributed transactions out-of-the-box. You can add this functionality using third party framework like Atomikos (just googled, I've never used it).
But remember, microservice with JTA ist not "micro" anymore :-)
Here is a small overview over available technologies and possible workarounds:
https://www.baeldung.com/transactions-across-microservices
If you can afford to write to the legacy system later (i.e. allow some latency between updating the microservice and the legacy system) you can use the outbox pattern.
Essentially that means that you write to the microservice database in a transactional way both to the tables you usually write and an additional "outbox" table of changes to apply and then have a separate process that reads that table and updates the legacy system.
You can also achieve something similar with a change data capture mechanism on the db used in the microservice(s)
Check out this answer on "Why is 2-phase commit not suitable for a microservices architecture?": https://stackoverflow.com/a/55258458/3794744
I am using AxonIQ AxonFramework version 4.5.3 with Spring Boot and custom event store.
I'm using MongoEventStorageEngine and configured a separate MongoDB database for the EventStorage.
I am doing some business logic with my business database through a microservice. In the same microservice, I've configured the custom EventStorage.
But a few tables (viz. association_value_entry, saga_entry, token_entry) are getting created on my business database which is a PostgresDB.
Why is AxonFramework creating new tables in my business database as I have already configured a separate MongoDB database for EventStorage. All the related database objects for Axon to work should be ideally created in the EventStorage database rather than in my business database.
The tables you are mentioned should be part of your 'read' model (I believe that is what you called business database).
They are not used for Event Storage or Event Sourcing but rather to specific things that are controlled on client side. For example, token_entry, among other things, is the table where your app keep track of the tokens and events it already consumed - you can read more about it here. Similar to the saga tables, where Sagas are stored on the client side having nothing to do with the Event Store - you can read more about it here.
We need to find best way to address distributed transaction management in our microservices architecture.
Here is the Problem Statement.
We have one Composite microservice which shall interact with underlying other 2 Atomic microservices (Which are meant for specific purpose obviously) and have separate database e.g. We can consider these 2 microservices as
STUDENT_SERVICE (STU_DB)
TEACHER_SERVICE (TEACHR_DB)
Here in Composite Service Usecase is like user (Administrator) can assign a Teacher to a student for the specific course etc.
I wonder how can we address this problem in one transaction as each servie (STUDENT_SERVICE and TEACHER_SERVICE ) has separate DB and all should happen in one transaction either commit or rollback.
Since those 2 services are separate and I see JTA would not be of help as it is meant for having these 2 applications (services) deployed on same application server!
I have opted out JTA as mentioned above
//Pseudo Code
class CompositeService{
AssignStaff(resquest){
//txn Start
updateStudentServiceAPI(request);
UpdateTeacherServiceAPI(request);
//txn End
}
}
System should be in consistent state after api execution
This is a tricky question even it's not obvious at the first sight.
The functionality you call for is understood to be an anti-pattern for microservice architecture.
Microservice architecture is in general a distributed system. Transactions in distributed systems are hard (see https://martin.kleppmann.com/2015/09/26/transactions-at-strange-loop.html). Your application consists from two services.
The JTA is a Java API for ACID style transactions. ACID transactions usually requires locks to be established in databases. As the transaction spans over multiple services (in your case there are two) then a failure of one service can block processing of the other service. In such case you are loosing the advantage of the microservice architecture - loose coupling and Independence of the services. You can end up of building a distributed monolith (see nice article https://blog.christianposta.com/microservices/the-hardest-part-about-microservices-data/).
Btw. there are several discussion on the topic of transactions in microservices here at Stackoverflow. Just search or check e.g.
Distributed transactions in microservices
Transactions in microservices
Transactions across REST microservices?
What are your options
(disclaimer: I'm a developer for http://narayana.io and presented options are from perspective of Java EE and Narayana. There could be other projects providing similar functionality. Plus, even Narayana integrates nicely with Spring you will possibly need to handle some integration issues.)
you really need to run the ACID style transaction in your project - aka you insists you need the transaction behaviour in way you describe. Then you need to span transaction over services. Then if services communicate over REST you can consider for example Narayana REST-AT (http://jbossts.blogspot.com/2011/03/rest-cloud-and-transactions.html, start looking into quickstart here https://github.com/jbosstm/quickstart/tree/master/rts)
you relax your requirements for atomicity and then you can cosider some transaction model relaxing the consistency (you are fine to be eventual consistent). You can consider for example LRA (https://github.com/eclipse/microprofile-lra/blob/master/spec/src/main/asciidoc/microprofile-lra-spec.adoc). (Unfortunately the spec and implementation is still not ready but PoC could be run on current state.)
you want to use a different approach for transaction processing completely. Then you can investigate on event sourcing. You would deploy e.g. Apache Kafka and send events for updates to the event store. Each service will reads those events and updates independently the DBs.
Since I am trying hard to understand the microservice architecture pattern for some work, I came across the following question:
It's always said that a microservice usually has its own database. But does this mean that it always has to be on the same server or container (for example having one docker container that runs a MongoDB and my JAR)? Or can this also mean that on one server my JAR is running while my MongoDB is located somewhere else (so two containers for example)?
If the first one is correct (JAR and database within one container), how can I prevent that after some changes regarding my application and after a new deployment of my JAR my data of the MongoDB is resetted (since a whole new container is now running)?
Thanks a lot already :-)
Alternative opinion:
In 99% of real life cases you musnt have a single container that runs
database and the application, those should be separated, since one
(db) is keeping state, while the other (app) should be stateless.
You don't need a separate database for microservice, very often a separate schema is more than enough (e.g. you dont want to deploy a separate Exadata for each microservice :)). What is important is that only this microservice can read and write and make modifications to given tables others can operate on those tabls only through interfaces exposed by the microservice.
First of all each Microservice should have its own database.
Secondly it's not necessary and also not recommended to have the Microservice and its database on the same container.
Generally a single Microservice will have multiple deployments for scaling and they all connect to a single Database instance which should be a diff. container and if using things like NoSql DB's its a database cluster.
Yes, Each Microservice should have its own database and if any other Microservice needs data owned by another microservice, then they do it using an API exposed by Microservices. No, it's not at all necessary to have the Microservice and its database to be hosted on the same server. For Example - A Microservice can be hosted on-premise and its database can live in the cloud like AWS DynamoDB or RDS.
I'm writing an application that has to communicate across 3 different platforms. Two expose their DB via a REST API (no jdbc driver) and one is a native JDBC connection (ex: Derby, MySQL, Oracle, etc).
My problem is that I have no way of assuring any ACID'ity when updating data, given that the three should be updated at the same time.
I've tried reading up on Spring XA but it seems as both 2PC and 1PC require some form of transactional backends. Given that 2 of my 3 destinations are REST APIs, I don't have any transactions. Just a save/update option.
Are there techniques I can use to ensure that the 3 sources are synchronized and that I don't run into inconsistent states if ever a write fails (ie: REST endpoint unavialble, etc)?
A transaction example would be:
Read from DB
Write to REST-1 endpoint
Update DB
Write to REST-2 endpoint
Is there some form of XA I could employ to wrap everything in such a way I can be assured of consistency?